1/GPT-4 live this week!

I spent this month getting back up to speed with AI, transformers, large language models, etc.

Everything happening at the moment is driven by a handful of fundamental constraints.

Here's how to think about modern AI from first principles.

2/The world is full of relationships between things, aka non-linear mappings.

Image -> caption
Question -> answer
Protein sequence -> structure

In theory, if we could program a computer to calculate these relationships, we could solve all sorts of important problems.

3/In practice, many of these relationships are far too complex and nuanced.

It's ~impossible for a human programmer to hand write an algorithm that works at the level we need.

This leads us to machine learning, and in particular deep learning.

https://xkcd.com/1425/

4/It turns out you can approximate any non-linear mapping with a neural network.

You take your data, multiply by a matrix, and apply a non-linear function. And repeat, over and over.

The algorithm is trivial. But where do you get the matrices?

https://en.wikipedia.org/wiki/Universal_approximation_theorem

5/Answer: learn them from data.

You collect examples of input/output pairs.

You run your network over an example input, calculate the results, and compare it to the actual output.

You then tweak the matrices to reduce the error, using the chain rule.

https://en.wikipedia.org/wiki/Chain_rule

6/In deep learning terminology - we have layers, weights, back propagation, etc.

But under the hood, it is just maths. Matrix multiplication, non-linearity, data and the chain rule.

7/This sounds magical. What is stopping us using this universal approximation technology everywhere? Three things:

1. Model capacity
2. Data coverage
3. Learning efficiency

These three constraints are a powerful lens to understand the drivers of AI today.

Let's see how.

8/1. Model capacity. Neural networks learn the structure in the world.

Each layer learns concepts at higher levels of abstraction. Every network has a limit to how much structure it can learn.

Bigger = better.

There are ~two ways to improve this.

9/Hardware - build dedicated chips and infrastructure optimised for deep learning (e.g. matrix multiplication).

This leads us to GPUs, TPUs, distributed training, etc.

This is the arms race in AI hardware.

10/Algorithms - design networks that come with some of the structure of the world "built in".

Example: ConvNet for translation invariance and spatial locality.

Nowadays we get a helping hand from transformers, which essentially learn their own architecture. Mind-blowing.

11/2. Data coverage. Neural networks cannot learn what they are not taught.

Often the bottleneck is collecting enough data to represent all the nuance and complexity inherent in the domain.

We're talking millions of annotated examples. A few solutions here.

12/Do the hard work.

Pay someone to label data - an expert, or mechanical turk, etc.

You can accelerate this by clustering the incoming data and prioritising anything novel.

13/Find a way for your system to collect its own labels, as the output of a simulation (either virtual or real world).

This is essentially how reinforcement learning works.

14/Find a way to use the data as its own label.

This is the secret of large language models.

The genius insight here is that "next token prediction" is equivalent to all sorts of tasks, given the right prompt.

This enables you to scale to insanely large datasets, e.g. the web.

15/3. Learning efficiency.

There is all sorts of nuance in how to find the global minimum of your objective function with minimal compute resources.

Vanishing/exploding gradients, local minima, over fitting.

16/And a whole mystic arts of what combination of tricks you need to use in practice.

Examples: batch norm, momentum, skip connections, learning rates, initialisation schemes.

This is an active area of research.

17/Back to GPT-4.

What you've got here is...

- Insane model capacity (100T parameters)
- Insane data coverage (the entire web)
- A world class team who know how to get it learning efficiently

...and in turn, a very powerful neural network.

18/What does this mean for an ML researcher? A few thoughts:

1. Embrace the bitter lesson.

Focus on improvements that drive a step change in model capacity or data coverage. (Whilst making sure it learns well.)

Everything else is just noise.

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

19/2. In turn, much of modern ML is engineering.

Your toolkit includes e.g. distributed training, or gradient optimisation, and multi-track architecture design.

And the constraint is often the amount of hardware and data available to you.

No wonder big tech dominates here.

20/3. Think about the limits of models like GPT-4.

Example: I get reliably wrong answers for basics maths with 20+ digit numbers.

To me, this suggests GPT-4 has learned a bunch of heuristics for maths, without learning the underlying structure of how maths works.

21/i.e. symbolic/abstract reasoning.

It's not clear to me if this is fixable with better prompts (e.g. chain of thought), or more parameters, or if we need a new paradigm.

There are compelling arguments on both sides.

https://nautil.us/deep-learning-is-hitting-a-wall-238440/

22/But this is super relevant for AI + science.

We want our learning algorithms to be accurate, and in particular in novel domains.

The only way to do this is to model how physics, chemistry, etc actually work. No heuristics!

23/My gut feeling is that this debate will shape the future of science for the next few decades.

Alongside the evolving tradeoffs in those three constraints.

Certainly an exciting time to be around!

Deep learning from first principles

1/Is it possible to be successful without compromising on your values? If not, what do we even mean by success?

I’ve struggled with this question a lot. Recently I’ve come up with a framework that has helped me.

I call it “the easy way and the hard way”.

2/The world is full of ambitious people who neglect things many of us care about.

Marriage, family, health, relationships, employees, society, so on.

It feels like you have to make a choice. Be ambitious. Or have a nice life. Not both.

3/I see this dichotomy play out in my own life.

If I treat my employees with respect … I don’t make those hard business decisions.

If I spend time with my family … I have less time for my work.

If I go to the gym ... I miss an important networking event.

4/This sucks. It makes me pessimistic.

If the only way to be successful is to give up these things … well for me, that’s not success.

And frankly, not for anyone.

5/We all have the same basic needs as human beings.

Think about Maslow's hierarchy.

If our drive to succeed leads us to neglect our emotional connections … how can we worry about self actualisation?

6/Well I don’t think you can.

So is the world full of miserable ambitious people ... and happy unambitious people?

No - but I think this is a surprisingly common point of view.

I've gone through phases thinking this way.

7/These thoughts hit me hard early in 2021.

After half a decade of startup stress and burnout, I realised something simple.

Every time I let my ambition get in the way of my integrity, morality, health, family, happiness, and so on - I regretted it.

It's never worth it.

8/I made a personal commitment never to compromise on what I value for my ambition.

And that pushed this topic to the forefront of my mind, because I realised how much harder this is going to be.

I am ambitious, and I want to believe I can follow my dreams.

9/You see, there are often shortcuts to get ahead in life.

Treat your employees with respect … or just order them around.

Take care of your physical health … or just ignore it.

This is the easy way. Empirically, it works 😬

10/There is also the hard way.

You follow your ambition, but without compromises.

And accept that it will take longer to get to where you want to go, because you constrain your available moves.

And sometimes you have to say no to big opportunities.

11/Why would you do things the hard way?

The day to day reality is that you feel like a sucker.

You'll watch your peers race ahead of you, while making painful decisions to hold back on your professional life.

12/But the hard way is all about optimising for the long term.

The reason we value things like integrity or family is because they matter. Today, and especially as you get older.

Don't believe it? Go ask them!

https://www.youtube.com/watch?v=r6WZuLKOa6k&ab_channel=NoahKagan

13/“After I get successful, I’ll fix those things”

I used to think this way, but it doesn’t work.

What if you haven't "made it" by 30? 35? 40?

The pressure that "my life starts when..." mentality puts on your day to day work is unbearable.

14/On top of that, many of the things you value may be broken irreparably by then.

Usually, it happened a long time before "it happened".

The hard way is not a moral choice, but a selfish one. This is simply what brings fulfilment in the long term.

15/In this sense, the hard way is the only way.

The easy way is just a mirage.

But it is so so hard, because the world is full of people who take the easy way and find temporary success. FOMO!

And somehow the people who take the hard way never seem to get the same press.

15/Fortunately, they are out there.

And what wonderful inspiration they are.

Take care of those connections!

16/What does it take to follow the hard way?

At one level, just that. Commit to it, the rest will follow.

For me, it has lead me to topics like - discipline, following my heart, thinking long term, enjoying the journey, personal productivity, &more.

17/But I didn't write this thread because I've figured it out. Far from it.

I wrote it because I think the world is full of amazing people who believe they have to hold back on their ambition.

That it isn't impossible to live by values AND be ambitious.

I believe you can.

18/In many ways, this is the only thing that matters and yet we don't talk about it at all.

There is a total lack of content and role models here. Or at least I haven't to find them.

This is almost the exception the proves the rule:
https://www.youtube.com/watch?v=tvos4nORf_Y&ab_channel=TEDxTalks

19/But I'm passionate about it and if it resonates, I'd love to hear from you.

Thanks to everyone who've helped me work through versions of these ideas.

You know who you are.

The easy way and the hard way

1/📢 Some news! I'm leaving my role as community lead at @DuneAnalytics, and joining @epochbio as a machine learning engineer.

Working at Dune was an awesome experience. I got to work with some of the best in the business.

Here are my top three learnings:

2/Lesson #1: The future is crypto, and it doesn't matter what industry you are in.

It will take time, there will be scams, and the technology may not be recognisable in 10 years time. But the future is already here, it's just not evenly distributed.

3/@agaperste talks about crypto as a "new language" and I agree.

It's a totally new way of thinking about the world, and just like living in a foreign country, those first few months can be painful.

It took me 3+ months to feel comfortable. Worth it.

https://mirror.xyz/web3datadegens.eth/1ZMEuGH0I_dpBYteNJHZt7MnkNF-nuBeZibsgintDfM

4/I love the fact that crypto gives us an excuse to think about things from first principles - like reputation, value or ownership.



5/A great way to make the transition is to find your niche. For me that is decentralized science.

I may be leaving Dune, but I'm excited to continue to contribute to DeSci and watch the space evolve - shout out to @DesciLondon!

6/Lesson #2: Community is the new advantage. What do I mean by that?

In a "traditional" company, if you have work to do, you hire people to do it.

7/At Dune, when we hit the limitations of what one person can do, we try to find a way to loop in the community to scale that work.

Our role shifts from "creator" to "curator".

8/Everyone wins. Dune can scale efficiently, and our community has an opportunity to advance their careers.

9/It takes a few ingredients to pull this off.

You need a strong sense of community identity, forums for building shared emotional connection, and a permissionless reputation system.

But it's amazingly powerful.

https://twitter.com/dvdkll/status/1618277905871826946

10/Lesson #3: The secret to scaling is people.

My experience at @GyanaHQ taught me about the importance of product-market fit.

But what happens when you find PMF and start to scale? What becomes the bottleneck?

11/I believe the answer is people.

For almost any problem a scale up faces, the meta-solution is always - hire an exceptional person and they will solve it for you.

12/This dovetails nicely into a community-first approach, because (1) there is less pressure to scale headcount at expense of quality and (2) your community becomes your greatest source of talent.

13/And that feels like a good time to say - I'm really grateful to everyone I've worked with at Dune.

What a fantastic, talent bunch of people you are. Look forward to when our paths cross in future!

14/ Thanks to @hagaetc @mewwts @ryan_caldbeck @0xBoxer @agaperste @andrewhong5297 @sk0ji @thomasrtgoss @afletsas @brnt @Dot2DotSeurat @SusmeetJain @cryptofreedman @m3thos @xsvfat @akhilasraju @augustiner_h @vegardstikbakke @NilesLawrence @beccy_ingram @SchezHugo + many more

15/Finally - why @epochbio?

As part of a personal mission - to accelerate our transition to permissionless in silico science. 🧑‍🔬 🧑‍🔬🧑‍🔬

More on that soon.

Leaving Dune

1/Spoke about Decentralized Science (DeSci) and @DuneAnalytics at our IRL meet-up on Monday 🧙‍♀️🧪

What does science look like when we remove the traditional gatekeepers of universities, publishers, labs and committees?

It's crypto, so a network ofc 🧵

2/Context: before working in startups, I studied science + philosophy and dropped out of a PhD in biosciences.

So I have an axe to grind here 🪓

You have been warned... 😂

3/What is science? A network of scientists + capital + tools.

🧑‍🔬-🧑‍🔬 Scientists exchange ideas;
🧑‍🔬-🔬 Design + run experiments; and
🧑‍🔬-💰 Fund it with capital

Through this iterative process, we learn how the world works and build technology to shape it.

4/What happened to universities etc? We'll get to that...

But one of the perks of crypto is to get to think about things from first principles.

And like many things, the scientific process is at its heart a network.

5/Two questions we can ask about this network.

1. How do we build these links? (Connection)
2. Who gets to take part? (Reputation)

We can't just let anyone play around with a $1m mass spectrometry machine.

6/In science today (TradSci), you won't be surprised to hear that these problems are solved by trusted institutions.

Scientists, capital and tools connect through publishers, labs and funding bodies.

These centralized entities hold (most of) the power.

7/Ditto reputation. People at Dune know I like to talk about power laws, and science is no difference.

There is a well defined "reputation journey" that every scientist must tread (approx numbers for UK).

The gatekeepers are universities, publishers and committees.

8/Why this is bad:

1. Gatekeepers constrain innovation (example: look at closed source vs open source software)

2. It does not scale to modern challenges (which require coordination of $Bs and thousands of scientists)

9/Brief interlude on commercial labs, e.g. DeepMind.

Their solution is to 100x the capital investment and sidestep all the intermediaries.

But in the process, things are becoming even more centralized. Just a band aid fix IMO.

10/And so ... DeSci.

Decentralized science is a vision for a scientific process without the intermediaries.

Where individual scientists are free to coordinate, find capital and run experiments without getting approval.

11/How does DeSci enable us to build a permission-less network for science?

To start with, we can replace those trusted institutions with the primitives of the modern web + crypto.

Protocols, DeFi, DAOs, etc.

12/As for reputation, we can look to our peers. 🧑‍🔬🧑‍🔬

If you do great work, people will upvote it.

This is how it works on Github (for software engineers) and indeed Dune (for crypto analysts).

(And who knows what the scientific journey looks after that.)

13/DeSci is nascent, but growing.

No doubt this bear market will be a great time to focus on building the foundations.

My brief overview (very WIP). And a few favourites...

14/🌐 @deworkxyz = any scientist can contribute to a research project and get paid for it.

🧮 @lab_dao = a protocol/marketplace for scientists to use cloud compute and robotic lab equipment.

💸 @vita_dao = fund research based on open governance.

15/How can we quantify all this? With @DuneAnalytics ofc! A few ideas:

- Funds raised via token auctions + NFT collections
- Open governance deployment of funds
- Protocol activity + economics e.g. LabDAO (after launch)

Already playing around with ideas. Watch this space...

16/A final thought 💭

The current scientific ecosystem takes a huge toll on the mental health of researchers.

Imagine a world where you didn't have to put up with a bad supervisor, ignorant publisher, or vindictive grant reviewer.

LFG! 🔥🧑‍🔬🔭💸

17/Oh and a bunch of useful links:

- desci.world - online “hub” @DeSciWorld
- desci.global - IRL events
- app.dework.xyz/hubs/desci - DeWork hub
- ethereum.org/desci - Ethereum docs

Londoners: come to desci.london in Jan!

Intro to DeSci + Dune

1/Chatting with someone today about startup tech stacks.

A very painful lesson I learned at @GyanaHQ is that 99% of online advice on tech stacks is irrelevant/actively harmful for building your own thing.

Let me explain.

2/There is a huge bias in big cos/scale ups for technologies that enable teams to coordinate painlessly and without politics.

The classic example of this is the Bezos memo about services.

https://gist.github.com/chitchcock/1281611

3/IMO the killer feature for many popular technologies today is that they make it easy for person A + person B to collaborate without talking.

React -> components
Docker -> containers
K8s -> services
GraphQL -> schemas

Now you can scale your org without coordination problems.

4/ For big cos + scale ups, this is great!

For startups, this is irrelevant. You are 3 people in a garage/Slack chat.

Abstractions add complexity, the opposite of what you need when finding product-market fit.

5/“OK Dave, if you’re right about this then why do so many startups build their MVP on React/GraphQL/k8s”

That is a great question and something that frustrated me for a long time.

For a while, this is exactly what we did at @GyanaHQ.

Here’s how I think about it now.

6/People building startups need to learn their trade somewhere.

If you’ve worked in a big co/scale up, you’re the target market for React/GraphQL/k8s.

It might be all you know. It’s totally natural you will build your own thing on it.

7/If you’re self taught (like me), your only option is to learn online.

The problem is, 99%+ written about technology stack choices on the internet is designed to sell these solutions to big cos.

(Sell != financial transaction FYI.)

They don’t have startups in mind, at all.

8/If by default, all experience and advice in the world points you in this direction ... that's what you will do.

So it seems the only way to learn the lesson is by actually building and discover the horrors of complexity for yourself.

http://steve-yegge.blogspot.com/2007/12/codes-worst-enemy.html

9/Well, unless you work with someone who experienced this first hand.

There does appear to be a kind of “oral tradition” for startup tech stacks.

Ask the right people, and they’ll you that “obviously Ruby on Rails” is the best way to build your MVP.

I see this on HN a lot.

10/"Alright, easy to dunk on React/GraphQL/k8s but what do you recommend instead?"

IMO many, many startups could be a Django/Rails monolith on Heroku and sprinkles of JS. Maybe BigQuery.

And in turn get to market 10x faster.

I know this, because it's what happened to us.

11/I turned our experience at @GyanaHQ into an opinionated guide for building B2B Saas MVPs on Django.

This got us to product-market fit for a no-code analytics tool with only 20k LOC.

(Today I'd say Hotwire -> HTMX/Alpine.)

https://davidkell.substack.com/p/davids-opinionated-guide-for-building

12/A few other resources that helped us on the way:

• https://www.saaspegasus.com/guides/modern-javascript-for-django-developers/ by @czue
• https://boringrails.com/
• https://boringtechnology.club/
• Docs for Django + Hotwire

13/The thing is, once you’ve internalised this lesson - you will discover plenty of communities for your chosen stack.

E.g. @saaspegasus for Django builders

They just tend to be quieter because there is no marketing team and everyone is too busy building.

14/"What if I'm not building a B2B Saas"

Here is a more general guide about making pragmatic technology choices in a startup.

TLDR:

• walk backwards from the UX
• use mature technologies
• have clear evaluation criteria

https://davidkell.substack.com/p/technology-decisions-for-startups

15/To be clear, I love (as an example) React + ecosystem.

E.g. building the workflows feature in Gyana would have been impossible without https://reactflow.dev/

And certain products are necessarily all in on React (e.g. Notion, Height).

Everything in moderation.

16/Final note about Javascript.

If there was a mature JS framework with the ergonomics of Ruby/Django, it would be the go to startup MVP stack.

One language, isomorphic code, ...

But AFAICT we're not even close, and it feels like we're 3-5 years away.