Rethinking AI: Beyond AGI – The Cognitive Cone

Artificial General Intelligence (AGI) is a vague concept, often debated over what truly qualifies as "general" intelligence. Some might argue that systems like ChatGPT have achieved AGI because they handle many tasks. However, these systems, like all intelligent ones, have limitations and aren't universally "clever." In other words, AGI is a poor metric for measuring progress and therefore cannot effectively guide development.

In his 2019 paper, "The Computational Boundary of a 'Self': Developmental Bioelectricity Drives Multicellularity and Scale-Free Cognition, Michael Levin articulates this idea:

"Any Self is demarcated by a computational surface – the spatio-temporal boundary of events that it can measure, model, and try to affect. This surface sets a functional boundary—a cognitive 'light cone' which defines the scale and limits of its cognition."

Michael Levin’s concept of the cognitive cone, as visualized in the diagram, provides a powerful framework for understanding and guiding AI progress. It reframes intelligence not as a binary achievement but as a continuum defined by the breadth and depth of a system's perceptive and actionable range. Here’s how this perspective can shape how we think about AI development.

This as tons of useful application regarding AI.

Full post there: 
https://damien-henry.com/writings/rethinking-ai-beyond-agi-the-cognitive-cone


Rethinking AGI: The Cognitive Cone

How much is a company an AI company?

Let's figure out!

This thread classifies businesses into 10 levels.

This post contains real insights, plus just a bit of shitposting at the end. Because, well, I'm not making the rules.

Let's go 👇

Level 1: The company doesn't use AI.

→ Their competitors, suppliers are already using AI and it’s already changing it's position in the ecosystem.
→ Some employees are using Claude, ChatGTP, or other tools for work without telling their boss.
→ In fact, all the tools we are using use AI under the hood. Your inbox, your camera, your social feeds depend on learning algorithms.

In other words, it's impossible to run a business strictly without AI, even if someone wanted to.

Level 2: The company uses AI officially.

→ The company subscribes to a couple of SaaS.
→ For instance, it may use @clipdropapp for image editing or @heyjasperai for text generation.
→ This may company encourages employees to use Claude or ChatGPT for work.

Most companies are at level 2 today.

Level 3: The  compagny implements AI workflows

→ This company restructures its workflows around AI to maximize automation.

Most companies are becoming level 3 while you are reading this thread. They are restructuring the way they are organized to maximize positive AI impact.

Level 4: The company is an AI-first company

→ It is a SaaS, and their product wouldn’t exist without AI.
→ It is likely doing generative AI, but it can be something else.
→ It has a good understanding of the market and injects some "know-how" through prompt engineering to differentiate themselves from vanilla AI providers like OpenAI.

Hundreds of AI-first startups are founded every day; there is tough competition for Level 4 companies.

Level 5: The compagny owns proprietary data

→ The company found a way to capture important business data and leverage them through AI.
→ It can offer a differentiated service than most competitors, especially vanilla AI providers like OpenAI won't be able to offer.

Having exclusive data can be a moat.

Level 6: The company AI is learning by itself!

I learned this concept from @ericschmidt during a chat when I was working at @google. He was an active and early AI evangelist. For him, this was “a new moat” that can make a small startup beat a big company.

→ The idea is that you can learn things from your users. For instance, you can use thumbs up/thumbs down feedback to retrain your models.
→ The more users you have, the better your model becomes. The better your model becomes, the more users you'll have.
→ To operate at this level, you need to train or at least fine-tune your own models.

OpenAI is famous for using RLHF, and that's a clear advantage they have today: they have more users.

Having a product able to learn by itself was the initial meaning of “AI first company” in the pre-OpenAI days.

This is what @cyrildiagne, @jblanchefr, and myself had in mind when we created @clipdropapp.

Clipdrop is level 6.

Level 7: The company owns its AI infrastructure.

→ Training models were too expensive and sensitive. To stay ahead, the company owns data centers or enters into multiple-year partnerships with another player that owns some.

→ OpenAI is almost at level 7 with it's relation with Microsoft.
→ Stability reached level 7 for a moment with a costly partnership with AWS. It led to a phenomenal impact (stable diffusion) but dried up their runway.

The GAFAM are all at level 7.

Level 8: The company owns its AI hardware.

→ GPUs are too expensive and prevent scale. The company didn’t want to be too dependent on a single provider, @nvidia.
→ Level 8 companies are creating their own hardware and own the full stack.

@google has been operating at level 8 since 2015 with TensorFlow and the TPU.

@meta is also operating at level 8 and introduced its first-generation chip, called the MTIA last year.

@tesla is trying to reach level 8 and actively developing its own custom AI chips, primarily for its "Dojo" supercomputer.

Level 9: The company AI is self-sufficient.

→ Energy became the main cost of operation. The compagny needed to control electricity costs and start investing in electrical plants.

No company has reached level 9 yet, but there were rumors about Google wanting to invest in nuclear plants.

This shouldn't be a surprise: all huge businesses consume tons of energy and want to reduce their bills.

Level 10: The AI compagny rules

→ The main limitation of applying AI everywhere was regulations.
→ Investing massively in lobbying was not enought.
→ Datacenter are getting attacked by ungratefull humans despite autonomous drone swarn to defend them.

Only solution: buying Iceland  (or conquer it).
There are tons of free energy there, and datacenter are easy to cool.

Create your own state, define your own rules.
Own satelites for conectivity, they can’t be shut down.

Only @elonmusk is on a good trajectory to reach level 10 for now.

The 10 Levels of Being an AI Company

I created the Google Cardboard almost ten years ago but quit VR soon after to focus on Machine Learning.

VR is based on a ridiculous misunderstanding.

Let me explain why in this thread.

The VR promise is to bring “a sense of presence” to computing.

But the “sense of presence,” seeing things as if they were there, adds little value beyond the wahoo effect it generates.

The misunderstanding is between "immersion" and a "sense of presence."

Listening to a good story is immersion.

Our brains' capacity to immerse ourselves in someone else idea is a superpower that allows us to co-create.

Reading a book is real immersion.

When immersed in a book, our senses are disconnected from our bodies, and we no longer feel our direct environment.

Reading a book is decoding black symbols on a white page. Our brains decode these abstract symbols, creating a virtual world inside our brains. There is no need for brain implants; we can already read the “Matrix”.

We are immersed when we are focused on our inner thoughts, controlled by some external signals.

VR takes the problem the wrong way. It tries to immerse us by fooling our sensory system.

VR is lying to our bodies. That’s why most VR experiences are unpleasant after a couple of minutes.

Our perceptions fade away when we are immersed. For instance, while reading a book in the tube.
On the other end, VR uses our body's senses as the medium, distracting us from the story, the message, and the real thing we want to be immersed in.

Ironically, VR is a bad medium because the only immersion that matters is the immersion of our minds.

OK, that's all for today.

There would be so much more to say about VR, but I'll keep it for next time!

The Misunderstanding of VR: Immersion vs. Sense of Presence

Does an apple exist when no one watches it? Donald D. Hoffman thinks it doesn’t, and his book explains why.

Here is a short summary. 🧵👇

The book explains that what we see and feel is not the physical world. For instance, we see colors, not wavelength.

The book's main argument is that natural selection made senses and perception valuable for an agent to survive, not made for describing the “True” physical world.

It explains that what we usually call reality is like the icons on our computer screens. A helpful user interface that presents the right amount of abstraction to maximize efficiency.

It shows that the concept of space-time is also likely a construction by natural evolution to make sense of our surroundings.

They don't exist if no one perceives them like a color that exists only in the mind of an observer.

All this theory matches the latest founding in quantum mechanics.

Finally, the end of the book depicts a theory about consciousness and the universe. You don't have to believe in it to find it enriching, poetic and beautiful.

I recommend this book 100% even if it didn’t explain some side effects of its theory: while space-time may be only a construction favored by evolution, it is so because these abstractions make sense at the scale of our body.

Our senses and our capacity to move co-evolved. So even if space-time does not exist, navigating along its abstract dimensions is still the only practical way to escape a predator. If there were other ways, evolution would have found it.

So even if space-time does not exist, our body and mind are still stuck inside it. Can we call something nonexistent if we can’t escape it?

Again, I 100% recommend this book. A must-read if you are interested in consciousness and its relationship with the physical world in the context of natural evolution.

The summary page is here:
https://dh7.net/the-case-against-reality-donald-d-hoffman


The Case Against Objective Reality

I was curious about this paper:
https://arxiv.org/pdf/2202.05826.pdf

It explains how creating an algorithm able to solve complex tasks while being trained on simple ones only.

The idea is to use multiple time the same block until it finds the solution.

The more complex the problem, the more you iterate before finding the solution.

The paper was not clear enough for me to understand how it really works, but I finally understood from another paper:

https://proceedings.neurips.cc/paper/2021/file/3501672ebc68a5524629080e3ef60aef-Paper.pdf

There is a trick so that it won't generalize well to any kind of problem, at least for now.

For the maze, it generates intermediate solutions like this.

This way, a fully convolutional network can learn on small labyrinths, and it became intuitive why it scales if you run it iteratively.

Not that impressive after all.

Stable Diffusion is also based on training a network to solve a small part of a problem, and it's by running it through several steps that you can generate an interesting issue.

It feels that this way of training network is more profound and is underused.

Generalizing from Easy to Hard Problems with Recurrent Networks

#ImageGen Video is just released, but I believe that #Phenaki also from @GoogleAI is much better and will lead the path while ImageGen and Make-a-video from Meta AI are a dead end.

This thread explains why.

MakeAVideo (and imageGen) learn "text to images" and next frame prediction independently.

While #Phenaki can learn from text and video pairs.

https://twitter.com/dh7net/status/1576658264959242242?s=20&t=phEPJEtbOLxDevH35KE5Fw

One of the primary challenges for generating videos from text is to get a compressed representation of videos.

They solved that by using an encoder-decoder.
It encodes a video in a list of tokens and can decode these tokens into videos.


I'm assuming it first learns to encode tokens and decode them without prompts.

Then the network learns to predict masked tokens, guided by a prompt.

The video generation is done like an RNN, by predicting tokens one after each other with a prompt as guidance for each step.

This method is better than MakeAVideo or ImageGen because it can generate arbitrary long videos, with multiple prompts to describe a full story.

https://twitter.com/dh7net/status/1575747858585866240?s=20&t=phEPJEtbOLxDevH35KE5Fw

Also this method will soon benefit from ImageGen which much better for super- resolution:
https://twitter.com/doomie/status/1577715163855171585?s=20&t=r_uy34KTCww7AVYIN3SVSg

You can expect more Breakthru in the coming weeks!

Phenaki

Meta AI revealed last week a new tech that can generate a video from a text prompt: https://makeavideo.studio/

How does it work?

This 🧵 to explain it!

But first, let's think about the challenge.

There are billions of images with alt-text on the web.
This helps creates a huge dataset, and it's the key to training networks that understand the relationship between image and text.

But there is no equivalent for videos.

The trick is that Make-A-Video learns the relationships between image & text, and movement from videos without the text.

This way, Make-A-Video generates videos from text without the need for paired text-video data.

Here is the architecture.

We can see it takes the text as an input, and generate the video as an output.

Let's have a closer look

The first part "Prior P" is the part doing the Text to Image.
They train it on paired text-image data and do not fine-tune it on videos.

It doesn't output an image directly but a latent representation of it. (it would needs a decoder to make an image out of it).

But instead of decoding one image, the spatiotemporal decoder output a sequence of images: 16 RGB frames, each of size 64 × 64.

That's the real innovation of this paper IMO.

After training on images, they add and initialize the temporal layers and fine-tune them over unlabeled video data.

That's the key to this paper: learning text to image and movement separately.

It is also its main limitation, but I will came on that later.

What is coming next is more classical, and is about transforming the 64x64 16 keyframes into a video

First step is frame interpolation from 16 frame to 76 frames

Then 2 networks increase the images to 256 × 256 and 768 × 768 respectively.

The first is spatiotemporal super resolution, to keep the consistency between frames over time.

The second one is doing super resolutions frame per frame and output the final video.

While this method is amazing, there are a couple of limitations:

The number of output images are fixed by the network, so it means all the generated video have the same duration.

But main bottle neck is that text is used only to generate images during the trainning.
So I don't think this tech can be used to describe a movement.

For instance I bet it can't generate things like "a ball moving left to right".

That's it for today, but there is another paper that seems to overcome this limitation.

Stay tuned :-)


Make a video

#Dalle, #StableDiffusion, etc.. are amazing to imagine things like avocado chairs, but they can't draw specific objects.

This is a major blocker for those who want to use this tech for advertising a real product.

But this will change.

👇 This thread to explain how.

A good solution would be to show #StableDiffusion a couple of images of your product (or your pet in this example) and get an identifier (a name) for it.

So you can use this new identifier (name) in your prompt.

This is precisely what a new paper from @GoogleAI is doing.

It's named Dream Booth: https://dreambooth.github.io/

It is surprisingly simple to understand how it works.
Let me explain:

First, you need a generic text2image network (in Red) and a couple of new images.

The goal is to get a new network (in blue) that will be able to draw everything, including your specific product/pet with a unique identifier.

So you can use this unique identifier when you need to draw exactly the thing you want to. (The thing/product/pet that was in your images)

How to create this new network (in yellow there)?

The idea is to start from a generic one and to teach it 2 tasks simultaneously.
1) learn what the generic network was doing (upper part)
2) learn to reconstruct specifics images (bottom part)

And that's it! Deceptively simple.

While this method is not perfect, given the progress of research, it's easy to bet these technics will become mainstream sooner than later.

If you are interested, keep an eye on what we are doing at clipdrop.co ;-)

Dream Booth

How https://clipdrop.co/relight works?

A short thread to explain how realtime relighting of a picture is possible. 

First,  we trained a deep neural network to predict a depth map and a normal map from a picture. 

The depth map allows generating a 2.5D model (almost 3D, but we are only adding depth to the picture plane)

The normal map is then applied to the 2.5D model. 

A normal map can be seen as a derivative of the depth map, but in practice, it adds a lot of information about how light should be reflected or not with much better precision than the depth map.

More info if you are curious:
https://en.wikipedia.org/wiki/Normal_mapping

Then we use @threejs  to add virtual light to the scene. 

And that's it!

That's the way you can create pictures like these ones:
https://twitter.com/onurxtasar/status/1567407984392962048?s=20&t=a8RoqIOTfneVHU398cHJeA

You can try to relight your own pictures there:
https://clipdrop.co/relight

And have more technical info on our blog here:
https://clipdrop.co/blog/relight-technicalities

Edit:

This thread is gaining popularity, so I'll make more content like this one. 

Follow me or @clipdropapp if you are interested.

ClipDrop Relight

I'm leaving Google and I'm starting an exciting adventure!

Stay tuned!

I'm now COO of http://Clipdrop.co :-)

I have the privilege to work with the 2 very best co-founder you can think of: @cyrildiagne &amp; @jblanchefr 

I can't be more excited by the idea of ​​helping people through machine learning. There are so many things waiting to be done!