How https://clipdrop.co/relight works?

A short thread to explain how realtime relighting of a picture is possible. 

First,  we trained a deep neural network to predict a depth map and a normal map from a picture. 

The depth map allows generating a 2.5D model (almost 3D, but we are only adding depth to the picture plane)

The normal map is then applied to the 2.5D model. 

A normal map can be seen as a derivative of the depth map, but in practice, it adds a lot of information about how light should be reflected or not with much better precision than the depth map.

More info if you are curious:
https://en.wikipedia.org/wiki/Normal_mapping

Then we use @threejs  to add virtual light to the scene. 

And that's it!

That's the way you can create pictures like these ones:
https://twitter.com/onurxtasar/status/1567407984392962048?s=20&t=a8RoqIOTfneVHU398cHJeA

You can try to relight your own pictures there:
https://clipdrop.co/relight

And have more technical info on our blog here:
https://clipdrop.co/blog/relight-technicalities

Edit:

This thread is gaining popularity, so I'll make more content like this one. 

Follow me or @clipdropapp if you are interested.

ClipDrop Relight