How clipdrop.co/relight works?
A short thread to explain how realtime relighting of a picture is possible.
First, we trained a deep neural network to predict a depth map and a normal map from a picture.
The depth map allows generating a 2.5D model (almost 3D, but we are only adding depth to the picture plane)
The normal map is then applied to the 2.5D model.
A normal map can be seen as a derivative of the depth map, but in practice, it adds a lot of information about how light should be reflected or not with much better precision than the depth map.
More info if you are curious:
en.wikipedia.org/wiki/Normal_mapping
Then we use @threejs to add virtual light to the scene.
And that's it!
That's the way you can create pictures like these ones:
twitter.com/onurxtasar/status/1567407984392962048?s=20&t=a8RoqIOTfneVHU398cHJeA
You can try to relight your own pictures there:
clipdrop.co/relight
And have more technical info on our blog here:
clipdrop.co/blog/relight-technicalities
Edit:
This thread is gaining popularity, so I'll make more content like this one.
Follow me or @clipdropapp if you are interested.