ML in Research VS in a Startup

•

I've been working on #AI in the best #research labs like @MetaAI and have 200+ citations incl @MIT I've been also bringing AI to the thousands of users in @get_suggestr #product The gap can't be larger! Here are 8 important lessons, every ML engineer need to know to become CTO:

1/ No one cares how big your NN is, how innovative the approach is, or what's the cross-val split. Basically, anything that will fail your paper review means zero in real life. The only question business asks is "will it work or not?" (if yes: what does it take to implement?)

2/ The easiest way to make smth work is to use smth you understand very well and can deploy fast. Hence, the best solutions irl are often simple algos, hard-coded heuristics, and off-the-shelf solutions like @huggingface The fewer points of failure it has – the better.

Pro tip: very complex algorithms that are used as a black box to produce primitives (e.g. embeddings from #transformers) work very well in a combination with simple logic on top of them.

3/ The easiest models to debug are the ones that are simple and that you understand very well. If the good model fails in 2% of unpredictable cases and crashes – it can't be used. Less accurate but bullet-proof heuristics will win.

4/ There is no such thing as a controlled environment! Everything should work for any user at any time of the day. Forget about the parameters that have to be adjusted or limited knowledge domains. The world is unlimited, and you should be prepared for that.

Pro tip: we did everything to go away from absolute values to sorting, rankings, and percentiles because these are things that let you sleep well at night.

/5 You should know the problem user solves better than a solution! Only by going deep into the domain and trying to solve it from the first principles, you'll be able to produce the best solution, account for all the edge cases, and see the whole picture.

Throw away all the variables and parameters you don't understand. Here less is more. For ex, we would recreate CF algorithm and find the best variations for our use case. It's like understanding the code VS what it does. You should be able to explain your algo without formulas.

/6 Everything has its price, so you can't research endlessly for the sake of research. The last 10% of accuracy takes 90% of the time, so try to concentrate on the first 10% of efforts and focus on the impact/effort ratio and not AUC. Use the elbow method.

7/ ML doesn't exist in a vacuum. To train the model you need data, to store the data you need storage, to run the inference you need resources, infra, back end logic. You can't add @PyTorch model when the first one is on @TensorFlo because dependencies will brutally kill you.

You can't use any data, you can't utilize all the resources of the server, and you can't afford crashes because it will take down all the front end (ideally you don't build it like that). You should think about your neighbors because a hole in a wall is a problem for all.

8/ ML doesn't exist in a moment of time. And it runs more often than just during train and test runs. If it won't run at night then you will wake up and run to your laptop. Production algorithms need maintenance, and the less maintenance your model needs – the better.

In a conclusion: a business doesn't value the Novelty, Coolness, or even the Beauty of your algorithm, what it values are the Simplicity, Transparency, and Effectiveness of your solution times the Cost of deployment and maintenance The same btw can be said about the engineer 😉