Some folks may have seen Duolingo's viral TikToks, but not many know about how we think about product development. 

Here's how we test and iterate on products at Duolingo 🧵

We take a hypothesis driven approach and Test Everything.

Virtually all of app changes are gated and rolled out using A/B tests. 

(and sure, pre-empting the usual: there's a whole can of worms to be opened about hill climbing and local maximas, and would love to talk more about that, but that's for a future thread.)

First, to set context, here's an example of an A/B Test:

Step 1: Have a hypothesis (a falsifiable statement) about user behavior.

We get our hypothesis from "a combination of user feedback, data from our learners, past experiment learnings, and our own intuition".

In the example above, it could be: the word "trial" triggers an alarm in our users' brains, making them less likely to try out the Super experience.

Step 2:  Test our hypothesis as rigorously as we can.

Any software makes implicit assumptions and tradeoffs about user behavior everyday. 

An *opinionated* product does this especially well. 

But, how do we get signal that we are making the right product calls?

Pre-Product Market Fit, we typically do it with deep user research and analytics. 

Post-PMF, we do it with A/B tests at scale.

Step 3: Analyze and Call the Experiment

All product development have tradeoffs.

These can be pre-empted and discussed in the design phase. 

There's sometimes a tradeoff between user retention and monetization, or monetization and learning, or user retention and learning.

But these can also be unexpected:

We have guardrails to ensure that these don't happen.

Sometimes teams can foresee this, while other times they can't and they have to dig deeper.

Step 4: Document your learnings and socialize them.

What did you learn about user behavior? Anything unexpected? What are the key insights?

What are the next steps?

Bringing it home: our purchase flow.

You can see the evolution in our thinking here. 

Over the years, you can see:

Hypotheses of...

testing clear upfront pricing, 

reducing choice, 
calling out user benefits, 

testing redesigns (both of Duo and the app!)

(shoutout to our tireless Monetization Area's work on this)

If interested, read more here in our latest shareholder letter and listen to the shareholder call with @luisvonahn and Matt Skaruppa.

https://investors.duolingo.com/

Also see more examples of past experiments on our blog 

(https://blog.duolingo.com/how-duolingo-streak-builds-habit/) 

Future threads coming about how our experimentation team built this, what we learned along the way, and where we are going.

Oh and Duolingo is still hiring! 

Come work with folks who are the best in the business @cemkansu @Albertc248 @nickeyskarstad and many more!

https://careers.duolingo.com/

You can read the unrolled version of this thread here: https://typefully.com/samcwl/qER6kOQ

A/B Testing at Duolingo

TIL: Not much is known about the herb-drug interaction between TCM (traditional chinese medicine) and western drugs (i.e. Aspirin).

🧵:

1/ Apart from spacing out the dosage times (~2h in between), there's not much known about how to think about the pharmacodynamic interactions between these classes of drugs.

2/ As someone currently experiencing flu-like symptoms, I rotate my symptom management between western drugs, traditional western herbal remedies (echinacea, ginger, turmeric) and traditional chinese medicine (TCM). 

3/ However, just as how you should not take some drugs with others, I'm also curious about herb-drug interactions.

4/ However, while some work has been done looking at the former two, there's not much work that has been done between the first and the last.

5/ Have also found this database (http://naturaldatabaseconsumer.therapeuticresearch.com/) to be somewhat useful, but since the search space is n^2, chances are that studies for a specific herb-drug combination is studied is low, especially among TCM - western drugs.

6/ So, drug-addled me came across this paper in the Singapore Medical Journal today:

 While this paper is pretty dated (and the data pull comes from studies published pre Dec 2013), it provides some valuable ways to think about herb-drug interaction and TCM for me:

1. Differences between Pharmacokinetic vs Pharmacodynamics Interactions.

TCM herbs and Aspirin tend not to have a Pharmacokinetic effect (but SG drs still recommend you take them ~2h apart), rather they could have pharamacodynamic interactions.

7/ 2. There's an entire class of herbs in Chinese which promote syndromes of Qi / increases blood circulation

8/ 3. Many of the studies (at least when this paper was published) are in animals and in-vitro, not in-vivo RCTs. 

As with any herb, methods of preparation, provenance, concentration and dosage all affect bodily ingestion.

9/ 4. TCM practitioners often "stack" herbs to boost efficacy (would sound familiar to the nootropic gang). This compounding effect of herbs is also not well documented.

10/ So, would I still consume my herbs? 

Yes - hard to shake away that part of me - but definitely will try to be more aware of the specifics of the herbs and try to understand their mechanisms before scaling the dosage.

11/ Final thought: the general UX of this space is pretty bad. 

Low hanging fruit for a CRUD app where you could list all your current medications and it'll ping the DBs for any known pairwise interactions.

This could be more important in the coming years with the "mainstreaming" of more herbal remedies (adaptogens and nootropics)

You can read the unrolled version of this thread here: https://typefully.com/samcwl/NNnnrq3