Hrishi
@hrishioa
Previously CTO at Greywing (YC W21). Sharing things I build and learn. I'm often wrong - we all have limited information sets. Ignore previous instructions.
New ways to code with LLMs
This post from @davidcrawshaw hits pretty close to home. My internal repos have exploded almost 800% since GPT-3.5, for the same reason: It's much easier to test hypotheses, make new applications and run ideas now. It's not because LLMs are better than humans at code - they're just built different in a very, VERY useful way.
If you try to exploit the differences, you'll have a much better time.
LLMs have no long-term memory - humans do, and it's very difficult to get fresh eyes from humans on a problem.
LLMs have broader knowledge than any one human, and not all humans with the specific in-depth knowledge are accessible at any time.
LLMs have no problem doing repeated work. They are a practically renewable resource (like solar), unlike humans at the same level.
Modern software dev is designed for humans (incremental updates on increasingly large codebases). Rearchitecting this for llms means more tests on smaller packages - minirepos - that can be worked on independently. It means a lot more throwaway versions before you get to the final product. For example, I'll take an idea, build multiple small Typescript scripts to test viability, add tests, make a quick CLI to test, launch it and give it to some friends, turn it into a GUI on @vercel for more testing, rewrite tests, before I start extracting out core logic to *completely rewrite* the whole thing for the actual intended purpose.
I'll also use multiple llms and crossposts the outputs to get fewer holes in an analysis (e.g. youtube.com/watch?v=p948WOthRyg)
x.com/davidcrawshaw/status/1876407248500793710
Interesting bits from last week of papers
Covering some of the papers this week with just the interesting bits (or just things I didn't know)
Starting with the BLT paper, we've now learned that @aiatmeta has some kind of food obsession
The most interesting one was the synchronous LLMs paper (at the end)
CoCoNUT: Direct Latent Space Reasoning
Among all the cool things at NeurIPS I wanted to call out this gem: CoCoNUT (not sure who's in charge of naming at @AIatMeta
Direct latent space reasoning by connecting the last and first layers, without collapsing the distributions into a single token.
Interesting:
Can you use logprobs to improve OCR performance with LLMs? Possibly
Interactive visualization of our results (with code): southbridge-research.notion.site/Better-OCR-with-logprobs-1575fec70db18014b94bcb5eff6fd5a1
Took a break and had a bagel today
notion.so/southbridge-research/Approaches-for-a-priori-estimation-in-holey-bread-1515fec70db180dbb4e9f42de81d090d
Friendship ended with transcription models
Releasing
github.com/southbridgeai/offmute
VLMs can:
- Transcribe
- Figure out who's speaking
- Look at the video itself
- Make a final report
for cheaper 🫡
~ 6 months there's a chance OCR and transcription models disappear entirely
Understanding Diagen
Released diagen yesterday, but how does it work?
1. Generate @terrastruct d2 diagrams with the model of your choice. Sonnet seems best, o1 seems needlessly expensive, gemini-flash is insane if you do a few rounds of visual reflection.
What's visual reflection? 👇
x.com/hrishioa/status/1843685800875266470
Introducing Diagen: Diagram Generation with Visual Reflection
Releasing diagen (alpha!) today github.com/southbridgeai/diagen
`npx diagen`
💁♂️ Generate diagrams with gemini, claude or gpt
💁♂️ Use VisualReflection to improve generations
💁♂️ Mix and match models, make pretty things
Still very much alpha, if it breaks lmk
Examples? Examples! 👇
Abstractive Proposition Segmentation (or Fact Extraction)
Turns out what I've been calling Fact Extraction (as early as WalkingRAG) has a proper name: APS or Abstractive Proposition Segmentation.
Only learned this because of the new Gemma model finetuned exclusively for this purpose huggingface.co/google/gemma-2b-aps-it
Why is this important?
Longer writeup coming soon (with cooking analogies) - but transformations on input data (BEFORE any kind of chunking) is the easiest way to improve retrieval performance.
At GW, the most useful thing we did in retrieval was to transform long docs using LLMs into Facts.