We've had a lot of success for complex tasks with LLMs using a method I don't think I've seen out there yet.
Not sure what to call it but a codename is 'Reversed Retrieval-Application'.
I'll make a detailed post later, but here's the skinny:
The standard way of working with complex data - let's say information retrieval, classification, or generation - is:
1. Provide all the context ('I am me')
2. Define task ('generate a calendar')
3. Define facts ('today is 12 Aug')
4. Define methodology (e.g. 'step-by-step')
This works pretty well for a lot of things, but for our most complex tasks we've previously needed GPT-4-level intelligence to be able to do it this way at 96% pass.
The new method gives me 98% on our dataset with claude-instant, and 100% with Claude-2.
How it works is simple - you reverse the order, and force the LLM to generalize.
Instead of providing say the date, the calendar info, etc. you provide one piece of information, and ask it to generate working steps that 'someone else with the context can use to get the answer'.
This significantly lowers prompt and inference complexity, and generates - I'm as surprised as you are - much tighter output with no loss of detail.
The LLM has less to be confused by, and has to generate working steps that can work with any (presumed) input combination.
You then feed this intermediate step - and only it - to another prompt with more context, and incrementally merge.
It's weird that this works, but I really can't argue with the results I'm seeing.