Rendered at 01:56:18 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
randysalami 4 hours ago [-]
This article mainly argues around harnesses but a breakthrough for me was understanding why harnesses. Without a harness and only a raw LLM, exploring and modifying codebases is not feasible. It’s just text in, text out, and has no way of handling the context of the codebase (if large enough). Plug an LLM into a harness, and all of the sudden it has an exosuit that lets it funnel its thinking into discrete actions. It can now use strategies to manage that large context in pieces.
verdverm 2 hours ago [-]
Depending on how we define harness, it's at its core a loop around llm calls with the tool field populated and a function per to use them after a round trip, easily fits into a single file in a crude setup. I think everyone should try writing a simple harness just for the learning experience, it gets thorny rather quickly.
Another aspect to this is model training, since bots are now being trained with a harness, and that has the tools it gets most burned in to weights. In my own harness, I tried Read,Edit,Write tools which always put the latest text of a file upfront, only one copy, to reduce context window growth as the bot tended to reread a file before/after edits. While this greatly slowed window growth, cache hits dropped to ~50%. If you give the bot a Grep and Read with Offset/Limit, this aligns with training and reduces window growth, even with the read-before-write pattern. You can see the pattern of grep-read when it does the default "use bashisms for everything instead of the dedicated tools"
I'm now exploring ways to remove the need for a shell tool completely, for various reasons, but notably limiting potential for loss of control. Early experiments hint this will not work well due to the heavy shell usage in training, but I'm not throwing in the towel just yet!
digdugdirk 46 minutes ago [-]
I know this sounds ridiculous, but try fine tuning a small local shell -> plain language model, then using the output of that for whatever you need on your end. You'll get the gist of what the frontier model is trying to accomplish, and can handle it with whatever harness workflow you want.
verdverm 41 minutes ago [-]
I'm not sure I follow, can you expand? Why would I want to go from shell to plain language? I don't have a use for an explanation of the shell command that I am aware of. I do sometimes enable a policy-tuned llm to add a non-deterministic check, especially when they try that "command didn't work, let me put it in a script and run that"
If this is in relation to removing the need for shell, my idea is more like inspecting common files like Makefile or package.json to extract and dynamically generate tools for the llm call.
LaurensBER 3 hours ago [-]
Wait until you realize that deep down everything on a computer is text. We juist build "harnesses" to manipulate in-memory "text" in different ways.
One level deeper is all numbers though.
And eventually it's all 0 and 1s!
verdverm 32 minutes ago [-]
there is an "everything is a file descriptor" in there somewhere too
devmor 32 minutes ago [-]
This is too reductive of a statement to have any meaning. You have re-defined a class of program to encompass all programs. The term “harness” exists to differentiate it from programs that are not harnesses.
ashkankiani 2 hours ago [-]
Quantum computers have entered the chat
Terr_ 3 hours ago [-]
"It was just a dream, Bender--there's no such thing as 2."
jswelker 11 minutes ago [-]
I say I am reading the text, but really it's a simple loop moving eyes left then right then repeat.
/s
zby 4 hours ago [-]
I don't know - all of these cache hit rate advocacy is kind of goodhearting. Cache hit rates are less important than the accuracy of LLM answers and people seem to forget about it. For example I've seen articles how sub-agents poison the cache. Sure you can fill up your context with irrelevant data to have high cache hit rate - but that degrades the LLM work - sub-agents are about providing the LLM with a clean context - just the information needed to do its job, no distraction.
All of the main providers now offer cache management that can work with sub-agents - but it was kind of bolted on - and all of them have different apis - so nobody uses them anyway.
ccgibson 3 hours ago [-]
I agree with you, accuracy of answers and the ability to complete task is the most important thing. But once a system can achieve a goal, throughput and cost come into consideration. In this regard, I think cache rates help make things both faster and cheaper.
hedgehog 4 hours ago [-]
The thing that surprises me is that any of the popular harnesses break cache this way. Is it just an artifact of too much LLM code and not enough tests?
ccgibson 3 hours ago [-]
One of the main things I've noticed is that several of the techniques are well intentioned for other reasons. For example, when we debug low cache rates with users probably culprit #1 is usually some memory system. Many of those memory systems are trading a stable prefix for the ability to augment the current turns request with some new memory data. For some -- this may be the right trade-off, but the impact is slower responses times due to the extra prefill compute.
So when a memory system promises better LLM performance the other trade-offs aren't necessarily well understood (at the expense of caching).
Features like OMP trying to clean up history to shrink context I think are also well intentioned. But I found these trade-offs interesting hence wanting to share.
verdverm 54 minutes ago [-]
if you can, where do you work where this is something you help customers with?
LaurensBER 3 hours ago [-]
It's easy to break the cache and there's a number of very valid usecases (mostly context management such as compaction or tool shaking).
Not to mention that it can be very strategic to break context. E.g, after chasing a bug for a while it does make sense to summarise all the approaches that did not work so that you have more context available for newer approaches.
Getting 99% cache hit rates while providing a good user experience is the hard part!
hedgehog 3 hours ago [-]
Compaction for sure, but putting git status in the system prompt? Why? (also I'm not sure about common approaches to compaction, they make context window management tricky and cost more generated tokens)
verdverm 56 minutes ago [-]
the fun ones are putting the current date in the system prompt, iirc OpenCode has fixed this, such that if your session spans day change, your cache does not bust
ccgibson 5 hours ago [-]
I'm the CEO of Neuralwatt, an inference cloud that bills on measured GPU energy. I wrote the post.
The harness, alongside the model, decides how much prefill work gets wasted. Across clients, uncached input ranged from 1.6% (deepseek-harness) to 11.1%. A mostly uncached request uses roughly twice the energy of a cached one of the same size.
Past 128k tokens, a turn that re-reads over half its prompt costs about as much prefill as 600 normal turns at that depth.
Every cache break in our lab came from a harness editing something it had already sent, such as changing the date in the system prompt.
Another aspect to this is model training, since bots are now being trained with a harness, and that has the tools it gets most burned in to weights. In my own harness, I tried Read,Edit,Write tools which always put the latest text of a file upfront, only one copy, to reduce context window growth as the bot tended to reread a file before/after edits. While this greatly slowed window growth, cache hits dropped to ~50%. If you give the bot a Grep and Read with Offset/Limit, this aligns with training and reduces window growth, even with the read-before-write pattern. You can see the pattern of grep-read when it does the default "use bashisms for everything instead of the dedicated tools"
I'm now exploring ways to remove the need for a shell tool completely, for various reasons, but notably limiting potential for loss of control. Early experiments hint this will not work well due to the heavy shell usage in training, but I'm not throwing in the towel just yet!
If this is in relation to removing the need for shell, my idea is more like inspecting common files like Makefile or package.json to extract and dynamically generate tools for the llm call.
One level deeper is all numbers though.
And eventually it's all 0 and 1s!
/s
All of the main providers now offer cache management that can work with sub-agents - but it was kind of bolted on - and all of them have different apis - so nobody uses them anyway.
So when a memory system promises better LLM performance the other trade-offs aren't necessarily well understood (at the expense of caching).
Features like OMP trying to clean up history to shrink context I think are also well intentioned. But I found these trade-offs interesting hence wanting to share.
Not to mention that it can be very strategic to break context. E.g, after chasing a bug for a while it does make sense to summarise all the approaches that did not work so that you have more context available for newer approaches.
Getting 99% cache hit rates while providing a good user experience is the hard part!
The harness, alongside the model, decides how much prefill work gets wasted. Across clients, uncached input ranged from 1.6% (deepseek-harness) to 11.1%. A mostly uncached request uses roughly twice the energy of a cached one of the same size.
Past 128k tokens, a turn that re-reads over half its prompt costs about as much prefill as 600 normal turns at that depth.
Every cache break in our lab came from a harness editing something it had already sent, such as changing the date in the system prompt.
Happy to discuss the methodology.