I read through the skill file of graphify and ran some of the commands it was telling the agent to run. The vibe I got is there was a lot of structured information returned that the model (and harness) isn't set up to work with effectively.
It seems like a really interesting tool and maybe future models will know how to use these tools more effectively, but yeah I remain skeptical.
The more I think about it, I suspect that either the training data or the finetuning process (or both) will have to incorporate these tools to really get benefit from them.
I was demo'ing Gemini 4 Argon last night and watching it use codemode to basically generate structured docs about the codebase for itself
so your experience matches my own limited experience of starting to run evals of my own harness, which is that these specialized tools can help weak models but on the frontier they just make it slower and more expensive
The memory invalidation based on tying memories to pieces of code (e.g. functions) is the big thing here, I think, though I see why it wouldn't be front and center.
I've always found memory systems icky, because it's difficult to tell whether a memory is still valid. Also, discoverability is always a concern (when is a memory relevant). This being both a code-graph and a memory system (seemingly unrelated things), actually addresses this in a novel way.
Hi, most memory tools get evaluated on benchmarks like LoCoMo, LongMemEval and BEAM, plus how many tokens they use
We compared a pre-release build of Graphicle's memory with 7 other memory tools (Mem0, Hindsight and others) on three public question sets. We wrote down the test rules before scoring anything, and used one open model to read and judge the answers for every tool, so it's the same setup for all of them. Details and caveats are in docs/benchmarks.md
What we haven't done yet is the thing you're really asking: does a coding agent actually finish more tasks with these tools? We could use the same agent and model on the same tasks, with and without the tool, and compare how many tasks pass, tokens used, tool calls and time.
Curious what you're using today, and what result would convince you?
I'm not looking to be convinced about this tool as much as I am genuinely looking for a way to eval these kinds of tools in general
I have only just begun playing around with evals for a harness I'm working on and so far I have just been doing runs of converting https://github.com/realworld-apps/realworld from one language to another
not super pleased with it
the unsatisfying answer I think is that crafting good eval environments for long runs takes a lot of time, and there are many things you'd want to verify that you cannot simply lint or test for
Has anyone actually gotten good results out of this?
For the evals I've run (albeit, not super scientific), these memory systems did worse than the raw harness given ripgrep.
I read through the skill file of graphify and ran some of the commands it was telling the agent to run. The vibe I got is there was a lot of structured information returned that the model (and harness) isn't set up to work with effectively.
It seems like a really interesting tool and maybe future models will know how to use these tools more effectively, but yeah I remain skeptical.
The more I think about it, I suspect that either the training data or the finetuning process (or both) will have to incorporate these tools to really get benefit from them.
I was demo'ing Gemini 4 Argon last night and watching it use codemode to basically generate structured docs about the codebase for itself
so your experience matches my own limited experience of starting to run evals of my own harness, which is that these specialized tools can help weak models but on the frontier they just make it slower and more expensive
This has been a general trend in my experience. The more structure you give it the worse it performs. And it also creates more work for you.
The memory invalidation based on tying memories to pieces of code (e.g. functions) is the big thing here, I think, though I see why it wouldn't be front and center.
I've always found memory systems icky, because it's difficult to tell whether a memory is still valid. Also, discoverability is always a concern (when is a memory relevant). This being both a code-graph and a memory system (seemingly unrelated things), actually addresses this in a novel way.
who out there is running evals of giving agents these kinds of tools and how are you doing it?
Hi, most memory tools get evaluated on benchmarks like LoCoMo, LongMemEval and BEAM, plus how many tokens they use
We compared a pre-release build of Graphicle's memory with 7 other memory tools (Mem0, Hindsight and others) on three public question sets. We wrote down the test rules before scoring anything, and used one open model to read and judge the answers for every tool, so it's the same setup for all of them. Details and caveats are in docs/benchmarks.md
What we haven't done yet is the thing you're really asking: does a coding agent actually finish more tasks with these tools? We could use the same agent and model on the same tasks, with and without the tool, and compare how many tasks pass, tokens used, tool calls and time.
Curious what you're using today, and what result would convince you?
I'm not looking to be convinced about this tool as much as I am genuinely looking for a way to eval these kinds of tools in general
I have only just begun playing around with evals for a harness I'm working on and so far I have just been doing runs of converting https://github.com/realworld-apps/realworld from one language to another
not super pleased with it
the unsatisfying answer I think is that crafting good eval environments for long runs takes a lot of time, and there are many things you'd want to verify that you cannot simply lint or test for
one thing you could do is run it against FrontierHarness eval and see how it stacks up?
Yeah learn how to do even the basics of science, please. Control groups!! Confounders!!
Letting another LLM decide quality probably isn't the only meterstick you want, either.
How is the benchmarks of this tool? Did you programm the tool within help of AI?