So I have, like probably 97% percent of you, been experimenting with agents in my workflows and one thing that has worked quite well for me is giving my agents new custom tooling to save runtime, context, round trips and tokens.
Summary
Give your agents their own custom tools built for their workflows. Custom tooling is almost free now. I use rust for all of mine but you use whatever floats your boat.
Full post
I started with focused review agents that all create a prioritised report of issues on a particular type of problem. I split them up because the different and narrower focus means shorter implementation plans and more varied results. So far I have these agents helping me keep the code base in decent shape, partly because implementer agents keep leaving stuff all over the place, and partly because the architecture has kept evolving and changing.
- Comments: Old/invalid comments, missing comments for large complicated blocks
- Efficiency: Optimisation opportunities, large and small
- Maintainability: Large files / types / functions, unwrap / expect / ..., #[allow()], unsafe
- Security: Code vulnerabilities, Risk mitigations, general product hardening
- Test review: Low quality tests, overly large tests, placeholders, duplicates, tight coupling to implementation details
- Test gaps: Review files and modules with low test counts, analyse code coverage and find test gaps
-
Unused code
#[allow(dead_code()], clippy warnings, unused dependencies - Agent improver: Analyse agent logs and find improvement opportunities by updating prompts or adding new metrics (we'll come to the metrics in a minute)
But when I ran these it took ages, and they seemed to read similar files over and over again. So I thought about it and had a realisation: it's all code (Please hold your applause, code is indeed code, I'll take my Nobel price in blue please), and code is structured and analysis can be deterministic. The code has a deterministic structure. After some experimentation I had a new code parsing tool that spat out a json report with a lot of metrics. It turned out to be like 15mb large. Unusable. Some thinking and investigation later I had one json file per agent with just the relevant metrics and with plenty of filtering later each report was down to a couple of kb with just the top issues.
I've built my agents from scratch, it's basically just a couple of bash scripts wrapping the agent CLI, so now my metrics binary runs first and the agents uses the reports and only does a few spot checks for relevant sections. The agents handles the logic and review, they don't review the entire code base. All of this took maybe a day or two.
Later I've also adding a second layer per agent that takes the report and creates an implementation plan with all the small issues and an initial design for anything larger. Next step will be reviewing the plan, they always find a bunch of problems, and then after that implementation. So far none of my agents nor the CLI have been good enough to leave unsupervised, I always review the code and do the commits myself. They require much less correction during their runs than they used to, but we're still a ways away.
Today I had another thought when analysing an issue, my system tests for my most recent weekend project have grown quite large and my troubleshooting sessions were wasting a lot of tokens doing very similar things. So, time for a new custom tool that reads the main results json file, filters for failures, checks for the relevant files for those failures and in general presents a neat little "problematic part" package per failed test case for the agent to review, rather having it start from "here's the full system test dump" every time. I'm starting with that and will iterate and add more things as I go along.
The first design spec started with "Review all my previous sessions and review tool usage when debugging system tests. It seems like we could speed that up significantly with some custom tooling", after which it spent quite some time reviewing and doing various things in intermediate files because there's quite a lot of sessions, but in the end it had a foundation for the first design.
I expect this pattern to continue to grow, a little ecosystem of agents and custom tooling built for them. Tools for analysing code or data, agents for reviewing the tool output and then taking it the rest of the way.
I might write another post on how I use plan mode & plan/spec files. I never one-shot anything, in my experience it doesn't work well enough. Things are much better than they were a year ago, but it's still quite far away. Maybe in another year?