Pranav Page
← Notes

code

My agent got less agentic every time I fixed it

A day building a field-data copilot on Gemma 4 at Hacktoberfest, and the agent harness I should have borrowed.

I spent a Saturday at FlytBase’s office for the AI Hackers Collective’s Hacktoberfest. The brief: build an AI system that answers questions about a dataset and draws visualizations to go with its answers. Open-weight models were encouraged, so I stuck with Gemma 4 all day.

I went in confident. I build agents, retrieval systems and evals at work. I left with two prizes, and with the sense that another participant had solved the same problem far more simply.

A wall of screens showing the opening slide, a teal Hacktoberfest 26 banner beside photos of an oil field crowded with pump jacks and a single pump jack in a field. The brief on the big screen: an oil field, and the records that go with it.

Tools for everything

The dataset was an oil field’s records (wells, inspections, repairs, operator notes, manuals) in directories of JSONL files. Rather than a RAG pipeline, I wanted an agent that could query the data, decide which operations it needed, and answer with a visual.

I built it in LangGraph as a ReAct loop: for each kind of question, I broke my manual approach into small deterministic operations and exposed each as a tool. Evals came along too: questions with expected answers, scored with DeepEval.

A handwritten notebook page. Under “Q1. WELL-0001” a list of steps: find 0001, read metadata, find well type, load welltype T200.md, find inspections, latest inspection, find the temperature delta, also read notes, check the last inspection, find inspected at, by user id and method. Below, “Q5 – how many days was WELL-0001 down?” with steps: find 0001, read, T200, inspections in 2025, how many, go to the dataset guide to find the number of days of downtime. My notes for two sample questions, each broken into steps small enough to become tools.

Working around a small model

Gemma 4 was capable, but I couldn’t trust it with arithmetic or dates, and in a multi-step workflow one slip spoils the answer.

So I kept moving reasoning out of the model and into tools. One final question asked whether every gas report got a response in time: within 24 hours before 1 January 2027, and 12 hours after. A gas_reports tool did the matching, the hours and the deadlines. The model only presented the rows: 26 reports, 9 missed.

Sometimes the fix was to take a capability away. When I gave a tool a switch to count workover-rig waits separately, the model flipped it and gave the wrong breakdown. With a small model, every option you expose is another way to be wrong.

I also stopped trusting the final answer. A plain function, not a second model, checked that every ID and number in it came from a tool result, and sent it back once if not:

# verify.py, trimmed
unsupported = [v for v in values if not _matches(v, evidence_numbers)]
if unsupported:
    problems.append(f"these numbers are in no tool result: {unsupported}. "
                    "Take every number from a tool result; "
                    "do not compute sums or averages yourself")

Flow diagram. A question goes to Gemma 4, which picks a tool. The deterministic tool handles dates, thresholds and SOP rules, saves its rows under a result ID, and returns them to Gemma. Gemma writes a draft answer. A checker written in code asks whether every ID and number is in a tool result. If yes, the answer goes out with citations and a visual. If no, the draft goes back to Gemma to be rewritten once. The model picks tools and writes. Code does the arithmetic and the checking.

Less agent, more space monkey

The evals made progress measurable. A retrieval failure meant the fact wasn’t in any tool result, so I fixed the tool. A generation failure meant the fact was there but missing from the answer, so I fixed the prompt. Most failures were retrieval. Over the afternoon the pass rate went from 4 to 15 of 23 cases. But look at where the arrows go:

Loop diagram. 23 eval cases, each with expected IDs, numbers, visual and rubric, feed into running the agent. The run is scored on fact recall, visual match and an LLM judge. Failures are labelled retrieval, generation or wrong visual. Every label goes to “me”, and I change a tool or add a prompt rule before the agent runs again. Every path back to the agent went through me.

The more failures I dug into, the more I over-engineered the tools, until almost every final question had a tool shaped exactly like it: gas_reports, bypasses, regulator_letter_spills, status_conflicts, recalled_pumps. I was making my agent less agentic. It wasn’t driving any more. It was a space monkey,1 working the one lever it had been trained to work. The answers were right, but it wasn’t really an agent.

The UI

The other half of the brief was visualization, and I treated it the same way: as something to make correct rather than something to explore.

I started with a Streamlit app at two in the afternoon and replaced it 40 minutes later with a Next.js chat built on assistant-ui, talking to the LangGraph server. A “Talking as” picker set the user, so the copilot knew who was asking and kept each person’s threads and memory separate.

Visuals came from a fixed catalog of seven: status card, table, line, bar and stacked bar charts, map and heatmap. The model never sent data. It called show_visual with a type, a title, column names and the ID of a tool result, and the server filled in the rows. A bad spec came back to the model as an error to fix, and the UI showed “Visual not shown” rather than a broken chart.

The copilot answering “Is WELL-0001 producing oil right now? When was it last inspected, and by whom? Show me its status card.” A collapsed line reads “2 tool calls”. The status card shows WELL-0001 as Producing, type T200 in Southwest Block 23, a temperature delta of 28.5 °C against a 15 °C threshold, last inspection INS-0001-0216 on 2027-01-14 by Sarah Mitchell by drone, and four recent operator notes. Its header reads “1 rows · r_3900”. Below it the answer says WELL-0001 is producing and was last inspected on 2027-01-14 by Sarah Mitchell using a drone. Q1 from my notebook, answered with a status card filled from tool result r_3900. The model never typed a number into it.

Every chart matched the database. But it was a rendering protocol, not a dynamic interface: the model could only pick from components I’d built. The same space-monkey pattern again.

How the day went

Chart of the share of questions answered correctly through the day, above a timeline of ten numbered checkpoints. Blue line: the eval-set score starts at 3 of 10 at 13:25, dips to 4 of 23 at 15:37 when an AI judge and all 23 cases are added, then rises to 6 of 23 at 15:58 and 15 of 23 at 16:25. From 16:15 the background is shaded: test data out, panic mode. There are no eval runs after that. An orange diamond at 18:06 marks the final questions on the test dataset: 10 of 11 fully correct, 1 partly. Scores and times from the eval history and the git log.

The test dataset landed at about 4:15. By 5 I was in panic mode, building tools for data the copilot had never seen. It finished with 10 of 11 final questions fully correct.

The harness I should have borrowed

Another participant started from Pi, an existing open-source agent harness, and adapted it to the problem. That was my aha moment of the day. I’d spent hours on plumbing (the loop, data access, file and database tools) that his harness already had. An extensible harness also lets an agent’s capabilities grow without someone hand-designing every operation. It freed him to work on the interesting half: AG-UI and dynamic interfaces.

The irony is that I’d already drawn this line well for my own work. I built the copilot with about 20 parallel coding-agent workspaces: 28 pull requests between noon and 6 pm. I was happy to borrow infrastructure to write the code. I just didn’t think to do it for the agent.

Next time

I came away runner-up in the overall competition, with a special prize for the best use of open-source AI. Probably because nobody was more troubled by Gemma 4’s capabilities than I was. The haul: two Hacktoberfest T-shirts, two Arduino Unos, loads of stickers and a wireless keyboard.

A black Portronics wireless keyboard with round keys, a folded green Hacktoberfest T-shirt with a colourful pixel pattern, and two boxed Arduino UNO Q boards, laid out on a marble counter. Part of the haul. The stickers didn’t make the photo.

Next time I’d start from an extensible harness and spend the day on evals, model limits and the interface. I still think handing unreliable reasoning to deterministic tools is sound. What I’d change is where I draw the line between infrastructure I build and infrastructure I borrow.

Footnotes

  1. In Fight Club, Tyler Durden’s recruits are “space monkeys”: each one does the small job he’s trained to do, without needing to understand any of it. ↩