llm generated agent.md files are harmful. handwritten ones on average don’t have an effect on ability to complete a task
thread article:
agents keep signing off on commits wrong. we added it to the agents.md and now they don’t
benchmarks notoriously don’t measure the likelihood that the results would be merged by maintainers, just that they passed a test
the old style “repo map” agents.md files that every tool used to create is no longer useful, but of course things that would otherwise be repeated prompts are still good. I cut the ones at work down from hundreds of lines each to tens. without those last few lines the amount of iteration I do - both with agent and when reviewing PRs - goes up significantly
your research paper:
thread article:
benchmarks notoriously don’t measure the likelihood that the results would be merged by maintainers, just that they passed a test
the old style “repo map” agents.md files that every tool used to create is no longer useful, but of course things that would otherwise be repeated prompts are still good. I cut the ones at work down from hundreds of lines each to tens. without those last few lines the amount of iteration I do - both with agent and when reviewing PRs - goes up significantly