Haskell for all: A sufficiently detailed spec is code

Tell your favorite coding agent to build Symphony in a programming language of your choice:

Implement Symphony according to the following spec: https://github.com/openai/symphony/blob/main/SPEC.md

I asked Claude Code to build Symphony in a programming language of my choice (Haskell2, if you couldn't guess from the name of my blog) and it did not work. You can find the result in my Gabriella439/symphony-haskell repository.

Not only were there multiple bugs (which I had to prompt Claude to fix and you can find those fixes in the commit history), but even when things "worked" (meaning: no error messages) the codex agent just spun silently without making any progress on the following sample Linear ticket:

Create a new blank repository

No need to create a GitHub project. Just create a blank git repository

In other words, Symphony's "vain attempt at verbal precision" (to use Dijkstra's words) still fails to reliably generate a working implementation3.

This problem also isn't limited to Symphony: we see this same problem even for well-known specifications like YAML. The YAML specification is extremely detailed, widely used, and includes a conformance test suite and the vast majority of YAML implementations still do not conform fully to the spec.

Symphony could try to fix the flakiness by expanding the specification but it's already pretty long, clocking in at 1/6 the size of the included Elixir implementation! If the specification were to grow any further they would recapitulate Borges's "On Exactitude in Science" short story:

…In that Empire, the Art of Cartography attained such Perfection that the map of a single Province occupied the entirety of a City, and the map of the Empire, the entirety of a Province. In time, those Unconscionable Maps no longer satisfied, and the Cartographers Guilds struck a Map of the Empire whose size was that of the Empire, and which coincided point for point with it. The following Generations, who were not so fond of the Study of Cartography as their Forebears had been, saw that that vast Map was Useless, and not without some Pitilessness was it, that they delivered it up to the Inclemencies of Sun and Winters. In the Deserts of the West, still today, there are Tattered Ruins of that Map, inhabited by Animals and Beggars; in all the Land there is no other Relic of the Disciplines of Geography.

Slop

Specification work is supposed to be harder than coding. Typically the reason we write specification documents before doing the work is to encourage viewing the project through a contemplative and critical lens, because once coding begins we switch gears and become driven with a bias to action.

So then why do I say that this is a misconception:

Misconception 2: specification work must be more thoughtful than coding work

The problem is that this sort of thoughtfulness is no longer something we can take for granted thanks to the industry push to reduce and devalue labor at tech companies. When you begin from the premise of "I told people specification work should be easier than coding" then you set yourself up to fail. There is no way that you can do the difficult and uncomfortable work that specification writing requires if you optimize for delivery speed. That's how you get something like the Symphony "specification" that looks superficially like a specification document but then falls apart under closer scrutiny.

In fact, the Symphony specification reads as AI-written slop. Section 10.5 is a particularly egregious example of the slop I'm talking about, such as this excerpt:

linear_graphql extension contract:

32dae5fa5a7e8149b533f14cdc500909

Purpose: execute a raw GraphQL query or mutation against Linear using Symphony's configured tracker auth for the current session.

32dae5fa5a7e814aaa69cc21684fca74

Availability: only meaningful when tracker.kind == "linear" and valid Linear auth is configured.

Preferred input shape:

Copy

{
  "query": "single GraphQL query or mutation document",
  "variables": {
    "optional": "graphql variables object"
  }
}

32dae5fa5a7e815eb7c0db21fe423529

query must be a non-empty string.

32dae5fa5a7e8141b59deb4e169e8b9e

query must contain exactly one GraphQL operation.

32dae5fa5a7e81379007ebb04a35a389

variables is optional and, when present, must be a JSON object.

32dae5fa5a7e8114af75f292b0f47fab

Implementations may additionally accept a raw GraphQL query string as shorthand input.

32dae5fa5a7e8157b3def4e1088123e7

Execute one GraphQL operation per tool call.

32dae5fa5a7e8190be9dc2a1cf8d0923

If the provided document contains multiple operations, reject the tool call as invalid input.

32dae5fa5a7e818a8158f7877658752a

operationName selection is intentionally out of scope for this extension.

32dae5fa5a7e81b19d0bf95d71c2cfea

Reuse the configured Linear endpoint and auth from the active Symphony workflow/runtime config; do not require the coding agent to read raw tokens from disk.

Tool result semantics:

1. transport success + no top-level GraphQL errors -> success=true
2. top-level GraphQL errors present -> success=false, but preserve the GraphQL response body for debugging
3. invalid input, missing auth, or transport failure -> success=false with an error payload

32dae5fa5a7e810ca218e1590cdf5977

Return the GraphQL response or error payload as structured tool output that the model can inspect in-session.

That is a grab bag of "specification-shaped" sentences that reads like an agent's work product: lacking coherence, purpose, or understanding of the bigger picture.

A specification document like this must necessarily be slop, even if it were authored by a human, because they're optimizing for delivery time rather than coherence or clarity. In the current engineering climate we can no longer take for granted that specifications are the product of careful thought and deliberation.

Conclusion

Specifications were never meant to be time-saving devices. If you are optimizing for delivery time then you are likely better off authoring the code directly rather than going through an intermediate specification document.

More generally, the principle of "garbage in, garbage out" applies here. There is no world where you input a document lacking clarity and detail and get a coding agent to reliably fill in that missing clarity and detail. Coding agents are not mind readers and even if they were there isn't much they can do if your own thoughts are confused.

Footnotes

32dae5fa5a7e812784ebce7def134fda

If you're wondering why the code snippet isn't highlighted that's because I'm preserving the GitHub-flavored markdown exactly as I found it in the original document. All the code snippets are explicitly annotated as text (one of many signs that this document is AI-generated). In fact, this is likely an example of the model obeying the letter rather than the spirit of the request. I'm pretty sure what happened is the human asked the AI to convert an initial draft of the project to a prose specification and the AI decided that labeling the code snippets as text made them more like prose rather than code. ↩

32dae5fa5a7e813da96ae61337109b15

People often tell me "you would get better results if you generated code in a more mainstream language rather than Haskell" to which I reply: if the agent has difficulty generating Haskell code then that suggests agents aren't capable of reliably generalizing beyond their training data. ↩

32dae5fa5a7e8144878ce235c3d8bf56

You can also try this same exercise yourself if you think I held it wrong. ↩