Zen and the Art of AI-Assisted Development

There's been something missing for me from the majority of conversations around AI-assisted development: a focus on quality. I don't mean quality of software output (that's important too). I mean using the broad idea of quality to centre and focus our discussions around AI-assisted development, and perhaps AI and automation in general.

There's also been so much fear-mongering around recursive self-improvement (RSI) towards artificial general/superintelligence (AGI/ASI) - but what does "improvement" actually mean here? "Movement towards greater quality" seems to me to be a reasonable understanding.

But what is quality?

Robert Pirsig had a fair amount to say on the topic in Zen and the Art of Motorcycle Maintenance. In fact he went even further to define a metaphysics of quality (MoQ) in the lesser-known sequel to ZAMM, Lila.

If I take Pirsig's view on quality, assuming it reflects how reality actually works, I can make a few conjectures:

1. Full technological automation is only really possible for cases where what constitutes "quality" can be precisely defined and measured, or where quality doesn't matter.
2. We'll always need humans in the loop to act as LLMs' output quality control.
3. RSI towards AGI/ASI is impossible.

To understand why, we'd need to understand a little more of Pirsig's ideas.

Pirsig on Quality

Pirsig argues in ZAMM that quality precedes thinking, is neither subjective nor objective, and is ultimately the generator of subjects and objects.

To attempt to illustrate by way of an example: let's say a person wants to ride a motorbike, but the motorbike's not working. Is the "low quality" of the situation in the subject, or the object?

Many people would say it's totally subjective, because some people would not care about the motorbike, and some would even be happy about it not working (e.g. if they could sell it for scrap). What if, however, they want to ride the motorbike, replace a broken part, and it starts working? The object changed, not the subject, and the situation became higher-quality for the subject.

Pirsig effectively says that it's the wrong question. A better question is, what drew the subject to want to ride the motorbike, and to fix it, in the first place?

This sort of dynamic is happening all over the place at various levels of detail, from the microscopic to the galactic. So much so that Pirsig concludes that what we intuitively understand to be quality is actually the mysterious driving force behind, and guiding hand of, evolution itself.

Quality and AI

Are AI agents - harnesses driven by LLM output - capable of detecting quality in the same way that we can?

Putting aside theoretical arguments, just pragmatically I've found that they're not capable of detecting quality in the same ways that I do.

The way I typically work with agents is to use a "work back and forth with me" skill, similar to Matt Pocock's grilling or wayfinder skills. Even with the latest frontier models, I frequently have to choose against or suggest entirely different options to those that the LLM proposes, and even suggest entirely different questions to what it comes up with. In other words, my own sense of quality frequently diverges from that of the LLMs'.

A more reliable way to coerce the agent into producing something even vaguely resembling a quality result is to tightly constrain its operating parameters: define hard pass/fail tests against which its outputs end up being measured. But constraining an agent, a computer program, is not the same thing as it sensing quality.

A looser evaluation is to have someone manually review the agents' output. Looser than that, and we're in full on vibe-coding territory - superficially it might look good, but under the hood it might be a total mess that will come back to haunt its creator later on.

This is why I say that we'll never genuinely be able to remove humans from the software development process. We are the agents' quality detection apparatus. We only think they can detect quality because we love to anthropomorphize them, and they're very convincing mimicks of quality detectors.

Recursive Self-Improvement

Some research has been done on RSI, and so far it looks like it's entirely possible when the goals are very clearly articulated (think: benchmaxxing). See El's excellent analysis of this.

However, what does it even mean to just "improve" towards "intelligence"? What even is "intelligence"? How can we recursively self-improve towards something we can't define? How do we know whether we're "getting there", and whether what's being built isn't just a convincing illusion?

As I understand it, intelligence is closely related to the ability to sense quality. The greater the nuance in one's ability to sense quality, the greater the intelligence. We seem to call incredibly nuanced intelligence "wisdom" - intelligence with depth, that considers many steps ahead and/or the context at varying degrees of detail.

If that's true, and agents are fundamentally incapable of detecting quality, the idea that AI can recursively self-improve not only towards intelligence, but superintelligence, is essentially magical thinking.

Automation and Quality

It seems as though investors have pinned their hopes, and perhaps the fate of the global economy, on the short-sighted idea that AI can automate most, if not all, human work. For that to happen, we would need fully aligned AGI that's cheaper than humans for all human work.

If Pirsig's ideas regarding quality are true, is that even possible with LLMs? Is that even possible with any non-biological technology that we could possibly create?

From what I understand of automation, it's most successfully applied to areas where the quality of its outputs can be automatically and precisely assessed.

Think of factories producing the same parts over and over again. Each part has very specific tolerances for its properties, and exceeding any of those tolerances can be considered a defect. Much of the time, assesment of those properties can be automated too, so automated QA is possible.

Now think of writing software, or prose, and how different those activities are from factories producing parts. What automated measures are there to assess the quality of any piece of software, or any prose? We can only ever hope to build specific measures of quality for specific software (e.g. by way of tests). The measures that apply to one system won't apply to others. And who comes up with those measures in the first place? Who determines the questions to ask that drive those measures, and where do those questions come from?¹

This would mean that the idea of a fully automated "software factory" is DoA (if quality matters, of course).

How do we retain our wisdom?

This is going to be one of the biggest challenges that the tech industry's going to face in the coming years. Given how easy it is to defer quality judgements to LLM-based coding agents (the "easy button"), junior developers can opt to never exercise their problem solving skills enough to develop the wisdom to be able to assess quality. Senior developers can be tempted to go for the easy button to the extent that their quality assessment skills rot over time.

Assuming we don't destroy ourselves as a species, I'd bet that the requisite wisdom to assess software quality is going to be even more important in 3 to 5 years' time than ever before.