How would they know what superior code is? They're trained on all code. My expectation and experience has been that they write median code in the best-case scenario (small greenfields projects, deeply specified, etc).
How do you determine flawlessness? How you even approximate a guarantee of it? To what specification is flawlessness judged, and can you precisely and completely relay that spec to your friendly local robot more efficiently than it can vendor / import existing libraries?
> This is a good thing. I don’t need to focus on oil refineries when I fill my car with gas. I don’t know how to run a refinery, and don’t need to know.
No, it isn't. If you do not know what the project does, you are unable to even review the changes proposed by a coding agent.
Consequently, we already have a long track record of LLMs introducing vulnerabilities.
LLMs are great at speeding up small tasks but the trade-off is that developers become progressively clueless.
Next iterations of models will have to deal with that code, and it would be harder and harder to fix bugs and introduce features without triggering or introducing more defects.
Biological evolution overcomes this by running thousands and millions of variations in parallel, and letting the more defective ones to crash and die. In software ecosystems, we can't afford such a luxury.
An example: it had a complete interface to a hash map. The task was to delete elements. Instead of using the hash map API, it iterated through the entire underlying array to remove a single entry. The expected solution was O(1), but it implemented O(n). These decisions compound. The software may technically work, but the user experience suffers.
If you have particular performance requirements like that, then include them. Test for them. You still don’t have to actually look at the code. Either the software meets expectations or it doesn’t, and keep having AI work at it until you’re satisfied.
How deep do you want to go? Because reasonable person wouldn't have expected to hand hold AI(ntelligence) to that level. Of course after pointing it out, it has corrected itself. But that involved looking at the code and knowing the code is poor. If you don't look at the code how would you know to state this requirement? Somehow you have to assess the level of intelligence you are dealing with.
Since the code does not matter, you wouldn’t need or want to phrase it in terms of algorithmic complexity. You surely would have a more real world requirement, like, if the data set has X elements then it should be processed within Y milliseconds. The AI is free to implement that however it likes.
Even if you specify performance ranges for every individual operation, you can’t specify all possible interactions between operations.
If you don’t care about the code you’re not checking in the code, and every time you regenerate the code you’re going to get radically different system performance.
Say you have 2 operations that access some data and you specify that each can’t take more than 1ms. Independently they work fine, but when a user runs B then A immediately, there’s some cache thrashing that happens that causes them to both time out. But this only happens in some builds because sometimes your LLM uses a different algorithm.
This kind of thing can happen with normal human software development of course, but constantly shifting implementations that “no one cares about” are going to make stuff like this happen much more often.
There’s already plenty of non determinism and chaos in software, adding an extra layer of it is going to be a nightmare.
The same thing is true for every single implementation detail that isn’t in the spec. In a complex system even implementation details you don’t think you care about become important when they are constantly shifting.
That's assuming no human would ever go near the code, and that over time it's not getting out of hand (inference time, token limits are all a thing), and that anti-patterns don't get to where the code is a logical mess which produces bugs through a webbing of specific behaviors instead of proper architecture.
However I guess that at least some of that can be mitigated by distilling out a system description and then running agents again to refactor the entire thing.
> However I guess that at least some of that can be mitigated by distilling out a system description and then running agents again to refactor the entire thing.
The problem with this is that the code is the spec. There are 1000 times more decisions made in the implementation details than are ever going to be recorded in a test suite or a spec.
The only way for that to work differently is if the spec is as complex as the code and at that level what’s the point.
With what you’re describing, every time you regenerate the whole thing you’re going to get different behavior, which is just madness.
You could argue that all the way down to machine code, but clearly at some point and in many cases, the abstraction in a language like Python and a heap of libraries is descriptive enough for you not to care what’s underneath.
The difference is that what those languages compile to is much much more stable than what is produced by running a spec through an LLM.
Python or a library might change the implementation of a sorting algorithm once in a few years. An LLM is likely to do it every time you regenerate the code.
It’s not just a matter of non-determinism either, but about how chaotic LLMs are. Compilers can produce different machine code with slightly different inputs, but it’s nothing compared to how wildly different LLM output is with very small differences in input. Adding a single word to your spec file can cause the final code to be unrecognizably different.
And that is the right assumption. Why would any humans need (or even want) to look at code any more? That’s like saying you want to go manually inspect the oil refinery every time you fill your car up with gas. Absurd.
Cars may be built by robots but they are maintained by human technicians. They need a reasonable layout and a service manual.
I can’t fathom (yet) having an important codebase - a significant piece of a company’s IP - that is shut off to engineers for auditing and maintenance.
It’s not about optimizing for performance, it’s about non-deterministic performance between “compiler” runs.
The ideal that spec driven developers are pushing towards is that you’d check in the spec not the code. Anytime you need the code you’d just regenerate it. The problem is different models, different runs of the same model, and slightly different specs will produce radically different code.
It’s one thing when your program is slow, it’s something completely different when your program performance varies wildly between deployments.
This problem isn’t limited to performance, it’s every implicit implementation detail not captured in the spec. And it’s impossible to capture every implementation detail in the spec without the spec being as complex as the code.
I don’t think it is possible to solve without AGI. I think LLMs can augment a lot of software development tasks, but we’ll still need to understand code until they can completely take over software engineering. Which I think requires an AI that can essentially take over any job.
Testing is not a proof a software system is correct. Also, if tests are generated as well, there's no trust in how anything works or if the tests are covering important aspects.
And there was a time when using libraries and frameworks was the right thing to do, for that very reason. But LLMs have the equivalent of way more experience than any single programmer, and can generate just the bit of code that you actually need, without having to include the whole framework.
As someone who’s built a lot of frontend frameworks this isn’t what I’ve found. Instead I’ve found that you end up with the middle ground choice which while effective is no better than the externally maintained library of choice. The reason to build your own framework is so it’s tailor suited to your use cases. The architecting required to do that LLMs can help with but you have to guide them and to guide them you need expertise.
I would like a more reliable way to activate this "way more experience."
What I see in my own domain I often recognize as superficially working but flawed in various ways. I have to assume the domains I am less familiar are the same.
> can generate just the bit of code that you actually need
Design is the key. Codebases (libraries and frameworks not exempt,) have a designed uniformity to them. How does a beginner learn to do this sort of design? Can it be acquired completely by the programmer who uses LLMs to generate their code? Can it be beneficial to recognize opinionated design in the output of an LLM? How do you come to recognize opinion?
In my personal history, I've worked alongside many programmers who only ever used frameworks. They did not have coding design sensibilities deeper than a social populist definition of "best practice." They looked to someone else to define what they can or cannot do. What is right to do.
reply