So do you, like, work at an actual company? One with actual customers and business partners, who need deliverables finished by deadlines that may be months or years out, and who need up-to-date info on whether the project is on track for completion or not? Companies are made or broken, billions earned or lost, because of these schedules and timelines. So the pure jewel version of Agile is incompatible with the business world. Agile is only accepted by business inasmuch as it promises better visibility and control over schedules and costs.
I get that the "pure jewel" version of Agile was some programmers attempting to do the Eric S. Raymond "package our rebellious hacker values in a way that's acceptable by corporate management and subvert the organization from within" thing. But what corporate management found was that it's an ill fit unless adapted to business needs. And even then, what Milt Bryce proved in the frickin' 70s was that for organizational information systems, BDUF actually saves you time and money in the long run. Agile was the standard practice in Bryce's time: almost no planning, analysis, or design, let the programmers do what they want, make changes piecemeal as users got frustrated with the system. His PRIDE methodology (the first software methodology to be marketed as such) was positioned as an alternative and involved lots of analysis and design cycles before a single line of code is written.
Don't know about the OP, but I do work in a large enterprise co that claims to be "agile". And it suffers for exactly the reasons brought up by the OP.
A recent project was being developed by the "product organization" for 18 months as pure slideware that never went anywhere (included Product Managers, Designers and Software Architects). When engineers were finally made accountable (partially on me pushing to do so lest it remain slideware), they were able to throw almost all of it away and ship something of value in 3 months. Of actual business and customer value. The organization still opposes involving them from the start and makes them mostly waste time filling up useless JIRA tickets so someone (not sure who) can claim it's all tracked and recorded.
Product Owners and Product Managers are generally mostly a hindrance: they are there to stop work from happening, or when it happens, to stop it from shipping because it's missing this and that. They claim to be agile, but can't react to any reality check nor figure out to iteratively ship something.
And they believe they are to be the source of all ideas, with engineers there mostly to estimate work and then build it according to that estimation.
True agile produces software of higher quality faster than any other approach. Unfortunately, corporations don't buy that and instead write hundreds of pages of docs on how to do agile their way.
Tim Bryce’s work, such as his “Is PRIDE too rigid?” article, uses analogies with traditional engineering (like jet engine design) to argue that skipping critical design steps can lead to costly rework later, but it stops short of offering rigorous empirical proof or extending the idea to the realm of software.
"Big Design Up Front" in software is the whole reason the Agile movement came about. To put BDUF into Agile is to miss the point completely--they are fundamentally incompatible. Any company that insists on "lots of analysis and design cycles before a single line of code is written" will be left in dust by competitors if all their programmers don't quit first.
Oh, and the Agile "movement" of the early 2000s was really nothing more than angry programmers looking to return to the good old days when management was so mystified by the complexity of the machine or the software it ran that programmers could do what they wanted, absent a reasonable semblance of discipline, organization, or accountability.
Tim Bryce also told the story of how he took a COBOL course in the 70s. While the programmers went straight to the keypunch and encountered numerous compiler errors, requiring recompiles, Tim pulled out his "trusty template", actually flowcharted the solution first, and then wrote the code, and it compiled and ran first time, every time. He said "That course taught me more about programmers than it did about programming."
Tim's father Milt explains the industrial approach behind PRIDE here:
Industrial products go through phases of specification, analysis, design, development, implementation, and review, in that order. Can you give me a reason why information systems (which, if you'd been paying attention, are much more than just software) shouldn't be the same? "Agile" sounds like skipping to implementation before the analysis or design have been completed -- going straight to the keypunch before you've documented and flowcharted exactly what you want to achieve. Of course what you're going to get is a buggy incomplete mess. But Agile says "ship the buggy incomplete mess and use your users' feedback to iterate on it!" That just annoys the users, as they just want something that works, but they have to go through many cycles of not-working software to get there.
With analysts (who are emphatically NOT programmers) in the loop, employing a structured, disciplined approach to systems design and implementation, the requirements can be clarified before a single line of code is written, leading to a better result the first time, saving the organization time, money, and frustration. It's just like building a car, or a skyscraper. You need to start with a clear design so the builders know what to build. Anything else is wasting time, money, and effort.
Another way of putting it is: Agile in its original form serves programmer needs while PRIDE serves management needs. Who actually runs the organization? Who has a big-picture view of what the organization's information needs are? Hint: It's not the programmers.
Relevant Bryce's Laws:
"Remember, it's ready, aim, fire; any other sequence is counterproductive."
"A system is a product that can be engineered and manufactured like any other product."
I get that the "pure jewel" version of Agile was some programmers attempting to do the Eric S. Raymond "package our rebellious hacker values in a way that's acceptable by corporate management and subvert the organization from within" thing. But what corporate management found was that it's an ill fit unless adapted to business needs. And even then, what Milt Bryce proved in the frickin' 70s was that for organizational information systems, BDUF actually saves you time and money in the long run. Agile was the standard practice in Bryce's time: almost no planning, analysis, or design, let the programmers do what they want, make changes piecemeal as users got frustrated with the system. His PRIDE methodology (the first software methodology to be marketed as such) was positioned as an alternative and involved lots of analysis and design cycles before a single line of code is written.