Implicit to this discussion is the belief that growth inevitably means more people, and that the collateral damage of more people is worth the cost of spending so much time to corral said people. So industry clings to frameworks with a single blessed path, languages that provide only one way to do things, caps expressiveness (just mention macros and people start frothing from the mouth).

This isn't easily fixable, but I'd like technologists to at least be able to perceive the extent to which the surrounding business culture has permeated into technical culture. I'm probably considered old AF by a lot of you (41) but I'm becoming increasingly interested in tools/methodologies that enable fewer devs to do a lot more, even if it means that there are sharp edges.

Fewer devs doing a lot more has always been my goal, and in hardware we would call this a vertical scaling versus horizontal scaling problem.

Horizontal scaling always incurs communication overheads faster, and as we know, developers have high communication overhead to begin with.

Human scaling is its own problem set and companies like to ramp projects up and then down again. Most of those people only think about how to get more people on board, not how to run the ship once attrition starts to condense responsibilities and reduce redundancy, especially for unpleasant tasks.

I’ve been trying to beat the idea of ergonomics into a smart but mule-headed developer (too old not to know better) for years now and it wasn’t until a combination of team shrink and having to work with code written by a couple of people who emulate his development style that all of a sudden he’s repeating things I said four years ago (of course with no self awareness of where those ideas came from). It remains to be seen if it affects how he writes code, or whether he starts practicing Do As I Say, Not As I Do, but I’m a short-timer now so it’s mostly an anthropological study at this point.

It seems natural and correct to me that the business concerns and technical concerns would be mixed together. Engineers should understand the business needs of the organization and account for them, just as leadership should understand technical needs.

As projects get larger, the problems become more organizational in nature.

Conways law: Tech follows communication patterns (.i.e org setup)

hence, if you want a certain architecture, you likely need to execute a "reverse conway law" org first. To get the org into the target config, the software will follow.

It’s okay for one team to have several logical units of code. In fact from a Conway First model, it’s helpful because it allows horse trading when it turns out the team competencies don’t match responsibilities very well. I’m not dismantling my team, I’m just transferring module ownership to another team.

What does not work is splitting ownership of many tings for a long time. It’s too many plates to spin. But some things can be shared by a few people without tipping over into anarchy.

Conway's law is an adage that states organizations design systems that mirror their own communication structure. It is named after the computer programmer Melvin Conway, who introduced the idea in 1967. His original wording was:

  Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.

  — Melvin E. Conway

https://en.wikipedia.org/wiki/Conway's_law

> My take: do Microservices all you want, but don't organize teams around the services!

Exactly. In order to reap the benefits of modular/Microservices architecture, you need teams to be organized around product/feature verticals.

IMO it’s not about the internals of a product/service architecture, but about encapsulating, isolating, and formalizing inter-organizational dependencies.

Organizing teams around microservices makes ... a lot of sense?

Also - talking of redesign should be a major lift for any product/service that has real paying clients. The risk of breaking is huge, and the fact that during that time you won't be delivering incremental value is also going to look bad.

One of the principal engineers at Amazon called that “Org-atecture”.

Amazon is a good example of a huge organization absolutely crippled by micro-service fiefdoms driven by sr.mgr whims and not strong technical guidance from such PEs.

Depends entirely on the org. My org actually broke SOA (thousands of services and some intense technical debt) and is now in a years long process of moving toward a centralized service orchestration model. This couldn’t have happened without PE, Sr PE and DE involvement.

Now that sounds interesting. I assume you're not counting each API endpoint as a service, so can you shed any more light on this? The scale sounds mind-boggling. Thousands of separate services being pulled (more) together. Can you give an idea of the size or scope of these services?

>1000 services, each with many APIs :)

Amazon's warehouses have incredible domain complexity which drives the complexity of the software architecture. It's also one of the oldest parts of the company with a direct lineage going back 20+ years. (for example, our inventory databases still have the same schema and are still using a RDBMS, albeit RDS Postgres instead of Oracle).

About five years ago we started rearchitecting to a centralized orchestration service which is being used to abstract this complexity from process teams. This is to avoid projects where 20+ service teams must be engaged just to launch a new solution.

And look how much damage it's done to their business.

Without the organizational benefits of a microservice you are just just making intermodular calls really slow, and fail intermittently.

> What happens if you need to redesign the architecture to meet new needs?

How often does major reorganization happen? For most companies the answer is never.

The "what if" argument is what leads to all sorts of premature optimizations.

I've had three reorgs in the past six months. Each one changed our projects and priorities.

Yeah, didn't get a lot done...

> How often does major reorganization happen? For most companies the answer is never.

For most companies – regularly.

The kind of reorg we're talking about here is not the typical musical chairs, but a full realignment. This happens a lot less frequently in my experience.

reorgs usually happen higher up the tree in larger companies, been through a bunch and its never affected any part of our work.

Reorgs rarely happen.

But the need for them does :)

> but don't organize teams around the services

This is actually the whole point (see: Reverse Conway).

indeed, you might as well build a distrubuted monolith if your priority is absolute flexebility and ability to re-design on the fly

"This has been my major criticism of them; you cement the design of the app by creating an organizational structure around your software components."

Conway's Law is basically "You don't have a choice." You will cement the design of your app around your organizational design. (At least, beyond a certain org size. If you have only 4 engineers you don't really have the sort of structure in question here at all.) je42 narrowly beats me to the point that if this is a problem you can try to match your organization to your problem, but that takes a fairly agile organization.

"What happens if you need to redesign the architecture to meet new needs? That's right; it's not going to happen, because people will fight it tooth and nail."

Unfortunately, in the real world, this is not so much "a disadvantage to the microservice approach" as simply "an engineering constraint you will have to work around and take account in your design".

Despite what you may think after I've said that, I'm not a microservice maximalist. Microservices are a valuable tool in such a world but far from the only tool. As with everything, the benefits and the costs must be accounted for. While I've not successfully rearchitected an entire organization from my position as engineer, I have had some modest, but very real success in moving little bits and pieces around to the correct team so that we don't have to build stupid microservices just to deal with internal organizational issues. I don't mean this to be interpreted as a defeatist "you're doomed, get ready for microservices everywhere and just deal with it"; there are other options. Or at least, there are other options in relatively healthy organizations.

But you will have code that matches the structure of your organization. You might as well harness that as much as you can for the benefits it can provide because you're stuck with it whether you like it or not. By that I mean, as long as you are going to have teams structured around your services whether you like it or not, go in with eyes open about this fact, and make plans around it to minimize the inevitable costs while maximizing the benefits. Belief that there is another option will inevitably lead to suboptimal outcomes.

You can't engineer at an organizational level while thinking that you have an option of breaking the responsibility and authority over a codebase apart and somehow handing them out to separate teams. That never works long term. A lot of institutional malfunctioning that people correctly complain about on HN is at its core people who made this very mistake and the responsibility & authority for something are mismatched. Far from all of it, there are other major pathologies in organizations, of course. But making sure responsibility & authority are more-or-less in sync is one of the major checklist items in doing organization-level engineering.

You never want code with shared ownership to tolerate feature creep. That’s impossible to keep on the rails. If you’re going to use SOLID anywhere, it’s in shared code.

If your org doesn’t suffer feature creep willingly, I believe that means you can have a lot more flexibility with respect to Conway’s Law. A low-churn project can maintain a common understanding. Not all of them will of course, but it’s at least possible. Feature factories absolutely cannot, and you shouldn’t even try.

What really kills a lot of projects though is an inverted dependency tree. And by inverted I mean that the most volatile code is at the bottom of the call graph, not the top. In this case every build, every deploy, or in the microservices scenario every request, can have different behavior from the previous one because some tertiary dependency changed under you. Now you need all sorts of triggers in your CI/CD pipeline to guarantee that integration tests are rerun constantly to figure out where the regressions are being introduced. Your build pipeline starts to look like one of those server rooms with loose wires everywhere and a bucket under the AC unit. Nothing is stable and everything is on the verge of being on fire at a moment’s notice.

I mentioned this elsewhere, but I think it's a good idea to read the actual text of conways law:

> Any organization that designs a system (defined broadly) will produce _a design whose structure is a copy of the organization's communication structure_.

Organization is the primary method by which we organize communication (indeed, that's its only point). That's why architecture follows organization. It is possible to "break" that boundary by organizing other forms of communication, for example the feature teams of SAFE (not a proponent of safe, just an example).

This is the "reverse Conway's law" that some others are referring to, and I alluded to. I've had success doing it in small places here and there. I'm yet to convince a management organization to do a really large-scale reorg because of some major pathology, though. Though I haven't tried often, either. Really just once. The costs are automatically so high for such a proposal that it is difficult for mere engineering needs to get to net-positive because it's hard to prove the benefits.

Part of the issue is that not everyone understands the trade-offs or perhaps believes the trade-offs so you get teams/leaders haphazardly ignoring expert guidance and crafting careless micro-services around team boundaries that do not line up with value.

I find the responsibility & authority angle is very helpful in these conversations. It doesn't take a lot of convincing (in a reasonably healthy organization) to say that these should be broadly aligned, and that if you can show people a place where they are either currently grossly out of sync, or are going to be out of sync if we move forward with some particular iteration of the plan, they start thinking more about how to bring them in sync.

The good news is, pretty much everyone has been on at least one side of that balance being broken (that is, having responsibility without authority, or being subject to someone else's authority without responsibility), and a lot of people have been on both, and once you bring it to their attention that they're creating another one there is a certain amount of motivation to avoid it, even if it is a bit more work for a them. You can actually use people's experiences with bad organization to convince them to not create another one, if you are careful and thoughtful.

> This has been my major criticism of them; you cement the design of the app by creating an organizational structure around your software components.

That will always happen, and I've seen it happen in a monolith too (expressed as module ownership/ownership over part of the codebase).

It's inevitable in almost all organizations.

How do you organize teams if not around services? As the GP points out, the whole point of SOA is to scale people. Yes, this makes rearchitecture hard, but that is always the case at scale. The problem of territoriality and resistance to change needs other solutions (tldr; clueful management and mature leadership level ICs who cut across many teams to align architecture vision and resolve disputes between local staff engineers)

Exactly my question. The teams are going to be organized around something. Even if not intentionally, it'll be true in practice, as people are going to end up knowing things. And if you fight that, it only gets worse.

Long ago I consulted for one company that reconstituted teams for every new project, so people got swapped around every few months, usually ending up in a new chunk of the code. Whatever the theoretical benefits, it meant that nobody felt a sense of ownership for any of the code. Combine that with aggressively short project deadlines and an absolute refusal to ever adjust them and the code rapidly dropped in quality. After all, you'd be on something else soon, so any mess wasn't your long-term problem. And why should you bother given that the code you started with wasn't pristine?

As far as I can tell, the best it gets is organizing around the most durable features of the business environment. E.g., audiences and their real-world needs. Next, long-lived solutions to those needs. And then you find the common and/or large needs of those teams and spin up service teams around them. And in the background, some strong culture around what the priorities really are, so that service teams don't get so self-important that they become the tail wagging the dog.

But I love all the war stories here and would love to hear more, as I think there are no perfect global solutions to organizational problems. I think it's mostly in the details of circumstance, history, and the people involved.

Organize around delivering value to the customer. This is the obvious split for different products, and it is natural for something like AWS where each service is a relatively small team. Sometimes the customer is internal and they need an API, and internal customer decides when they are satisfied, not the API passing tests. SOA will arise, but services need to be big rather than micro, have defined internal customers and their own team. Better to have teams with frontend, backend, designer, etc. who collectively complete features, rather than hoping a bunch of services integrate into a solution for the customer at the end. It is much easier to get the architecture right when people talk first and often about what the customer needs more than implementation details. I think once you have decided how data is going to be persisted and what the customer needs to accomplish, the architecture falls into place. The application code should be stateless so it is never a scaling issue. When the code is isolated by customer feature, the blast radius for future changes is much smaller.

Microsoft, Apple, Amazon I think do a good job at preaching the customer first model and organizing around that.

Organize teams around products. The point is to increase value delivered to the customer through the product, not to produce left-pad-as-a-service.

Agreed. I've worked at a scarce few companies who organized this way, but it seemed to work better than "frontend team, backend team, devops team." When each piece of the stack is a different team, it creates unnecessary hostility and miscommunication. Aligning engineers around products, such that your iOS dev and your backend dev are on the same team, seems to run more smoothly. YMMV.