The big tech companies (ie. FANG) all have huge demand for principal+ engineers and know how to scope and support them properly. The size means they have lots of them which reduces the snowflake nature of the role and makes career development, salary support, etc… more standardized. Since they are large, there can be some variance, but overall they are a good option if you are interested. I don’t know where in EU you are, but most of FANG has roles in EU and some even do remote as well.
Smaller companies tend to want principal engineers in theory, but as you are finding they can often struggle with how to utilize them properly. Often you see them playing a Chief Architect role or something similar which is more hybrid with PM than pure tech. So getting PM skills could help set you up for this direction as well.
Finally, you have said you did management for a bit before, so you should know if this is an interesting path or not. This is a very different role, so I would recommend you do this only if you have the passion and desire to do this full time. Most principal engineers I know have done management for a bit and gained a lot of skills related to it, but ultimately wasn’t what they wanted to do.
The problem at the big (especially FAANG) companies is that to be effective you need to really buy in to the company's tech stack and development ethos. People who were promoted to those roles naturally do; you probably don't. In fact, if you have a lot of experience elsewhere you're likely to have opinions that conflict with the company zeitgeist, and that leads to conflict with its champions. Similarly, effective principal/staff/+ work requires a lot of strong connections to people in many teams, which again tends to favor internal promotions. This is not competition for its own sake. It means that when you're compared to your home-grown peers every review cycle it will be difficult for you to keep up let alone stand out (even for a while after the typical one-year grace period which isn't long enough at this level). This is discouraging, and in some cases can leave you in a permanent bind w.r.t. team trust or possibility of an internal transfer. Many thrive despite all this, but many also end up seeing it as lost time and opportunity. Quite a few end up going back to where they were, while others opt for a smaller company.
P.S. I'd argue that, for all their advantages within the company, those internal promotions are usually over promotions. People's attachment to that tech stack and development ethos, and lack of experience with any other, means there's an even sharper drop in their value going elsewhere than for outsiders coming in.
you need to really buy in to the company's tech stack and development ethos.
You need to be of the mindset that you're hired to help the company. The company has a certain tech stack and development ethos, so you're hired to help them with that. Just because you know there are better ways to do it, your job is still to help them do it their way.
It can be possible to get them to change development ethos, but this is a big deal and uses a lot of political capital. If you can really convince most people that it's better, you'll be seen as a senior tech leader for sure. But if you're optimizing for the best performance reviews -- in other words, the incentives the company has set for you -- then it's usually better just to work within the system.
"To be a leader, you need to have followers." So leadership isn't having the best product, e.g. the way Google is a leader in search. It's more like class elections in high school, it's a popularity contest. Your job is to figure out what people are complaining about or advocating for, and do those. Most likely, everybody is used to the development ethos and just thinks it's the only or obvious way to do things. So nobody is really complaining about it.
Mostly I agree with everything you say, especially about needing followers before you can lead, but I think this part deserves more discussion. At staff level (if not slightly before), "help the company" and "do it their [historical] way" are often not the same thing. As I said in another sub-thread, at that level you're hired to bring knowledge and skills and perspective that the org doesn't already have (or have enough of). Unlike lower levels, pulling in some direction is part of the job in this case. I think EMs at all levels understand this and often support it quite well. The problem I've seen is other high-level engineers who never knew anything but the current way and believe that it's generally the only way (except of course for the one part they personally understand and want to change). This sometimes leads to hiring people and then thwarting their efforts to do what they were hired for.
> uses a lot of political capital
> ...
> optimizing for the best performance reviews
That's the problem. These two should be aligned. You should reward what you want to see more of, and I think people using their best professional judgment qualifies. Relying on the continual presence of people who will sacrifice their own career/financial progress to make needed change (as I mentioned in another sub-thread) is not an effective or ethical strategy. I won't even say whether I believe I'm in that category myself, but I certainly saw other people who got tired of lying down across barbed wire so other people could run past them.
To put a more positive spin on what you are saying, very senior ICs carry the company’s technical culture. They have influence in shaping it, but they aren’t hired or promoted to buck it. If you want that kind of influence you need to climb the EM ladder up towards VP Engineering or equivalent. However, in that case you don’t get to spend 30%, or any, time programming. On the third hand you can go to a start up and wear a ton of hats, but you probably won’t get high cash comp.
Agreed. There's a lot of opportunity there if people are willing to adapt to local norms, even if that means setting aside past (often hard-won) lessons. Some can. Some can't. Most don't really know if they can or not until they try. That's all fine, but I do take issue with this.
> you need to climb the EM ladder
No. Many of these companies take pride in being "engineer first" but that's a false claim if engineers are discouraged from challenging the local orthodoxy too much and only high-level execs may do so. It's too easy for territoriality and NIH to set in, or for real progress to be replaced with mere churn. Didn't we learn these lessons with older tech giants like IBM or AT&T or DEC? They had the same pattern of people replacing one internal system with an almost identical one, because reaping credit and promotions that way was easier than fighting for true change. They had the same pattern of people who had learned those habits too well becoming DEs or fellows and using the same "guardians of the culture" excuse to enforce conformity for its own sake. And look where it got them.
Obviously those who wish to challenge the status quo need to balance that with productive work within the existing paradigm, and strong claims require strong evidence (which a VPE is unlikely to have BTW), but that's exactly why there should not be additional barriers. I was not the first or only person at Facebook to observe that the whole thing would come crashing down if not for an ever-changing cast of engineers determined to do the right thing despite the effect they knew it would have on their PSCs. In a true engineer-first culture challenges to the status quo would be encouraged and engaged, but in my experience that wasn't always the case. Corporate ossification wasn't only a problem for prior generations.
EMs are engineers even if you don’t respect them because they don’t write code anymore. This is different than old school tech companies where managers were businessmen and engineers were thought of similarly to assembly line workers.
The Dilbert dream of no hierarchy (vice a hierarchy made up of engineers) has never worked beyond small companies.
A truly flat org is communism of corporate cultures—-great on paper, a disaster in practice. The dysfunction at these places isn’t because they haven’t flat org’ed hard enough or because of evil, devious middle management subverting the purity of the system—-it’s because the idea is bad in the first place.
Let's not turn this into an exercise in moving goalposts and constructing strawmen, OK? I never expressed any disrespect of EMs, nor did I propose a flat organizational structure. You specifically mentioned going up to Vice President of Engineering level, which is quite different than a line EM, and I responded to that. Your absurd invocation of communism aside, that's way over on the old-fashioned authoritarian/hierarchical end of the organizational spectrum.
That’s where some decisions should be made. For example, creating a new programming language. The answer is almost always “no, that’s a horrible idea” the determination otherwise should be made by the person ultimately responsible for all engineer execution.
Some, yes. Look at those goalposts go! Staff engineers are hired to bring skills and knowledge and perspective not already present. All I'm saying is that they should be able to exercise those assets, and all too often that is discouraged. I'm beginning to wonder if your accusation about disrespecting EMs is just projection of your own disrespect for higher-level ICs.
From my original reply:
“They have influence in shaping it, but they aren’t hired or promoted to buck it.”
I’m not sure we disagree that much, maybe over where to draw the line, or maybe over how we talk about roughly the same outcomes. I’m content to leave the discussion here. Cheers.
A top-down org where employees don't have the power to vote out management is exactly that - an authoritarian structure. That's why is called privately owned.
> start up and wear a ton of hats, but you probably won’t get high cash comp.
I'm not sure start-ups are any freer on the dogma, so you'll still run into this unless you find a start-up that matches your own. Worse still -- start-ups have serious pressures that make debating dogma look like not carrying the load.
If you are in early enough the decisions haven’t been made yet, so by definition they can’t be set in stone. You could have a technical co-founder that wants to decide everything without discussion but in that case you probably don’t want to be there for many reasons.
> Similarly, effective principal/staff/+ work requires a lot of strong connections to people in many teams, which again tends to favor internal promotions.
This might be a smaller factor for doing effective work itself, the number of new faces at all levels at big big companies might offset this. And generally, ppl working under this principal/staff+ engineer usually follow along.
But promotions would require connections to many ppl in many teams.
Other big non-FANG companies also need Staff/Principal Engineers. You can find them in AI consulting companies, banks, insurance, and pretty much everywhere when you know what to look for. This includes most European cities if you have a EU passport.
Yes. The progressives were fighting against the mask mandate imposed by the lynching-friendly Republican mayor of SF. They were saying that mask mandates are unscientific, and turned out to be right:
> A study then in 1919 concluded that mandatory mask mandates did not make any difference on epidemic
A couple of months ago, in a public chatroom for DBAs, one of GitLab's engineering leads made some very insulting comments about Facebook engineers & eng practices. Pretty ironic.
I want to feel bad for GibLab, but it's really, really hard when they hire people like that.
Oh wow, some interesting stuff in here. Looks like they use MySQL as a queue for scheduling the ORC Peons? Would have loved to hear more about why they did that.
We use it because it works well for us. We've put a lot of work into making MySQL scale for us to the point where it's a very well supported system and one of the main choices for a lot of storage decisions.
We even use MySQL as a queue for Facebook Messenger. More details about this:
Because it works perfectly fine? I tend to write anyone off who scoffs at simple database-as-queue designs without understanding what the scaling requirements are. You can use a database as a job queue for 10s of thousands of jobs per day without any sweat.
Please forgive me, but I don't understand your hostility.
I understand that there are perfectly legitimate reasons for using a database as a queue. If you frequently need to look at and rearrange your jobs while they're in flight, chances are a pure FIFO structure probably doesn't work that well for you anyways. If the enqueue is contingent on a transaction committing, probably makes a lot of sense for the job to be in the same database. You don't need to tell me -- I've seen more than a few in production.
But an actual message queue also "works perfectly fine" based on the information provided, and I would imagine that a company like Facebook probably already has a few of those lying around. It would have been a conscious choice to use MySQL as a queue.
I think Production Engineering is especially interesting to me as I think we do a very good job of lowering the wall between ops and dev, mostly we build tools and do evangelism to help SWEs own their own services, but above all we do whatever is necessary to keep the site up in the most scalable way we possibly can. This includes engaging with our product engineers, not just infrastructure, and helping them understand the impact of product on infra and vice versa. That's one thing I especially like about PE, there is very little of the ops/dev divide, it's more of a partnership with PE and SWE both helping to own the service in production. (disclosure, I work at FB as a PE)
(I am a manager of the Data-Perf team at FB which deals with lots of different DB technologies)
MySQL (specifically InnoDB) is extremely efficient as a storage backend compared to PostgreSQL.
There are a few features that make InnoDB better in many cases:
1. Change buffering for IO bound workloads: If you are IO bound, then InnoDB change buffering is a huge, huge win. It basically is able to reduce IO required for secondary index maintenance by a huge amount.
2. InnoDB compression: When you are space constrained (say using flash storage), then being able to compress your data is a big win. In our case, it reduces space by around 40% which translates directly to 40% less servers required. While you could do something like run PG on ZFS with compression, for an OLTP workload, you want the compression in the DB so that it can do a lot of work to minimize the compressing and decompressing of data.
3. Clustered index: The InnoDB PK is a clustered index. This makes a lot of query patterns (such as range scans of the PK) very cheap. Combined with covering indexes (which PG now has too!), you can really minimize the IO required by properly tuned queries.
There are a variety of smaller things as well, such as InnoDB doing logical writes to the redo log vs. PG doing full page writes, so on very high write systems, the REDO log bytes written will be dramatically less. Also MySQL replication has traditionally been more flexible than PG, but PG has made some great strides recently, so I don't know if I would maintain that position still.
Your smaller points don't make much sense to me... PG doesn't have a REDO log, that's just not the way it's architected. If you mean WAL, a patch went out in 9.4 to prevent updates to a page from rewriting more than necessary, so they're not doing full-page writes each time. Clustered index also makes inserts slower--it's again not a straight win for MySQL (and what query patterns other than range scans of the PK does it make cheaper?). Finally, while obviously it's not the same as InnoDB compression, TOAST does a rather good job in practice, and Postgres's indexes are quite efficiently compressed (especially with new changes in 9.4). I can't speak to (1), but it's not at all clear to me that any of these advantages put MySQL ahead in the long term, and certainly not for all workloads.
Right, PG calls the REDO log the WAL (for most purposes they are the same thing). I did not know that 9.4 can do partial page writes to the WAL now. Guess I will have more reading to do, thanks for pointing it out! A nice blog post by a colleague recently showing how large writes to redo logs matter is (not about PG, but why it is significant in the context of size of entries):
As far as when clustering a table is useful, see the CLUSTER command in PG. It is roughly the same places you would want to do it, except it is automatically maintained. You do need to realize what is going on to minimize impact on inserts, but in a lot of cases, data in inserted in generally ascending order so it mostly 'free'. This does make GUIDs really bad for PKs in InnoDB.
Clustered indexes are like covering indexes (which PG got recently). You don't quite realize how useful they are until you get access to it ;)
InnoDB compression for us is primarily for non-lob objects, so TOAST is quite a bit different than the cross-row compression that we get. We will normally do compression of large objects outside of the DB whenever possible.
I'm not saying that InnoDB is always better than PG, but in a lot of cases that I have tested it with, it is indeed better. PG has come a long way recently, including options such as covering indexes to close the gap.
Thanks a lot for the detailed reply, this is why I love HN!
1-2 years back I saw a benchmark stating that PG works better than MySQL on I/O constrained AWS instances, so I didn't expect this ("I've learned something today").
I think Postgres is a better default choice because you are likely to be safer and for it to behave properly with your data. In certain cases, apparently Mysql might be faster for certain workloads, and with certain tradeoffs. That's the kind of informed decision you can make once you're scaling to that size.
There is nothing that prevents these from being implemented for PostgreSQL. In fact, they have already done a lot of optimizations that close the gap, such as covering indexes. Before covering indexes, InnoDB was even further ahead of them.
Of the ones mentioned, I would guess compression is the most complex. For InnoDB this took many years to get the point of it being usable and efficient. Many naive implementations can cause huge overheads for CPU which makes it unusable.
