This advice only works at the team level. Beyond that, it’s borderline fantasy. At senior and leadership levels, promotions aren’t rewards for effort, they’re forward looking bets made by people above you, and those people have incentives, blind spots, and egos. Reporting to an insecure director or a clueless manager will quickly disabuse you of the “just do great work” myth.
Also, if the company isn’t growing, none of this matters. You can operate like a CTO all you want and all that happens is more work gets dumped on you for the same pay. Take on stretch work if you’re hungry for it and it’s explicitly acknowledged as next-level responsibility. Otherwise, you’re just volunteering to be exploited.
I’ve spent a decent chunk of my career wrestling with time sync — NTP/PTP, GPS, timezones, all that fun stuff. For real world network time infrastructure, where do we actually hit diminishing returns with clock precision? Like, at what point does making clocks more precise stop helping in practice?
Asking partly out of curiosity, I have been toying with a future pet project ideas around portable atomic clocks, just to skip some of the headaches of distributed time sync altogether. Curious how folks who’ve worked on GPS or timing networks think about this.
For network stuff, high security and test/measurement networked systems use precision time protocol [1], which adds hardware timestamps as the packets exit the interface. This can resolve down to a couple nanoseconds for 10G [2], but can get down to picosecond. The "Grandmaster" clock uses GPS/atomic clocks.
For test and measurement, it's used for more boring synchronization of processes/whatever. For high security, with minimal length/tight cable runs, you can detect changes in cable length and latency added by MITM equipment, and synch all the security stuffs in your network.
and that precision is really important. For instance, when working with networked audio, which usually has a temporal resolution of packets that can be between 100us and 10ms (so abysmally slow in computer-time), non-PTP network cards are basically unusable.
My understanding is that precise measurement of time is the basis of all other measurements: space, mass, etc. They are all defined by some unit of time. So increasing time precision increases potential precision in other measurements.
Including of course information - often defined by the presence or absence of some alterable within a specific time.
We invent new uses for things once we have them.
A fun thought experiment would be what the world would look like if all clocks were perfectly in sync. I think I'll spend the rest of the day coming with imaginary applications.
They couldn't stay synced. There's a measurable frequency shift from a few cm of height difference after all. Making a pair of clocks that are always perfectly in sync with each other is a major step towards Le Guin's ansible!
For other readers' info, clock stability is crucial for long-term precision measurements, with a "goodness" measured by a system's Allan variance:
https://en.wikipedia.org/wiki/Allan_variance
This is true, but atomic clocks are about a million times more accurate than any other measurement device. For all pratical purposes, they are never the limiting factor.
I use fairly precise time but that's because I control high speed machinery remotely. The synchronization is the important part (the actual time doesn't matter). At 1200 inches per minute, being a millisecond off will put a noticeable notch in a piece.
Ptp and careful hardware configuration keeps things synced to within nanoseconds
For most applications, clock precision of synchronization isn't really necessary. Timestamps may be used to order events, but what is important is that there is a deterministic order of events, not that the timestamps represent the actual order that the events happened.
In such systems, ntp is inexpensive and sufficient. On networks where ntpd's assumptions hold (symetric and consistent delays), sync within a millisecond is acheivable without much work.
If you need better, PTP can get much better results. A local ntpserver following GPS with a PPS signal can get slightly better results (but without PPS it might well be worse)
I guess very few systems have better absolute time than a few microseconds. Those systems are probably exclusively found in HFT and experimental physics.
This past week I tried synchronizing the time of an embedded Linux board with a GPS PPS signal via GPIO. Turns out the kernel interrupt handler already delays the edge by 20 us compared to busy polling the state of the pin. Stuff then gets hard to measure at sub microsecond scales.
Yes, I'm aware of some of these developments. Impressive stuff, just not the level of precision on achieves tinkering for a few days with a basic gnss receiver.
If your board’s SOC has a general purpose timer (GPT) then you can often have it count cycles of a hardware clock and store the value every interrupt pulse from a GPIO. I designed a GPS disciplined oscillator like this where we had an ADC generate a tuning voltage for a 100 MHz OCXO (which was a reference clock for microwave converters) which we divided down to 10 kHz and fed into the GPT, along with the 1pps from a GPS module, and the control loop would try to adjust it until we got 10K clock cycles for every pulse. This kind of synchronisation gets very accurate over a few minutes.
Even just triggering an GPT from an GPS PPS input counting cycles of an internal clock you could use a GPT to work out the error in the clock, and you only need to query it once a second.
Sure, I was specifically talking about computer system clocks. Also with an oscillator _absolute_ time offset doesn't matter, unless you want to synchronize the phase of distributed oscillators and then things quickly get non-trivial again.
Intel Ethernet pps input pin works much much better for this. See how the open timecard mini folks do it. Easy to get sub microsecond even on cheap embedded. Most m.2 of Intel chipsets expose it (for example) as well.
Another commenter mentioned that this is needed for consistently ordering events, to which I'd add:
The consistent ordering of events is important when you're working with more than one system. An un-synchronized clock can handle this fine with a single system, it only matters when you're trying to reconcile events with another system.
This is also a scale problem, when you receive one event per-second a granularity of 1 second may very well be sufficient. If you need to deterministically order 10^9 events across systems consistently you'll want better than nanosecond level precision if you're relying on timestamps for that ordering.
The article makes a fair case for sticking with OTel, but it also feels a bit like forcing a general purpose tool into a domain where richer semantics might genuinely help. “Just add attributes” sounds neat until you’re debugging a multi-agent system with dynamic tool calls. Maybe hybrid or bridging standards are inevitable?
Curious if others here have actually tried scaling LLM observability in production like where does it hold up, and where does it collapse? Do you also feel the “open standards” narrative sometimes carries a bit of vendor bias along with it?
I think standard relational databases/schemas are underrated for when you need richness.
OTel or anything in that domain is fine when you have a distributed callgraph, which inference with tool calls does. I think the fallback layer if that doesn't work is just say Clickhouse.
I bought into the “leader vs. manager” Kool-Aid for years—until I actually had to manage. Spoiler: the dichotomy is a myth. Once you’re in the seat, it’s all gray area. You’re not just “inspiring people,” you’re stuck between upper leadership and your team, juggling chaos while trying to keep the ship afloat.
Any manager who’s been in the trenches knows the real game is shielding your team while still getting things done. Be as much of a “leader” as you want, but without authority and accountability, you’re just cosplaying. The rah-rah leadership Kool-Aid is mostly there to keep people inspired while the actual decisions happen in rooms you’ll never be invited to.
Simon Sinek is far from the worst of the genre, but these aphorisms are much easier to say than they are to exemplify in practice. I’ve worked with more than one LinkedIn influencer with millions of followers, and their primary talent turned out to be saying inspiring things. They had enough experience and the sheen of big tech credibility to be taken seriously by aspirational juniors, but when you actually had hard stuff to solve they are not who you’d want to be in the trenches with.
Out of curiosity, what's keeping you in slack ecosystem? Why not leverage Discord and run on your own server? Wouldn't that be a much economical alternative to begin with?
Is this deeply coupled with Chroma or something we can fork out for other applications? I would love to explore this for lightweight change data capture on S3.
Absolutely! We took care during the build to address chroma’s needs, but I believe the only requirement is chroma storage. Long term it’ll appear on crates.io as a standalone package.
I’m a bit underwhelmed by the quality of articles coming out of Netflix. 100 Million records / entity is nothing for Redis — even without RAW hollow-style compression techniques used (bit-packing, dedup, dict encoding is pretty standard stuff) [1].
Framing this as a hard-scaling problem (tudum seems mostly static, please cmiiw if thats not the case) feels like pure resume-driven engineering. Makes me wonder: what stage was this system at that they felt the need to build this?
Unpopular opinion: in 2025, nobody should be reaching for an ORM first. They’re an anti-pattern at this point. The “abstraction” it promises rarely delivers—what you actually get is leaky, slow, and a nightmare to operate at scale.
The sane middle ground is libraries that give you nicer ergonomics around SQL without hiding it (like Golangs sqlx https://github.com/jmoiron/sqlx). Engineers should be writing SQL, period.
I think you're fundamentally misunderstanding the primary purpose of an ORM. It's in the name - Object Relational Mapper. It's meant to ease the mapping from SQL query results into objects in your code, and from objects back to SQL queries. Doing this manually at scale when you have a large number of tables is also a nightmare.
There's no rule saying you can't integrate your own manually written SQL with an ORM, and in fact, any production-ready, feature-complete ORM will allow you do it, because it's effectively a requirement for any non-trivial use case.
Well said. Please never be silent over this fact. It's important to educate people on what an ORM is, what it means and especially what it doesn't mean. Especially in times where VC-baked companies misinform and manipulate people about that, like Prisma is doing
the trouble is that even if people embrace that thinking, the ORM encourages them to pull entities out of the db and do a bunch of computation in the server that would be much faster to do in the db.
”If you're doing OLAP…SQL isn't wholly adequate for this, it's hard work to get the SQL right and if there's joins involved it's not hard to accidentally fan out and start double counting.”
I feel this in my bones. Anytime someone in the business changes something in a source system, the joins create extra rows. Trying to address this with SQL is like plowing a field with a spoon.
And I don’t think ORMs are the answer. Just imperative code. The ability to throw in a few lines of sanity checking outside of SQL is such a massive boost to reliability and data quality, when the source systems are moving underneath your feet.
In my experience, LATERAL is a massive quality-of-life improvement when it comes to double-counting specifically; where a normal subquery is not possible, a correlated subquery in FROM is much nicer than a CTE+JOIN.
> The sane middle ground is libraries that give you nicer ergonomics around SQL without hiding it (like Golangs sqlx https://github.com/jmoiron/sqlx). Engineers should be writing SQL, period.
The blog suggests that an ORM for OLAP would do exactly that
Is it a variation of: "I suffered when I was young, so everyone must suffer as I did?"
SQL is terrible, however you slice it. It's a verbose bass-ackwards language. ORMs that remove the need to deal with SQL for 99% of trivial cases are great.
SQL remains the only way to efficiently perform MANY computations in the database precisely because it's lingua franca for the database planner. If you're not writing SQL, it doesn't mean that you're unable to cover 1% of the queries, it only means that you're leaving 99% of performance on the table. You can tell a bad programmer by their refusing to use views and materialized views. Not to mention normalisation! I'm yet to see a coder using ORM produce a sane schema. And don't get me started on aggregates. Relational databases represent relations, not objects, period.
> If you're not writing SQL, it doesn't mean that you're unable to cover 1% of the queries, it only means that you're leaving 99% of performance on the table.
Honestly? You're spewing bullshit. In most apps most of SQL is trivial. It's typically limited to simple joins (with indexes) with simple filters. Anything more complicated, and it's usually not suitable for OLTP. Heck, all our SQL is linted to not have full-table scans.
This kind of SQL is perfectly auto-generated by ORMs.
Those multi-page queries that required mystic DB knowledge for placing hints, burning incense, and paying $1000 per hour to Oracle consultants? They're entirely useless in modern software stacks. Because you can either keep the entire working set in RAM so these queries can be trivially rewritten, or you just won't use regular SQL for it.
> You can tell a bad programmer by their refusing to use views and materialized views.
You can tell a bad programmer by them using DB in a way that requires materialized views. It typically ends with moving app logic into SQL, and may even lead to "SELECT '<td>' + row.cust_name + '</td>'".
I have a ton of respect for José Valim and the Elixir core team, I have to say: Elixir just doesn’t mesh well with the kind of infrastructure tooling that’s become standard today. The ecosystem has been growing impressively and there’s a lot to admire, but its philosophy often feels at odds with containerized, Kubernetes-based deployments.
Elixir promotes a "do it all in one place" model—concurrency, distribution, fault tolerance—which can be powerful, but when you try to shoehorn that into a world built around ephemeral containers and orchestration, it starts to crack. The abstractions don’t always translate cleanly.
This opinion comes from experience: we’ve been migrating a fairly complex Elixir codebase to Go. It’s a language our team knows well and scales reliably in modern infra. At the end of the day, don’t get too attached to any language. Choose what aligns with your team’s strengths and your production reality.
If you can't get Elixir to work well in k8s, you're simply doing it wrong. Yeah you can build a lot of what k8s offers on the BEAM on a bare server but you don't have to at all. Elixir offers modern telemetry that works really well with pretty much any monitoring tool, and give you a lot of introspection into the internals of the VM if you need it. "Let it fail" works just fine in k8s, you can run a supervision tree in a container and have it manage process restarts internally just fine, its just an added layer of fault tolerance. Sure, you can't easily have long running stateful processes if you're doing a lot of blue-green deployments, but you probably don't need to do that most of the time.
If you don't know Elixir and the BEAM well, of course you're going to have a bad time. That's true of any language.
phoenix live view is effectively built around the assumption that the server process never dies and never restarts. its a very dubious model for a production deployment.
> what happens when the server restarts / connection is lost / server dies?
> you lose all of the current client state but you can work around this by persisting all the client state somewhere.
> phoenix live view is effectively built around the assumption that the server process never dies and never restarts.
Not exactly, it's built to hold state in memory by default but doesn't assume how you want to handle deploys or server restarts. There's a built in restore mechanism for forms and it's trivial to shuffle state off to either the client/redis/your db[1]. You'd have the same problem if you were storing all your state in memory for any web application regardless of your library choice. Or you conversely have the problem that refreshing the page trashes client state.
So there are two thinks here, you don't have to use live_view to use elixir or phoenix, and if you do you just need to actually think about how you're solving problems. The state can get trashed anywhere for any number of reasons. Tossing on the client and forgetting about it just moves the problem.
> You'd have the same problem if you were storing all your state in memory for any web application regardless of your library choice.
But that's the thing - traditional server-side web applications don't do this. The the stateless request/response cycle of traditional server-rendered apps is a _huge_ advantage from a system design standpoint. State is bad. State is hard to manage correctly over time. Elixir makes it possible to manage this in-memory state relationship better than other languages, but it's still difficult to do at scale regardless.
LiveView turns stateless web applications into stateful web applications, and this is a problem most folks aren't considering when they see developer experience improvements the LiveView project claims. This is _the_ specific tradeoff that LiveView makes, and I wish folks wouldn't handwave it away as if it were trivial to manage. It's not. It's a fundamentally different client/server architecture.
Source/disclaimer: I work at a large company with a large LiveView deployment, we have spent a ton of time and money on it, and it's a pain in the ass. I still love Elixir, I just think LiveView is oversold.
I guess my quibble is that I’ve talked to Jose and Chris about this and I don’t think they’re overselling it. The caveats are in the docs and they expect you to understand the tools.
And realistically there are cases where I’d use another tool.
the way live-view is sold is some kind of alternative to writing javascript. if you have a very rich client side application then you can easily end up with a bunch of state which would be in the clients memory but now is in the servers memory. but if its in the servers memory the data ends up being much more fragile because any failure of the client or the server means the live view state is lost. whereas if the data was just in client memory then it would only be lost if there is a problem in the browser. ideally people would write applications that are tolerant to both faults. but i think in practice live-view practitioners just ignore the problem and then you are left in a worse situation.
i realize there is still a dataloss problem when there is state on the client but there is a lot of simple stuff that causes problems if you are effectively reloading the page if the server disappears due to a redeploy or a crash.
for example i'm reading an email in my message client. i've scrolled half-way down the email. but now the server crashes i reconnect to live view and my scroll position when reading the email is reset.
when i look at live view i feel like its written by people that have never deployed something in production. of course this is not really true but i feel like it would be better if live-view people were more honest about the tradeoffs. its a very complicated situation and having some bad outcomes might be worth the increase in developer productivity but i feel like live-view people just ignore the bad outcomes.
also, take a cloud deployment. websockets seem to be an inherent problem in cloud deployments especially AWS. as far as i know AWS does not expose some event to a instance that is part of a load balancer that it is about to die. ideally if you have a websocket instance with a client and you realize you are scheduled to be reaped you would message the client that it needs to reconnect. then the client would reconnect to a server that would not be reaped and everything would be dandy. but AWS doesn't seem to have this functionality (or its not easy to implement!) but more importantly live view does not expose any kind of hooks so you can have 'safe' server redeployment out of the box.
I'm not sure who from the core group of developers is overselling it in the way you're claiming. Yeah, I wouldn't try to build figma or gmail with liveview, but I'd be fine building an app for a bank for example. Tools exist in a mesh of other tools and problem spaces where they may or may not be appropriate.
it really depends on what you are doing. if you have a complex application then the automatic recovery might not be enough for a fault free user experience. i feel like live view is sold as some kind of magic where the developer does not have to write javascript and you can just implement the server side purely in elixir as you would javascript without writing to a database, or some session storage. but if you do this then you are in a worse situation than javascript when it comes to network faults, or server processes restarting.
Sure, if your a company which is fully bought into ephemeral containers running on Kubernetes then Elixir is probably not going to be a great fit for you. I have once introduced it into a company like that and eventually the same thing happened, the application got rewritten in a language that fits the paradigm better.
If I were starting a new company today though I'd probably go with Elixir, and then I simply wouldn't bother with containers, Kubernetes, and schedulers. Just run some servers and run the application on them, let them cluster, and generally lean into the advantages.
most people/companies who think they need kubernetes don't actually need it. you are not google or amazon. and with erlang/elixir that's likely even more true.
> if your a company which is fully bought into ephemeral containers running on Kubernetes then Elixir is probably not going to be a great fit for you
As a community, we have got to stop saying this stuff. It's false. Nothing about Elixir or k8s precludes using the other. Elixir applications can and do run just fine in containers and k8s deployments. In fact, they're complementary: https://dashbit.co/blog/kubernetes-and-the-erlang-vm-orchest...
Elixir has a Ruby on Rails approach which is kind of all in one. Not everyone’s cup of hot chocolate.
Erlang is low level, lightweight processes and message passing - perfect for micro-services and containerisation.
What Erlang lacks are high level web-oriented packages, i.e. markdown conversion, CSS and JavaScript packaging, REST (not quite true: cowboy is fantastic) - for which though Erlang was never intended.
However the cool thing is that you can combine both in the same project, allowing you to have high level Elixir and low-level process management in one project. This is possible because both use the same virtual machine - BEAM.
Erlang isn't really that much lower level than Elixir. The Elixir syntax is just a bit different and they've improved a few things like string handling. Processes are still processes.
We look at K8s more like "the Cloud Operating System". Many of its capabilities are more valuable for other runtimes, but that doesn't mean that Elixir is a bad fit with it.
For example, the Erlang VM clustering can make use of K8s for Service Discovery. You can do ephemeral containers and use readiness probes to create a " hand over" period where new instances can sync their data from old, about-to-be-replaced instances.
Did you do hot updates? I ser that is mention in the post, but I thought the community has walked away from it? Or at least that its mixed feelings about it?
As a non-native English speaker, I’ve often struggled to communicate nuance or subtlety in writing—especially when addressing non-technical audiences. LLMs have been a game-changer for me. They’ve significantly improved my writing and made it much easier to articulate my thoughts clearly.
Sure, it can be frustrating that they don’t adapt to a user’s personal style. But for those of us who haven’t fully developed that stylistic sense (which is common among non-native speakers), it’s still a huge leap forward.
Also, if the company isn’t growing, none of this matters. You can operate like a CTO all you want and all that happens is more work gets dumped on you for the same pay. Take on stretch work if you’re hungry for it and it’s explicitly acknowledged as next-level responsibility. Otherwise, you’re just volunteering to be exploited.
reply