> the dots aren’t connected between a lot of the aspects presented
that's on me (author); I tried to cut the content down to a manageable post size that covered some interesting stuff - but probably dropped the connective tissue in the process. We'll keep this in mind for next time.
Author here; RAM is so much more expensive than disk though. Two 500G M.2 NVMes for a RootFS in RAID1 are basically max ~$150 which is I think much cheaper than a single 64G DDR5 ECC RAM module [I don't have exact numbers on me, but ECC RAM is pricey]. It's also a lot harder to debug when things go wrong because you lose the machines state if everything was ephemeral.
We run a thin base OS on the boxes and then VMs on top which we consider more ephemeral. The frequency of needing to update that base OS is v. low.
I think there's a case for building a custom PXE booted RAMdisk image to replace the install though; something like what Equinix Metal (formerly Packet) do with Tinkerbell (https://github.com/tinkerbell); they call it an OS Install Environment, but the idea is a small lightweight linux install agent that can DD a golden image onto the disk (vs. an install each time).
Are you guys still hiring? Would be keen to have a chat about this and see if anything comes out. Sent one of you guys a LinkedIn invite but no response yet.
Yep! We have two roles open https://railway.com/careers#open-positions - Datacenters and Infra that might be of interest to you. Idk if many of us use LinkedIn so not the best way to get in touch. Can I share the email on your HN profile link with our in-house recruiter?
Yes please. The reason I wanted to informally reach out by LinkedIn is that I'm more interested in a chat to see if my skills align and if there's potential, even for short-term work - what you guys are doing sounds cool and I've played around with some of it in the past (thus the Buildroot suggestion - that was my approach to a similar requirement).
author here; Claude use here was pure laziness - and personally, I found it quite funny that it worked. We could sample pixels and try and build that detection, but $<1c per run to write a prompt and get some json was too hilarious not to ship to prod.
Maybe it needs to have more complex logic/detection and we need something more complex down the road. But it's like easy and cheap OCR for now.
what was kinda funnier was that I tried to get Claude to generate its own Go client code to upload the image and run the prompt; it totally totally hallucinated on that part :).
It's a fair question. What Oxide are building is cool, but it's too custom/monolithic for us to risk. We're more likely to look at OCP racks/chassis down the road.
We currently pass on our cloud egress costs to users via the current pricing. We'll be publishing a pricing update soon as part of our migration - and egress [and some other things] will be coming down.
We have a distributor we work with - just because it makes import/export a lot easier. But we get to interface directly with Supermicro for the technical/design stuff, and they're super awesome. If you're looking in the US, reach out to their eStore - really great fuss-free turnaround and all direct.
the good news on this is that we've got a tonne of deep-dive material on networking and whitebox switches we cut from this post. We'll definitely be talking more about this soon (also cos' BGP is cool).
correct; I think the first version of our tool sprung up in the space of a couple of weekends. It wasn't planned, my colleague Pierre who wrote it just had a lot of fun building it.
There's a fork called nautobot that tries to add-in automation. Most things we wanted to do with either meant we had to go writing django plugins and trying to interface with their APIs (and fight with the libs). Overall just hammering together a small custom service ended up being way faster/simpler.
All valid points - and our ideas for Gen 2 sound directionally similar - but those are at crayon drawing stage.
When we started, we didn't have much of an idea about what the rack needs to look like. So we chose a combination of things we thought we could pull this off. We're mostly software and systems folks, and there's a dearth of information out there on what to do. Vendors tend to gravitate towards selling BGP+EVPN+VXLAN or whatever "enterprise" reference designs; so we kinda YOLO'ed the Gen 1. We decided to spend extra money if we could get to a working setup sooner. When the clock is in cloud spend, there's uh... lots of opportunity cost :D.
A lot of the chipset and switch choices were bets and we had to pick and choose what we gambled on - and what we could get our hands on. The main bets this round were eBGP to the hosts with BGP unnumbered, SONiC switches - this lets us do a lot of networking with our existing IPv6/Wireguard/eBPF overlay and a debian based switch OS + FRR (so fewer things to learn). And ofc. figuring out how to operationalise the install process and get stuff running on the hardware as soon as possible.
Now we've got a working design, we'll start iterating a bit more on the hardware choice and network design. I'd love for us to write about it when we get through it. Plus I think we owe the internet a rant on networking in general.
Edit: Also we don't use UniFi Pro / Uniquity gear anywhere?
