I don’t lose bags either, but airlines do. The AirTag let me tell United which building in Houston in ended up in (after getting lost at SFO), and refute their gaslighting multiple times that it was heading my way. Worth its weight in gold, literally.
While it’s hardly insightful that SQL is useful, I would have liked to read more about what the actual workload involving duckdb on a local machine looked like. I’m fully on board that local or single vm workloads can do an awful lot, but I’ve never been particularly satisfied with the pipelines I’ve seen (including my own). Usually they’re piles of scripts and intermediate data files sitting around and are hard to make idempotent and understand if you aren’t the author.
Also fwiw there’s no such thing as an M4 Ultra chip. That detail was either a mistake or hallucinated.
Original author here -- thank you for your thoughtful comment.
You're absolutely right that saying "SQL is useful" isn't exactly novel. My goal with the blog post was to describe the practical impact of leaning into SQL (and DuckDB) at our company.
I'm not the SQL expert on our team (that's my colleague Kian) but I've seen the difference he's made with his expertise. A lot of the work we migrated into SQL was originally implemented as the kind of multi-step pipelines you described: we used multiple libraries, wrote intermediate files, and had to translate data between different formats.
Kian recently rewrote a large stage of our pipeline so it runs entirely inside a single SQL script. It's a complicated script to be sure, but that's because the logic it implements is complex. And with CTEs, temp tables, and DuckDB's higher-order functions, it ended up being dramatically clearer than the original sprawl of code. More importantly, it's self-contained, and easy to inspect. Consolidating the logic into one place made a big difference for us.
And thank you for catching my error about the CPU type. We recently moved from an M2 Ultra servers to M4 machines, and I mistakenly conflated the two when I wrote "M4 Ultra." I've corrected the post.
I’ve spent a lot of time with both, and hands down the wired one is far more flakey. Granted I think that’s more a Mazda software issue, but a solid 10% of the time I get “CarPlay failed” and the only way to fix it is to turn the car on and off. Never once had an issue with wireless in a Hyundai.
In my Mazda the wired CarPlay also seems to fail a lot. But whenever I rent a car with wireless CarPlay it's been fine. Take this one anecdote for what it's worth.
Looks like Django 6 is getting an offical task system.
There’s no real world brokers or workers supported (at least built in), but still centralising around a standard interface might make things nicer than the celery tentacle monsters Django apps eventually tend to mutate into.
I'm curious, why is that? I've heard that sentence before but usually from people who want to write their own task system and end up partially implementing what celery implements just worse.
Celery is large and complex now and edge cases always show up at scale but that is usually not a reflection of the platform quality. The custom implementations I've seen are no where near what celery is capable of and can cater to so haven't seen the edge cases yet but that doesn't mean they implemented bug-free code and celery hasn't.
After asking about it, the issue always went towards a hand-wavey "performance". What is your experience on that front?
I'll choose the small, simple, reliable tool over the larger framework every time, if I can get away with having the smaller feature set. I'd go out of my way to make the simpler tool work if I could, to keep the overall cognitive load and complexity of the project down.
I never considered writing my own, I just want a tool to plug in and work so I can focus on the application. I also never thought performance of this part of the stack would be a bottleneck for my use cases.
In contrast, I am happy to use Django over Flask because you can't get away from needing lots of web framework features for a nontrivial web app. The bigger framework is usually justified, especially if it's high quality like Django. But spawning tasks (for my use cases) is just an aspect of the project that can have a simple interface and it doesn't justify adopting a big framework. Time I've invested in Django (going back to 2009) is still paying off for me today, but there'd be much lower ROI on the time I'd have to put into Celery. Unless my projects really needed its complexity, but I think far fewer projects actually require that in practice than their architects tend to think.
My experience is that Celery is fragile. It loses the connection with RabbitMQ sometimes and it needs a restart to recover. I never had those problems with Rails and Sidekiq + Redis.
My understanding is that if you just need to return a response to the client as quickly as possible, but are ok with then processing your task directly after that, then it's still usable today.
But if you want to schedule a task further in the future, then a new backend will be needed for that.
I think the bigger use case is being able to (backoff) retry failing API calls to 3rd party services. AFAIUI the new tasks package doesnt offer this in v1 which is a deal breaker for my project, at least.
For me I think it works well as is because my use case is sending several different emails after POST'ing to a view, which, there is no need to make the user wait for in my case, as they don't care about the status of the mail delivery.
But I realize there are many other usecases too that will need proper workers.
I guess the idea is first to provide a generic interface to connect various backends to and let the community develop those. Users of Django should then be able to swap one out for another. Maybe one will emerge as a quasi-standard or maybe it will be like database backends where different backends serve different purposes.
Introducing JIT features has a lot of opportunities beyond numerical numpy/numba vectorisation. There’s endless amounts of hot loops, data shuffling, garbage collection, and monomorphisation that could be done in real world python that would benefit a lot, much like V8 has done for JS.
I guess my point is that truly performant python code, at least for number crunching, uses vectorized numpy functions instead of loops, and the overhead on type checking for those is fairly minimal. I have a PR in on a compute heavy python program that I tried using numba to jit. Timing was within margins on using numpy and numba (even though the numba code could exit the loop early, which was why I was trying it) except with numba I'd be adding a dependency and it's more work to maintain the algorithms myself instead of relying on numpy.
I think of the JS code I've seen, it's mostly written in JS. So making JS faster makes JS faster. With python, the fast code is written outside python. It's too late by like 20 years. The world won't rewrite itself into native python modules
It’s true that phones cameras are miracles of technology, especially considering their size. But I take a modern Fuji traveling because the modern phone camera look is so over-processed and distinct. There’s no faking the real optics a large aperture and sensor gives, the portrait mode on phones is still a poor imitation of the real thing.
Fuji then has the whole film simulation system with all their colour science from the last century. It’s a ton of fun, and the jpgs it produces are distinct and beautiful, and I believe better than 99% of people could achieve from post processing the raws, myself included.
The middle-age guy part is accurate though, I got it as a thirtieth present.
Yeah it really was a social phenomena. Ten years ago conferences were swarmed with docker employees, swag, plenty of talks and excitement.
The effort to introduce the concepts to the mainstream can’t be understated. It seems mundane now but it took a lot of grassroots effort and marketing to hit that critical mass.
Oxc js a rust toolkit made by Void0, the mob that makes vite and vitest. These are highly regarded and performant additions to the Js/Ts ecosystem.
Oxlint is a alternative to eslint built on Oxc. It has suffered from not supporting the additional level of type-based linting that typescript-eslint can provide. They’ve now addressed that by patching and wrapping Microsoft’s new go-based typescript compiler.
Hopefully they are up to the task of continually keeping up to date with the go compiler’s internals, and/or Microsoft exposes a programmatic interface for the new compiler’s parser and type-checking.
I also wonder if or how plugins will be possible for this go+rust combination linter – they’re a pretty important part of the eslint ecosystem they’re trying to upend.
That site is misleading. Even putting aside regulations for many of the elements, everything after plutonium does not occur in nature and is made synthetically in individual atomic amounts that usually decay in hours to milliseconds. There is no way to see them, let alone sell them as a commercial product.
