but isn't this like a lot of other CS-related "gradient descent"?
when someone invents a new scheduling algorithm or a new concurrent data structure, it's usually based on hunches and empirical results (benchmarks) too. nobody sits down and mathematically proves their new linux scheduler is optimal before shipping it. they test it against representative workloads and see if there is uplift.
we understand transformer architectures at the same theoretical level we understand most complex systems. we know the principles, we have solid intuitions about why certain things work, but the emergent behavior of any sufficiently complex system isn't fully predictable from first principles.
that's true of operating systems, distributed databases, and most software above a certain complexity threshold.
yeah, but isn't the whole point of claude code to get people to provide preference data/telemetry data to anthropic (unless you opt out?). same w/ other providers.
i'm guessing most of the gains we've seen recently are post training rather than pretraining.
Yes, but you have the problem that a good portion of that is going to be AI generated.
But, I naively assume most orgs would opt out. I know some orgs have a proxy in place that will prevent certain proprietary code from passing through!
This makes me curious if, in the allow case, Anthropic is recording generated output, to maybe down-weight it if it's seen in the training data (or something similar)?
yup, bezos said "we will be able to beat the cost of terrestrial data centers in space in the next couple of decades". presumably this means they'll need huge ass radiators, so its all about bringing down launch costs since they'll need to increase mass.
they should just acquire one of the many agent code harnesses. Something like opencode works just as well as claude-code and has only been around half of the time.
I used opencode happily for a while before switching to copilot cli. Been a minute , but I don't detect a major quality difference since they added Plan mode. Seems pretty solid, and first party if that matters to your org.
true on the naming, but i think geometric/clifford algebra has its own mysterious aura precisely because it can be framed as "suppressed" or "overlooked".. plus it genuinely does have elegant mathematical structure backing up the hype
funny thing is quaternions had that exact same energy in the computer graphics community for years. after ken shoemake introduced them to CG in 1985, there was a long period of "why are we using euler angles like cavemen when this exists??". now quaternions are well known tooling for people in graphics and the mystique has worn off at least in that community.
>EDIT: More interestingly, I find an issue, what do I even DO? If it's not related to integrations or your underlying data, the black box just gave nonsensical output. What would I do to resolve it?
Lots of stuff you could do. Adjust the system prompt, add guardrails/filters (catching mistakes and then asking the LLM loop again), improve the RAG (assuming they have one), fine tune (if necessary), etc.
Deep SSMs, including the entire S4 to Mamba saga, are a very interesting alternative to transformers. In some of my genomics use cases, Mamba has been easier to train and scale over large context windows, compared to transformers.
solid analysis but i think you're missing the logical endpoint here: this doesn't end with companies "relearning scarcity"... it ends with the permanent contractor-ification of various types of work at these tech companies (not just tech roles, but other types of roles at these companies). already, contractor-to-employee ratio has gotten higher and higher at these companies in recent years and I expect this to continue.
ZIRP (especially the "double tap" ZIRP in 2021/2022) created this monster (bootcamp devs getting hired, big tech devs making "day in the life of" tiktok vids).
contractors give:
instant scale up/down without layoff optics
no benefits overhead
no severance obligations
easy performance management (just don't renew)
this mirrors what other industries typically do after large restructuring waves ... manufacturing got temp agencies and staffing firms as permanent fixtures post-rust belt collapse. tech is just catching up to the same playbook.
when someone invents a new scheduling algorithm or a new concurrent data structure, it's usually based on hunches and empirical results (benchmarks) too. nobody sits down and mathematically proves their new linux scheduler is optimal before shipping it. they test it against representative workloads and see if there is uplift.
we understand transformer architectures at the same theoretical level we understand most complex systems. we know the principles, we have solid intuitions about why certain things work, but the emergent behavior of any sufficiently complex system isn't fully predictable from first principles.
that's true of operating systems, distributed databases, and most software above a certain complexity threshold.
reply