I did a lot of playing around early on for this with LLMs.

Some early testing I found that injecting a "seed" only somewhat helped. I would inject a sentance of random characters to generate output.

It did actually imrpove its ability to make unique content but it wasn't great.

It would be cool to formaile the test for something like password generation.

I bet you the predictions are largely correct but technology doesn't care about funding timelines and egos. It will come in its own time.

It's like trying to make fusion happen only by spending more money. It helps but it doesn't fundamentally solve thr pace of true innovation.

I've been saying for years now that the next AI breakthrough could come from big tech but it also has just a likely chance of comming from a smart kid with a whiteboard.

Well, the predictions are tied to the timelines. If someone predicts that AI will take over writing code sometime in the future I think a lot of people would agree. The pushback comes from suggesting it's current LLMs and that the timeline is months and not decades.

Yes scaling is always capitol hungry but the innovation itself is not

All of moltbook is the same. For all we know it was literally the guy complaining about it who ran this.

But at the same time true or false what we're seeing is a kind of quasi science fiction. We're looking at the problems of the future here and to be honest it's going to suck for future us.

And this was just about how to decide an auto accident case. With the experiment varying the circumstances.

My summary is still: seasoned judges disagree with LLM output 50% of the time.

I don't think the elite think all voters are dumb more like they think they're easy to manipulate to vote for something (which is largely true). Anecdotally I easily get manipulated by the type of information I consume. I occassionally catch it after the fact or a conversation with others but there's no telling how much I've just accepted that's manipulated.

From that angle it's a game of who has the money, power, and diatribution to enact this manipulation.

Twitter being a prime example. Is Elon "right"? Maybe but the main point is it doesn't matter as he has the distribution.

If you have money but low to no distribution -> you do what gary is doing. Maybe he'd be interested in removing rights to vote but someone like Zuck would NOT because he has outsized ability to influence as he sees fit.

Good people are moral for a million bucks but almost none are for a billion.

lol pretty much. their reservation price was more than met.

Although considering Brin's interactions with female employees etc, no surprise really. They were full of it from the off. Page is better at hiding it.

It's basically continual learning. This is beyond a hard problem it's currently an impossible one. I know of no system that solve CL even at small scale let alone large models.

Annoyingly, they have SOME inherent capability to do it. It's really easy to get sucked down this path due to that glimmer of hope but the longer you play with it the more annoying it becomes.

SSI seems to be focused on this problem directly so maybe they discover something?

So, surprising, that is not completely true - I know of 2 finance HFT trading firms that do CL at scale, and it works - but in a relatively narrow context of predicting profitable actions. It is still very surprising it works, and the compute is impressively large to do it - but it does work. I do have some hope of it translating to the wider energy landscapers we want AI to work over…

Nah, it works - let's just call it personal experience.

During covid almost every prediction model like that exploded, everything went out of distribution really fast. In your sense we've been doing "CL" for a decade or more. It can also be cheap if you use smaller models.

But true CL is the ability to learn out of distribution information on the fly.

The only true solution I know to continual learning is to completely retrain the model from scratch with every new example you encounter. That technically is achievable now but it also is effectively useless.

Yes and no - the ones that exploded - and there were many - got shut down by the orchestrator model, and within 2 weeks it was now a new ensemble of winners - with some overlap to prior winners. To your point, it did in fact take 2-3 weeks - so one could claim this is retraining...

Schmidhuber solved it at a small scale: https://arxiv.org/abs/2202.05780 .

Ehhh KNN doesn’t have a training phase, so it’s really more that the concept of continual learning doesn’t apply. You have to store your entire dataset and recalculate everything from scratch every time anyway.

Yes, that's basically the point. You get 'free' continuous learning just by throwing the new data into the pool. Needing an explicit training step is a weakness that makes CL hard to make work for many other approaches.

For any practical application KNN will need some kind of accelerated search structure (eg Kd-tree for < ~7 dimensions) which then requires support for dynamic insertions. But this is an engineering problem, not a data science problem, it works and is practical. For example this has been used by the top systems in Robocode for 15+ years at this point, it's just academia that doesn't find this approach novel enough to bother pursuing.

>Needing an explicit training step is a weakness that makes CL hard to make work for many other approaches.

On the other hand, not having an explicit training step is a huge weakness of KNN.

Training-based methods scale better because the storage and runtime requirements are independent of dataset size. You can compress 100TB of training data down into a 70GB LLM.

A KNN on the same data would require keeping around the full 100TB, and it would be intractably slow.

Feature engineering is a thing, you don't need the full data source for KNN to do the search in. It is already used extensively in RAG type lookup systems, for example.

Bandits?

Spaced repetition algos

Hit a nerve huh?

I guess I'm ignorant but why do we continue overspending worldwide?

Like do these groups all have heavy metrics about how budget is growing so we actually know anything is in balance here?

I guess I've never seen proper studies around:

"We project tax revenue will grow X amount with this level of confidence therefore we know we can consistently be within Y range of debt growth forever safely"

Or are we all just hoping we all die before someone has to comes in and cuts services spending in half?

I think you're already getting at the "why", which for many situations you can view it as a leveraged investment. This site shows education as the biggest expenditure, and there's some argument about that resulting in an overall return on investment. If that is really the case then there's a large opportunity cost in not overspending.

The analysis is not always done sufficiently (or without bias) but that's the idea.

Sure but that's kinda my point. If they did this analsysis why isn't it blasted everywhere?

I've never seen a politician mention this kind of (we spend x for y returns) discussion or a doc on it holistically.

It's obvious to most people to invest money can give you more in return but that's not really discussed except peicemeal for one program at a time.

It's not blasted everywhere because economics doesn't really resonate with people, I think. It comes up during government budgeting sometimes

Lots of Californians do not benefit from social programs but support them anyway.

Do they have a choice?

Yeah, they can vote against it. Most don't