That's a bit of an uncharitable summary. In bases 8, 12, 16, 24 and 32 their model achieved 99.7% accuracy. They would never expect it to achieve 100% accuracy. It would be like if you trained a model to predict whether or not a given number is prime. A model that was 100% accurate would defy mathematical knowledge but a model that was 99.7% would certainly be impressive.
In this case, they prove that the model works by categorising inputs into a number of binary classes which just happen to be very good predictors for this otherwise random seeming sequence. I don't know whether or not some of these binary classes are new to mathematics but either way, their technique does show that transformer models can be helpful in uncovering mathematical patterns even in functions that are not continuous.
A pocket calculator that would give the right numbers 99.7% of the time would be fairly useless. The lack of determinism is a problem and there is nothing 'uncharitable' about that interpretation. It is definitely impressive, but it is fundamentally broken, because when you start making chains of things that are 99.7% correct you end up with garbage after very few iterations. That's precisely why digital computers won out over analog ones, the fact that they are deterministic.
Category error. You want 100% accuracy for an impossible problem. This is a famously unsolved conjecture. The only way to get the answer is to fully calculate it. The task was to make a guess and see how well it could do. 99.7 is surprisingly good. If the task was to calculate, the llm could write a python program, just like I would have if asked to calculate the answer.
There is a massive difference between an 'unsolved problem' and a problem solved 'the wrong way'. Yes, 99.7% is surprisingly good. But it did not detect the errors in its own output. And it should have.
Besides, we're all stuck on the 99.7% as if that's the across the board output, but that's a cherry picked result:
"The best models (bases 24, 16 and 32) achieve a near-perfect accuracy of 99.7%, while odd-base models
struggle to get past 80%."
I do think it is a very interesting thing to do with a model and it is impressive that it works at all.
The problem here is deterministic. *It must be for accuracy to even be measured*.
The model isn't trying to solve the Collatz conjecture, it is learning a pretty basic algorithm and then doing this a number of times. The instructions it needs to learn is
if x % 2:
x /= 2
else:
x = x*3 + 1
It also needs to learn to put that in a loop and for that to be a variable, but the algorithm is static.
On the other hand, the Collatz conjecture states that for C(x) (the above algorithm) has a fixed point of 1 for all x (where x \in Z+). Meaning that eventually any input will collapse to the loop 1 -> 4 -> 2 -> 1 (or just terminate at 1). You can probably see we know this is true for at least an infinite set of integers...
Edit: I should note that there is a slight modification to this, though model could get away with learning just this. Their variation limits to odd numbers and not all of them. For example 9 can't be represented by (2^k)m - 1 (but 7 and 15 can). But you can see that there's still a simple algorithm and that the crux is determining the number of iterations. Regardless, this is still deterministic. They didn't use any integers >2^71, which we absolutely know the sequences for and we absolutely know all terminate at 1.
To solve the Collatz Conjecture (and probably win a Fields Metal) you must do one of 2 things.
1) Provide a counter-example
2) Show that this happens for all n, which is an infinite set of numbers, so this strictly cannot be done by demonstration.
Most primality tests aren't 100% accurate either (eg Miller Rabin), they just are "reasonably accurate" while being very fast to compute. You can use them in conjunction to improve your confidence in the result.
Yes, and we know they are inaccurate and we know that if you find a prime that way you can only use it to reject, not confirm so if you think that something is prime you need to check it.
But now imagine that instead of it being a valid reject 0.3% of the time it would also reject valid primes. Now it would be instantly useless because it fails the test for determinism.
It's uncharitable because the comment purports to summarise the entire paper while simply cherry picking the worst result. It would be like if asked how did I do on my test and you said well you got question 1 wrong and then didn't elaborate.
Now I get your point that a function that is 99.7% accurate will eventually always be incorrect but that's not what the comment said.
LLMs are not calculators. If you want a calculator use a calculator. Hell, have your LLM use a calculator.
>That's precisely why digital computers won out over analog ones, the fact that they are deterministic.
I mean, no not really, digital computers are far easier to build and far more multi-purpose (and technically the underlying signals are analog).
Again, if you have a deterministic solution that is 100% correct all the time, use it, it will be cheaper than an LLM. People use LLMs because there are problems that are either not deterministic or the deterministic solution uses more energy than will ever be available in the local part of our universe. Furthermore a lot of AI (not even LLMs) use random noise at particular steps as a means to escape local maxima.
I think they keep coming back to this because a good command of math underlies a vast domain of applications and without a way to do this as part of the reasoning process the reasoning process itself becomes susceptible to corruption.
> LLMs are not calculators. If you want a calculator use a calculator. Hell, have your LLM use a calculator.
If only it were that simple.
> I mean, no not really, digital computers are far easier to build and far more multi-purpose (and technically the underlying signals are analog).
Try building a practical analog computer for a non-trivial problem.
> Again, if you have a deterministic solution that is 100% correct all the time, use it, it will be cheaper than an LLM. People use LLMs because there are problems that are either not deterministic or the deterministic solution uses more energy than will ever be available in the local part of our universe. Furthermore a lot of AI (not even LLMs) use random noise at particular steps as a means to escape local maxima.
No, people use LLMs for anything and one of the weak points in there is that as soon as it requires slightly more complex computation there is a fair chance that the output is nonsense. I've seen this myself in a bunch of non-trivial trials regarding aerodynamic calculations, specifically rotation of airfoils relative to the direction of travel. It tends to go completely off the rails if the problem is non-trivial and the user does not break it down into roughly the same steps as you would if you were to work out the problem by hand (and even then it may subtly mess up).
>A pocket calculator that would give the right numbers 99.7% of the time would be fairly useless.
Well that's great and all, but the vast majority of llm use is not for stuff you can just pluck out a pocket calculator (or run a similarly airtight deterministic algorithm) for, so this is just a moot point.
People really need to let go of this obsession with a perfect general intelligence that never makes errors. It doesn't and has never existed besides in fiction.
yeah it's only correct in 99.7% of all cases, but what if it's also 10'000 times faster? There's a bunch of scenarios where that combination provides a lot of value
Ridiculous counterfactual. The LLM started failing 100% of the time 60! orders of magnitude sooner than the point at which we have checked literally every number.
This is not even to mention the fact that asking a GPU to think about the problem will always be less efficient than just asking that GPU to directly compute the result for closed algorithms like this.
Correctness in software is the first rung of the ladder, optimizing before you have correct output is in almost all cases a complete waste of time. Yes, there are a some scenarios where having a ballpark figure quickly can be useful if you can produce the actual result as well and if you are not going to output complete nonsense the other times but something that approaches the final value. There are a lot of algorithms that do this (for instance: Newton's method for finding square roots).
99.7% of the time good and 0.3% of the time noise is not very useful, especially if there is no confidence indicating that the bad answers are probably incorrect.
I'd recommend you watch the video which is linked at the top of the Reddit post. Everything matches up with an individual learner who genuinely got stung.
The command it supposedly ran is not provided and the spaces explanation is obvious nonsense. It is possible the user deleted their own files accidentally or they disappeared for some other reason.
> My view is that the approach to building technology which is embodied by move fast and break things is exactly what we should not be doing because you can't afford to break things and then fix them afterwards.
There are a lot of people who don't know stuff. Nothing wrong with that. He says in his video "I love Google, I use all the products. But I was never expecting for all the smart engineers and all the billions that they spent to create such a product to allow that to happen. Even if there was a 1% chance, this seems unbelievable to me" and for the average person, I honestly don't see how you can blame them for believing that.
I think there is far less than 1% chance for this to happen, but there are probably millions of antigravity users at this point, 1 millionths chance of this to happen is already a problem.
We need local sandboxing for FS and network access (e.g. via `cgroups` or similar for non-linux OSes) to run these kinds of tools more safely.
Codex does such sandboxing, fwiw. In practice it gets pretty annoying when e.g. it wants to use the Go cli which uses a global module cache. Claude Code recently got something similar[0] but I haven’t tried it yet.
In practice I just use a docker container when I want to run Claude with —-dangerously-skip-permissions.
I think the general public has a MUCH better grasp on the potential consequences of crashing a car into a garage than some sort of auto-run terminal command mode in an AI agent.
These are being sold as a way for non-developers to create software, I don't think it's reasonable to expect that kind of user to have the same understanding as an actual developer.
I think a lot of these products avoid making that clear because the products suddenly become a lot less attractive if there are warnings like "we might accidentally delete your whole hard drive or destroy a production database."
Google (and others) are (in my opinion) flirting with false advertising with how they advertise the capabilities of these "AI"s to mainstream audiences.
At the same time, the user is responsible for their device and what code and programs they choose to run on it, and any outcomes as a result of their actions are their responsibility.
Hopefully they've learned that you can't trust everything a big corporation tells you about their products.
This is an archetypal case of where a law wouldn't help. The other side of the coin is that this is exactly a data loss bug in a product that is perfectly capable of being modified to make it harder for a user to screw up this way. Have people forgotten how comically easy it was to do this without any AI involved? Then shells got just a wee bit smarter and it got harder to do this to yourself.
LLM makers that make this kind of thing possible share the blame. It wouldn't take a lot of manual functional testing to find this bug. And it is a bug. It's unsafe for users. But it's unsafe in a way that doesn't call for a law. Just like rm -rf * did not need a law.
- sell software that interacts with your computer and can lead to data loss, you can
- give people software for free that can lead to data loss.
...
the Antigravity installer comes with a ToS that has this
The Service includes goal-oriented AI systems or workflows that perform
actions or tasks on your behalf in a supervised or autonomous manner that you
may create, orchestrate, or initiate within the Service (“AI Agents”). You
are solely responsible for: (a) the actions and tasks performed by an AI
Agent; (b) determining whether the use an AI Agent is fit for its use case;
(c) authorizing an AI Agent’s access and connection to data, applications,
and systems; and (d) exercising judgment and supervision when and if an AI
Agent is used in production environments to avoid any potential harm the AI
Agent may cause.
Didn't sound to me like GP was blaming the user; just pointing out that "the system" is set up in such a way that this was bound to happen, and is bound to happen again.
There is also https://en.wikipedia.org/wiki/HumancentiPad (which is almost surely an homage to the movie) which was 2011 and tied in all kinds of tech-aspects like licensing and iPads.
> So really, why wouldn’t I blow that up and start over?
I really wish more people would mention the option of dual booting. Use another separate SSD to install your linux OS and that way you always have the option of going back to your Windows install. You can even reserve some programs for Windows and do everything else with linux.
There's really no need to approach it with a "screw it" attitude. You'll probably get yourself in too deep with that approach.
Windows has a rather famous bad habit of nuking any other OSes installed on the same drive, so you really do need an extra separate drive, which is inconvenient if you don't already have a separate drive.
Yeah this happened to me at least once, and I had to spend several hours with low-level recovery software to get my files back. This was the catalyst that finally got me to ditch windows for good 7 years ago.
This used to happen to me. But there was a time I just had two different drives. With different OS in it and used the bios selector to boot what I wanted
I tried to do this, but booting into a different OS depending on the task just turned into a chore. I tried going full Linux but despite the claims, many games don't "just work" out of the box. Many require tweaking, at minimum. Of the top 100 games in Proton, only 9% are "Tier 1," and reading reviews, even that doesn't guarantee a flawless experience. (https://www.protondb.com/dashboard) On top of this, kernel level anti-cheat games are not supported at all, and trying to run them in VMs result in permanent bans. Worse still, many peripherals have zero driver support. I have Fanatec wheels and pedals and could not get them to run in Linux.
I could live with using Linux for web browsing, but because it doesn't do the other stuff I like, I ended up just staying in Windows and eventually uninstalled Linux.
Mirrors my experience very closely. I really /want/ to use Linux since I really do like GNOME over the Windows Desktop, but if half my peripherals don't work (in this case an Elgato mic where my friends would say I'm either exploding their eardrums or too quiet, and my CREATIVE USB DAC) I'm just very demoralized when trying to use Linux to play games. I've tried the dual boot route, but a spontaneous Discord message of "hop in loser, we're going gaming!" turns into "let me reboot my machine and then Windows update and now my game has to update and it's now 10pm I have to go to bed" just makes me stick with Windows full time.
I'm still really rooting for the Year of the Linux Desktop, and it does continue to get better and better, but I'll keep rooting from the sidelines.
I very much agree. The driver support won't happen until Linux reaches a much larger install base, but that has some hurdles. First, unless Valve creates some kind of kernel-level anti-cheat for SteamOS, we'll never see some of the most popular online FPSs. Frequent cheating is a red line for most players.
Second, Linux needs a standardised *and enforced* application installation method like .exe. One should never, ever, for any reason, ever, need to use the CLI to install an application. Yet there are so many applications out there which require the use of guides/manuals and the CLI to install, configure, or use. This is partly a dev preference, partly to save time, and partly because it's difficult to build and maintain distinct UI for different distros which each have their own quirks. People often ask, "why don't they release this on Linux?" But that's not actually what they're asking. They're really asking, "why don't they release this software on 20 distinct operating systems?" Each distro might have 0.1% of their total addressable market. Unfortunately, even if SteamOS enforces some kind of package manager like flatpak, that's not going to force devs to use it. It would need to be Linux-wide, and that will never happen. So we're left with fragmented install methods across multiple package managers, and a huge headache for people who just want their OS to get out of the way.
This is the way. I've been dual booting with Ubuntu for almost 20 years now and my main finding these days is just how easy it has become and how rarely I need to switch to Windows. Sure, it happens and the option is always there, but Ubuntu as a daily driver is solid.
You weren't the target audience. The target audience was manager types tired of being told no by engineers. Always listen to the quiet parts left unspoken/unacknoeledged.
It's the same thing they tried to sell with low/no-code.
The problem is that the engineer turning what you want into code isn't normally the bottleneck. I would say about 50% of my job is helping people specify what they want sufficiently for someone to implement.
Non-technical people are used to a world of squishy definition where you can tell someone to do something and they will fill in the blanks and it all works out fine.
The problem with successful software is that the users are going to do all the weird things. All the things the manager didn't think about when they were dreaming up their happy path. They are going to try to update the startTime to the past, or to next year and then back to next week. They are going to get their account into some weird state and click the button you didn't think they could. And this is just the users that are trying to use the site without trying to intentionally break it.
I think if managers try to LLM up their dreams it'll go about as well as low/no-code. They will probably be able to get a bit further because the LLM will be willing to bolt on feature after feature and bug fix after bug fix until they realize they've just been piling up bandaids.
I am cautiously optimistic that there will be a thriving market for skilled engineers to come in and fix these things.
Reminds me of the pre-GitHub days, when I had to use CM tools designed to appeal to project and CM managers, not to the poor developers who had to use them every day. Anybody else remember Harvest?
"Professional riders number roughly three to six thousand worldwide, while professional drivers number roughly twenty to forty thousand across major sanctioned series."
Perhaps it's worth posing the question: what sorts of "engineers" might feel threatened by agents? Those doing engineering, or those who spend their careers wading in the shallows? Competent designers with deep comprehension, or, at best, the superficial pedants?
According to the CEO of Medium, the reason is because their founder, Ev Williams, was a fan of typography and asked that their software automatically convert two hyphens (--) into a single em-dash. Then since Medium was used as a source for high-quality writing, he believes AI picked up a preference for em-dashes based on this writing.
I guess one possible avenue of thought is that when I opened the linked article, I had a few seconds to start reading before I got one full screen modal dialog, followed by another 1/5th height popup dialog on top of that to click away.
Not that most websites are any better. My favourites are basically the ones that just show a default "sorry but this content is blocked in your region" text.
Just now I opened a medium site and before I could even start reading I was hit with a popup to download the mobile app, some other popup that I ignored (cookies I guess), and within a second or two, a full screen modal asking me to subscribe. Often I also get a pay wall. All within seconds of opening the site. If that’s not hostile, I don’t know what is.
Needless to say, I closed the tab. No content is worth dealing with that over.
Sure plenty of other sites do it too but “other people do it” doesn’t mean it’s not hostile nor does it excuse the behavior. Medium is and has always been one of my most hated sites because a lot of tech people post there, a lot of medium links are submitted to HN, yet it’s a horrible place for the reader.
Probably the most annoying thing on the web lately is Cloudflare and all the "mysteriously verifying that you're a real human" junk.
Probably the second most annoying thing on the web today is when you click a link that looks interesting but the page you land on almost immediately says you have to do or pay something to actually read the thing the referring page implied. I don't even start reading a Medium article now if I can see that pop-up below - it's just an instinctive reaction to close the tab. I wish people wouldn't link to articles in walled gardens and search engines would remove those articles from their index - or if that's not reliable then exclude entire sites. Those walls break the whole cross-linking model that made the web the success it is and they waste people's time on a global scale.
I recognise that my position may be somewhat hypocritical because I'd rank AI slop as #3 and maybe #1 and #2 are making some kind of attempt to avoid supporting AI slop. But then I'd propose a more draconian solution to that problem as well - one involving punitive penalties for AI companies that scrape others' content without permission to train their models and possibly for anyone else using models that are tainted.
“Probably the second most annoying thing on the web today is when you click a link that looks interesting but the page you land on almost immediately says you have to do or pay something to actually read the thing the referring page implied.”
If you feel you’re entitled to everyone else’s labor - I dunno what to tell you.
On the other hand, if you value your own time so little that the only amount you're willing invest in the quality of what you read is $0 - I also don’t know what to tell you.
Either way, I hope you figure it out.
Medium (at least what it is today) tries to bring down the friction of making valuable content available at a reasonable price.
The alternative solutions the
web has been to come up with is to take the valuable content and lock it up in hundreds of silos (Substack, etc), leave residual low value content marketing available, and then cover most everything else with a browser melting level of “adtech”
If you feel you’re entitled to everyone else’s labor - I dunno what to tell you.
You're perfectly entitled to keep your content commercial if you want. Just don't put it in the same place as the freely available material that everyone else was working with and then complain when people find you irritating. Some of us are content to share our own work for free on the web and to enjoy work that is offered freely by others. We're all doing it right now on HN and many of us run non-commercial blogs of our own too. And we made the web an interesting and useful place long before sites like Medium came along and tried to centralise and commercialise it.
I remember a time when Google search would downrank you if you showed different content to the user then you showed to Google. I wish we had that functionality back.
I remember hearing an argument once that said LLMs must be capable of learning abstract ideas because the size of their weight model (typically GBs) is so much smaller than the size of their training data (typically TBs or PBs). So either the models are throwing away most of the training data, they are compressing the data beyond the known limits, or they are abstracting the data into more efficient forms. That's why an LLM (I tested this on Grok) can give you a summary of chapter 18 of Mary Shelley's Frankenstein, but cannot reproduce a paragraph from the same text verbatim.
I am sure I am not understanding this paper correctly because it sounds like they are claiming that model weights can be used to produce the original input text representing an extraordinary level of text compression.
> If I am understanding this paper correctly, they are claiming that the model weights can be inverted in order to produce the original input text.
No, that is not the claim at all. They are instead claiming that given an LLM output that is a summary of chapter 18 of Mary Shelley's Frankenstein, you can tell that the input prompt that led to this output was "give me a summary of chapter 18 of Mary Shelley's Frankenstein". Of course, this relies on the exact wording: for this to be true, it means that if you had asked "give me a summary of chapter 18 of Frankenstein by Mary Shelley", you would necessarily receive a (slightly) different result.
Importantly, this needs to be understood as a claim about an LLM run with temperature = 0. Obviously, if the infra introduces randomness, this result no longer perfectly holds (but there may still be a way to recover it by running a more complex statistical analysis of the results, of course).
Edit: their claim may be something more complex, after reading the paper. I'm not sure that their result applies to the final output, or it's restricted to knowing the internal state at some pre-output layer.
> their claim may be something more complex, after reading the paper. I'm not sure that their result applies to the final output, or it's restricted to knowing the internal state at some pre-output layer.
It's the internal state; that's what they mean by "hidden activations".
If the claim were just about the output it'd be easy to falsify. For example, the prompts "What color is the sky? Answer in one word." and "What color is the "B" in "ROYGBIV"? Answer in one word." should both result in the same output ("Blue") from any reasonable LLM.
Even that is not necessarily true. The output of the LLM is not "Blue". It is something like "probability of 'Blue' is 0.98131". And it may well be 0.98132 for the other question. Certainly they only talk about the internal state in 1 layer of the LLM, they don't need the entire LLM values.
The point I'm trying to make is this: the LLM output is a set of activations. Those are not "hidden" in any way: that is the plain result of running the LLM. Displaying the word "Blue" based on the LLM output is a separate step, one that the inference server performs, completely outside the scope of the LLM.
However, what's unclear to me from the paper is if it's enough to get these activations from the final output layer; or if you actually need some internal activations from a hidden layer deeper in the LLM, one that does require analyzing the internal state of the LLM.
The LLM proper will never answer "yes" or "no". It will answer something like "Yes - 99.75%; No - 0.0007%; Blue - 0.0000007%; This - 0.000031%" etc , for all possible tokens. It is this complete response that is apparently unique.
With regular LLM interactions, the inference server then takes this output and actually picks one of these responses using the probabilities. Obviously, that is a lossy and non-injective process.
If the authors are correct (I'm not equipped to judge) then there must be additional output which is thrown away before the user is presented with their yes/no, which can be used to recover the prompt.
It would be pretty cool if this were true. One could annotate results with this metadata as a way of citing sources.
Why do people not believe that LLMs are invertible when we had GPT-2 acting as a lossless text compressor for a demo? That's based on exploiting the invertibility of a model...
I was under the impression that without also forcing the exact seed (which is randomly chosen and usually obfuscated), even providing the same exact prompt is unlikely to provide the same exact summary. In other words, under normal circumstances you can't even prove that a prompt and response are linked.
I'm under the impression that seed only effects anything if temperature > 0. Or more specifically that the LLM given a sequence of input tokens deterministically outputs the probability for each possible next token, and then the only source of randomness is in the procedure for selecting which of those next tokens to use. And that temperature = 0 means the procedure is "select the most likely one" with no randomness at all.
The seed and the actual randomness is a property of the inferencing infrastructure, not the LLM. The LLM outputs probabilities, essentially.
The paper is not claiming that you can take a dump of ChatGPT responses over the network and figure out what prompts were given. It's much more about a property of the LLM internally.
> First, we prove mathematically that transformer language models mapping discrete input sequences to their corresponding sequence of continuous representations are injective
I think the "continuous representation" (perhaps the values of the weights during an inference pass through the network) is the part that implies they aren't talking about the output text, which by its nature is not a continuous representation.
They could have called out that they weren't referring to the output text in the abstract though.
To a small degree, yes. GZIP knows that some patterns are more common in text than others - that understanding allows it to compress the data.
But that's a poor example of what I'm trying to convey. Instead consider plotting the course of celestial bodies. If you don't understand, you must record all the individual positions. But if you do, say, understand gravity, a whole new level of compression is possible.
Imagine that you have an a spreadsheet that dates from the beginning of the universe to its end. It contains two columns: the date, and how many days it has been since the universe was born. That's very big spreadsheet with lots of data in it. If you plot it, it creates a seemingly infinite diagonal line.
But it can be "abstracted" as Y=X. And that's what ML does.
I don't think it's the same thing because an abstraction is still tangible. For example, "rectangle" is an abstraction for all sorts of actual rectangular shapes you can find in practice. We have a way to define what a rectangle is and to identify one.
A neural network doesn't have any actual conceptual backing for what it is doing. It's pure math. There are no abstracted properties beyond the fact that by coincidence the weights make a curve fit certain points of data.
If there was truly a conceptual backing for these "abstractions" then multiple models trained on the same data should have very similar weights as there aren't multiple ways to define the same concepts, but I doubt that this happens in practice. Instead the weights are just randomly adjusted until they fit the points of data without any respect given to whether there is any sort of cohesion. It's just math.
That's like saying multiple programs compiled by different compilers from the same sources should have very similar binaries. You're looking in the wrong place! Similarities are to be expected in the structure of the latent space, not in model weights.
For sure! Measuring parameters given data is central to statistics. It’s a way to concentrate information for practical use. Sufficient statistics are very interesting, bc once computed, they provably contain as much information as the data (lossless). Love statistics, it’s so cool!
> That's why an LLM (I tested this on Grok) can give you a summary of chapter 18 of Mary Shelley's Frankenstein, but cannot reproduce a paragraph from the same text verbatim.
I have not known an LLM to be able to summarise a book found in its training data, unless it had many summaries to plagiarise (in which case, actually having the book is unnecessary). I have no reason to believe the training process should result in "abstracting the data into more efficient forms". "Throwing away most of the training data" is an uncharitable interpretation (what they're doing is more sophisticated than that) but, I believe, a correct one.
I think you are probably right but it's hard to find an example of a piece of text that an LLM is willing to output verbatim (i.e. not subject to copyright guardrails) but also hasn't been widely studied and summarised by humans. Regardless, I think you could probably find many such examples especially if you had control of the LLM training process.
> Wouldn't that mean LLMs represent an insanely efficient form of text compression?
This is a good question worth thinking about.
The output, as defined here (I'm assuming by reading the comment thread), is a set of one value between 0 and 1 for every token the model can treat as "output". The fact that LLM tokens tend not to be words makes this somewhat difficult to work with. If there are n output tokens and the probability the model assigns to each of them is represented by a float32, then the output of the model will be one of at most (2³²)ⁿ = 2³²ⁿ values; this is an upper bound on the size of the output universe.
The input is not the training data but what you might think of as the prompt. Remember that the model answers the question "given the text xx x xxx xxxxxx x, what will the next token in that text be?" The input is the text we're asking about, here xx x xxx xxxxxx x.
The input universe is defined by what can fit in the model's context window. If it's represented in terms of the same tokens that are used as representations of output, then it is bounded above by n+1 (the same n we used to bound the size of the output universe) to the power of "the length of the context window".
Let's assume there are maybe somewhere between 10,000 and 100,000 tokens, and the context window is 32768 (2¹⁵) tokens long.
Say there are 16384 = 2^14 tokens. Then our bound on the input universe is roughly (2^14)^(2^15). And our bound on the output universe is roughly 2^[(2^5)(2^14)] = 2^(2^19).
(2^14)^(2^15) = 2^(14·2^15) < 2^(16·2^15) = 2^(2^19), and 2^(2^19) was our approximate number of possible output values, so there are more potential output values than input values and the output can represent the input losslessly.
For a bigger vocabulary with 2^17 (=131,072) tokens, this conclusion won't change. The output universe is estimated at (2^(2^5))^(2^17) = 2^(2^22); the input universe is (2^17)^(2^15) = 2^(17·2^15) < 2^(32·2^15) = 2^(2^20). This is a huge gap; we can see that in this model, more vocabulary tokens blow up the potential output much faster than they blow up the potential input.
What if we only measured probability estimates in float16s?
Then, for the small 2^14 vocabulary, we'd have roughly (2^16)^(2^14) = 2^(2^18) possible outputs, and our estimate of the input universe would remain unchanged, "less than 2^(2^19)", because the fineness of probability assignment is a concern exclusive to the output. (The input has its own exclusive concern, the length of the context window.) For this small vocabulary, we're not sure whether every possible input can have a unique output. For the larger one, we'll be sure again - the estimate for output will be a (reduced!) 2^(2^21) possible values, but the estimate for input will be an unchanged 2^(2^20) possible values, and once again each input can definitely be represented by a unique output.
So the claim looks plausible on pure information-theory grounds. On the other hand, I've appealed to some assumptions that I'm not sure make sense in general.
> That's why an LLM (I tested this on Grok) can give you a summary of chapter 18 of Mary Shelley's Frankenstein, but cannot reproduce a paragraph from the same text verbatim.
I have some issues with the substance of this, but more to the point it characterizes the problem incorrectly. Frankenstein is part of the training data, not part of the input.
reply