Just a heads-up: this is not the first time somebody has to explain Markov chains to famouswaffles on HN, and I'm pretty sure it won't be the last. Engaging further might not be worth it.
I did not even remember you and had to dig to find out what you were on about. Just a heads up, if you've had a previous argument and you want to bring that up later then just speak plainly. Why act like "somebody" is anyone but you?
My response to both of you is the same.
LLMs do depend on previous events, but you say they don't because you've redefined state to include previous events. It's a circular argument. In a Markov chain, state is well defined, not something you can insert any property you want to or redefine as you wish.
It's not my fault neither of you understand what the Markov property is.
By that definition n-gram Markov chain text generators also include previous state because you always put the last n grams. :) It's exactly the same situation as LLMs, just with higher, but still fixed n.
We've been through this. The context of a LLM is not fixed. Context windows =/ n gram orders.
They don't because n gram orders are too small and rigid to include the history in the general case.
I think srean's comment up the thread is spot on. This current situation where the state can be anything you want it to be just does not make a productive conversation.
> As far as anybody can tell, mathematics is way older than literature.
That depends what you mean by "literature". If you want it to be written down, then it's very recent because writing is very recent.
But it would be normal to consider cultural products to be literature regardless of whether they're written down. Writing is a medium of transmission. You wouldn't study the epic of Gilgamesh because it's written down. You study it to see what the Sumerians thought about the topics it covers, or to see which god some iconography that you found represents, or... anything that it might plausibly tell you. But the fact that it was written down is only the reason you can study it, not the reason you want to.
No, the argument is even dumber than that. The person who writes a poem hasn't created any literature.
The person who hears that poem in circulation and records it in his notes has created literature; an anthology is literature but an original work isn't.
> No, the argument is even dumber than that. The person who writes a poem hasn't created any literature.
Sure they have, by virtue of writing it down. It becomes literature when it hits the paper (or computer screen, as it were).
(Unless you mean to imply that formulating an original poem in your mind counts as "writing", in which case I guess we illustrate the overarching point of value in shared symbols and language and the waste of time in stating our original definitions for every statement we want to make)
> Unless you mean to imply that formulating an original poem in your mind counts as "writing"
You're close. I'm making the point that, in modern English, no other verb is available for the act of creating a poem.
Here's a quote from the fantasy novel The Way of Kings that always appealed to me:
>> "Many of our nuatoma -- this thing, it is the same as your lighteyes, only their eyes are not light--"
>> "How can you be a lighteyes without light eyes?" Teft said with a scowl.
>> "By having dark eyes," Rock said, as if it were obvious. "We do not pick our leaders this way. Is complicated. But do not interrupt story."
For an example from reality, I am forced to tell people who ask me that the English translation of 姓 is "last name", despite the fact that the 姓 comes first.
Similarly, the word for writing a poem is "write", whether this creates a written artifact or not. And the poem is literature whether a written artifact currently exists, used to exist, or never existed.
(Though you've made me curious: if the Iliad wasn't literature until someone wrote it down, do you symmetrically believe that Sophocles' Sisyphus is no longer literature because it is no longer written down?)
> I'm making the point that, in modern English, no other verb is available for the act of creating a poem.
You literally used another perfectly acceptable verb in modern English besides “writing” for the act of creating a poem in the very sentence making this claim, which somewhat undermines the claim.
> Literature, by definition, appeared later than writing.
Literature, by strict defintion, appeared no earlier than writing, but it is only a tentative conclusion from which surviving writing has been found and understood that it appeared later than writing.
> I have no idea why you think the geometric series has anything to do with this -
IgorPartola is perfectly right to mention geometric series, you can easily use a geometric progression to construct a shape with infinite perimeter and finite area, e.g. by gluing together rectangles with height one and width decreasing in geometric progression. With a bit more thought you can also construct a smooth shape having this property.
> your glued together rectangles will have perimeter 2*(1+2)
No. You should think through that perimeter calculation one more time, preferably while drawing a picture.
Here's a hint: the perimeter of a rectangle is no less than its height; you can glue so that the perimeter of each rectangle contributes at least 1 to the perimeter of the union.
I think you're both right. But there are two ways to do what you said and you didn't specify which one.
First, a rectangle of height 1 and width 1/2. The perimeter is 1 * 2 + 1/2 * 2, two sides of height 1 and two sides of width 1/2.
You "glue" the second rectangle. As one may understand this, you glue them by putting them one beside the other standing up, i.e. you glue them along one of the heights. Sorry for the crude ascii art:
Now you have a single rectangle, height 1, and width 1/2 + 1/4. The perimeter is 1 * 2 + (1/2+1/4) * 2. The "added perimeter" in this step is just 1/4 * 2 = 1/2.
Go on doing that and for a rectangle of width 1/n, you only add 2 * 1/n to the perimeter. In the end you get a single rectangle with height 1 and width 2. The perimeter is 2 * 1 + 2 * 2.
---
Now, maybe, you may want to specify that you glue the rectangles along their widths, not their heights.
That way, the resulting shape when you add the second rectangle is not a rectangle but an irregular shape with 6 sides. Sorry for the crude ascii art again:
The added perimeter now is exactly 2 * 1 on each step. Now the final perimeter is infinite but the area is not.
But you didn't specify this option over the other one. And, honestly, if we talk about putting rectangles in a sequence, I think it's more common to think of the rectangles as standing up side by side with their heights together as in the first option. For the second option I would describe the rectangles as having a fixed width of 1 and decreasing heights.
> You're cherry-picking your own examples. It worked in Iowa City.
Indeed, it worked in Brisbane (a metro area comparable to Baltimore in the U.S.) and Lanzhou (comparable to Boston-Cambridge-Newton): congestion was reduced, the environment benefited, and usage increased in many cities that dislodged from that equilibrium and switched to a free-of-charge or symbolic-charge model.
I don't think GP's claim stands, for transit cities big or small.
Further cherry picking. Brisbane's free buses are only the "city loop". The rest of the transit system is fare based. It also has not stood the test of time yet.
> Brisbane's free buses are only the "city loop". The rest of the transit system is fare based
With all due respect, I expect more effort than Googling "are buses really free in Brisbane", then copy-pastig the AI summary. Symbolic charges were mentioned for a reason, both cities have a fixed "fare" of about 30 US cents on their networks.
If you think there are examples of GP's claim that "every major city that tries free transit at scale will eventually snap back to it", feel free to substantiate it by naming major cities which tried the Brisbane-Lanzhou model and snapped back.
> both cities have a fixed "fare" of about 30 US cents on their networks.
What form of corruption-induced lobbying is this now? A sizable advantage of making it actually free is to remove the huge cost of the fare collections infrastructure.
If you remove the fare collection infrastructure, you remove beneficial usage tracking infrastructure too.
There might also be other "social engineering" benefits to having a fixed symbolic charge, as some people argued in this thread. I don't know about that, but I don't think it's _just_ lobbying.
> If you remove the fare collection infrastructure, you remove beneficial usage tracking infrastructure too.
Most of the cost of collecting fares is actually the money. You need machines that can process currency, which are expensive and often requires network infrastructure and middlemen and contractors, and then they have to be secured against theft or card skimming etc., and you need customer service and billing and tech support when the machines break and all the rest of it.
If all you want is to track usage you can just put a simple pedestrian counter at the door and you're not actually disrupting anything if it's offline for a week because you're just looking for statistical sampling anyway.
> There might also be other "social engineering" benefits to having a fixed symbolic charge, as some people argued in this thread. I don't know about that, but I don't think it's _just_ lobbying.
Ambiguous "social engineering benefits" are the sort of thing that implies it is lobbying, because there is no good way to prove or disprove it but it gives someone something to claim is their reason when the real ones are less sympathetic, i.e. they're trying to get the collections contract (or have read a study funded by someone who does) or they just don't like spending money on transit but know that won't be a convincing argument to someone who does.
That's not quite true, what they can measure with the "tap on tap off devices" is people's movement patterns (point to point). That is valuable data that you can't really get just by counting people on and off or taking cash.
In Brisbane I think our ticketing system cost overhead is maybe 10%?
The cost of the programme rolling out new ticketing infra (the first major ticketing system upgrade in ~15 years since we first got integrated ticketing, going from a stored-value smart card to also being able to tap your credit card) is roughly the same amount of money as the annual revenue from fares.
So hundreds of millions of dollars to be saved then, and that's just for the periodic upgrades, which are an up-front cost and therefore cost you even more in terms of time value of money.
Then as long as the system is in place you need to pay ongoing costs to repair and maintain the equipment, enforcement against anyone who skips the fare, payment network fees, customer service for anyone with payment issues or damaged cards, connectivity service for anything that needs to be networked, etc.
And the overhead percentage depends on the fares. If it was ~10% when the fare was $5, what is it when the fare is $0.50? Well:
> If fare revenue is now only about $20 million per year, does it even cover the cost of fare collection? The current ticketing system rollout was expected to cost $371m, but ended up at $434m – which appears to cover operations for 17 years from 2018… so $25.5m per year. [0]
The point of buses is to replace cars, not short walks.
If you make it so that everybody who could walk 5min takes a bus, the bus will have to stop more often - and for longer - which makes it worse for the people who can't just walk 5min.
The trick is to balance the system so that buses (and other forms of transit) are cheaper - and approximately as convenient - as cars, without making them cheaper and more convenient than walking (for those who can still walk).
Fares don't necessarily need to be about financing the system. They can be about setting the correct incentives, and ensuring people value the service they're getting.
It's very unlikely people are actually going to take the bus for a 5 minute walk : the wait time for the bus is going to be on that order of magnitude and you'd need your route to be perfectly aligned and have perfect stop placement for that to happen.
Most likely, you will have extra trips because people won't feel the need to justify the fare.
That's not true on a major avenue that serves 10 different routes, which combined have a frequency of one every couple of minutes.
Also, it doesn't help to make bus stops more spaced, and you may not want a bunch of express routes that skip most stops, because another purpose of buses is to help people with difficulty moving (like the elderly), for whom it's not a 5min walk.
You just want to make the service available, and as good as possible, without incentivising people who could just walk to use it.
Because the actual goal is to displace cars (not walkers, or cyclists, or…)
> If you make it so that everybody who could walk 5min takes a bus, the bus will have to stop more often - and for longer - which makes it worse for the people who can't just walk 5min.
... Eh?
I often hit the leap card weekly cap (24 eur) in Dublin. This absolutely does not lead me to take a bus instead of walking for five minutes, because that would be _insane_. Like, maybe there are a few people who despise walking to an unreasonable extent and do this, but it would not be common. If it was, you'd see people doing it anywhere which has a fare cap (ie. most cities, these days).
It also hasnt worked in other places. Like Estonia. The data for "invest in capacity and speed" is much better then the for "reduce fares". So if you have extra money, the evidence on what to do is 100% clear.
If you're looking for return on investment, then cycle infrastructure is the way forwards. Each mile travelled by bike actually benefits society (less illness etc) whereas each mile travelled by car costs society.
> For every £1 invested, walking and cycling return an average of around £5-6
> A study of New York concluded that, in terms of health: “Investments in bike lanes are more cost-effective than the majority of preventive approaches used today.”
People that walk or bike are also more likely to do small shopping locally. This benefits the local economy and gets less money to international big box retailers, which generally pay less taxes.
If you drive by a small market you often won't park your car to go there. Cars and trucks destroy streets fast. Having less of them keeps repairs less frequent. Infrastructure for walking and biking can exist for multiple decades or even millennia
So you’re saying that people should do shopping locally, spend more, and waste more time to prop up inefficient businesses who don’t benefit from economies of scale? Tell that to voters and then try to win an election. Society exists to make our lives easier, not to prop up as many businesses and employees as possible as a make-work make-taxes program.
Oh my god the obsession with 'economics of scale' is such narrow minded nonsense.
Shopping locally is more efficient, because the distribution network distributes things locally much more efficiently then a bunch of housewives in their SUVs.
Sure its cheaper for the shop providing the food, but for the society its more expensive. What you are completely missing is the massive cost of all the infrastructure and the massive subsidizes therein to create these centralized stores. And then the massive cost in time and capital investment for every users to buy a car to pick that stuff up.
You are also ignoring the massive waste this creates because people only go shopping every 1-2 weeks. And you are ignore the lack of fresh foods in the food system because of this behavior. That of course Americans eventually pay in their medical systems.
If you actually do some research you will see the systematic bias that is in the zoning code, infrastructure cost calculation and services. Walmart often consumer more in just police services then they pay in taxes. Walmart is systematically gaming the property rights system to pay almost no taxes. And yet Walmart uses a huge amount of land and requires a huge amount of public infrastructure to sustain itself.
If you really need to do some massive pickup of stuff for a party or something there are still larger stores you can go to as well, just not for everyday stuff.
Actually having a shop where I can locally pick up fresh food every day or every other day is actually much more convenient and saves far more time. And I know this is crazy for Americans to consider, but as a society it would be nice if people without a car could also buy food sometimes. This video points this out:
Cities only exist because of economies of scale! Public transit, road maintenance, utilities delivery, public recreation facilities, and the plethora of small and large businesses are more efficient in a city because of economies of scale.
What people don’t benefit from is laws to artificially benefit small businesses at the expense of the consumer. Here in New York, we have this stupid law that one corporate entity can only have one liquor retail license. This law was created at the “behest” of the lobby of liquor store owners. The end result of this is that liquor is more expensive than my hometown of Vancouver, despite NYS collecting a significantly lower tax rate than BC province. That money all flew into private coffers, and the consumer still gets bent over in the end.
I also take issue with the implication that Walmart incurring policing costs is bad for society. The implication is that Walmart should either have private police or be a shoplifting free for all. The former is a bad idea because Walmart police don’t have the same responsibility or accountability to the public as public police. The latter is a bad idea because society will collapse without property rights.
> Cities only exist because of economies of scale!
Sure if you want to have an intellectual debate about what the economics of scale means then that's fine but your point about economics of scale about suburban super-markets was still wrong and that was the context of my critic.
The rest of your post is irrelevant to my point. I have not advocated for any policy specifically to help small business. Small shops in cities can and are operate by major cooperation. There is no contradiction between large companies and small/urban locations. Not sure why you are even bringing this up. Are you so 'America'-brained that you think large companies can only exist in large commercial zones right of highway exchanges?
> I also take issue with the implication that Walmart incurring policing costs is bad for society.
I didn't say the issue is that its incurring policing cost, I said the cost it incurs is higher then the taxes its paying. The whole point of taxes is that they finance the operation of a geographical area. And everybody living or operating in that area should help finance that area.
If somebody operates in that area that incurs more cost then benefits then that somebody should only continue to be doing so if people consider it a 'social good'. And supporting Wallmart a highly profitable company, clearly doesn't fall under that.
So designing policies so that a multi-billion $ company can show up and extract value from your town is nonsensically stupid.
In fact you are stealing from other business and people in your area to give more profits to wallmart.
> The implication is that Walmart should either have private police or be a shoplifting free for all.
No 'the implication' is that when a community does land use, infrastructure and tax planning it should consider facts, and consider cost to provide services and infrastructure for to those areas.
The fact is, most communities make most of their money in the 'down town' that is true even for very small town and even villages.
What you are proposing is basically that a community should finance, build and maintain a lot of public infrastructure, then finance continue police and other services far away from where most people actually live to protect cooperate property (and specifically the parking lot) all while Wallmart does everything in its power to pay the absolute minimum back to the community it is in. Both by local tricks and by tricks on a federal level.
I would favour walking. Its far easier to get people to do it and most people can do it. A lot of infrastructure already exists and its cheaper and easier to improve.
If you provide reasonably public transport its far easier to walk. I drive into town, then park and walk, where I currently live because busses are infrequent. I never even owned a car in London because it has good public transport (to be fair, I probably would have if I lived in a suburb and had kids).
It's not XOR; you can do both. Most people who cycle also walk at times.
Why exclude cycling? You're range is ~5x walking - in the time and effort required to walk a mile, you could easily cycle 5 miles.
It doesn't have to be a universal panacea to be valuable. Walking isn't possible for everyone, including some who can cycle, and it's not useful for longer distances like 5 miles, ~2.5 hrs walking or 30 min cycling slowly.
16 minutes per mile (3.75mph) (5 miles in 80 minutes) is basically my 'fast walk' time in the city. My wife who is shorter has a hard time keeping up with that.
Unfortunately stop lights/etc. will also make this longer. My "best" pace where I actually do get tired is about 15 minutes per mile. Few can keep up a 4mph pace even on a dedicated walking trail with no elevation changes for any material length of time.
Most folks average probably greater than 20 minutes per mile, especially for longer walks. 2.5 hours is definitely on the extreme end, but close to 2 hours is probably more realistic for most.
The quote I was replying to was about return on investment. Money spent on cycling infrastructure is money not spent on walking and public transport. Spending money is definitely XOR.
I am arguing that walking for shorter distances and providing public transport for longer distances is a better use of money in most places.
> Why exclude cycling? You're range is ~5x walking
Your range for walking plus public transport is far greater.
One big problem I see with cycling is low uptake in most places - in most (not all, to be fair) places I know cycles lanes are mostly empty.
> Walking isn't possible for everyone, including some who can cycle
You are right that money is XOR. But cycling takes people of roads and improves the overall system while being very cheap. Also, you don't need dedicated super dutch style infrastructure to encurrage cycling. Making it safe for walking and cars to interact, ALSO makes it much, much better for cycling. Its a mistake in thinking that 'cycling' infrastructure is only dedicated cycling lanes. Encuriging cycling almost always pays off, in pretty every systematic measurment ever done on the topic. And if you reach Dutch level it pays of a gigantic amount. Its really low investment high return.
Cycle lanes actually look empty often because they are so much more efficent. They have done some studies on paths that were always empty and the threwput is usually not as bad as people think. To be sure the place you are refering to could be totally empty, it happens, but its not the norm even in the US.
Also, of course in a lot of places, specially in the US where they are really bad at cycling infratructure, they believe that all it takes is for one road to have a cycle lane and then magically people on bikes show up. They just want to get in on the 'hype' and were pressured to 'do something'. You need to actually be able to reasonably safely go from A to B between places that are vaguly useful. Dedicate cycling lanes make sense in places where there is no alterantive and you need cars to go resonably fast. But generally its more about your car infrastructure and how fast and dangerous your cars are. Improving cycling always goes hand in hand with making cars less dangerous, and that always pays off.
It takes some up front investment and actual planning, but its not like that infrastructure is worthless after a few years. Cycle lanes once built basically never get destroyed, so putting it in speculatively while having a long term plan makes a lot of sense.
while the majority of [the savings] consists of traditional transport decongestion benefits, around a fifth are arising from e.g. health, journey quality and safety.
Cycling doesn't replace cars, it just reduces the cost of cars!
Once you're too old, the health benefits are less clear e.g. my mother dangerously broke bones after falling off her bike (I think cause was overloading herself with a grubber in a backpack).
Yes I know. I'm a huge, huge fan of cycling infrastructure. And I agree that it is the highest value.
But even if you have that you still need high quality public transport. Its not either or. And if you are going to invest in public transport, investing in capacity, speed and convenience. Is a better investment then not making people pay.
Odd hill to die on, but if you wish to argue that Iowa City is a serious transit city, but Brisbane and Lanzhou are not, feel free to state your definition of serious transit city. These cities are bigger than Iowa City and their public transport share of journeys to work is higher than any similarly-sized U.S. metro area.
Beware: if there are no true Scotsmen left, and your definition of serious transit city excludes everything apart from ~10 European cities, the conclusions that one can draw from the policies of serious transit cities will be so limited that they will in fact be useless.
A mascot is an animal figure that represents a product or sports team. For example, the penguin named Tux is the mascot of Linux, and the mascot for the Brisbane Broncos rugby team is the horse named Buck the Bronco.
A Markov chain [1] is a discrete-time stochastic process, in which the value of each variable depends only on the value of the immediately preceding variable, and not any variables in the past.
LLMs are most definitely (discrete-time) Markov chains in this sense: the variables take their values in the context vectors, and the distribution of the new context window depends only on what was sampled previously context.
A Markov chain is a Markov chain, no matter how you implement it in a computer, whether as a lookup table, or an ordinary C function, or a one-layer neural net or a transformer.
LLMs and Markov text generators are technically both Markov chains, so some of the same math applies to both. But that's where the similarities end: e.g. the state space of an LLM is a context window, whereas the state space of a Markov text generator is usually an N-tuple of words.
And since the question here is how tiny LLMs differ from Markov text generators, the differences certainly matter here.
>LLMs are most definitely (discrete-time) Markov chains in this sense: the variables take their values in the context vectors, and the distribution of the new context window depends only on what was sampled previously context.
When 'what was previously sampled context' can be arbitrarily long and complex and be of arbitrary modality, that's not a markov chain. That's just being funny with words. By that logic, humans are also a markov chain.
No, context windows are not arbitrarily long and complex. The set of possible context windows is a large finite set. The mathematical theory of Markov chains does not depend at all on what the elements of the state space set look like. The same math applies.
You argue LLMs are Markov chains because the context window is a 'large finite set.' But the physical configuration of the human brain is also a large finite set. We have a finite number of neurons and synaptic states; we do not possess infinite memory or infinite context.
Therefore, by your strict mathematical definition, a human is also a discrete-time Markov chain.
And that is exactly my point: If your definition is broad enough to group N-gram lookup tables, LLMs, and Human Beings into the same category, it is a useless category for this discussion. We are trying to distinguish between simple statistical generators and neural models. Pointing out that they both satisfy the Markov property is technically true, but structurally reductive to the point of absurdity.
I find mask-wearing men hiding in the shadows quite scary.
But not nearly as scary as the person who generalizes "I personally dislike something" to "it must therefore be unhealthy for society and other individuals to enjoy it".
The latter tends to kill more innocents in the long run.
What an idiotic comment; especially the last line!
The psychological effects of prolonged exposure to extreme gratuitous violence in horror films (the sub-genre of torture/sadism/slasher) are well known. The effects are quite detrimental (particularly in youngsters/teens) and in many cases long-lasting. They include Persistent Anxiety, Fearfulness, Avoidance, Obsession, Desensitization towards Violence (especially towards Women), Lack of Empathy, Nightmares, Sleeplessness etc.
In my comment's defense, it at least didn't mistake an article from the Edina High School student newspaper for a scientific study conclusively demonstrating the effects of prolonged exposure to extreme gratuitous violence in horror films.
I purposefully chose the "Edina High School" article for you.
I might also add that the details there are from certified Psychologists/Teachers employed by the School (AP Psychology teacher Heidi Mathers and Edina High School psychologist Samantha Bialozynski). It is highly relevant since it talks about the effects on school students/teens who are the primary market for the horror film genre. These psychologists have more data and are in a better position to observe behavioural changes day-to-day than in outside studies.
I don't think you are capable of understanding more nuanced studies done by NIMH etc. However, i did give a well-respected and often-cited study by Kristen Harrison and Joanne Cantor in the other link. You are encouraged to read the original study (mentioned in the article) for edification.
> Who are the "people at the top" you speak of? Are they just an amorphous blob of executives and politicians?
I don't think you're asking a serious question.
What kind of answer would you accept? It's not like you're going to change your mind if they say that e.g. the Cyvorefrx family from Palm Beach is one of the people on the top, right?
Nor is this question an effecive rhetorical device to convince onlookers: they'll rightfully ignore it just like people ignord "Who are these Guantanamo Bay torturers you speak of? Are they just an amorphous blob of guards and soldiers?"
It's not a serious question because the preceding statement wasn't serious either. It's just a vague anti-elite statement.
At the very least, it's not the CEO's job to keep unemployment rate low. That's the job of politicians and central bankers. To blame unemployment on "people at the top" shows a fundamental misunderstanding of how society is structured. To take your Guantanamo Bay example, it's like blaming it on terror on "the military industrial complex". Is it vaguely directionally correct? Yeah. Is it a cogent statement? No.
But CEOs do influence politicians and central bankers. Heck you have people like Elon openly trying to influence elections in many countries using his money and fame.
reply