An LSTM takes a series of values and uses a combination of gates to determine critical information to hold on to or forget as a sequence unfolds. This is a compressive technique that removes the requirement of having all previous sequence information at the time of a particular inference.
This paper "compress sequence information into an anchor token" which is then used at inference time to reduce the information required for prediction as well as speed up that prediction. They do this via "continually pre-training the model to compress sequence information into the anchor token."
I don't care about learning Mandarin, I want to find out how this guy's motivation system works and then download it into my brain.
Doing a PhD and learning Mandarin as a side project?! Doing hours of Anki practice and new note taking, some of it while running on a treadmill? There's just a crazy amount of drive (and what sounds like an epic memory) here.
I don't think people consider base motivation enough when thinking about processes and this guy won some kind of biological and/or upbringing lottery.
Do not underestimate the urge to procrastinate (by still doing productive things, like learning Mandarin) while pursuing a PhD.
I am not sure if this will be the author's experience too, but pursuing a PhD will often leave you exhausted without any hope of ever finding "the final missing ingredient" to solve the problem you are currently tackling. So turning to entirely unrelated problems, however productive they may seem to outsides, suddenly becomes an attractive alternative in order to procrastinate.
>I find that when someone's taking time to do something right in the present, they're a perfectionist with no ability to prioritize, whereas when someone took time to do something right in the past, they're a master artisan of great foresight.
It is so underrated, that I have been led to put off doing more structured procrastination until I have more time. If more people had told me how great it could be, I would be doing it now!
Probably using a QMK firmware-based keyboard where you can access different layers and shortcuts.
I'm using one right now (though mine runs off ZMK which is similar but wireless) which is a split with just 42 keys. The rest--numbers, symbols, function keys, etc. are all under layers. The layout is dynamic because holding down different keys makes the layout 'change' as you do so. Holding down the left spacebar and pressing 'Z' sends 'F1' to the computer while holding down another key on the right half turns my WER/SDF/XCV keys into a Numpad, etc.
Yes, both keys send the same key code to the computer, however, pabloescobyte said they’re using ZMK, so the left/right space bar distinction is happening on the level of the keyboard controller.
I am currently learning to color grade, am an active bedroom musician, enjoy cooking and learning about food science, and am training for my first half marathon alongside my PhD. The side project thing is definitely real.
I’m not sure it’s procrastination. Years ago, when struggling with maths , I learned juggling ( 5 balls, tricks etc ) and ended up spending quite a bit of time on it every single day.
In practice, it made me feel very good, more relaxed, because I was able to learn something new and make progress rapidly - self confidence was back. The maths soon got unstuck and life became good.
I think an increasingly big difference isn't so much drive as it is sort of the inverse, lack of distraction. A lot of people's attention is just permanently dispersed. What's very effective is just cut all low quality or non necessary media or apps out of your life or limit them to say 15 minutes a day.
Then when you have inevitably nothing to do you can either throw in 10 minutes of doing anki flash cards or doing nothing, and that'll lower the bar to learn immensely. If I had to guess one thing that Isaak doesn't do is scroll for hours through newsfeeds or TikTok. With myself and most people I know that's by far the biggest thing to eat time.
> Doing hours of Anki practice and new note taking, some of it while running on a treadmill?
I studied with Anki on long 1hr walks and it worked incredibly well for me. I’d definitely recommend trying it!
Some things I learned were DS/algos, Greek alphabet pronunciation (so that I could read math symbols), the periodic table/chemical properties, and misc LeetCode interviewing stuff.
Essentially I'd have a question like "invert a binary tree", "implement union-find", or "structure of a topological sort". All of these are small enough that I can keep them in my head.
For small algorithms I would just regurgitate the code line-for-line in my head. For more complex ones I would just go over the structure (not actual code), e.g. I know that topological sort can use a stack or queue, you need to track indegree, add nodes to the stack/queue when their indegree == 1, and so on.
I also used this to help learn (and really understand) common runtimes which helped me when deriving the runtimes of my own algorithms.
Since I started with premade decks I had to look at lot of things up during my walks, but that slowed down as towards the end of my studying.
I did all this to prep for interview (which I wrote about on my blog: https://sjer.red/blog/2024/job-hunt/) -- I would say it worked fairly well for me though you definitely need to pair it with LeetCode or similar.
I emigrated twice within ten years (move, learn the language, find a job, than find a job, move, learn the language). I sometimes wonder how does it feel not having to run and push all the time just not to fall behind.
The price for the motivation could be higher you're willing to pay.
I'm guessing for someone learning a new language is relaxing and therefore helps recharging the person after hours of intense PhD work - things like enjoying daily progress, discovery of foreign culture, the euphoria of being able to read and watch new stuff...
Learning a new language is a stressful grind. I've studied CJK at a similar pace as this article, and it's equal parts exhaustion as it is elation. Not for the feint of heart.
Thanks, that was a great read and resonates a lot with me. Looking back, most of my learning new languages, getting in shape, playing the guitar, and making new friends was structured procrastination.
For what it's worth I've been through periods of this multiple times with different things and it never really feels like discipline is required (aside from on some _really_ busy days), you just want to do it if you're actually interested.
I likely couldn't force myself to learn, say, Spanish, if I tried despite it being technically far easier. It's just not interesting to me in the same way.
It’s interesting that you mention motivation/drive on a post like this. I have similar thoughts whenever there are posts about personalized learning technology or improving public education.
> Doing a PhD and learning Mandarin as a side project?!
his matriculation year is 2024 (and fall classes haven't even started) so he's doing a PhD like the pre-med kids were "doing" med school freshman year. people that brag like this don't finish - there were a few in my cohort too that washed out after quals.
Author also graduated high school from Austria early, and finished a Berkeley math degree in 2 years. I’d say author is gifted.
That said, technical PhDs often require a combination of raw mental horsepower, persistence and luck. (Working for the right advisor in a promising area)
I brought about the same smarts as my peers but they graduated in 5 years whereas I did 8 years because I didn’t have the most promising area of research plus I got unlucky.
Not really - I’d say he just really wants to. Just like when you meet someone and want to spend time with them, are binge watching a show, playing a video game, reading a book, writing code : you want to do that thing again, here learn mandarin.
I’m a lot more familiar with Japanese than with mandarin (and use srs), and it’s worth noting than srs essentially turns anything into an extremely satisfying game : you learn quickly, you really wonder how far you can go, so you keep playing.
I believe that’s the killer combo : really wanting to do something, and having that something turned into a game to avoid giving up.
"it" is the youth. The guy looks to be mid-20ies. Back then in those years i could go for 3 days without sleep while working, studying, drinking, etc. and many of my friends and classmates at the University were similar.
You're doing classic strawman. The other 2 guys arguing against me are also doing strawman. I can only wonder why that my lighthearted recall of my youth struck so much nerve in you and those other 2 guys.
Man, have you ever been 20smth years old? Do you really remember it as the time when you were thinking about "heavy price" beforehand and were suppressing your "machismo"?
The point you’re making is ridiculous though, because what Isaak has done is clearly an unusual accomplishment. You’re actually just making excuses for yourself, being too old to have the motivation. It’s a less helpful explanation than just plain curiosity - and that’s available to all age groups.
Yes, yes, and yes? People use youth to excuse all sorts of unhealthy behavior. Some of my friends smoked cigarettes but most of us didn't because why would we harm ourselves on purpose like that?
How we know of that unhealthiness? Because some people have explored those options, and we learned the result. One can say those people did public service for our species.
>People use youth to excuse
You're either doing strawman by substituting excuse for explanation, or you're arguing against evolution and natural selection which placed the peak of exploration/exploitation behaviors ratio at the youth time, be it our species or say wolves or cats, etc. I think there are a lot of good reasons why that peak is at the youth time.
>Some of my friends smoked cigarettes but most of us didn't because why would we harm ourselves on purpose like that?
Well, cigarettes is established harm. Took decades of years and millions of people, and as result we call it an obvious harm today. It looks though to me that there are greater harm out there which can be observed these days - the more and more suppressed exploration instinct which over the generations will result in tremendous negative impact on our civilization, and bringing it back wouldn't be so easy as say quiting smoking (did it myself 22 years ago after a decade of pack-a-day smoking)
> How we know of that unhealthiness? Because some people have explored those options, and we learned the result. One can say those people did public service for our species.
We've known cigarettes are unhealthy for a very long time. We'd know sooner if it weren't for cigarette company lobbying. And we didn't need to wait for people to die of cancer, a look at the label is enough lol.
> or you're arguing against evolution and natural selection which placed the peak of exploration/exploitation behaviors ratio at the youth time
This is at least as fallacious if not more so. I don't have a name for this odd "because of evolution" argument but I hear it a lot when discussing sociology. "Kids smoke cigarettes because cavemen wanted to explore the horizon" isn't a coherent causal explanation at all.
> Well, cigarettes is established harm.
Exactly, and very well established by the time I was a teenager in the 2010s.
> . It looks though to me that there are greater harm out there which can be observed these days - the more and more suppressed exploration instinct which over the generations will result in tremendous negative impact on our civilization
Ok so what I'm getting is that you have a pet theory that kids not smoking these days is a symptom of an overall rejection of some form of naturalism that you believe is overall harming our society somehow. Is that the case?
>I don't have a name for this odd "because of evolution" argument but I hear it a lot when discussing sociology
sociology without evolution/natural selection is like physics without Newton's laws - a religion.
> "Kids smoke cigarettes because cavemen wanted to explore the horizon" isn't a coherent causal explanation at all.
both have the same major component - they are different manifestations of the drive to disobey authority and the established order of things. There are 2 ways of dealing with such things - 1. to convince against and to explain harm, etc vs. 2. to suppress, to force into submission.
>Ok so what I'm getting is that you have a pet theory that kids not smoking these days is a symptom of an overall rejection of some form of naturalism that you believe is overall harming our society somehow.
You picked an easy example (borderline strawman) - smoking (well, it is easy today, after 300+ years it took to establish the harm of tobacco smoking) where the 1. is easy. In many situations the 1. isn't that easy and it requires approach specific to the issue at hands, and the society more and more it seems drives toward applying the 2. as a general solution for all the issues, be it cigarettes or horizon exploration. That general drive of widespread application of the 2. is, i think, very harmful for our civilization.
Imagine you've had an idea bouncing around in your head, or even an emotion, for a long time and you've never been able to express it. Then one day you push a button and a piece of art captures what you've been feeling perfectly.
It's not the craft that drives attachment in this case but the emotional resonance of something that you think should exist finally existing.
AI mentioned above is not at the level of capturing and expressing ideas or emotions beyond "a sad rock song about a breakup". Try guiding it to express any clearly formed musical idea.
Author's attachment is to a large degree based on the false notion that they somehow contributed to the creation process.
The generic, frigid, un-interesting "product" that is produced by said AI is why no one other than the prompter is moved by the result.
That’s just not true. I’ve used Suno to generate songs where I have provided all the lyrics. Those lyrics came from a combo of LLMs and my steering/direct edits and then I ran the lyrics through Suno multiple times until I got something I wanted.
I can agree that:
> "a sad rock song about a breakup"
Is probably not going to capture or express your ideas or emotions because you haven’t given it enough. In contrast, writing the lyrics or giving the model a ton more context can absolutely produce something that captures and expresses your ideas and emotions.
At the end of the day I don’t make music for the masses (hell, I’ve only generated a handful of final songs that I’ve liked) but the people I have made them for (or the ones just for me) have enjoyed them quite a bit.
I’m not a songwriter nor am I a musician and I never will be. That’s not where my skills lie and it’s not a skillset I want to learn and hone. AI/LLM tools give me the ability to express myself in a medium that previously was effectively impossible and it makes people I care about smile and that’s good enough for me.
Providing lyrics is called "writing lyrics". If you think that pasting lyrics into a prompt makes you somehow more involved in the process of writing music I don't know what to say.
> can absolutely produce something that captures and expresses your ideas and emotions
The right analogy here is to imagine an infinite museum where you can wander until you find a piece that expresses your emotions. It has nothing to do with the act of your expression, and everything to do with you resonating with a piece produced by someone/something else.
> At the end of the day I don’t make music for the masses
Fair. But you also don't "make music" for yourself. At most you write lyrics.
It's not closer to producing that it is to composing. In fact, I would say it's closer to composing in the sense that you can at least add lyrics and pick a genre.
Production requires specifying very precise requirements, which the current gen AI is unable to follow. Even at the most fuzzy production level like "a song with strings and a choir", Suno will generate something completely irrelevant. And if you will try to go deeper -- use a classic Moog synth line in the chorus -- don't expect to generate something meaningful.
I won't argue that in the most broad sense, prompt engineering is a creative process. Picking which shoes to wear to work is also a creative process. My argument is that this has barely anything to do with the process of music composition or production. You can literally reuse the same prompt to generate an image or a poem.
I was just about to try it, but the idea of allowing Firefox access to my audio/video to talk to a machine-generated person gave me such a bad feeling, I couldn't go through with it even fuelled by my morbid curiosity.
I did it. The demo is kinda cool. If they want to steal an unshowered, back-lit, messy hair picture of me, go for it. I can't imagine it'd be that useful right now.
It left me speechless after commenting on a (small) text on my hoodie – this made it feel super personal all of a sudden (which is amazing for an AI of course)
I've encountered this in the Pittsburgh area with AHN. Trying to get a provider results in printed lists of outdated numbers, practices not accepting patients, etc.
Frankly I think we need to start breaking laws. A startup needs to offer straight up good care and fuck the web of infinite regulations which support America's for profit health failure.
Doctors can lose their licenses pretty easily so it's going to have to be a straight tech play. Offer as-good-as-possible care entirely outside of the medical profession. AIs are getting good enough that despite the obvious errors they make they are still better than the nothing-burger of care we get here.
I don't see how the law is the problem here. The problem sounds like insurance companies that dodge their duties. We need them to be held accountable. We likely also need the general cost of doing business in healthcare to go down - which they are partly responsible for due to throwing up so many barriers for them to actually pay for anything.
Steelmanning the OP (which I’m not sure I agree with), it’s possible they are alluding to regulatory capture, implying the laws are crafted to benefit for-profit companies first and patients second.
However, I’m not sure going in the direction of less regulation would help. It’s like saying “The for-profit healthcare companies have too much power, so let’s just give them more power.”
It's not so much a capture as a hodgepodge. There are a great many interests, and even in a good faith environment it would be a challenge to get everything right.
Add to that the fact that it's not a good faith environment. There are many forces, not even connected to the industry, who fight to lower prices at any cost. Even if it means finding out too late that you're not actually buying anything at all.
The laws and regulatory environment are "the best compromise people were able to get at the time" rather than any kind of cogent plan.
> We need them to be held accountable. We likely also need the general cost of doing business in healthcare to go down
These two statements are at odds with each other.
> responsible for due to throwing up so many barriers
To me, it's obviously lack of competition that's the problem, you don't want to punish crappy providers, you want to subsidize new ones so the market is flooded with options.
Which can be done right after we solve the monopolization problem in health care service providers, medical equipment providers, and "pharmacy benefit managers."
My dream project is a non-profit smart-contract based insurance system: with double blind analysis of claims by doctors who have nothing to win from approving/rejecting a specific claim, a system were there is no asshole middleman insurance racket company.
I know this tech is disliked in HN, but I am positive that it is possible build something like that, due to the "trustlessness" capabilities.
It's not like starting up a food truck. A health care operation is vast, and no existing providers are going to break the law and risk losing their licences.
You would need to create an entire parallel network. It would cost tens of billions, possibly hundreds.
Most large systems can absorb a certain level of free-loaders and remain stable by shifting that burden to other people. What you’re advocating is a collective action problem and will just make matters worse for everyone else. Unless you have a way to create a critical mass of similar behavior, it’s tantamount to a selfish action that benefits you to the expense of others.
Unless the system gets worse, there will never be enough willpower to change it for the better.
As it stands, the system is already awful for the majority of people, with outcomes like this that become commonplace.
> it’s tantamount to a selfish action that benefits you to the expense of others.
Not unlike having great insurance paid for by your company, while others (just at the cutoff of govt subsidy for the plan) suffer the most and have to pay thousands of dollars for routine care.
It's the ultimate "fuck you got mine", only applied to something that most people can't live without. And while "fuck you got mine" is okay in the context of luxury items, it is not in the case of medicine/housing/food.
So maintaining the status quo is just as, if not more, selfish than protesting the system with a non-payment. But again, keeping the status quo is just letting the wound fester at this point.
2. Provides more health care to more people at lower costs (this requires health insurance folks to lose their jobs en masse. Sorry not sorry kind of a thing)
3. Is politically tenable to be enacted within the current generation (e.g. in time for Millenials to benefit from it in retirement)
Me too, I just don’t think what you suggested is it (or a net positive for anyone but yourself).
My current preference is to 1) start with covering veterans completely at the VA and 2) have Medicare for all phased in over decades by gradually lowering the qualifying age. The first is generally politically feasible and will help identify appropriate problems of scale and the second is slow enough to allow the system to adapt but also help the current generation of younger workers by the time they tend to need more healthcare
Astrophysics peeps, why is it that the danger seems to be only tied to the mass of the object. Given p=mv is (relative) speed essentially the same for all of them or otherwise inconsequential?
The range of mass is typically much larger than the range of velocities. There is an upper bound to the speed for the vast majority of things which are dangerous (escape velocity of the solar system). However there are orders of magnitude differences in mass.
Adding a little context for those not familiar with orbital mechanics: the speed of a circular orbit is directly related to its distance from the barycenter (i.e. the Sun). The asteroid belt is between the orbits of Mars (orbiting at 24km/s) and Jupiter (13km/s). Whilst an asteroid that strays far enough to hit Earth (orbiting at 30km/s) is by definition not in an exactly circular orbit, nor one always between Mars and Jupiter, the difference in speeds isn't that great. That accounts for ddahlen's point about the limited range of velocities.
A very rough calculation of mine involving a hypothetical asteroid in a elliptical orbit extending as far as Jupiter and right down to Earth, assuming no difference in orbital inclination to Earth and no significant gravitational perturbations, would result in a relative speed of 5km/s. The actual impact speed would be greater due to Earth's own gravity, adding an extra 11km/s.
Not all asteroids are from the asteroid belt, but I am under the impression that visitors from the outer solar system (which could be as fast as the upper bound that ddahlen mentions) are much more infrequent than stray asteroid belt objects, so the median impact speed would still be relatively slow.
Earth's orbital diameter is: ~30 x 10^7 km
number of seconds in a year is: ~ π x 10^7 s
so, Earth's orbital velocity is: ~30 km/s
speed of light, c, is: ~30 x 10^4 km/s
so, Earth's orbital diameter is: ~ 1,000 light seconds
and, Earth's orbital velocity is: ~ 0.0001 c
We have spotted a grand total of 2 interstellar objects, they were moving faster, but are many orders of magnitude less numerous then the local stuff.
Just doing some back of the envelope calculations, looks like Omuamua was moving about 165,000km/hr (relative to Earth) when it was about at Earths orbital distance.
This speed is not actually a crazy number, it is a lot faster than the majority of things which could hit us, but there are geometries of things in our solar system which can reach these relative velocities. (For example things in retrograde, IE: reverse orbits) can lead to basically escape velocity + earths velocity.
I'm not an astrophysicist, but I was part of a D&D group with one a few years ago and the topic came up (outside the game). In practice the speeds fall into two fairly tight clusters (asteroids and comets), but you don't even need that to justify focusing on mass. There's a hard lower bound on everything, and also a hard upper bound on any object that is part of our solar system, and it works out to at most a factor of 40ish in kinetic energy between the slowest and fastest possible impacts. The masses of objects of interest have a much wider range.
Everything coming in speeds up when it falls to earth.
Tiny stuff burns up completely in the upper atmosphere, where the pressure is low, because they have low surface area per mass -- the atmosphere can stop them entirely. Their terminal velocity is low. (That is, when the velocity through air is high enough that the drag prevents gravity from speeding up the object any further.)
Medium objects have a higher terminal velocity get deeper into the atmosphere before exploding. Fragments from these (which now have higher surface area per mass) can then be slowed further by the atmosphere and make it to the surface, but not so dramatically. Bits of the Chelyabinsk impactor fall into this category.
Big objects have a high terminal velocity. They make it to the ground largely intact... and without being slowed as much by the atmosphere. That gives you craters and bad days for being a dinosaur.
Not into astrophysics, but I guess "faster asteroids experience higher forces when going through the atmosphere, therefore burning faster". Another explanation could be "most of asteroid's speed comes from Earth's gravity, not from it's initial state".
These are just random guesses though, so I could be completely wrong.
Anyone have reliable numbers on the file sizes here? Doom.exe from my searches was around 715k, and with all assets somewhere around 10MB. It looks like the SD 1.4 files are over 2GB, so it's likely we're looking at a 200-2000x increase in file size depending on if you think of this as an 'engine' or the full game.
