Really sad to see. Emad pivoting to posting crypto nonsense on twitter made me think the writing is on the wall for Stability, but I still didn't expect it so soon. I expect they'll pivot to closed models and then fade away.
> Really sad to see. Emad pivoting to posting crypto nonsense on twitter made me think the writing is on the wall for Stability
Open-source AI is a race to zero that makes little money and Stability was facing lawsuits (especially with Getty) which are mounting into the millions and the company was already burning tens of millions.
Despite being the actual "Open AI", Stability cannot afford to sustain itself doing so.
Wow much of what he’s saying in that video is a lie with regards to the history of latent diffusion, creating an open source GPT3, etc. Just taking credit for a bunch of work he didn’t have much to do with.
You know it never ceases to amaze me how even the most respected fall prey to this money laundering scheme. If people even spent some time to read about Tether they would not touch this stuff. It's blood money.
You should probably post some citations to that. Tether most probably have backing for every single dollar they have, and then plenty more. What about it is “blood money”?
I mean, is AI a less sketchy space in 2024 than crypto/blockchain in 2024? Two or three years ago sure, I guess, but today?
The drama around OpenAI is well documented, there are multiple lawsuits and an SEC investigation at least in embryo, Karpathy bounced and Ilya's harder to spot than Kate Middleton (edit: please see below edit in regards to this tasteless quip). NVIDIA is pushing the Dutch East India Company by some measures of profitability with AMD's full cooperation: George Hotz doesn't knuckle under to the man easily and he's thrown in the towel on ever getting usable drivers on "gaming"-class gear. At least now I guess the Su-Huang Thanksgiving dinners will be less awkward.
Of the now over a dozen FAANG/AI "boomerangs" I know, all of them predate COVID hiring or whatever and all get crammed down on RSU grants they accumulated over years: whether or not phone calls got made it's pretty clearly on everyone's agenda to wash all the ESOP out, neutron bomb the Peninsula, and then hire everyone back with at dramatically lower TC all while blowing EPS out quarter after quarter.
Meanwhile the FOMC is openly talking about looser labor markets via open market operations (that's direct government interference in free labor markets to suppress wages for the pro-capitalism folks, think a little about what capitalism is supposed to mean if you are ok with this), and this against the backdrop of an election between two men having trouble campaigning effectively because one is fighting off dozens of lawsuits including multiple felony charges and the other is flying back and forth between Kiev and Tel Aviv trying to manage two wars he can't seem to manage: IIRC Biden is in Ukraine right now trying to keep Zelenskyy from drone-bombing any more refineries of Urals crude because `CL` or whatever is up like 5% in the last three weeks which is really bad in an election year looking to get nothing but uglier: is anyone really arguing that some meme on /r/crypto is what's pushing e.g. BTC and not a pretty shaky-looking Fed?
Meanwhile over in crypto land, over the same period of time that AI and other marquee Valley tech has been turning into a scandal-plagued orgy of ugly headlines on a nearly daily basis, the regulators have actually been getting serious about sending bad actors to jail or leaning on them with the prospect (SBF, CZ), major ETFs and futures serviced by reputable exchanges (e.g. CME) have entered mainstream portfolios, and a new generation of exchanges (`dy/dx`, Vertex, Apex, Orderly) backed by conventional finance investments in robust bridge infrastructure (LayerZero) are now doing standard Island/ARCA-style efficient matching and then using the blockchain for what it's for: printing a consolidated, Reg NMS/NBBO/SIP-style consolidated tape.
As a freelancer I don't really have a dog in this fight, I judge projects by feasibility, compensation, and minimum ick factor. From the vantage point of my flow the AI projects are sketchier looking on average and below market bids on average contrasted to the blockchain projects, a stark reversal from even six months ago.
Edit: I just saw the news about Kate Middleton, I was unaware of this when I wrote the above which is in extremely poor taste in light of that news. My thoughts and prayers are with her and her family.
I and many many people around me use ChatGPT every single day in our lives. AI has a lot of hype but it’s backed by real crap that’s useful. Crypto on the other hand never really did anything practical except help people buy drugs and launder money. Or make people money in the name of investments.
MS vs Sun was about MS doing their usual EEE thing and adding parts to MS Java that weren't part of the spec (which is why it was also a trademark lawsuit).
Google vs Oracle is about whether APIs are copyrightable and Google won.
IME NVLink would be overkill for this. Model parallelism means you only need bandwidth to transfer the intermediate activations (/gradients + optimizer state) at the seams and inference speed is generally slow enough that even pcie x8 won't be a bottleneck.
48 gb suffice for 4-bit inference and q-lora training of a 70b model. ~80 GB allows you to push it to 8-bit (which is nice of course), but full precision finetuning is completely out of reach either way.
Though you're right of course that pcie will totally suffice for this case.
Calling him "the person hired to build Stable Audio" seems a bit misleading? He was in a executive position (VP of product for Stability's audio group). An important position, but "person hired to build" to me evokes the image of lead developer/researcher.
I think that also helps in understanding his departure, since he's a founder with a music background.
It isn't unusual for those in leadership positions to use such phrasing when talking about projects and products. It's not a "taking credit" from the engineers sort of thing, but rather about the leadership of the engineers.
Managing a group of people is not synonymous with doing the actual knowledge work of researching and developing innovations that enabled this technology. I find it hard to believe that the contribution of his management somehow uniquely enabled this group of engineers to create this using their experience and expertise.
A captain may steer the ship, but they're not the one actually creating and maintaining the means by which it moves.
> A captain may steer the ship, but they're not the one actually creating and maintaining the means by which it moves.
And yet virtually everyone will go along with a statement like "The captain sailed the ship across the ocean" or "Captain Kirk charted the Gamma Quadrant" or whatever, so I'm not sure how this serves as an objection to the original phrasing.
Depending on the work, an equal level of technical competency can be very beneficial for leadership to have, if not required in some situations. To debate his contribution as a counterfactual exercise without the context of his leadership is entirely non-productive. HN is always quick to dismiss leadership, despite evidence of good and poor leadership being tautologically debated about nearly constantly here. A talented group of engineers can self-organize, but their collective technical expertise does not translate to business acumen.
The crux of the debate appears to rest on the fact that context matters. If an engineer says to another engineer "I built the spam detection system", it is understood that they mean they either wrote the code or had some direct part in producing it. If an executive says to another executive "I made the Mac", neither is interpreting that as them literally building the thing. They know they are in leadership, the meaning is assumed to be "as a leader".
Ed here. Saw this thread and thought I'd weigh in.
Agreed, I wouldn't say I was hired to build Stable Audio. Crazy talented team of research engineers / software engineers / designers did the building.
Also wanted to clarify that I didn't quit due to concerns around the training data used for Stable Audio. I was proud of the approach we took to training data - a rev share with rights holders. I quit because of the prevailing view on training data at the wider company, as documented in its public response to the copyright office, where it argues that training on people's work without consent is fair use.
FWIW, I am a rightsholder for a number of published songs and recordings. I once spent $12k of my own money on a record and made about $1000 back.
I have spent more blood, tears and money on art than most of you would find even remotely bearable.
I not only consider my songs to be fair use for training a model but I would also honored if my works were included and influenced further musicians in a way that my records probably never will.
The best songwriters I know have other careers and keep on going otherwise. If you actually care about musicians you should make it a habit to go see local live music!
Thanks for setting the record straight! People tend to put a singular face to things, especially if they have strong feelings about it. Leads to a lot of misrepresentation, especially when the context doesn't accompany the message.
HN is usually allergic to recognizing the value of leadership, which always struck me as ironic considering it's leadership that made plenty of startups work.
Why would you want it to be divorced from success? Seems like popularity/commercial success is the best measure of "being good" societally that we have for art.
People making art/games only for themselves is fine of course, but I don't see why society should subsidize it (as long as we have finite resources - beyond that everything goes of course).
> Why would you want it to be divorced from success?
Because art is not inherently successful, and using commercial success as a metric on whether it's "good" is philistinism. Art that is "bad" is at the very least interesting in more than one dimension. I have come to appreciate art that to most would be off-putting because where it excels and interests me, it does so in such unique ways I would never expect. There's no telling where those seeds of ingenuity will grow to and who they will inspire.
Commercially successful art is that which is "popular enough". The value of art is completely subjective, so what other measure do we have?
If there's "bad art" that appeals to you and others enough, then it isn't bad art. There's lots of art that critics and the general public would consider poorly made that is still commercially successful because it has its fans.
"Real" bad art is art that almost no one cares for enough to shell out money. Why should we subsidize that? Just writing this off as philistinism isn't convincing in the least.
It's not clear to me how you can say that in good faith.
There are just too many instances of artists who died destitute only to be considered one of the greats many years after the fact.
And the ones that didn't were bankrolled by a wealthy individual.
thirdly, many artists draw things they don't really want to because that thing sells better than what they enjoy drawing. It's optimizing for a different thing than what the other poster is claiming should be optimized for.
You mention great artists who died destitute - but we consider them great because they are posthumously popular. We can't look into the future to see what stands the test of time, so current popularity and commercial success seem like the best proxies.
>thirdly, many artists draw things they don't really want to because that thing sells better than what they enjoy drawing. It's optimizing for a different thing than what the other poster is claiming should be optimized for.
Since the proposal isn't about UBI in general, why should we care about artists in particular? Many (most?) people work jobs where they do things they don't really want to because it's more useful to others than what they'd actually like to do.
Bear in mind that "art" is not just "paintings and poems". Novels are art. Movies and television shows are art. Music (of all types) is art. Games, indeed, are art.
Basically anything that you might turn to for entertainment and aesthetic pleasure is art, and as humans we need it in our lives—both consuming and, in a great many cases, creating.
But many forms of art have always been commercially non-viable. Many if not most of the best-known painters and sculptors from times past fell into one of two categories: those who had wealthy patrons, and those who died destitute.
Why should we say that only those artists whose work meets some arbitrary standard of commercial success are allowed to create art without spending most of their waking hours doing something else, or else having to constantly worry whether they'll be able to live to see another year?
Because we don't live in a world with UBI, so we have to decide how to allocate finite resources. The alternative to using some arbitrary standard of commercial success is some other arbitrary standard (unless you introduce UBI).
If we want to optimize for enrichment of people's lives through art, "how many people are willing to pay for it" seems like the best proxy we have. Not to mention that people with some financial success will be less costly to subsidize.
Like, if I had to choose between subsidizing two musicians projected to make half of living wage and a single musician earning nothing, to me that's a pretty obvious choice. The musician earning nothing might be a true visionary who will be recognized in the future, but the odds are far more likely that the music they produce just doesn't hold much appeal.
Now, commercial success is of course not a perfect measure for "enrichment", but, again, what's the alternative? Currently it's some government body deciding, though this holds the quite elitist assumption that the taste of the art establishment is better.
Unless you go for UBI, art subsidies will be finite. You need a measure to decide who gets subsidized and who doesn't. We're not talking about people doing art as a hobby after all, but about the government paying people to make art.
I'm not saying artists should optimize for money but financial sustainability. It seems like a good target because it indicates popularity and means you need less subsidies. You don't need to make _all_ the money or even break even, but if nobody is interested in paying you, the odds are good that you're not enriching many peoples lives.
financial stability is a different word for money.
no one is saying the government needs to specifically pay for art, the original discussion was about how to determine the worth of art. You think money is the right proxy, the other poster thinks fulfillment is the right proxy.
yes the other poster was hinting at UBI and using artists as a proxy for people being able to work on what they find interesting. you strawmanned that into an argument that governments are being asked to specifically pay for art.
I've definitely seen edge computing used in the context of IoT to refer to compute done on sensing devices. The less narrow meaning at least isn't really "new".
(I assume this question is about whether models need all those layers during training, even if they don't need them during inference)
Yes. There's the so called "lottery ticket hypothesis". Essentially the idea is that large models start with many randomly initialized subnetworks ("lottery tickets"0 and that training finds which ones work best. Then it's only natural that during inference we can prune all the "losing tickets" away, even though we need them during training.
It's kind of an open question how large this effect is though. As the article mentions, if you can prune a lot away, this could also just mean that the network isn't optimally trained.
I figured this must be a well-known property of neural networks. I'll do some reading on the lottery ticket hypothesis. That is almost exactly what I was thinking when reading the article: sure after you have trained it you can prune the layers that aren't used. But I wasn't sure you could know/guess which layers will be unused before you train.
It strikes me as an interesting open question since if it is the case that you need big networks for training but can use significantly smaller "pruned" networks for inference there are many, many reasons why that might be true. Determining which of the possible reasons is the actual reason may be a key in understanding how LLMs work.
