With 1TB of RAM you can run nearly anything available (405B essentially being the largest ATM). Llama 405B in FP8 precision fits in H100x8 which is 640GB VRAM. Quantization is a very deep and involved well (far too much for an HN comment).
I'm aware it "works" but I don't bother with CPU, GGUF, even llama.cpp so I can't really speak to it. They're just not even remotely usable for my applications.
> tokens per second
Sloooowwww. With 405B it could very well be seconds per token but this is where a lot of system factors come in. You can find benchmarks out there but you'll see stuff like a very high spec AMD EPYC bare metal system with very fast DDR4/5, tons of memory channels, etc doing low single-digit tokens per second with 70B.
> ill get a GPU too but not sure how much VRAM I need for the best value for your buck
Most of my experience is top-end GPU so I can't really speak to this. You may want to pop in at https://www.reddit.com/r/LocalLLaMA/ - there is much more expertise there for this range of hardware (CPU and/or more VRAM limited GPU configs).
literally 90% of AI startups right now are some combination of the above and their "edge" is just a long system prompt that tells them to "be this niche and output response that mimics an expert from that niche"
This was never going to work out because OpenAI is literally watching everybody and how they are using it including prompts. AI wrapper startups serve these functions:
- wrappers prop up NVIDIA which are owned by the people giving VCs the money to invest in wrappers who are long Nvidia too.
- wrappers give OpenAI RSU holders a fuzzy warm feeling and they are able to see what the market is doing with their product so they can pull "Amazon Basics"
- IRS, attorneys, accountants, politicians love them all because they all get to squeeze milk from the fume
While the space is hot and most will fail, I disagree with the general negative sentiment. The same could be said for most/all startups. They all can be distilled into some simple implementation.
Exactly. "All SAAS for b2b is just an SQL wrapper". That's until they do something distinguishing, make implementation easier/faster/cheaper or come up with a new technology. Yes the playing field is level now, but what can they do in 1,3,10 years? How will new models play into this?
> The same could be said for most/all startups. They all can be distilled into some simple implementation.
Hard disagree. There are lots of niches and businesses are complex. If everything was CRUD the people who are using no-code app builders would win against the ones who code, which hasn’t happened despite continuous attempts for 20-or-so years.
Now, I don’t know if GP is right, but assuming that it’s true (many prompt wrapping startups), then yes absolutely they are completely owned if they sit on top of an existing “platform” which can see everything they’re doing. That’s much worse even than selling IaaS on top of aws.
It was the same during the blockchain hype peak – you’d see lots of startups that forked some existing project and replaced the name of a token, and created a website to drive hype.
Not sure if you are trying to prove my point or not. Indeed businesses are complex. With every wave there are a large number of copycats and business with no moats. Many factors go into who win and who does not. Unlike blockchain and whatever you are trying to prove. There is already value being proven with the current wave in AI. Overvalued in the near term but undervalued long term.
Not sure anyone was trying to say no code apps would win.
We can already see that it’s possible to create comparable models in the LLM space. You could just as easily run your business off OpenAI models through azure. You have a hard sell trying to convince that azure is going to steal your data and roll a product of their own. Might as well tell us aws is stealing your web app.
> Not sure if you are trying to prove my point or not.
Yeah maybe we’re talking over each other. My point is that it all depends on what you build. A wrapper with an elaborate prompt does not too me sound like moat (I can be wrong). That’s where the bigger fish would not leave money on the table and eat them all.
That said, I suspect the company in question has much more moat than that. End user products make or break based on details in UX, even if they’re “doing the same thing” from 1000m away. And once the dust settles, the tech that’s integrated well into products win anyway, no matter the VC hype and valuations.
> you can bet this isn't the exit most parties wished for. Although by this stage the founder(s) is probably happy to find a buyer who will retain their team.
I've seen a startup that was bootstrapped 10 years ago but took on too much money and ended up getting acquired for "undisclosed sum" but at the same time everybody knew the investors were just recouping their money while founders got nothing (they had sold their equity for raising more money as their costs were way up but nobody was buying their product). Cue local news reporters don't know the nuances so they'll just announce "Startup X acquired by Y, wow!"
Founders place a lot of emphasis on getting "acquired" but there is roughly 95% chance of you selling at breakeven where the terms leave you with a year's worth of salary as a junior SWE
Get ready to see a lot more of these "acquired" news coming out in the near future. something around less than 1% chance of getting more money than you put in.
Startup has horrible odds especially if you are building a simple GPT wrappers
just tried it out and it struggled with the bubble caption and adopting other drawing styles but oh god yes this is awesome because an image like this would take forever for me to do and if even arranging someone to commission it is expensive
starving artists are going to famish now, not sure how to feel about it
I’m not sure what saar means here but these images are fairly standard and a drop in the bucket compared to the hideous number of porn fine tunes published daily on civit ai if that’s what you’re looking for
