As a society we have undertaken massive efforts to reduce all of those. Certainly debatable if it's been enough but ignoring the new thing by putting zero effort in while it's still formative seems short-sighted.
Just from the abstract I don't see why not, it's just replacing the feed forward network that's part of all of these models with a very sparse one. The bigger problem is you seemingly have to retrain the model, so you couldn't just drop in llama2 weights from meta and have it work. Which makes it much more limiting. Something that used existing weights would be a lot more practical (like quantization for example).
For BERT, I can see this being useful if you had to make a really fast embedding model. There was a discussion about a fast embedding use case not long ago https://news.ycombinator.com/item?id=37898001
It certainly could, and I wouldn't be surprised if the authors want to try it out on those. You do have issues of past improvements often not quite enhancing more powerful models nearly as much. I'd expect this to possibly not work as well, something like the bigger models ending up with more polysemantic neurons because they're given more ''incentive'' (training time, neuron count, dataset size which they're encouraged to be able to reconstruct) to extract as much possible. This might make so the method performs worse due to this intermingling.
(See the transformer circuits website for that)
(Though I expect there's ways to recover a good chunk of extra lost throughput/accuracy, maybe by doing extra steps to directly steer the training towards breaking apart polysemantic neurons)
There are two issues here -- for one, in big transformers, more compute is in the attention layers, while this work improves only feed-forward layers, which are more important for smaller models and smaller sequence lengths. Second, in many typical scenarios LLM inference is memory bandwidth bound, I'm not sure if it's possible to utilize their approach to reduce required memory bandwidth.
Yes it might. "Reduction of number of neurons" is not static here, unlike traditional pruning approaches, here they still keep all weights, but the network dynamically selects which sub-portion of them to use. There is a related discussion of this in section 3.2 (page 4), but they don't think they mention actual memory bandwidth requirements/wins of their implementation, and probably there can be different tradeoffs for different devices.
Kinda reminds me of all the early Twitter clones where you could post 141 or unlimited or whatever size posts. Just thinking that mathematically bigger is immediately better in all dimensions seems like a common fallacy.
8 I would say is a bit early, there is some pretty gory scenes, and adult themes like "working girls" you might not want to be explaining in detail. I would say 10-12, mine is in the middle.
Although I had to laugh at https://en.wikipedia.org/wiki/Firewalker_(The_X-Files) which is about a contagious alien(?) virus which I remember being super scary. The whole idea of viruses, pandemics and quarantine didn't really phase the kids at all. How times have changed.
Different situation but I have a 6yo with downs and this rings true to me.
He’s always trying to crack a joke. He’ll say loudly “good night Uschi!” (his grandmother’s nickname) to my wife with a grin on his face like he made the funniest joke. Not Seinfeld over here but we laugh pretty hard.
Wrble (https://www.wrble.com) | (Technical Content Marketing: Remote Always) (Developer Customer Success: Remote Always) | Full-time
Come help make logging better. New tech makes logging cheaper and faster, help us spread the word and make customers successful.
At Wrble we have an engineering-first culture and are looking to hire people with technical backgrounds that are tired of the software-grind and want to look at other adjacent positions.
Both positions are fully remote now and in the future.
It's unfortunately true with a ton of things: hotels, rental cars, etc. Taxes, franchise fees, other taxes, facilities charges, etc. can all add on 20% or more. Especially if you're comparison shopping, you really need to verify whether you're looking at a base fee or the final tally.
Airbnb does this all the time. Their advertised prices don’t include cleaning fees.
If Airbnb included those fees the pricing would often come out higher than a hotel in many cases. And hotels include cleaning costs as part of the nightly rate
If you search for a certain date range, Airbnb will calculate and display the full price for each property on the SERP. It makes sense to compare asymptotically when you don't know the length of stay. The cable and telephone companies billing practices on the other hand are purely deceptive with no utility for the consumer.
Resort fees aren't new. Perhaps they've become more common but I haven't noticed. The more common issue with hotels, which isn't really their fault, is you might have 5 different taxes added to the bill which can add up to a good 20% of the total bill.
I think requiring to advertise full price is even more market friendly than sneaking in more fees. This just incentivizes dishonest behavior. It’s hard to advocate against such a rule. The US health system is also in dire need of such a rule.
You can rent cars, houses, computers - what law would normally forbid those financial relationships?
Cable companies and other utility providers need special treatment in the law because they enjoy special advantages as monopolists. Instead of saying "anyone is allowed to dig up the road, bury cables and let the customers choose the provider with the best service" we often allow only one or two cable providers and add some special rules to make them treat their hostage customers better.
>Instead of saying "anyone is allowed to dig up the road, bury cables and let the customers choose the provider with the best service" we often allow only one or two cable providers and add some special rules to make them treat their hostage customers better.
Wouldn't installing conduit be a still-better option for consumers? Also, the government-granted ISP monopoly approach prohibits would-be providers that don't need to use cable.
Anyway, your dichotomy ignores the historical reality: local politicians subsidized builds by promising providers they didn't have to worry about competition, subsidies extracted from the future captured customers, subsidies that are in fact the very things people complain about now.
As far as I understand it, this is about saying a company can't charge you a rental fee for your modem when you're using your own modem you bought yourself. It does seem like a special law just for this shouldn't be necessary.
As a customer we're already paying for "renting" telcos' infrastructure which delivers the service to us. Modem is an essential piece in the infrastructure, without it consumer cannot get the service.
The fact that telcos itemize the modem means it's supposed to be a kind of elective, just as in older days when one could rent a "fancy" phone directly (with a voice-mail indicator!) from the provider. Those were add-ons, with extra fees.
I see no validation for not allowing customers to use own modems. Sure, the techical support fee will be gamed in such case. But at least this will clearly separate the infrastructure from the user equipment.
If modem indeed falls into user equipment category, then the rental must be elective. If it's infrastructure, then it has been already paid for in the delivery fees.
Companies can generally agree to sell you a product only as part of a package deal. You can't get the Happy Meal toy without paying for the burger and fries. So normally they could insist you can't get the cable service without paying for the modem rental.
Whether they are allowed call it a "meal deal" or a "convenience fee" or a "special discount" or a "rental fee" is more about marketing than economics.
>You can't get the Happy Meal toy without paying for the burger and fries.
I'm sure your general point stands, but you absolutely can. And in San Francisco, you technically can only buy the toy separately, though for much cheaper when paired with the meal.
Once again, just like with Covid, we can look to what New Zealand did for the obvious and correct answer:
Have the government lay fibre to every home and business, and allow any ISP to compete for service over that fibre.
Australia tried, and around 10% of homes got connected. But the conservative Liberal government got in power and changed it from fibre to DSL, a move that was universally derided as unfeasible both economically and technologically.
It was known to be a waste of tens of billions of dollars - but Rupert Murdoch doesn't want poor people to have access to information outside his propaganda network, so he handed the election to the Liberals (the conservatives) in exchange for destroying the network and salting the earth so it was hard for even a future government to fix.
Telstra, the big incumbent ISP with a monopoly in many areas, also had alot to do with it. Instead of gigabit fibre, we now have 25mbit DSL, which doesn't work when it's raining even lightly, or in strong winds.
This network was also slower to build, and far more expensive than the fibre.
But as long as it's built with fibre, it works. New Zealand's network is enourmously successful, and now anyone has access to gigabit fibre on their choice of ISP. It works both in theory and in practice, you just must not allow a conservative government to hand it over to monopoly corporations.
The common "government can't be trusted" rubbish is bunk, the government only screwed it up in Australia because that was their goal, the fibre networks have always been a success.
So as long as the government does it it’s ok as long as the government doesn’t do it incorrectly? Therein lies the problem, governmental failure is always a possibility as Milton Friedman observed, and if they fail there’s no penalty to failing other than the size of government increasing :) Even in America the issue is that we’ve over-regulated the internet industry to the point where even a company like Google has issues, although one could argue they did it for different reasons i.e. to force the existing companies to lay down fiber faster. Regulatory capture is the real issue.
This entire discussion is about privately held organizations systematically charging customers for services they did not receive with no oversight to prevent it. Your argument doesn't even make sense in this context.
Some would argue that those "privately held organizations" are able to avoid market pressures via regulatory capture, if not outright grants of monopoly by governments.
If you want to argue that a government-granted public monopoly is better for consumers than a government-granted private monopoly, I'd be inclined to agree. But there are other options that might be better still for consumers, which is what I think the above poster was trying to get at.
Way down they say "Optionally, configure encrypted logging with TLS." but don't mention it in the intro paragraphs, or that it's a best-practice, or that people should do it... just that it's "optional".
I'd suggest signing up for the service and looking at the actual user-facing setup UI (which is how users arrive at these pages). I think it's really hard to miss in the UI, but if you disagree or even just have suggestions how to make it even more obvious, email them to Papertrail; they're responsive.
