It is a very complex phenomenon, with no single driving force. The usual culprit is uncertainty, which itself can have a ton of root causes (say, tariffs changing every few weeks, or higher inflation due to government subsidies).
In more uncertain scenarios small companies can't take risks as well as big companies. The last 2 years have seen AI, which is a large risk these big companies invested in, pay off. But due to uncertainty smallish companies couldn't capitalize.
> The last 2 years have seen AI, which is a large risk these big companies invested in, pay off
LOL. It's paying off right now, because There Is No Alternative. But at some point, the companies and investors are going to want to make back these hundreds of billions. And the only people making money are Nvidia, and sort-of Microsoft through selling more Azure.
Once it becomes clear that there's no trillion dollar industry in cheating-at-homework-for-schoolkids, and nvidia stop selling more in year X than X-1, very quickly will people realize that the last 2 years have been a massive bubble.
No as you and I both know - I can't. Because it's a qualitative view, and not a quantitative one. I would need to know _when_, quite precisely, I will turn out to be right.
And I don't know, because I have about 60 minutes a week to think about this, and also good quantitative market analysis is really hard.
So whilst it may sound like a good reposte to go "wow, I bet you make so much money shorting!" knowing that I don't and can't, it's also facile. Because I don't mind if I'm right in 12, 24 or 60 months. Fwiw, I thought I'd be right in 12 months, 12 months ago. Oops. Good thing I didn't attempt to "make money" in an endeavor where the upside is 100% of your wager, and the downside theoretically infinite.
Your reasoning is correct if you think about negotiating options, or going all in on a trade, but its not quite right for stocks. The borrowing rates for MSFT and NVDA - even for a retail investor - are less than 1% yearly. So if your view is right you could hold a short on them for years. The market cap for these companies has already incorporated a large capex investment for AI DCs. As long as you use a reasonable rebalancing strategy, and you are right that their current investment in AI will not pay off, you will make money.
Mind you, this is a view that exists - a few large hedge funds and sell side firms currently hold negative positions/views on these companies.
However, the fact of the matter is, fewer people are willing to take that bet than the opposite view. So it is reasonable to state that view with care.
You might be right at the end of the day, but it is very much not obvious that this bet has not (or will not) pay off.
Markets are supposed to be decision makers, and having pricing information be public helps firms and other allocators take better decisions. The question at hand is if the flow of information today is so fast that it deters smart decision makers because of other participants who can extract their decisions from them before they can act on them. Wether you believe markets should be more or less opaque should be decided by where you sit on this spectrum imo.
Does anyone understand why they are taking the difference between transformers instead of the sum? It seems to me that in a noise reducing solution we would be more interested in the sum, as random noise would cancel out and signal would be constructive.
Of course, even if I'm right proper training would account to that by inverting signs where appropriate. Still, it seems weird to present it as the difference, especially seeing as they compare this directly to noise cancelling headphones, where we sum both microphones inputs.
The noise isn't truly random; it's just a matrix of small values that shouldn't be taken into account. Subtracting them cancels them out.
As pointed out by a different comment, it's actually the attention we are interested in that is cancelled out *if they are both equal*. This is what the paper mentions in its abstract;
> promoting the emergence of sparse attention patterns
In theory, it is quite clever, and their results seem to back it up.
I suspect that plus vs minus is arbitrary in this case (as you said, due to being able to learn a simple negation during training), but they are presenting it in this way because it is more intuitive. Indeed, adding two sources that are noisy in the same way just doubles the noise, whereas subtracting cancels it out. It's how balanced audio cables work, for example.
But with noise cancelling headphones, we don't sum anything directly---we emit an inverted sound, and to the human ear, this sounds like a subtraction of the two signals. (Audio from the audio source, and noise from the microphone.)
Oh! It's been a good while since I've worked in noise cancelling. I didn't know current tech was at the point where we could do direct reproduction of the outside noise, instead of just using mic arrays! That's very cool, it used to be considered totally sci fi to do it fast enough in a small headset.
Intuitively, audio is way more sensitive to phase and persistence because of the time domain. So maybe audio models look more like video models instead of image models?
I'm not really sure how current video generating models work, but maybe we could get some insight into them by looking at how current audio models work?
I think we are looking at an auto regression of auto regressions of sorts, where each PSD + phase is used to output the next, right? Probably with different sized windows of persistence as "tokens". But I'm a way out of my depth here!
It's the other way around - in hearing, phase is almost irrelevant. At medium frequencies, moving head by a few centimeters changes phase wand phase relationships of all frequencies - and we don't perceive it at all! Most audio synthesis methods work on variants of spectrograms and phase is approximated only later (mattering mostly for transients and rapid frequency content changes).
In images, scrambling phase yields a completely different image. A single edge will have the same spectral content as pink/brown~ish noise, but they look completely unlike one another.
Makes sense! My impression that phase matters from audio comes from when editing audio in a DAW or anything like that. We are very sensitive to sudden phase changes (which would be kind of like teleporting very fast from one point to another, from our heads point of view). Our ears kind of pick them up like sudden bursts of white noise (which also makes sense, given that they kind of look like an impulse when zoomed in a lot).
So when generating audio I think the next chunk needs to be continuous in phase to the last chunk, where in images a small discontinuity in phase would just result in a noisy patch in the image. That's why I think it should be somewhat like video models, where sudden, small phase changes from one frame to the next give that "AI graininess" that is so common in the current models
I have an example audio clip in there where the phase information has been replaced with random noise, so you can perceive the effect. It certainly does matter perceptually, but it is tricky to model, and small "vocoder" models do a decent job of filling it in post-hoc.
Imagine you are Goldman Sachs and a client wants to make a 100mm USD wire transfer to one of their accounts at Citibank. How does citibank know that the account at GS has the money to cover this transfer?
Right now, the way this works is essentially through a lot of trust and some guarantees by the fed. This has some downsides: because you need a lot of confirmations, it makes transfers take longer. Also, small players can't really get in on this system, so some regional banks are at a disadvantage.
How do you make this safer and more robust? GS obviously can't send info on all of its clients accounts and balances to Citi. You could imagine a protocol where the client/GS sends Citi a zkp to prove that the client has the money (as long as all inputs are agreed upon).
Of course, you don't really need zkps. You could also have the fed keep a database on all money in all accounts (like they do in Brazil), so that the bank only has to ask the central bank to give you an ok. But that is a whole lot of power in the hands of a central authority, as well as a single point of failure, which is something banking systems should avoid imo
> How does citibank know that the account at GS has the money to cover this transfer?
At the moment this is all handled with Swift, and I’m not sure you what you gain from adding ZKPs. Depending on the transaction you might send a Swift MT799 with a pre-advice letter, a proof of funds letter, or a blocked funds letter. Again depending on what you’re doing you might need a MT760 to send a bank guarantee or some sort of letter of credit, and finally a MT103 to initiate the actual transfer of funds.
At this point your counter party risk lies with the banking institution itself, and their willingness and ability to complete the transactions they have legally committed to, rather than the account holder, and this risk doesn’t go away with the addition of ZKPs.
But Swift is just a messaging protocol, right? It doesn't handle trust at all - like you said, you need an awful lot of documents for a single transfer.
I think what could be gained with a zkp protocol would be timeliness. Not needing to confirm if the client has funds in the other institution manually or from trusting their in house APIs would be pretty nice.
The Brazilian central bank has a system that does essentially that, and wires here (even for very large sums) take seconds to fill, instead of the usual 2 days for US interbank wires.
Swift is a messaging system, but it absolutely does manage trust. Swift messages can be used to transmit contracts and other financial obligations which are nonrepudiatable.
When using Swift, the financial institution crafts the content of the messages, some of which describe the state of their systems (like an account balance). So as a counterparty, you are trusting the institution, the jurisdiction the institution is based in, and the laws and enforcement in that jurisdiction.
If you introduce ZKPs, perhaps you could take the human out of the message authoring for some message types, but those messages would still be based on the state of systems controlled by that institution, and really a lot of the “trust” involved with Swift transaction is the trust that an institution will meet its future obligations (something ZKPs don’t help with at all). So the end result is that as a counterparty, you would still be trusting… the institution, the jurisdiction the institution is based in, and the laws and enforcement in that jurisdiction.
There are other payment systems that don’t have the same features that Swift has (like the ability to send bank guarantees, or proof of funds letters, etc…) like ACH and SEPA. But if those things are needed, you’ll just use Swift, or a different system entirely.
The delay in processing Swift transactions is also a feature not a bug for large institutions. If I send you a Swift payment for $100, unless one of us is on a watch list or something, it’ll just go through without any additional input required from either of us. But if I wanted to send you $1,000,000,000, at that level the banks want the opportunity to scrutinise the transaction for AML and anti-terrorism reasons. There is no definition of what a transaction that is involved in money laundering or financing terrorism looks like, so these checks cannot be automated in any way. If you want your transaction to go through, you have to answer whatever questions the bank officers ask, provide any material they ask for, and this can include literally anything they deem necessary. If the transaction is successfully completed it is not because you met some statutorily defined requirements, or somehow proved you weren’t money laundering or financing terrorism, it is because you convinced the appropriate bank officer that the risk of them being implicated in money laundering or financing terrorism was small enough to be acceptable in relation to processing your transaction. So ZKPs can’t help you here either.
In more uncertain scenarios small companies can't take risks as well as big companies. The last 2 years have seen AI, which is a large risk these big companies invested in, pay off. But due to uncertainty smallish companies couldn't capitalize.
But that's only one possible explanation!