Why is there extra data that's part of the transaction hash, but not part of the cryptographic signature? (This fact seems to be the source of the problem.)

This is probably not by accident, so what does this design accomplish? Why isn't every bit of the transaction signed?

Why is there any choice in how the signature is encoded? Why is there some sort of "script" (I'm not sure I understand this part) that's important enough to be hashed, but not important enough to be signed? This looks like a very deliberate design decision.

If the developers want to avoid transaction malleability, they will not be able to include the signature in the hash.

Would that be a problem? Since a transaction is already uniquely identified by the inputs, why not hash everything that's signed and use that as the identifier in (tx, index) pair?

Bitcoin has good/interesting reasons for using scripts inside transactions, but AIUI these are separate from the signature script.

1. Do the attackers (users who abuse this), modify all transactions they see or do they target those transactions that only involve their wallet?

2. Even given the relative ease an attacker would have with modifying a transaction, that transaction still needs to be accepted by the majority otherwise their modification is ignored. So given 2 transactions for the same bitcoin transfer, I would guess that 50% of the time their transaction is accepted. I guess that the attacker could modify the transaction, for example, 4 times and then it's an 80% chance that one of their modification is accepted (4 out of 5). Do I have that right?

This was a good article, I enjoyed it's breakdown with examples.

2) You don't have to make it to a majority, you only have to propagate your transaction to the miner who wins the next block prior to the original broadcaster propagating their transaction to miner of the next block. This is really easy to arrange in a P2P network by, e.g., controlling more nodes than the original party or by e.g. modifying the standard Bitcoin client to not sleep between attempts to propagate transactions or by e.g. communicating preferentially with nodes known to be controlled by or "close to" miners and not bothering to propagate to the 99.X% of nodes which are not.

If you control thousands of nodes, which you can trivially do ("rent a botnet"), you will ROFLstomp the propagation capabilities of most people originating transactions.

The ease of creating extra nodes is why Satoshi designed Bitcoin as "One CPU one vote" (for a transaction's correctness) rather than "One IP one vote." Unfortunately, it appears that Satoshi did not anticipate that the transaction hash is, effectively, the latter. That wouldn't be problematic except systems which interface with Bitcoin frequently rely on the transaction hash being stable but it is, in effect, written on water for the first hour or so.

Say you receive 10 Bitcoin as, I don't know, your salary. You then spend 0.01 Bitcoin on a sandwitch, leaving you with 9.99 in change. Because you cannot safely spend change until after the sandwitch transaction is fixed in the block chain, this freezes your 9.99 remaining Bitcoin for about an hour. They're still yours, you just can't spend them.

If you hypothetically try to spend one after 10 minutes (maybe you need to, I don't know, pay rent or do something people routinely do with actual money), you have two options:

1) If your counterparty is on the ball, they'll say "Nope, you don't have a single Bitcoin to your name right now." (You know you do, but he can't prove it, as 100% of your Bitcoin are currently in flight rather than being in accounts demonstrably under your control.)

2) If your counterparty is not on the ball, two outcomes:

a) If your sandwich transaction is mutated and the mutated version makes it onto the block chain first, or if it is mutated and the mutated version loses the race to be confirmed but the confirmation of your original transaction happens on a block that doesn't end up in the block chain due to reorganization [+], then your rent payment fails. Your counterparty will discover later that they don't have the money that they expected, even if they watched you send it.

b) If everything goes right, your transaction succeeds, and your counterparty does not realize that you just paid your rent with the Bitcoin equivalent of "The check's in the mail! Honest!" This continues happily until the check is, well, not in the mail.

Bitcoin is currently undergoing active attack, causing many transactions by people who don't understand the inner workings of the system to fail. This attack is capable of disrupting (a portion of) transactions worldwide on Bitcoin for about a few tens or hundred dollars a day in botnet renting cost and could be coded by a CS102 student, if you told them where to look and what to do.

[+] "What?" Glad you asked. See, the fundamental theory of Bitcoin is that miners throw immense amounts of hashing power to create "blocks" in a sequence. Each block references the last block. Each block also encodes transactions. However, everyone is racing to discover the N+1 block after N is released, so there can actually be multiple N+1 blocks. Only one of them will eventually win. (Bitcoin breaks ties based on height, so if N+1 #1 gets another block concatenated to it before N+1 #2, then it is much more likely to be the block which actually matters. #2 vanishes from consensus history, along with all of the transactions inside of it. That normally isn't a problem if you just rebroadcast your transaction, but if N+1 #2 had "your" transaction and #1 won with the mutated version of your transaction, then the transaction which you actually made is suddenly, retroactively, different from the one you think you made.)

The reason everyone keeps talking about "roughly an hour" is because blocks happen at plus or minus every ten minutes, and after N+6 blocks, it is vanishingly unlikely that a transaction in block N will be removed from history by block N being replaced by a totally new chain. You could, for lower risk transactions, count it as "Probably good enough!" after N+2, as long as you're OK with occasionally having your transactions retroactively vanish.

I agree that the current system of address management is impractical for some applications.

It should be possible to automatically organize addresses in a way that small amounts are sent from addresses that also have small balances, so that this problem does not occur. Your salary payment provider could split up the payment into 9 + 10 * 0.1 Bitcoin.

But it will take a while until the concept and the UI is there.

At least as long as people aren't using the selfish mining strategy.

If one or more large pools were doing selfish mining, you might see long parallel chains with some regularity.

This triggered an actual irlol. :D

edit: To clarify, once the mutated transaction is on the blockchain even once, the wallet should be able to figure out what the correct transaction is now. True, when it's not on the blockchain at all, the transaction can get rejected.

They effectively forked the sandwich payment, and your rent payment is only valid on your branch. The rent money will come back under your control because the transaction will be refused, but you'll have to make a new transaction (on top of the now-valid mutated branch) to actually pay it.

#1: What if your client kept your bitcoins spliced over several addresses ? So, each address kept just 10 BTC. Now, when you make a 99BTC transaction, you use up 10 addresses and 1 BTC change gets tied up for further confirmation. (adjust the numbers as convenient of course)

Thus, you basically can't spend your change until 6 confirmations but now it's far lesser than the 9.99 you surmised.

#2: (this one's better!) Or what if you just kept 10 separate addresses and used 1 at a time manually ? You'll tie up 1 address for up to an hour but the likelihood of making 10 transactions an hour is fairly low, and if you did want to do that, you could still split and proceed.

Entropy for BTC addresses is fairly high and each person on the planet can have far more than 10 so this is a non-issue in terms of availability.

Clients could manage these scenarios fairly easily if unconfirmed inputs/outputs were indeed to become a limiting factor.

This is how Bitcoin was intended to be used from the beginning, and how it's always been implemented in the reference client, but unfortunately due to a usability bug the most popular wallets these days reuse addresses and are teaching new users this incredibly destructive habit.

The problem started in the original client where it generated new private keys randomly. This meant effectively there was no way to implement a sane backup policy. Later on, they implemented the ability to generate private keys in batches of 100. This is better, but still problematic because your backup have to be refreshed after no more than 100 operations.

Obviously that was a usability disaster, and so wallet designers worked around that by just using the same private key over and over again. Other people worked out secure deterministic address generation so that you could just backup a single seed once, but by the time that got standardized (BIP32) the damage was already done.

Blockchain.info is the most popular wallet, and they've taught about a million newbies to treat single-use payment codes as permanent account numbers. It's a privacy and security catastrophe, but those issues are invisible to most people who only care about ease of use.

So that means that this unspent output issue coupled with the confirmation time lag is basically an implementation problem due to the way wallet systems have evolved.

It can be a pain to generate a new address for every transaction, even for automated software. I suppose these are the kind of issues that show us that bitcoin is still very much a work in progress and while it's always being worked on, it's still in 'beta'.

Seriously, how should Bitcoin replace real money if that remains true? There's a book store round the corner accepting bitcoin - so I go, select a book, pay and hang around an hour until I get to leave with the book? At least I'd have enough reading material...

#1: "Well if you buy a book with a credit card, the bookstore has no guarantee that that charge actually settles successfully."

#2: "Most people won't use Bitcoin directly. They'll keep their Bitcoin with a trusted provider. The trusted provider can verify capability to pay as quickly as a SQL transaction, via any mechanism mutually acceptable to it and the bookstore, which need not involve the blockchain. Providers which routinely allow their customers to steal from other members of the ecosystem won't be trusted for very long, and be forced to transact through the blockchain versus being being extended instantaneous credit."

The response to #2 is typically "We could call those providers 'banks'" and the Bitcoin community is, how shall I put this, divided by how much they like that outcome.

And regarding the creation of banks for bitcoin I find it interesting that we keep coming back towards old and tried solutions. As a bystander it's pretty educative, I feel I've learned quite a lot about how money works (or doesn't work) in all those bitcoin threads we've had lately.

The best guarantee you get when you run a card in the US is "As of the current instant in time, that card is valid and has sufficient credit on it to cover the amount you just charged", but that is very far from a guarantee that the charge will actually be settled without incident. But it's still good enough.

(And if you're scandalized by this, ask me about checks!)

(P.S. Your thoughts and response are always very thoughtful and appreciated).

You don't get to wield much power when your costumers are using you for buying coffee at Starbucks, but not when buying a car or another big purchase, or something shipped from somewhere (since it's reasonably to wait one hour for those to process).

Also, if I understood patio11s first post correctly, that large-purchase wallet would still need an hour after each purchase to clear properly [1]. Now, I don't run around and buy a couple of cars one after the other from different provides but it quite often happens that I need to book flights, hotel accommodations and rental cars for a business trip and I can't wait an hour after each step to go forward to the next one. More wallets here? Am I missing something?

[1] The ideal case seems to be to wait until a transaction is in the blockchain before sending the next one from the same wallet, but that could in theory take even longer.

I don't think book flights and hotel reservations would require you to wait until previous transactions clear, these kinds of businesses always have the option to cancel your booking if they notice a double-spend on your side.

Edit: in general, businesses that receive your payments days in advance of delivering the product, Bitcoin is a better alternative than Credit Cards which can have chargebacks weeks after or charge a high fee

In Electrum, you can right-click on a receiving address with 0.1 coins in it, and choose to spend coins only from that address ("Send from").

Same principle as not carrying around a wallet full of $1000 bills.

#2 just replaces the current system with the same system but another name and different actor, so I still don't get where the advantage lies :)

I can totally see how bitcoin can work in online transactions where credit cards are currently used, especially if fulfillment happens after some delay, but I don't see it replacing cash unless we mint little bitcoins with an embedded chip that contains a non-removable amount of satoshis. But that's very much not viable.

You covered #2 - yep, those are banks. And you'd be giving them a lot of control over the currency if this became the standard means of transacting.

#1 ... if bitcoin is to replace fiat, then it must replace fiat. I can go ahead and walk into a bookstore with a $20, bill and walk out with a book and $10 in change.

It's not a replacement for fiat if there's an hour before I can leave the bookstore, and during the same hour can't buy anything else with my remaining $10...

My answer is much simpler:

#3 It will be fixed.

And I know I'm not alone with that answer.

It's just the risk of third-party mutation, as under the current mischief-making malleability attack, plus the compounded risk created by chains of unconfirmed-transactions, that makes this a problem.

Long before the current attacks, I had my own outgoing payments stuck when they were inadvertently arranged as a dependent chain, and the first transaction never confirmed.

But once you (or the people writing your software) know of such risk, mitigating workarounds are easy and automatable:

You arrange for receipts of large amounts to be broken into smaller units, capping the amount of change that might be 'in flight'. (If necessary, you do this break-up yourself upon receipt, knowing there's a period of up to an hour before that completes.)

Or more generally, you keep small working amounts of mixed-size coins ready for short-term use, simply not expecting to spend a large new receipt instantly. (No one who is living 'hand to mouth' should be relying on Bitcoin for all their life's necessities, anyway. It's still beta!)

When the third-party mutated-transactions risk is contained – within a week or two, all major miners will probably blockade tampered transactions – this risk of stuck funds will evaporate... but the lessons and new software practices for handling corner cases will persist, ready for future similar situations.

It's interesting that patio11 is using receipt-of-salary as an example. Until your bank knows you, classic paychecks can be held for a day or more before full funds are available. Until clearing-practices were changed in the US a few years ago, large paychecks often had most of their funds held as unavailable for 7-10 days – even for checks between large American banks in the same city. This situation persisted for decades.

Further, the fraud of "check-kiting", which is vaguely like the interplay-of-timing-bugs that has tricked some bitcoin-payout systems, was possible in the traditional banking system, on and off, for decades.

Compared to that, are Bitcoin's temporary problems, perhaps involving an hour's holdup, and likely fixable within a week via rapid software iteration, that big of a deal?

When patio11 makes people aware of the transient gotchas and rough-edges, so they can adjust their near-term expectations, that's a public service.

But if these things are portrayed as inherent and insurmountable weaknesses that doom the system? That viewpoint is going to look like 1995 Clifford Stoll.

Also, with the difficulty increasing, block are going to be rarer and you'll have to wait way longer to get a confirmation right?

And what if 3 guys find the same block, at the same time, at 3 very different places in the world? Can't imagine what would happen.

Also, with the difficulty increasing, block are going to be rarer and you'll have to wait way longer to get a confirmation right?

Nope. The network adjusts the difficulty dynamically to target plus or minus ten minutes per block. The reason difficulty is increasing is to keep blocks coming in every plus or minus ten minutes rather than having ASICs generating them several thousand times a second, which is what would happen if the difficulty was still where it was when the entire Bitcoin network ran on Satoshi's Macbook.

And what if 3 guys find the same block, at the same time, at 3 very different places in the world? Can't imagine what would happen.

Feel free to reason it out. Miners each pick one of the three to try to build off. (Many Bitcoiners assume they pick randomly but I believe that to be a very problematic assumption.) Eventually, a miner wins the race. "His" ancestor block more than likely becomes the block which history adopts. The other two, more than likely, vanish and are never heard again. If your transaction was in one of them but not in the block which actually got adopted, you wait until it plays pingpong around the network or you retransmit.

This has happened hundreds of times before.

> You know you do, but he can't prove it, as 100% of your Bitcoin are currently in flight rather than being in accounts demonstrably under your control.

A minor nitpick: you can demonstrate the accounts are under your control, just not via a "normal" bitcoin transaction.

Did you have a different strategy for demonstrating that I control the Bitcoins that aren't fixed with 6+ confirmations yet?

1) A nation-state activity to destroy the credibility and usefulness of such currencies before such currencies become entrenched and their ability to track criminal activity through money transfer is seriously compromised. (depending on who you ask, 'following the money' is either #1 or #2 on the stop/solve crime toolbox)

2) A crime-syndicate increasing their 'profits' by getting the system to pay out in unanticipated ways. Because, at its heart, crime is all about the benjamins[1] so finding ways to increase profits without increasing risk is an ongoing activity.

3) Idle mathematicians/cryptographers/hackers demonstrating their skillz against what they perceive to be a dangerously flawed system. In the hopes that it will {die|improve|humor them}.

[1] US slang for currency.

I do not know if it is modified. Neither of them are confirmed. (I converted my Starbucks card balance to a meager fraction of a Bitcoin. Since then my $7.21 has plummeted to $5.69.)