The blockchain auto-paused two days ago when the working Merkle tree was produced incorrectly. I have debugged and fixed the issue in two ways. First, I identified a few more concurrency issues that can cause data inconsistency and fixed those. Second, I added a global lock to the blockchain to make concurrency issues all but impossible. I decided to rewind one block to the block before the concurrency issue occurred. Every user has been granted a free rewind survivor badge.
Although I realize it is very annoying that the blockchain was stopped for two days and we lost one block, at least the auto-pause feature worked and we didn’t lose more than that.
The fundamental issue that is that the database schema for the blockchain was not designed with concurrency in mind. This was my mistake. After solving quite a number of concurrency issues with the database so far, I realize now there are some approaches I could have taken that would have prevented most of these issues. My plan moving forward is to keep the global lock in place, but also to redesign the blockchain schema in parallel so that we can eventually lift the global lock.
With the global lock, we are probably limited to approximately 10 transactions per second. Once I can lift the global lock, the total limit on transaction volume will be much higher. 10 transactions per second should be adequate for the near-term future, so there should be plenty of time to redesign the blockchain schema.
So long as there are no more issues with the consistency of the blockchain, my next steps are: