Why I Had to Rewind the Blockchain

I had to rewind the blockchain to block number 109 because block 110 was invalid. I have done what I can to fix the errors that caused this problem, but please note that there may be more bugs and I may need to rewind the blockchain at any time. It will be a while before the software is mature enough to be able to guarantee no rewinds.

The fundamental issue was a concurrency bug. To ensure data consistency, I use features of our relational database (MySQL) such as transactions and auto-incrementing keys. However, for storing the current merkle tree, I decided to store that in memory, along with the number of transactions in the tree, because that is much more efficient than retrieving the merkle tree from the database every time it is needed (which is every swipe on the mining page). Unfortunately, I made a bug in this code, resulting in inconsistent data, leading to invalid blocks.

The fundamental issue (simplified) looks like this:

class NetworkState {
  private nTransactions: number;
  private merkleTree: MerkleTree;

  async setData() {
    this.nTransactions = await getNewTransactionCount();
    this.merkleTree = await getNewMerkleTree();
  }

  async getData() {
    return {
      nTransactions: this.nTransactions,
      merkleTree: this.merkleTree,
    };
  }
}

This code has a problem for network services. If multiple requests happen at nearly the same time, which is a regular occurrence in the app, then the number of transactions nTransactions will be out of sync with the merkle tree merkleTree. The await for each of those variables is not atomic.

The fix (simplified) is this:

class NetworkState {
  private nTransactions: number;
  private merkleTree: MerkleTree;

  async setData() {
    const { nTransactions, merkleTree } = await getNewBlockchainData();
    this.nTransactions = nTransactions;
    this.merkleTree = merkleTree;
  }

  async getData() {
    return {
      nTransactions: this.nTransactions,
      merkleTree: this.merkleTree,
    };
  }
}

By setting the variables nTransactions and merkleTree at the same time, without an await in between, this guarantees that the data is updated atomically and that each request will get consistent data.

I have made these fixes to the code, but I cannot guarantee there are not more bugs I haven’t yet identified. Therefore, I have also added another feature to help: several places in the code now perform sanity checks on the data, and if the data is invalid, mining will be paused automatically until I have a chance to investigate the issue and take action. Although pausing the mine is annoying, it is better than having to rewind by several days.

I will do everything in my power to avoid rewinding the blockchain, but I cannot guarantee it will not happen again. I will continue to work on the software to make it more reliable and to make it easier to fix bugs when they do occur.