Invalid TX corrupts the local node's blockchain state

[EnchanterIO]

Steps to reproduce:

1. Submit a valid TX (none 1)
2. Submit an invalid TX (e.g. too big of a size > 32kb) (nonce 2)
3. Submit a valid TX (nonce 3), this TX will fail due to a nonce mismatch in a state (nonce 2)

Additional details

The above behaviour is due to different states being maintained across the application. In this case, the issue is TX Pool forcing a new nonce and not unregistering invalid TXs that didn't get through.

How to reproduce

• Run a node
• Execute the workload truffle test and wait for first txs to be broadcasted

truffle test ./test/04_workload.js --network=sirius

• Shut down now in the middle of the test execution
• Wait for few minutes and boot the node again.
• Every new txs signed by the same account used to performance the test will provide the following error output

ERROR[02-27|15:16:05.557] Nonce not strictly increasing. Expected 6926 got 6928 engine=consensus module=ABCI

Solution

• Ensure an invalid TX doesn't get added to the pool
• If that happens because we have more validation logic on ABCI::checkTx() side than Ethereum in TxPool::validateTx(), remove the TX from the pool or reset the nonce.

Note: I will first submit a PR cleaning the codebase, adding doc adding more logging etc and then a separate PR with a fix