And we’re off to the races!
In a recent series of performance tests, BigchainDB topped out at 1 million transactions in 26 minutes. Impressive. We know.
Over the past several months, we’ve been running tests to get a sense of BigchainDB’s performance. It has been a long slog, because:
- When we started, our testing tools were fairly rudimentary, and we kept improving them over time.
- There are many configuration parameters, including Tendermint configuration parameters, and it turns out that some of them affect performance a lot.
- Tendermint has an unresolved issue where it gets slower after being under sustained high load. We hoped they’d resolve that, but so far they didn’t.
- Tendermint and BigchainDB kept changing. Sometimes the changes were due to issues uncovered in the course of performance testing, so there was a feedback loop.
About “Transactions per Second“
The meter, second and byte are standard-size things, so if we learn that Joey is 1.8 meters tall and Sara is 1.7 meters tall, then we will agree that Joey is taller than Sara.
Unfortunately, a blockchain “transaction” is not a standard-size thing. If we learn that Blockchain X can record 100 transactions per second and Blockchain Y can record 200 transactions per second, we can’t compare those numbers. The size of the transactions matters. One reason is that bigger transactions (with more bytes) take longer to write to disk. Therefore, when someone tells you that their blockchain can do N transactions per second, ask them the size of the transactions.
In the tests described below, all transactions were 765 bytes.
Some Performance Test Results
We have three performance test results to share today. BEP-23 describes the methodology and results in detail. Some test setup highlights:
- The tested BigchainDB network had four nodes running Ubuntu 18.04.1, BigchainDB Server 6a9064 (between 2.0.0b5 and 2.0.0b6), Tendermint 0.22.8 and MongoDB 3.6.3.
- All four nodes were in the same Azure data center, each one on a different compute-optimized Fsv2-series virtual machine.
- The virtual machine generating the test load was in the same data center. The test load was generated by our bigchaindb-benchmark tool.
- In the first two tests, one million transactions were posted to the BigchainDB network. In the last test, 16000 transactions were posted.
- As mentioned above, all transactions were 765 bytes.
The First Test
The first test tested a BigchainDB network with all BigchainDB nodes running standard BigchainDB Server without any special options turned on.
One million transactions were processed in about 56 minutes without any failure. 99.7% of transactions were finalized within 9.392 seconds, 95% within 5.143 seconds, and 68% within 2.855 seconds. The network finalized an average of 298 transactions per second, with a median value of 320 transactions per second.
The Setup of the Second and Third Tests
The second and third tests had BigchainDB’s “experimental parallel validation” feature turned on. As the name suggests, BigchainDB checked independent transactions (e.g. transactions involving different assets) for validity in parallel, allowing each node to use more than one CPU for validation at the same time.
You shouldn’t use the experimental parallel validation feature in production unless you understand what you’re doing. It uses a hack where BigchainDB tells Tendermint (DeliverTx) that every transaction is valid, before checking it, so BigchainDB can get the next transaction from Tendermint right away. That way, BigchainDB tricks Tendermint into feeding it transactions-to-validate as fast as possible. BigchainDB then validates all independent transactions in parallel. BigchainDB won’t keep the invalid transactions (in MongoDB), but Tendermint keeps all transactions and doesn’t know which ones were invalid. That means Tendermint can’t prove that a node was misbehaving. The reason we did this experiment was to see if parallel validation could help with performance. As we see below, it does. The next step is so see if Tendermint can support parallel validation officially, e.g. by sending transactions to BigchainDB in bulk. It might be okay to use the above-described hack for now, but you need to understand what you’re doing.
In the second and third tests, BigchainDB always responded to Tendermint’s CheckTX requests with “true,” i.e. it didn’t spend any time checking the transaction before putting it in the mempool (to replicate it among all the nodes for inclusion in an upcoming block). That’s okay because BigchainDB did check the transaction before including it in BigchainDB’s blockchain. Invalid transactions still didn’t get included there.
In the third test, we changed the flag skip_timeout_commit to true in ${HOME}/.tendermint/config/config.toml. We noticed a significant delay to create the last block of the test, and skipping the timeout commit seems to help.
Test 2 Results
One million transactions were processed in about 26 minutes without any failure. 99.7% of transactions were finalized within 2.033 seconds, 95% within 0.113 seconds, and 68% within 0.018 seconds. The network finalized an average of 636 transactions per second, with a median value of 636 transactions per second.
Test 3 Results
16000 transactions were were processed in about 18 seconds without any failure. 99.7% of transactions were finalized within 4.358 seconds, 95% within 4.099 seconds, and 68% within 3.081 seconds. The network finalized an average of 889 transactions per second, with a median value of 1102 transactions per second.
Dig Deeper
You can read all the details of how we set up and ran the tests, including our configuration files and scripts, in the BEP-23 folder on GitHub. That folder also includes more detailed test results, including charts.
Follow us on Twitter and LinkedIn to get the latest announcements. Visit our website to learn more about us and to Get Started using BigchainDB software today. Ask any technical questions you may have to our developers on Gitter. If you’re a developer using BigchainDB, we want to hear from you. Send us an email at contact@bigchaindb.com and tell us your story.
