Benchmarks

This page contains various benchmarks on the protocol.

Setup and background

Setup and reproducibility

We obtain the plots on this page by running the test_transfer_size() and test_withdraw_size() functions in the src/ directory of the bitcoin-ipc repository (see the Bitcoin-IPC repo page) at version v0.3.0. These functions create the Bitcoin transactions for all defined parameters and write the results to bench-plots/bench-transfer-sizes.csv and bench-plots/bench-withdraw-sizes.csv, respectively. The script bench-plots/plot.py is then used to generate the plots, which are stored in bench-plots/.

That means, the numbers reported are obtained from the actual transactions that the Bitcoin-IPC protocol would submit to Bitcoin. The results are independent of the hardware used for the benchmarks. The ipc repository (see the IPC repo and ipc-cli) is not involved in the benchmarks, as all relevant transactions are created by the bitcoin-ipc repository, hence the results are independent of the version of IPC or Fendermint. Moreover, the Bitcoin Core binary does not need to be running for the benchmarks, as the transactions are not actually submitted to Bitcoin (we create them and measure their size).

Overview of the protocol

For ease of exposition, we summarize here the parts of the protocol that are relevant for the benchmarks. An IPC L2 subnet (the source subnet) periodically batches outgoing transfers to other subnets (the target subnets) – all transfers are batched, even if they have multiple target subnets. These transfers are submitted to Bitcoin as two transactions, which we refer to as checkpointTx and batchTransferTx. Apart from transfers, they also encode multiple subnet functionalities, such as state checkpointing, stake updates in the subnet, and depositing and withdrawing funds to and from the subnet.

The checkpointTx contains one input UTXO (with value the total amount of all transfers and all withrawals) and one output UTXO for each target subnet and one for each withdrawal. It also contains an output UTXO locked with the hash of a script. The batchTransferTx spends this UTXO by revealing the script in the witness. The script contains the details of each transfer, that is, the recipient and amount

The vbytes metric

As most of our experiments measure the transaction size in vbytes, we provide here a brief explanation of how this unit is defined. In Bitcoin, a vbyte (virtual byte) is a unit used to measure transaction size for fee calculation in SegWit transactions, which distinguish between regular and witness data. Each transaction has a weight defined as weight = base_size * 3 + total_size, where base_size is the size of the transaction excluding the witness. Dividing this weight by 4 gives the size in vbytes. This means that witness data – where signatures are stored in SegWit transactions, and where we store details of each transfer – is effectively discounted by 75%, making it cheaper than non-witness data.

Overview of experiments

Our benchmarks measure the amount of data that is submitted to Bitcoin (and the corresponding amount of fees that must be paid) for each transfer between L2 subnets, for each deposit from Bitcoin to an L2 subnet, and for each withdrawal back to Bitcoin. Specifically, we report on the following.

• Experiments 1 and 2: The average size of a transfer, depending on different parameters (number of batched transactions, number of validators in source subnet).

• Experiment 3: The improvement in terms of Bitcoin throughput that Bitcoin-IPC provides.

• Experiment 4: The cost for depositing and withdrawing between Bitcoin and an L2 subnet.

• Experiment 5: The cost that subnet validators incur for checkpointing to Bitcoin.

• Experiment 6: The throughput and latency within an L2 subnet.

• Experiment 7: Further improvement on the transfer size that can be achieved with threshold signatures.

• Experiment 8: Benchmarks showing Bitcoin fees (repeating previous experiments using Bitcoin fees as the metric instead of vbytes)

Experiment 1: Average size of transfer vs total number of batched transactions

By batching transfers together, the amortized size per transfer becomes smaller than sending the transfer directly as a Bitcoin L1 transaction. In this experiment we quantify this improvement.

The next plot shows the amortized size (in vbytes) for a transfer against the total number of batched transfers (across all target L2 subnets). The plot shows one line per number of target L2 subnets (for 1, 2, 5, 10 target subnets). It also shows the cost of sending the transfer natively over Bitcoin, which is constant at 141 vB. Notice that the horizontal axis is in logarithmic form.

We observe that, using the Bitcoin-IPC infrastructure, independently of the number of target L2 subnets, the amortized cost per transfer drops. The more the target subnets, the more expensive the batched transfer is. This is because the checkpoint transaction contains one output UTXO per target subnet and the batch-transfer transaction contains (only once) the address of each target subnet. However, this additional cost essentially vanishes if we batch sufficiently many transfers.

Finding the break-even point

The protocol of Bitcoin-IPC becomes more efficient than native Bitcoin L1 transactions already by batching three or four transfers, depending on the number of target subnets.

Taking batching to the limits

We now explore how many transfers we can batch within a single Bitcoin transaction. Even though the maximum block size is 1M vB, the default policy of Bitcoin core restricts the size of individual transactions to 100K vB in order to prevent spam and oversized transactions. To reach this limit, we increase the batched number of transactions, as long as each of the checkpoint and batch-transfer transactions is at most 100K vB.

We find out that we can batch up to approximately 16,500 transfers within one pair of Bitcoin transactions. For one target subnet, the amortized transfer size converges to approx. 6.07 vB, for 10 target subnets to approx. 6.1 vB. This amounts to roughly 23x (141vB / 6.1vB) scalability improvement, or a throughput of over 160 tps on Bitcoin using Bitcoin-IPC (vs 7 tps on Bitcoin L1)!

Experiment 2: Average size of transfer vs number of validators in source subnet

So far we have assumed that the source L2 subnet has four validators. We now explore what happens when this number changes. The next plot shows the amortized size for a transfer against the total number of batched transfers, this time fixing the number of target subnets and varying the number of validators in the source subnet.

We observe that the amortized size increases with the number of validators. This is due to the additional signatures that must be included in the Bitcoin transaction, as Bitcoin-IPC currently uses multisigs. As before, this cost is smaller if we batch many transactions. Despite the overhead of Bitcoin multisigs, Bitcoin-IPC still leads to lower transaction costs than native Bitcoin. The break-even point now changes with the number of validators – for example, when the source subnet has 36 validators the batching pays off after at least eight transfers.

Experiment 3: Improving Bitcoin throughput by batching transfers

The Bitcoin network has an average throughput of approx. 1,667 vBytes per second (it finalizes one block of size at most 1K vB every approx. 10 minutes). When measured in tps (transactions per second), this corresponds to 7 transactions (of average size) per second.

The protocol of Bitcoin-IPC essentially compresses Bitcoin transfers, so that one transfer requires significantly less than 141 vBytes to be processed by Bitcoin — as we saw, approx. 6.1 vB per transfer. This effectively increases the throughput in terms of tps. In the following diagram we report the maximum throughput (in tps) we can achieve by batching transfers in Bitcoin-IPC.

If we batch very few transfers (such as one or two) we actually waste throughput, because these transfers could be directly implemented as Bitcoin transactions more efficiently. However, by batching three or four transfers (depending on the number of target subnets) the benefit of Bitcoin-IPC starts paying off.

We earlier found out that we can batch up to approximately 16,500 transfers within one pair of Bitcoin transactions, for an amortized transfer size of approx. 6.1 vB. This translates to a throughput of over 160 tps on Bitcoin using Bitcoin-IPC (vs 7 tps on Bitcoin L1).

Experiment 4: Cost of deposit and withdraw

In order to use the infrastructure of Bitcoin-IPC, users must deposit bitcoins from the Bitcoin mainnet to an IPC subnet. These funds can then be used in the subnet or transferred to other IPC subnets. Users can withdraw their funds from any IPC subnet back to the Bitcoin mainnet. In this section we report the cost for depositing and withdrawing.

Depositing from Bitcoin to a subnet

Users deposit bitcoins to a subnet by sending the desired amount to the multisig address controlled by the validators of the subnet. The cost of deposit is constant, independent of the number of validators in the subnet. As deposits are performed by users, they do not support batching.

Withdrawing from a subnet to Bitcoin

An IPC subnet periodically batches all withdrawal requests, in the same way it does with transfers. All withdrawals are included in the checkpoint Bitcoin transaction (as introduced earlier).

The following plot shows the amortized size of a withdrawal for a varying number of batched withdrawals. The current implementation supports batching up to 255 withdrawals per checkpoint transaction. For that number, the amortized size per withdrawal is 43.8 vB.

Experiment 5: Cost of checkpointing

An IPC subnet must periodically post a hash of its state to Bitcoin. This functionality is called checkpointing and it is performed regardless of the existence of outgoing transfers in the subnet. The checkpointing period, at which a subnet submits this checkpoint, is configurable and specified at creation of the subnet. Our implementation adds an OP_RETURN script (we currently need 78 bytes to encode the checkpoint) to the checkpoint transaction (this is the first of the two transactions that are submitted to Bitcoin, as explained in the beginning of this document), resulting to an overhead of 90 vB per checkpoint (on top of the cost for encoding the transfers, as mentioned earlier).

In this section we measure the cost incurred to a subnet by the checkpointing functionality. For varying values of the checkpoint period (see parameter --bottomup-check-period in Subnet creation and initialisation, we report the total amount of data (in vbytes) submitted by a subnet to Bitcoin. We remark that the --bottomup-check-period parameter is specified in number of subnet blocks, but in this experiment we report the approximate actual time it takes to create these blocks.

For a reasonable choice of the checkpoint period, such as 2 hours, the overhead due to the checkpointing is 1080 vB per 24 hours. The tradeoff here is that longer checkpoint periods incur less total overhead, but result is transfers and withdrawals being processed more slowly.

Experiment 6: Throughput and latency within an L2 subnet

The framework of IPC (which Bitcoin-IPC extends) allows each L2 subnet to run its own consensus protocol, as long as it implements certain interfaces and functionalities defined by IPC. We have maintained this feature in the design of Bitcoin-IPC, allowing subnets to use any efficient consensus protocol.

The current implementation uses Fendermint as the consensus protocol in all L2 subnets. Hence, the throughput and latency within an L2 subnet are determined by Fendermint, which is a fork of Tendermint Core. We do not repeat benchmarks on Fendermint or Tendermint Core at this point. Previous results report the throughput of implementations of Tendermint Core at about 500 tps. For more details we refer to this publication, this survey, or the report from the CometBFT team.

Experiment 7: Improving the transfer size using threshold signatures

As we saw earlier, the amortized transfer cost increases with the number of validators in the source subnet. Even though it becomes lower than the Bitcoin transaction size when we batch many transfers, ideally we would like that cost to be independent of the number of validators.

The following plot shows the amortized size of a transfer for different numbers of validators in the source subnet. Observe that the plot contains entries for one validator in the source subnet. This corresponds to an implementation of Bitcoin-IPC with threshold signatures, where only one signature is included in the Bitcoin transaction, independent of the number of validators.

The benefit of threshold signatures (which we plan to introduce in a future release of Bitcoin-IPC) is significant if we batch a small number of transfers, such as 1, 2, 5, or 10 transfers. For larger batches, consisting for example of 50 or 100 transfers, the benefit is smaller. For example, if we batch 50 transfers, then the amortized cost per transfer is 13 vB with threshold signatures and 48.5 vB with a multisig implementation among 100 validators. For batches of 100 transfers, the corresponding sizes are 9.5 vB with threshold signatures and 27.3 vB with multisig among 100 validators.

Experiment 8: Benchmarks showing Bitcoin fees

In this section we repeat some of the previous plots, showing the Bitcoin fees instead of the transaction size in the vertical axis. We assume a fee rate of 200 sat/vB, the median fee rate observed in the Bitcoin network at the last halving block14. For all conversions from satoshis to USD, we assume for simplicity 1 BTC = 100,000 USD.

Fee per transfer vs total number of batched transactions

We now show how, by batching transfers together, the fee per transfer becomes smaller than sending the transfer directly as a Bitcoin L1 transaction.

The next plot shows the fee (in satoshis) per transfer against the total number of batched transfers, for a varying number of target L2 subnets (1, 2, 5, 10 target subnets). It also shows the fee for sending the transfer natively over Bitcoin, which is constant at 28,200 satoshis, or 282 USD (200 sat/vB for an average-sized transfer of 141 vB). Notice that the horizontal axis is logarithmic. As we can see, the amortized fee per transfer converges to approx. 1214 satoshis (12.14 USD) for one target subnet, while for 10 target subnets it converges to approx. 1220 satoshis (12.2 USD).

Fees for checkpointing

We now report the fees incurred to a subnet by the checkpointing functionality, for varying values of checkpoint-period (the parameter is measured in number of subnet blocks, but in this experiment we convert to the approximate actual time it takes to create these blocks). The results are shown in the following plot. We find out that for a reasonable choice of the checkpoint- period, such as 2 hours, the fees due to checkpointing are approx. 33,000 satoshis per 24 hours, or approx. 33 USD.

Improving the transfer fee costs using threshold signatures

Finally, we repeat Experiment 7 measuring the improvement, in terms of fees per transfer, that threshold signatures can achieve when implemented in Bitcoin-IPC .

The next plot shows the fee per transfer for different numbers of validators in the source subnet. The plot contains a line for one validator in the source subnet, which corresponds to an implementation of Bitcoin-IPC with threshold signatures. For batches of 50 transfers, the fee per transfer is 2,600 satoshis (26 USD) with threshold signatures and 9,700 satoshis (97 USD) with a multisig implementation among 100 validators. For batches of 100 transfers, the corresponding sizes are 1,900 satoshis (19 USD) with threshold signatures and 5,460 satoshis (54.6 USD) with multisigs.

Even though in all cases the fee per transfer is lower than when submitting the transfer natively over Bitcoin, the improvement is bigger when threshold signatures are used.