Skip to content

rpc: allow writing UTXO set to a named pipe, introduce dump_to_sqlite.sh script

Placeholder Sebastian Falbesoner requested to merge github/fork/theStack/202412-dumptxoutset-allow_write_to_named_pipe into master

This PR is based on #27432 and slightly modifies the dumptxoutset RPC to allow writing the UTXO set dump into a named pipe, so that the output data can be consumed by another process, see #31373. Taking use of this with the utxo-to-sqlite.py tool (introduced in #27432), creating an UTXO set in SQLite3 format is possible on the fly and becomes a one-liner with a newly introduced script dump_to_sqlite.sh. E.g. for signet:

$ ./contrib/utxo-tools/dump_to_sqlite.sh "./build/src/bitcoin-cli -signet" ~/utxos.sqlite3
UTXO Snapshot for Signet at block hash 000000ddc3b251483cf1ebb23e2750ba..., contains 5705634 coins
1048576 coins converted [18.38%], 4.474s passed since start
2097152 coins converted [36.76%], 8.793s passed since start
3145728 coins converted [55.13%], 13.146s passed since start
4194304 coins converted [73.51%], 17.478s passed since start
5242880 coins converted [91.89%], 21.832s passed since start
{
  "coins_written": 5705634,
  "base_hash": "000000ddc3b251483cf1ebb23e2750ba2490701d0c547241b247a9beb85498d0",
  "base_height": 227678,
  "path": "/tmp/tmp.MFHEVqetv0/utxos.fifo",
  "txoutset_hash": "f29e524c999487cbd0cfca5201dce67c2c5e5c5eb115c63ad48c2239f23eea4c",
  "nchaintx": 8272649
}
TOTAL: 5705634 coins written to /home/thestack/utxos.sqlite3, snapshot height is 227678.

Note that the dumptxoutset RPC calculates an UTXO set hash as a first step before any data is emitted, so especially on mainnet it takes quite a while until the conversion starts and something is happening visibly.

The new script is quite minimal and PoC-y at this point, there are some potential improvement ideas:

  • better error handling (e.g. detect if bitcoin-cli exists, clean up tmpdir if bitcoin-cli execution fails etc.)
  • allow to pass through the rollback option (now we always dump at the current height, i.e. "latest" parameter)

Merge request reports