Skip to content

fix: contrib: allow multi-sig binary verification

Placeholder jamesob requested to merge github/fork/jamesob/2021-09-binary-verify into master

Related: https://github.com/bitcoin-core/bitcoincore.org/issues/793

This changeset improves the contrib/verifybinaries/verify.py script to account for the new binary verification procedure introduced in the 22.0 release; for details on this new process, see the note in https://github.com/bitcoin/bitcoin/issues/22634.

In short, instead of relying on a single signature from the lead maintainer attesting to the expected hashes of binary releases, this verify script now supports validating that a minimum threshold of trusted identities have signed the checksum file.

image

Both the threshold and identities to trust are configurable by the end user, but sensible defaults are provided: identities are inferred from local GPG trust and the builder-keys file and a minimum threshold of 4 trusted signatures is configured by default. There are options for overriding these.

Various improvements have been made to the script for allow for easier programmatic use; a --json option is introduced and logging output is now directed to stderr.

Automatic pubkey import

Pubkeys that are referenced in checksum signature files can now be automatically downloaded based on a user prompt. This behavior can be disabled with the --noninteractive flag to support CI use.

I have built in functionality for elevating local GPG trust of the imported keys, but it isn't quite working yet; I think instead of manually modifying the GPG trust database I have to use the --sign-key command. But this is optional functionality and fixing the broken process is the priority here, so I can enable this nicety in a follow-up.

Builder-key diffing

Builder keys (as listed in ./contrib/builder-keys/keys.txt) are used by default to establish trust in a pubkey signature. This can be disabled with --no-builder-keys.

If builder keys are locally supplied (by running this command from the root of the repository), they are diffed with the remote version obtained over HTTPS from Github. A diff is reported if it exists, e.g.

image

Granular JSON output

Because binary verification is now a gradient on the basis of an end user's trust in a set of pubkeys, users of this script may want a granular report on which signatures were used. Stderr logging can be parsed for this information, but a convenient JSON blob is also accessible using the --json flag:

% ./contrib/verifybinaries/verify.py 22.0-x86 --noninteractive --json 2>/dev/null

{
  "good_trusted_sigs": [
    "SigData('9D3CC86A72F8494342EA5FD10A41BDC3F4FAFF1C', 'Aaron Clauson (sipsorcery) <aaron@sipsorcery.com>', trusted=False, status='unknown')",
    "SigData('637DB1E23370F84AFF88CCE03152347D07DA627C', 'Stephan Oeste (it) <it@oeste.de>', trusted=True, status='unknown')",
    "SigData('9DEAE0DC7063249FB05474681E4AED62986CD25D', 'Wladimir J. van der Laan <laanwj@visucore.com>', trusted=True, status='unknown')",
    "SigData('0AD83877C1F0CD1EE9BD660AD7CC770B81FD22A8', 'Ben Carman <benthecarman@live.com>', trusted=False, status='unknown')",
    "SigData('152812300785C96444D3334D17565732E08E5E41', 'Andrew Chow (Official New Key) <achow101@gmail.com>', trusted=True, status='expired')",
    "SigData('D1DBF2C4B96F2DEBF4C16654410108112E7EA81F', 'Hennadii Stepanov (GitHub key) <32963518+hebasto@users.noreply.github.com>', trusted=True, status='unknown')",
    "SigData('590B7292695AFFA5B672CBB2E13FC145CD3F4304', 'Antoine Poinsot <darosior@protonmail.com>', trusted=True, status='unknown')"
  ],
  "good_untrusted_sigs": [
    "SigData('0CCBAAFD76A2ECE2CCD3141DE2FFD5B1D88CA97D', '.0xB10C <0xb10c@gmail.com>', trusted=False, status='unknown')",
    "SigData('28F5900B1BB5D1A4B6B6D1A9ED357015286A333D', 'Duncan Dean <duncangleeddean@gmail.com>', trusted=False, status='unknown')",
    "SigData('CFB16E21C950F67FA95E558F2EEB9F5CC09526C1', 'Michael Ford (bitcoin-otc) <fanquake@gmail.com>', trusted=False, status='unknown')",
    "SigData('6E01EEC9656903B0542B8F1003DB6322267C373B', 'Oliver Gugger <gugger@gmail.com>', trusted=False, status='unknown')",
    "SigData('74E2DEF5D77260B98BC19438099BAD163C70FBFA', 'Will Clark <will8clark@gmail.com>', trusted=False, status='unknown')"
  ],
  "unknown_sigs": [
    "SigData('82921A4B88FD454B7EB8CE3C796C4109063D4EAF', '', trusted=False, status='')"
  ],
  "bad_sigs": [],
  "verified_binaries": [
    "bitcoin-22.0-x86_64-linux-gnu.tar.gz"
  ]
}

image

Backwards incompatibility

Note that I have broken the interface; the script is no longer invoked in the same way and output differs. I initially tried to retain the old format, but I found it made less and less sense for the new signature scheme, and was not easily parsed sensibly. If there are users that rely on the old output structure, they can copy the script from an old version of the source tree.

Examples

Validate releases with default settings:

./contrib/verifybinaries/verify.py 22.0
./contrib/verifybinaries/verify.py 22.0-rc2-x86_64
./contrib/verifybinaries/verify.py bitcoin-core-0.13.0-rc3

Get JSON output and don't prompt for user input (no auto key import):

./contrib/verifybinaries/verify.py 22.0-x86 --json --noninteractive

Don't trust builder-keys by default, and rely only on local GPG state and manually specified keys, while requiring a threshold of at least 10 trusted signatures:

./contrib/verifybinaries/verify.py 22.0 \
    --no-builder-keys \
    --trusted-keys 74E2DEF5D77260B98BC19438099BAD163C70FBFA,9D3CC86A72F8494342EA5FD10A41BDC3F4FAFF1C \
    --min-trusted-sigs 10

Followups

New CLI interface

% ./contrib/verifybinaries/verify.py --help

usage: verify.py [-h] [--verbose] [--cleanup] [--noninteractive]
                 [--require-all-hosts] [--bitcoin-src-path [BITCOIN_SRC_PATH]]
                 [--skip-import-builders]
                 [--min-trusted-sigs [MIN_TRUSTED_SIGS]]
                 [--keyserver [KEYSERVER]] [--trusted-keys [TRUSTED_KEYS]]
                 [--no-builder-keys] [--json]
                 version

Script for verifying Bitcoin Core release binaries. This script attempts to
download the sum file SHA256SUMS and corresponding signature file
SHA256SUMS.asc from bitcoincore.org and bitcoin.org and compares them. The
sum-signature file is signed by a number of builder keys. This script ensures
that there is a minimum threshold of signatures from pubkeys that we trust.
This trust is articulated on the basis of configuration options here, but by
default is based upon a unionof (i) local GPG trust settings, and (ii) keys
which appear in the builder-keys/keys.txt file. If a minimum good, trusted
signature threshold is met on the sum file, we then download the files
specified in SHA256SUMS, and check if the hashes of these files match those
that are specified. The script returns 0 if everything passes the checks. It
returns 1 if either the signature check or the hash check doesn't pass. If an
error occurs the return value is >= 2. Logging output goes to stderr and final
binary verification data goes to stdout. JSON output can by obtained by
setting env BINVERIFY_JSON=1.

positional arguments:
  version               version of the bitcoin release to download; of the
                        format <major>.<minor>[.<patch>][-rc[0-9]][-platform].
                        Example: 22.0-x86_64 or 0.21.0-rc2-osx

options:
  -h, --help            show this help message and exit
  --verbose
  --cleanup             if specified, clean up files afterwards
  --noninteractive      if specified, do not block for user input
  --require-all-hosts   If set, require all hosts (https://bitcoincore.org,
                        https://bitcoin.org) to provide signatures. (Sometimes
                        bitcoin.org lags behind bitcoincore.org.)
  --bitcoin-src-path [BITCOIN_SRC_PATH]
                        specify path to bitcoin repository. Used to find
                        builder keys.
  --skip-import-builders
                        If set, do not prompt to import builder pubkeys
  --min-trusted-sigs [MIN_TRUSTED_SIGS]
                        The minimum number of good signatures from recognized
                        keys to require successful termination.
  --keyserver [KEYSERVER]
                        which keyserver to use
  --trusted-keys [TRUSTED_KEYS]
                        A list of trusted builder GPG keys, specified as CSV
  --no-builder-keys     If set, do not trust the builder-keys from the bitcoin
                        repo by default
  --json                If set, output the result as JSON

Merge request reports