Skip to content
This repository has been archived by the owner on Jun 17, 2020. It is now read-only.

Latest commit

 

History

History
413 lines (279 loc) · 16 KB

Daedalus-Installer-Update-Procedure.md

File metadata and controls

413 lines (279 loc) · 16 KB
Raw

Daedalus Installer Update Procedure

After the installers are built in CI, they need to be copied to the installer updates S3 bucket, and an update proposal needs to be submitted to the cardano-sl network. The update bucket itself is defined in configuration yaml file under the installer-bucket key. In each of the networks which may use this procedure (mainnet, testnet, staging), the configuration yaml file can be found as config.yaml which is a symlink to the underlying network's yaml file.

This is how wallet clients will self-update.

Intended Audience

This document is mainly intended for use by IOHK DevOps working from the mainnet, staging mainnet, or testnet deployers.

It can also be used for testing the update system against a developer cluster.

Running the scripts

ssh into the deployer get a nix-shell in the iohk-ops checkout.

# Change directory into the cluster the proposal will be made on
cd mainnet
nix-shell -A withAuxx

You can see the list of subcommands with:

io update-proposal --help

Each update-proposal subcommand has its own usage information accessible with the --help option.

Subcommand Synopsis
init Create template config file and working directory.
find-installers Download installer files from the Daedalus build.
sign-installers Sign downloaded installer files with GPG.
upload-s3 Upload installer files to the S3 bucket.
set-version-json Update the version info file in the the S3 bucket.
submit Send update proposal transaction to the network.

The commands are run in order with manual checks between each step.

Requirements

  1. The keys for core nodes should be in the keys subdirectory.

  2. You need to know the following things:

    • Daedalus revision to propose. Note: this should be the same revision as specified in the release candidate announcement and signed off by QA.
    • The lastKnownBlockVersion, according to the table below.
    • The voterIndex, according to the table below.
    • Signing key passphrase (optional).
    • IP address of a privileged relay (use io info).

1. Initialise params

io -c NETWORK.yaml update-proposal init [DATE] --revision REVISION --block-version VER --voter-index N --release-notes /dev/null

Where -c NETWORK.yaml can be omitted from the command if there is already a config.yaml file symlinked to the proper network configuration yaml file in the cluster directory.

Where DATE is a string which identifies the update proposal. By default it will be today's date in YYYY-MM-DD format. The other values should be set as required.

Where REVISION is the daedalus repo commit from the revision that the proposal will be submitted for. This is also the same revision that the io find-installers -r REVISION command would have used to generate the email for QA.

Where VER from --block-version can be obtained from the logs of the network the proposal will be made on, such as by examining: journalctl -f -u cardano-node -o cat for the block: v$VER string.

Where N from --voter-index is a number from 0 to 6, representing voting keys of the core nodes. Note that once a voter index is utilized for a proposal, the same voter index number cannot be utilized again for that epoch or the proposal will be rejected.

This command will create a template config file like update-proposals/mainnet-2018-04-03/params.yaml.

The directory that the config file is created in is called the work dir. It will contain logs, a node db, keys, installers, hashes, and other information generated by the update proposal.

Open params.yaml in the work dir and double-check that it has correct values. The contents will look similar to this:

voterIndex: 2
daedalusRevision: 32daff63e1fb1590cd7320e4253e61b2a47b0963
lastKnownBlockVersion: '0.1.0'

2. Download installer files from CI

io -c NETWORK.yaml update-proposal find-installers DATE

The DATE parameter is required to be specified and should be the same as what was reported after running the init step.

This will locate the CI builds for the previously configured revision and download their installer artifacts to the installers subdirectory of the work dir. It will then use the file program to check that they are actually installers for the correct platform, and calculate their hashes using both cardano-auxx and sha256sum.

After it has finished, inspect the following values in params.yaml:

  1. The grApplicationVersion value must be a value greater than all previous update proposals on the target cluster. This is recorded in Daedalus Installer History.

  2. The grApplicationVersion found should match what you have set in cardano-sl.

  3. The ciResultBuildNumber for all ciResults should be the correct build.

  4. The grCardanoCommit value should match cardano-sl-src.json in the Daedalus tree.

  5. The installer filenames should look normal, have the right network and versions, and correspond to the same build as was approved for release by QA (if updating mainnet installers).

  6. The installerHashes and installerSHA256 values should be present.

These values will also be summarised in the file wiki.md within the work dir.

Important: If an update proposal is made with the wrong applicationVersion, the update mechanism may fail and users will be required to intervene by manually installing an update.

When there is more than one build for the chosen revision

Sometimes there can be multiple builds corresponding to a given Daedalus revision.

In this case, find-installers will list the builds and then exit without downloading anything. You need to re-run the command with --buildkite-build-num or --appveyor-build-num arguments added.

3. Sign executables with HSM

From a local machine, use the release-build.nix file of the daedalus repo, at the same commit as the proposal, to sign the Windows executables:

nix-build release-build.nix --argstr buildNum BUILDNUM --allow-unsafe-native-code-during-evaluation

The build process will utilize ssh to the HSM signing server via server HSM which can be defined to the current HSM server in an ssh config file.

Once the Windows HSM signed binary has completed building on a local machine, remove the symlink on the deployer for the existing non-HSM signed binary, found in dir: update-proposal/$CLUSTER-$DATE/installers/

Replace the non-HSM signed symlink with the locally built HSM signed binary using scp to push the signed binary to the deployer; verify size and expected hash on the deployer.

With HSM signed binary or binaries now in place on the deployer, the installerHashes and installerSHA256 hashes for each HSM signed binary need to be updated in the params.yaml file found in dir: update-proposal/$CLUSTER-$DATE/

The correct hashes to update in the params.yaml file can be generated with the following commands:

For installerHashes hashes: cardano-auxx cmd --commands 'hash-installer $HSMBINARY'

For installerSHA256 hashes: sha256sum $HSMBINARY

4. Sign installer files with GPG

This step requires a signing key available on the deployer host.

io -c NETWORK.yaml update-proposal sign-installers -u [email protected] DATE

Where the the -u parameter (here shown as [email protected]) can be found by running gpg -K and obtaining the email address associated with the signing key.

If the signing key is protected with a passphrase, you will be prompted to enter it.

This will place detached signatures in .asc files within the work dir. There should be one .asc file produced for each binary. These will be uploaded to S3 at the same time as the installer files.

5. Upload installer files to the S3 update bucket

io -c NETWORK.yaml update-proposal upload-s3 DATE

This will upload the hashed installers to S3, under their original filename, as well as by their hash.

There will also be a file daedalus-latest-version.json added to the work dir with download links and SHA-256 hashes.

Hint: The destination S3 bucket is configured with the installer-bucket value in NETWORK.yaml. You can edit this for debugging purposes.

Troubleshooting

If you get an error like this: AesonException "Error in $['mainnet_staging_short_epoch_wallet_win64']: key \"infra\" not present"), ensure that the cardano-sl-auxx that you are using matches that cardano-sl version used by Daedalus.

If you get an error like the following, or a different s3 upload related error:

[ServiceError] {
  service    = S3
  status     = 403 Forbidden
  code       = SignatureDoesNotMatch
  message    = Just The request signature we calculated does not match the signature you provided. Check your key and signing method.
  request-id = Just $VALUE
}

The files can be uploaded manually with the aws cli, for example:

First, upload the binaries and their .asc files:

for x in update-proposals/$CLUSTER-$DATE/installers/*; do aws s3 cp $x s3://$INSTALLER_BUCKET/; done

Second, upload each binary again, this time naming the target binary as its installerHashes value. Example:

aws s3 cp update-proposals/$CLUSTER-$DATE/installers/$BINARY s3://$INSTALLER_BUCKET/$INSTALLERHASHES_HASH

6. Update version JSON

io -c NETWORK.yaml update-proposal set-version-json DATE

This will drop the previously created daedalus-latest-version.json into the S3 bucket. If done with the mainnet settings, it will have the effect of immediately updating the download links on The Daedalus Wallet download page.

7. Propose the update and vote in favor using majority stake

Find the private IP address of a core node. This is normally private info so don't leak it.

On the deployer, but in a different shell than the one planned to execute the proposal on, open a tunnel to the core node, port 3000 from the deployer with the following command:

nixops ssh -d $CLUSTER $CORENAME -L 3000:$CORE_INTERNAL_IP:3000

Then, execute the update proposal from the original shell

io -c NETWORK.yaml update-proposal submit DATE --relay-ip 127.0.0.1
    [--without-linux] [--without-macos] [--without-windows]

Exit the ssh session in the other shell once the proposal update has been submitted successfully to end the ssh tunnel to the core node.

8. Notarization for Mac Catalina

Steps for MacOS notarization (must be done on the MacOS guest signing servers):

  1. scp or otherwise push/pull the installer to mac-mini-<1|2>-signing

  2. ssh to the mac-mini-<1|2>-signing guest where the scp pushed the file above

  3. Switch to the appropriate user and path of the installer package if not already there (these notarize staple commands will run ok as nixos or root user)

  4. Check the sha256 is expected as from the deployer params.yaml update proposal file and/or direct sha256 of the package file on the appropriate deployer

  5. Do the notarization (the $USER and $PASSWORD are in lastpass vault under apple-notary-pass)

xcrun altool --notarize-app \
  --primary-bundle-id "io.iohk.daedalus.pkg" \
  --username "$USER" \
  --password "$PASSWORD" \
  --file "$DAEDALUS_INSTALLER_FILE"

  1. Optional: Check status: xcrun altool --notarization-history 0 -u "$USER" -p "$PASSWORD"

  2. Optional: Get further information:

# RequestUUID will have been provided in the initial cmd output
xcrun altool --notarization-info $REQUESTUUID -u "$USER"

More docs: https://developer.apple.com/documentation/xcode/notarizing_your_app_before_distribution/customizing_the_notarization_workflow?language=objc

Note from 2018-12-17 By default, installers will be proposed for Linux, Windows and macOS. Use the --without-OS flag to exclude certain installers from the update proposal.

This will generate a new node db, copy keys from the top-level keys directory, then "rearrange" the copied keys.

It will then send a transaction to the given relay.

Note the proposal ID which is printed at the end of the output.

If the update proposal was successful, the ratified proposal will take effect in k slots time, where k is the security parameter. On mainnet/staging/testnet with k=2160 and 20 second slot duration, this will be 12 hours.

8. Testing proposal acceptance

Useful generic search: keywords

  1. Search Papertrail for We'll request data for key Tagged (UpdateProposal,[UpdateVote]) and confirm it references the first 8 chars of the proposal ID from the previous step.
  2. Search Papertrail for Processing of proposal csl-daedalus: and confirm that
    1. the number following : matches the intended applicationVersion
    2. the correct UpId: is referenced
    3. the tags for the supported platforms are mentioned: tags: [win64, macos64, linux]
    4. matching lines end with is successful

Testing for proposal confirmation

12 hours after proposal acceptance, it should be confirmed.

Search papertrail for Proposal 6e2f23c1 is confirmed on a core node, using the first 8 characters of the proposal ID from the previous step.

9. Testing the update

Work in progress: DEVOPS-651.

This is also covered in how-to/test-update-system.md, section Check update taken by wallet.

10. Update the wiki

Copy the contents of wiki.md from the work dir and paste into Daedalus Installer History.

Also update any other documentation which is missing or out of date.

Block version table

The lastKnownBlockVersion reflects the current adopted protocol version of the network. Use the following values depending on the network where the update will be proposed.

Network lastKnownBlockVersion
mainnet 0.2.0
staging 0.2.0
testnet 0.1.0

Voter index

The update proposal procedure requires a voter index which selects the secret key used to sign the update proposal.

Note: The voterIndex needs to be ≥ 0 and < 7 (where 7 is the number of core nodes), otherwise you will receive the error Prelude.!!: index too large.

See also

Future automation

  1. The lastKnownBlockVersion parameter should be investigated to determine whether it is really needed to be configured, or can be determined from the configuration.

  2. When printing paths to work dir, show them as relative to the iohk-ops checkout, rather than as absolute paths.

  3. Automatically make sure that the version of auxx and tools used corresponds to the cardano-sl version of Daedalus. Currently, the version used is that built by the nix-shell.

  4. DEVOPS-816 Automated testing of update proposals.