# Release process (/docs/internals/release-process) Shipping is a single pipeline: **`release-train.yml`** runs on every push to `main`, builds the runtime once, and promotes that identical wasm through the networks. A second workflow, **`watch-mainnet-release.yml`**, watches the chain and publishes release artifacts once the mainnet upgrade actually executes. ## The release train [#the-release-train] ``` push to main └─ srtool build (once per train) └─ deploy devnet ───── smoke-check devnet └─ deploy testnet ── smoke-check testnet ── publish SDK rc to PyPI └─ propose mainnet upgrade (multisig) ``` ### Build once (srtool) [#build-once-srtool] The train builds the runtime with [srtool](https://github.com/paritytech/srtool) — a pinned Docker build environment that makes the wasm **byte-reproducible**. The artifact (`subtensor.wasm` + digest) is uploaded once and reused by every deploy job; promotion never rebuilds. Anyone can verify the hash independently by running `scripts/srtool/build-srtool-image.sh` followed by `scripts/srtool/run-srtool.sh` locally (see [Repository scripts](/docs/internals/scripts)). ### The ship lever: `spec_version` [#the-ship-lever-spec_version] Every deploy job first compares the built runtime's [`spec_version`](/code/runtime/src/lib.rs#L238) (`runtime/src/lib.rs`) with what the target chain is running, and **no-ops unless the build is newer**. Consequences: * A merge without a `spec_version` bump builds and then does nothing — this is deliberate, and it's why the Spec Version Check on PRs exists (label `no-spec-version-bump` to opt out knowingly). * Stale queued trains are harmless: they no-op at each guard. * Rollback is not a concept here — you ship a *newer* fixed version. ### Promotion gates [#promotion-gates] Human approval lives in GitHub **environment settings**, not workflow code: | Stage | Environment | Gate | | ------- | ----------- | -------------------------------------------------------------------------------------------- | | devnet | `devnet` | none — deploys automatically, then runs the devnet smoke suite | | testnet | `testnet` | environment reviewers (one-click approval), then smoke suite + SDK release candidate to PyPI | | mainnet | `mainnet` | environment reviewers **plus** the multisig ceremony below | Deploys to devnet and testnet are direct `setCode` transactions via the CI deploy key. After the on-chain `spec_version` is verified, the train moves the corresponding network mirror branch. That branch push triggers both Docker publishers from the exact deployed commit, updating the matching `:devnet` or `:testnet` tag in `ghcr.io/raofoundation/subtensor` and `ghcr.io/raofoundation/subtensor-localnet`. It does not move either package's `:latest` tag. The smoke suite then validates the live network while the independent image builds run. ### The mainnet multisig ceremony [#the-mainnet-multisig-ceremony] CI never upgrades mainnet directly. Before any on-chain action, the train reserves the immutable `v` tag at the exact source commit. Its final job then submits a **multisig proposal**: the CI key is one half of a 2-of-2 deployment multisig that holds a `SudoUncheckedSetCode` proxy on the sudo key. The triumvirate then approves the proposal 2-of-3 **out-of-band** — no GitHub credential can unilaterally change the mainnet runtime. Until they sign, nothing happens on chain. For rehearsal, the `mainnet-clone` PR label spins up a live clone of mainnet running your runtime with a public endpoint ([mainnet clone testing](/docs/internals/mainnet-clone)). ## The release watcher [#the-release-watcher] `watch-mainnet-release.yml` polls mainnet every 10 minutes. When an upgrade executes, it resolves the release-train artifact whose commit matches the immutable tag and whose runtime hash matches the finalized on-chain `:code`. It moves the protected `mainnet` mirror to that exact commit before cutting the release: 1. **GitHub release** `v` 2. **Production Docker images** tagged `v` and `latest` 3. **Python SDK + bittensor-core wheels** to PyPI (trusted publishing, with PEP 740 provenance attestations) 4. **Publishable Rust crates** to crates.io (auto-discovered; workspace crates are `publish = false` while they depend on the patched polkadot-sdk fork) 5. **Production website/docs** to Vercel This ordering means the release always reflects what is *actually running on mainnet*, not what was merged. PyPI is the terminal state for stable Python publication. The watcher requires the SDK wheel and source distribution plus the full `bittensor-core` Linux and macOS wheel matrix and source distribution. Every file must carry a PEP 740 attestation from this repository's mainnet release workflow. A failed or partial upload is retried on the next watcher run even if the GitHub release already exists or the release-train artifact has expired. Retries require the release target, immutable tag, protected `mainnet` mirror, and finalized runtime bytes to agree. The SDK's committed `X.Y.Z.dev0` version is stamped to `X.Y.Z` for this build. The release train rejects a base SDK or core version that already exists on PyPI before publishing release candidates. Releases through v432 predate automated stable Python publication and are handled as historical state. The watcher explicitly dispatches `docker.yml` and `docker-localnet.yml` because releases created with the default `GITHUB_TOKEN` do not emit another workflow event. The production build publishes both the immutable release tag and `:latest`. Independently, every merge to `main` publishes the moving production `:main` tag, so developers can prepare against merged features before they reach mainnet without changing the deployed `:latest` image. The localnet build publishes the immutable release tag in its separate package; localnet `:latest` remains an alias for the current `:main` branch image, while `:devnet` and `:testnet` follow their respective branches. ## Hotfixes [#hotfixes] The same pipeline applies: land the fix on `main` with a `spec_version` bump and let the train promote it. There is no side channel that skips devnet and testnet validation.