ShopperCove
Menu
All writingBlogTopicsCategoriesAboutRSS
Blog
Categories
Observability & SRE62All categories
About

Plate 11

  1. Blog

After xz: A Practical Supply-Chain Checklist for Solo and Small Teams

Turn the xz-utils backdoor lesson into action: tarball vs git diffs, SBOM, Sigstore Cosign, SLSA provenance, and a solo/small-team release gate you can run this week.

Aditya Challa·29 September 2026·9 min read

Summary
On this page
  1. Intro — what this post promises
  2. What xz taught that checklists must capture
  3. The artifact is not the git tree
  4. Social trust is not a control
  5. Detection still matters
  6. Threat map — map controls to failure modes
  7. Checklist stage 1 — consuming dependencies
  8. Pin by content, not by floating tag
  9. Diff the tarball (non-optional after xz)
  10. Generate an SBOM on every release candidate
  11. Checklist stage 2 — building with provenance in mind
  12. Prefer a hosted builder you do not personally admin
  13. Sign with Sigstore Cosign (keyless when possible)
  14. Checklist stage 3 — release and deploy gates
  15. What this checklist does _not_ claim
  16. Worked mini-lab (ran 29 Sep 2026 IST)
  17. More reading
  18. Stay in touch

Intro — what this post promises

In March 2024, malicious code in XZ Utils 5.6.0 and 5.6.1 (tracked as CVE-2024-3094) showed how a patient supply-chain implant can hitch a ride through a trusted compression library into SSH-adjacent code paths. CISA’s alert recommended downgrading to an uncompromised release (for example 5.4.6 Stable), hunting for related activity, and reporting findings.

Related links:

  • CISA’s alert

ShopperCove already covered the story of that backdoor: xz-utils backdoor technical deep dive. This post is deliberately not another narrative retelling. It is the practical checklist you run on your deps and your releases when you do not have a dedicated AppSec team.

Related links:

  • xz-utils backdoor technical deep dive

You will get:

  1. What xz actually taught practitioners (threat classes, not gossip).
  2. A staged checklist: consume deps → build → release → deploy.
  3. Concrete commands for tarball diffs, SBOM, Cosign, and SLSA-oriented provenance — run on a real dependency in lab.
  4. Honest limits for a solo or three-person team.

Lab honesty: Numbers below are from a ShopperCove lab on 29 Sep 2026 (IST): zlib v1.3.1 (git 51b7f2ab…) release tarball vs git archive; Syft 1.18.1 SBOMs; Cosign v2.4.1 local key sign+verify with a deliberate wrong-key failure. Keyless Cosign hit Sigstore device-flow OIDC and could not complete on this box (no browser identity / no CI id-token). This was one careful afternoon on a shared Linux host — not a full GHCR + admission rollout (that belongs in a dedicated CI workflow lab).


What xz taught that checklists must capture

The artifact is not the git tree

The xz implant was not “just a bad commit someone forgot to review.” Attackers invested in build-time obfuscation and distribution of compromised release artifacts. If your CI only checks out git and never compares what users download from a release page, you miss a whole class of attacks.

Social trust is not a control

Maintainer takeover / multi-year social engineering is outside the reach of npm audit. Your checklist should assume identity of the human can be wrong, and compensate with reproducible process, signed provenance, and artifact comparison.

Detection still matters

Andres Freund’s discovery path (anomalous SSH/CPU behavior during unrelated work) is a reminder: runtime anomaly signals and fuzz/vetting (see our Lily / fuzz-based defense post) complement static gates. Checklist ≠ silver bullet.

Related links:

  • Lily / fuzz-based defense post

Threat map — map controls to failure modes

Failure mode (post-xz lens)ExampleControl you can afford
Compromised release tarball ≠ gitHidden build scripts in distDiff git archive vs upstream tarball
Unsigned / re-tagged containerRegistry push from laptopCosign sign + admission/CI verify
“Provenance” forged on laptopFake attestation JSONHosted builder + SLSA Build L2+ goals
Dependency confusion / typoWrong package namePin + lockfile + scope allowlist
Transitive surpriseNew gzip/xz path in treeSBOM + SCA on every release
Silent runtime hookBackdoor only under rare triggersAnomaly alerts + staged canaries

Lab row (zlib v1.3.1): Release tarball had 0 files that were not in git. Four files existed only in git (.github workflows + .gitignore). All 253 shared files were byte-identical (cmp). Empty extras is a valid result — document it and keep the playbook.


Checklist stage 1 — consuming dependencies

Pin by content, not by floating tag

  • Lockfiles committed (go.sum, package-lock.json, Cargo.lock, poetry.lock, …).
  • Prefer digest pins for container base images (image@sha256:…) over mutable tags.
  • Record upstream release URL + checksum for critical native libs.

Diff the tarball (non-optional after xz)

Pattern we ran on zlib v1.3.1:

git clone --depth 1 --branch v1.3.1 https://github.com/madler/zlib.git
cd zlib
git archive --format=tar HEAD | tar -tf - | sed 's|^\./||' | sort > /tmp/from-git.txt

curl -fsSL -L -o zlib-1.3.1.tar.gz \
  https://github.com/madler/zlib/releases/download/v1.3.1/zlib-1.3.1.tar.gz
# SHA-256 we measured:
# 9a93b2b7dfdac77ceba5a558a580e74667dd6fede4585b91eefb60f03b72df23
tar -tzf zlib-1.3.1.tar.gz | sed 's|^zlib-1.3.1/||' | sed 's|/$||' | grep -v '^$' | sort > /tmp/from-tarball.txt

comm -13 /tmp/from-git.txt /tmp/from-tarball.txt   # only in tarball (extras)
comm -23 /tmp/from-git.txt /tmp/from-tarball.txt   # only in git
# Deeper: extract both trees; cmp each common file; diff -ruN

What we found:

Signalzlib v1.3.1 lab
Files in git archive257
Files in release tarball253
Only in tarball0
Only in git.github/workflows/{cmake,configure,fuzz}.yml, .gitignore
Content mismatches on common files0

Allowlist lesson: zlib’s maintainer release is a subset of the tagged tree (CI metadata stripped). That is the opposite of the xz-class failure (hidden extras in the dist). Still run the diff — an empty extras list is publishable evidence, not a skipped control.

Trap: GitHub’s auto-generated source archive (archive/refs/tags/v1.3.1.tar.gz) is a different blob (SHA-256 17e88863… in our lab) and still contains the CI files. Diff against the release asset distributors actually fetch, not the auto-archive.

Wall time once tools were present: about two minutes for clone + download + diff on this dependency.

Generate an SBOM on every release candidate

We used Syft 1.18.1. Inventory is the goal — not theater.

# Pinned in lab: syft 1.18.1
syft scan dir:./artifacts/from-tarball -o spdx-json > zlib-tree.spdx.json
# Bare zlib C sources: Syft reported no packaged components (expected).

syft scan dir:./mini-svc -o spdx-json \
  --source-name lab04-minisvc --source-version 0.0.1 > sbom.spdx.json
# Tiny Go release candidate (Go 1.24.4, github.com/google/uuid v1.6.0):
#   github.com/google/uuid @ v1.6.0
#   shoppercove.local/lab04-minisvc
#   stdlib @ go1.24.4

Store the SBOM next to the artifact you ship. An empty component list on a bare C tree is honest; the Go mini-service shows what a lockfile-backed release candidate looks like under the same tool.


Checklist stage 2 — building with provenance in mind

Prefer a hosted builder you do not personally admin

SLSA v1.0 Build track ladders roughly as:

Related links:

  • SLSA v1.0 Build track
LevelIntent (short)
Build L1Provenance exists (easy to forge)
Build L2Signed provenance from a hosted build platform
Build L3Hardened platform: builds isolated; signing material not available to user steps

Solo teams rarely “finish” L3 on day one. Aim first for L2-shaped habits: builds on GitHub Actions / GitLab CI / Google Cloud Build (or equivalent), provenance attached, verification in the same pipeline that ships.

Sign with Sigstore Cosign (keyless when possible)

Sigstore’s Cosign overview explains identity-based (“keyless”) signing: Fulcio issues a short-lived certificate binding an ephemeral key to an OIDC identity; Rekor records the signing event in a transparency log. That removes long-lived signing keys from your laptop’s ~/.ssh graveyard.

Related links:

  • Sigstore’s Cosign overview

Lab — keyless limit (honest): cosign sign-blob --yes on the zlib tarball entered Sigstore’s device-flow OIDC (oauth2.sigstore.dev). This lab box has no browser session and no GitHub Actions id-token, so keyless could not complete. In CI, use something in this shape (pin action SHAs in production):

permissions:
  id-token: write   # for keyless OIDC
  contents: read
  packages: write
steps:
  - run: cosign sign --yes ghcr.io/ORG/APP@${{ steps.digest.outputs.digest }}
  # verify later with certificate-identity + certificate-oidc-issuer

Lab — local key sign + verify (completed with Cosign v2.4.1):

cosign generate-key-pair
cosign sign-blob --yes --key cosign.key \
  --output-signature zlib-1.3.1.tar.gz.sig zlib-1.3.1.tar.gz
cosign verify-blob --key cosign.pub \
  --signature zlib-1.3.1.tar.gz.sig zlib-1.3.1.tar.gz
# -> Verified OK (Rekor tlog index 3003501942 for the zlib blob in our run)

# Deliberate failure with a different pubkey:
cosign verify-blob --key wrong.pub \
  --signature zlib-1.3.1.tar.gz.sig zlib-1.3.1.tar.gz
# -> exit 1 (invalid signature)

We also signed the SPDX SBOM and a tiny Go binary the same way — three Verified OK results, one wrong-key exit 1. Verification mindset from Sigstore verify docs: check identity (who) and issuer (which OIDC provider) for keyless; for local keys, treat key distribution as the trust root you are trying to eliminate in production.

Related links:

  • Sigstore verify docs

Checklist stage 3 — release and deploy gates

Minimum gate for anything internet-facing:

  • CI built from a protected branch / required reviews
  • SBOM artifact uploaded with the release
  • Image/binary signed; verify step fails the job on mismatch
  • Provenance attestation present for the production digest
  • Deploy path rejects unsigned digests (cluster admission policy or CD verify step if you are not on Kubernetes)
  • Dependency diff / advisory check for critical native libs this quarter

Measured time on this lab afternoon (not a multi-day CI migration):

ActivityThis lab
Install Syft 1.18.1 + Cosign v2.4.1~3 seconds
One zlib tarball vs git playbook~2 minutes
Syft SBOM (zlib tree + Go mini-svc)~10 seconds
Cosign keylessblocked (device-flow OIDC; needs CI id-token or browser)
Cosign local sign + 3× verify OK + wrong-key fail~13 seconds after keys exist
Wire Cosign in CI / admission denynot timed here — natural follow-on for a dedicated workflow lab

Steady-state costs shrink once the playbook is scripted; do not treat the table above as a week-long AppSec hire substitute.


What this checklist does not claim

Claim to avoidWhy
“SLSA L3 means unhackable”Levels reduce classes of tampering; they do not erase insider or zero-day risk (SLSA levels).
“Cosign replaces code review”Signatures prove who built what under which identity — not that the code is correct.
“SBOM finds backdoors”SBOMs inventory components; they do not semantically analyze build scripts. Our bare zlib tree correctly produced an empty component list.
“We would have caught xz automatically”Honesty: many small teams would not. The checklist raises the cost for the next attack. Our zlib diff showed no extras — the control still matters for the next dependency that does.

Related links:

  • SLSA levels

Worked mini-lab (ran 29 Sep 2026 IST)

Scenario: Critical native dep (zlib v1.3.1) consumed via release tarball; tiny Go service as a stand-in release candidate for SBOM + Cosign.

StepSignal in labStatus
Tarball vs git archive0 extras; 4 git-only CI files; 253 identical commonsDone
Syft SBOMzlib tree empty; mini-svc: uuid v1.6.0 + stdlib go1.24.4Done (Syft 1.18.1)
Cosign keyless signDevice-flow OIDC; no identity on lab boxBlocked — documented
Cosign local sign + verify3× Verified OK; Rekor indices recordedDone (v2.4.1)
Verify with wrong keyexit 1Done
Deploy with unsigned digestNot exercised (no cluster / no GHCR push this lab)Deferred

More reading

  • xz-utils backdoor technical deep dive
  • Not Git Yard / fuzz-based defense
  • About ShopperCove
  • CISA alert on xz-utils / CVE-2024-3094
  • SLSA Build levels
  • Cosign signing overview
  • Cosign verify
  • zlib v1.3.1 release
  • Syft

Stay in touch

  • Subscribe via RSS: https://www.shoppercove.com/feed.xml
  • About the author / method: https://www.shoppercove.com/about

supply chain securityxz utilssbomsyftcosignsigstoreslsatarball vs git

Lab evidence

What I found running this

Lab 29 Sep 2026 IST. zlib v1.3.1: tarball vs git 0 tarball-only extras, 4 git-only (.github/.gitignore), 253/253 common files identical. GitHub auto-archive SHA ≠ release asset. Syft 1.18.1: bare zlib tree empty components; Go mini-svc SBOM uuid v1.6.0 + stdlib go1.24.4. Cosign v2.4.1 keyless blocked (OIDC); local key 3× Verified OK + wrong-key fail.

Notes when a lab post goes up

Occasional email for new hands-on reviews. No sequence and no sponsors.

Related links

  • Plate 13

    Cosign + SBOM in CI: Sign and Attest a Container Image in One Workflow

    One CI shape: build an image by digest, Syft SBOM, Cosign sign + SBOM attest, verify success, and prove wrong-key/unsigned failure — the operational follow-on to after-xz.

    29 Sept 2026

  • Plate 80

    Lily in CI: Trying Fuzz-Based Backdoor Detection on a Real Repo

    Hands-on Lily (ASE 2026) on an owned C toy: rosa/lily 0.6.0 pin, directed catch of a localhost-bind trigger, clean refactor with zero flags, discovery miss, and CI cost math.

    30 Sept 2026

  • Plate 41

    React 19.3 View Transitions & Fragment Refs: Frontend Guide (2026)

    2 Oct 2026

On this page

  1. Intro — what this post promises
  2. What xz taught that checklists must capture
  3. The artifact is not the git tree
  4. Social trust is not a control
  5. Detection still matters
  6. Threat map — map controls to failure modes
  7. Checklist stage 1 — consuming dependencies
  8. Pin by content, not by floating tag
  9. Diff the tarball (non-optional after xz)
  10. Generate an SBOM on every release candidate
  11. Checklist stage 2 — building with provenance in mind
  12. Prefer a hosted builder you do not personally admin
  13. Sign with Sigstore Cosign (keyless when possible)
  14. Checklist stage 3 — release and deploy gates
  15. What this checklist does _not_ claim
  16. Worked mini-lab (ran 29 Sep 2026 IST)
  17. More reading
  18. Stay in touch
All writingBlogCategoriesTopicsAboutPrivacyRSS

© 2026 ShopperCove