expand openspec for extension delivery architecture - #452
Conversation
carillan81
added
semver: minor
docs: skip
peter-smith-phd
left a comment
peter-smith-phd
left a comment
There was a problem hiding this comment.
Thanks for doing this! As I've mentioned several times, the update process is quite sensitive and we need to test it very carefully. I'd suggest your next step is to generate a detailed test plan for the DevX team to review.
| - binary archive: `lstk-*` binaries and `lstk-extensions.toml` as siblings of `lstk` at the archive root; | ||
| - Homebrew: automatic via the cask's Caskroom staging of the whole archive (lstk ships as a **cask**, not a formula — no libexec involved); the cask's post-install quarantine hook is widened from the single `lstk` binary to the whole staged directory; | ||
| - npm: bundled files are copied into each **platform package** (`@localstack/lstk-<os>-<arch>`), where the real binary lives — not the wrapper package — via a post-processing step in the release job. | ||
| - **Pull the prebuilt closed-source bundled binaries from the private extensions repository's releases** into the release build context, **version-pinned via a pin file in this repo** (`bundled/extensions.version`), checksum-verified against the private release's manifest, authenticated with a dedicated read-only token, without exposing source in the public repository. |
There was a problem hiding this comment.
I'm not clear on the the bundled/extensions.version concept. I had been thinking that each lstk package would take the latest version of the bundled extensions. There would be no need for any "pinning". Was this a conscious change? Does it introduce a maintenance burden we don't currently have?
| - **Pull the prebuilt closed-source bundled binaries from the private extensions repository's releases** into the release build context, **version-pinned via a pin file in this repo** (`bundled/extensions.version`), checksum-verified against the private release's manifest, authenticated with a dedicated read-only token, without exposing source in the public repository. | ||
| - **Ship the hand-authored descriptions file** (`lstk-extensions.toml`), owned by LocalStack's private extensions repository, and **validate it at release time** (`scripts/check-descriptions.sh`) so a described-but-missing extension is a release-blocking error. | ||
| - **Update the `lstk`/`lstk-*` set as one unit** in `internal/update` for the self-managed binary channel (stage `.lstk-new` siblings, then rename, lstk last); Homebrew and npm replace the whole package — and therefore the whole set — via their package managers. | ||
| - **Guarantee update continuity**: `lstk update` keeps working for every existing install across the transition — a pre-bundling lstk updates cleanly into the first bundling release on all three channels (Homebrew and npm especially, where the updater shells out to the package manager), and a bundling lstk updates cleanly from an archive that carries no extensions (rollback). Bundled extensions are payload, never a precondition: their absence must not fail an update. |
There was a problem hiding this comment.
their absence must not fail an update.
Kind of... at least this makes sense for a rollback of the initial non-bundling -> bundling transition (that is, going back to a non-bundled-extensions release). But upgrading from a bundling lstk to another bundling lstk should require the extensions to be downloaded/installed correctly. They're not optional.
| `add-extension-mechanism` delivers the extension *mechanism* and bundled-directory *resolution*: lstk runs an `lstk-<name>` found next to its binary, ahead of `PATH`. It intentionally defers *distribution* so the first release can validate bundled extensions by manual placement. This change automates getting LocalStack's bundled extensions into the install artifacts and keeping them version-matched. The code for these decisions was prototyped during `add-extension-mechanism` and then removed from that change; this change re-introduces it. | ||
| `add-extension-mechanism` (PR #340) delivered the extension *mechanism* and everything lstk needs at runtime: bundled-directory resolution (`extension.BundledDir` — the directory of the symlink-resolved lstk executable, searched ahead of PATH), descriptions loading (`extension.LoadDescriptions` reads `lstk-extensions.toml` from that directory, degrading to an empty map on any failure), and help wiring (`cmd/extension.go`). It intentionally deferred *distribution* so the first release could validate bundled extensions by manual placement. This change automates getting LocalStack's bundled extensions into the install artifacts and keeping them version-matched with lstk. | ||
|
|
||
| The distribution code was prototyped during `add-extension-mechanism` and removed before the squash-merge; it is **not recoverable from git history** (the squashed commit contains only the runtime code and these openspec docs). Everything below is designed against the current tree. |
There was a problem hiding this comment.
I love it how Claude remembers everything that I did 😄 (or should that make me nervous?)
|
|
||
| The private extensions repository — the same source of truth that builds the closed-source binaries and hand-authors the descriptions file — publishes **tagged releases** whose assets are the per-platform binaries (`lstk-<name>_<os>_<arch>[.exe]`), `lstk-extensions.toml`, and a `checksums.txt` manifest covering them. | ||
|
|
||
| This repo carries a **pin file**, `bundled/extensions.version` — a single line naming the private release tag. Each lstk release therefore maps deterministically and reproducibly to one extensions bundle; bumping the pin is an ordinary reviewable PR (automatable later from the private repo's release workflow). `scripts/fetch-bundled-extensions.sh` reads the pin, downloads the assets (`gh release download`), **verifies each against the bundle's `checksums.txt`**, and stages them under `bundled/` with canonical names. It hard-fails when any lstk target platform has no matching asset (subject to an explicit not-supported allowlist), so platform gaps surface at pull time, not as an empty-glob failure inside GoReleaser. |
There was a problem hiding this comment.
The "reviewable PR" seems like an extra burden for the lstk maintainers. Is the expectation that somebody will review the extensions release and make a go/no-go decision on whether to start using them? I was thinking that we always release the latest extensions alongside each lstk release.
| This repo carries a **pin file**, `bundled/extensions.version` — a single line naming the private release tag. Each lstk release therefore maps deterministically and reproducibly to one extensions bundle; bumping the pin is an ordinary reviewable PR (automatable later from the private repo's release workflow). `scripts/fetch-bundled-extensions.sh` reads the pin, downloads the assets (`gh release download`), **verifies each against the bundle's `checksums.txt`**, and stages them under `bundled/` with canonical names. It hard-fails when any lstk target platform has no matching asset (subject to an explicit not-supported allowlist), so platform gaps surface at pull time, not as an empty-glob failure inside GoReleaser. | ||
|
|
||
| `internal/update` treats `lstk` and its bundled `lstk-*` set (binaries + the descriptions file) as one unit. For the self-managed binary channel, the extractor stages every new `lstk`/`lstk-*` member next to its destination (`.lstk-new` siblings) and renames each into place, so an interrupted update never leaves `lstk` and a bundled extension at mismatched versions. For Homebrew and npm, the package manager replaces the whole package — and therefore the whole bundled set — atomically. | ||
| The credential is a **dedicated fine-grained read-only PAT** (contents: read on the private repo only), stored as a repository/organization secret — not a reuse of the broader `PRO_ACCESS_TOKEN`. Least privilege, independent rotation. |
There was a problem hiding this comment.
Just to clarify, this "credential" is GitHub's access to allow the lstk repository actions to access the lstk-bundled-extensions releases?
| 1. **Discover** the set at the extracted archive root: the lstk binary, every executable `lstk-*`, and `lstk-extensions.toml`. An archive with no extensions yields a set of size one — today's behavior, byte for byte. | ||
| 2. **Clean orphans**: remove any `*.lstk-new` siblings left by a previously crashed update. | ||
| 3. **Stage**: copy each member into the destination dir (the running executable's directory) as a `<name>.lstk-new` sibling — same directory ⇒ same filesystem ⇒ each upcoming rename is atomic — and set 0755 on binaries. Any failure here removes the staged files and leaves the installation untouched. | ||
| 4. **Commit**: rename each `.lstk-new` over its final name — extensions and the descriptions file first, `lstk` itself **last**, so the load-bearing swap is the final act and "update reported success" implies the whole set committed. Windows keeps the existing rename-running-exe-to-`.old` dance for `lstk.exe` only; extensions are not running during `lstk update` and rename directly. |
There was a problem hiding this comment.
Migrating the lstk binary last seems to only matter if lstk refers to specific version of the extension. Given that it solely relies on the name, such as lstk-doctor, it doesn't really matter if lstk is first or last. The update of multiple files is simply going to be non-atomic, but since renames rarely fail (I've never seen it happen, when within the same directory), I'm not worried about this.
|
|
||
| 1. **Discover** the set at the extracted archive root: the lstk binary, every executable `lstk-*`, and `lstk-extensions.toml`. An archive with no extensions yields a set of size one — today's behavior, byte for byte. | ||
| 2. **Clean orphans**: remove any `*.lstk-new` siblings left by a previously crashed update. | ||
| 3. **Stage**: copy each member into the destination dir (the running executable's directory) as a `<name>.lstk-new` sibling — same directory ⇒ same filesystem ⇒ each upcoming rename is atomic — and set 0755 on binaries. Any failure here removes the staged files and leaves the installation untouched. |
There was a problem hiding this comment.
What's missing from this discussion is that there's only one extensions binary (called bundled-extensions), and everything else is a symlink to that binary. At least, that's how it works in Unix-based systems. I think for Windows there are restrictions around symlinking, so a different solution might be required (needs more investigation?)
|
|
||
| **The honest guarantee** (this is what the spec promises — not "atomic across files", which POSIX cannot deliver): no partially-written file is ever visible under a final name; the mismatch window is a handful of renames; an interrupted update is healed by re-running `lstk update` (the flow is idempotent). Momentary version skew inside that window is benign by contract: `LSTK_EXT_API_VERSION` bumps only on breaking changes. | ||
|
|
||
| **Additive-only**: the binary-channel update replaces and adds members but never deletes an `lstk-*` sibling absent from the new archive. Deleting safely requires knowing lstk *owns* the file — users may place their own extensions next to the binary, and the descriptions file is not an ownership manifest (the spec deliberately permits undescribed bundled binaries). A renamed/dropped extension therefore leaves its old binary behind on this channel only: it keeps working at its old version and shows name-only help (the replaced descriptions file no longer describes it, so help never disagrees with the shipped set). Deletion is deferred to the managed-extensions-directory work. Corollary: if a user manually placed an `lstk-<name>` in the install dir and a release later bundles that same name, the update overwrites it — that directory is lstk's install dir; PATH is the supported home for user-installed extensions. |
There was a problem hiding this comment.
I think we can solve this problem later, if we ever decide to remove an extension. For example, we could add custom rm code to explicitly remove the old extension. No need to worry about this now.
|
|
||
| ### Decision 5: Update continuity is a hard requirement — no existing install may be cut off from `lstk update` | ||
|
|
||
| The transition release (the first that ships bundled extensions) must be reachable by every in-the-field updater, and a later extension-free release must be reachable from a bundling one (rollback). Per channel: |
There was a problem hiding this comment.
This is the thing that makes me the most nervous. We need careful planning (i.e. solid and reviewed test plan) to convince ourselves this will work. It would be a really bad thing if 1000s of customers were stranded with an lstk that couldn't be upgraded anymore.
For example, if we introduce bundling in lstk version N, but a user goes on holiday for a month, then tries to upgrade from N-1 to N+3 in a single upgrade, are we sure the upgrade will work successfully?
2 participants
This PR includes only openspec docs for the extension bundling mechanism. It expands previous work (Thanks Peter!) and defines how the bunlding would work and the tasks to make it real.
This is intended to be the first draft to move forward the project. Any feedback, change, correction, pun or non pejorative expression will be very welcome!