rawpotion/forest
1
0
Fork 0
Code Issues 2 Pull requests 1 Projects Releases 7 Packages Wiki Activity Actions

Bundle versioned stdlib components with Forest CLI #1

New issue
Open
opened 2026-06-28 15:50:10 +02:00 by kjuulh · 0 comments
kjuulh commented 2026-06-28 15:50:10 +02:00
Owner
Copy link

Problem

Forest already has stdlib-like components in apps/forest/components, but installed releases do not ship them. Users must bootstrap/publish/update registry packages before local projects can use standard components like:

  • forest/sdk — currently 0.7.0
  • forest/deployment — currently 0.3.0
  • forest-contrib/build-rust|build-go|build-docker — currently 0.1.0

Current release tarballs contain only the forest binary. scripts/install.sh installs only that binary. Runtime resolution then depends on registry/CUE module availability.

Grounded findings

  • Project dependencies are parsed from forest.cue into DependencyType::{Versioned, Local} in apps/forest/crates/forest/src/services/project.rs.
  • Versioned deps are resolved through ComponentsService::sync_components and fetched from registry into ~/.cache/forest/components/<org>/<name>/<version>/.
  • CUE imports are separate from Forest dependencies. cue export is delegated to the external cue binary and relies on CUE_REGISTRY or vendored files under cue.mod/pkg.
  • Registry storage already supports multiple committed versions via components(name, organisation, version) plus per-version files/manifests/artifacts.
  • The repo source tree currently carries only one working copy per bundled component under apps/forest/components.
  • forest update can resolve latest/partial versions via ListComponentVersions, but normal forest run currently matches the version string declared in forest.cue directly.

Recommended design

Add an embedded/packaged Stdlib Component Provider that is checked before registry.

Resolution order:

  1. Local path dependency
  2. Stdlib bundled catalog
  3. User cache
  4. Remote registry

Introduce a versioned stdlib catalog, e.g.

apps/forest/components/stdlib/
  manifest.toml
  forest/sdk/0.7.0/...
  forest/deployment/0.3.0/...
  forest-contrib/build-rust/0.1.0/...

The catalog should expose the same concepts as registry:

  • org/name/version
  • kind: files, binary, deno, external
  • CUE files
  • manifest/descriptor metadata
  • per-platform binary hashes/paths where needed

Implementation plan

  1. Create stdlib catalog format

    • Add explicit versioned stdlib snapshots.
    • Stop relying on “whatever is latest in apps/forest/components/<org>/<name>”.
    • Keep a manifest with component list, versions, kind, file hashes, and binary platforms.

  2. Add StdlibProvider

    • Internal API equivalent to:
      • list_versions(org, name)
      • get_component_version(org, name, version)
      • get_manifest(org, name, version)
      • materialize_files(org, name, version)
      • materialize_binary(org, name, version, os, arch)
    • Hook it into ComponentsService::sync_components, ensure_versioned_dep_cached, forest add, and forest update.

  3. Materialize bundled components into existing cache

    • Reuse ~/.cache/forest/components/<org>/<name>/<version>/.
    • For binary components, write:
      • .forest/component/meta.json
      • forest.component.cue marker
      • content-addressed binary under existing binary cache layout
    • This keeps release prepare, forest run, Deno, templates, and command discovery mostly unchanged.

  4. Handle CUE stdlib imports offline

    • Before spawning cue, ensure bundled stdlib CUE modules are vendored into:
      • cue.mod/pkg/forest.sh/forest/sdk@v0/
      • cue.mod/pkg/forest.sh/forest/deployment@v0/
    • Apply to project parsing, publish, generate, global user config, and release prepare.
    • Keep CUE_REGISTRY fallback for non-stdlib imports.

  5. Update packaging

    • Current .woodpecker/release-build.yaml builds only forest and tars only the binary.
    • Either:
      • package components/stdlib beside the binary and install to a known data dir, or
      • embed stdlib files into the binary and materialize on demand.
    • For first cut, packaged data dir is simpler and safer for binary components.
    • Update scripts/install.sh and forest self update to install/copy stdlib assets.

  6. Lockfile/source tracking

    • Extend forest.lock with a stdlib source, e.g. 
      forest/sdk@0.7.0 stdlib:forest-0.2.4 sha256:...
    • This makes provenance clear and avoids pretending bundled components came from registry.

Acceptance criteria

  • Fresh install can parse a project importing forest.sh/forest/sdk@v0 without a Forest registry.
  • forest add forest-contrib/build-rust resolves bundled latest without registry.
  • forest run build works from a clean machine after install, using bundled build-rust.
  • forest update resolves stdlib versions from local catalog before registry.
  • Multiple stdlib versions can coexist in the bundled catalog.
  • Non-stdlib components still resolve from registry.
  • Release tarball/install/self-update preserve stdlib assets.
  • Tests cover offline CUE import, stdlib materialization, binary command discovery, and registry fallback.

Main risks

  • CUE imports and Forest dependencies are separate systems; both need stdlib handling.
  • Binary stdlib components need per-platform packaging, not just CUE files.
  • Current normal forest run does not appear to use forest.lock for partial/latest resolution; this should be fixed or avoided for stdlib specs.
  • Registry search/detail have latest-oriented assumptions; not blocking local stdlib, but relevant if stdlib is surfaced in UI later.
## Problem Forest already has stdlib-like components in `apps/forest/components`, but installed releases do not ship them. Users must bootstrap/publish/update registry packages before local projects can use standard components like: - `forest/sdk` — currently `0.7.0` - `forest/deployment` — currently `0.3.0` - `forest-contrib/build-rust|build-go|build-docker` — currently `0.1.0` Current release tarballs contain only the `forest` binary. `scripts/install.sh` installs only that binary. Runtime resolution then depends on registry/CUE module availability. ## Grounded findings - Project dependencies are parsed from `forest.cue` into `DependencyType::{Versioned, Local}` in `apps/forest/crates/forest/src/services/project.rs`. - Versioned deps are resolved through `ComponentsService::sync_components` and fetched from registry into `~/.cache/forest/components/<org>/<name>/<version>/`. - CUE imports are separate from Forest dependencies. `cue export` is delegated to the external `cue` binary and relies on `CUE_REGISTRY` or vendored files under `cue.mod/pkg`. - Registry storage already supports multiple committed versions via `components(name, organisation, version)` plus per-version files/manifests/artifacts. - The repo source tree currently carries only one working copy per bundled component under `apps/forest/components`. - `forest update` can resolve `latest`/partial versions via `ListComponentVersions`, but normal `forest run` currently matches the version string declared in `forest.cue` directly. ## Recommended design Add an embedded/packaged **Stdlib Component Provider** that is checked before registry. Resolution order: 1. Local path dependency 2. Stdlib bundled catalog 3. User cache 4. Remote registry Introduce a versioned stdlib catalog, e.g. ```text apps/forest/components/stdlib/ manifest.toml forest/sdk/0.7.0/... forest/deployment/0.3.0/... forest-contrib/build-rust/0.1.0/... ``` The catalog should expose the same concepts as registry: - org/name/version - kind: `files`, `binary`, `deno`, `external` - CUE files - manifest/descriptor metadata - per-platform binary hashes/paths where needed ## Implementation plan 1. **Create stdlib catalog format** - Add explicit versioned stdlib snapshots. - Stop relying on “whatever is latest in `apps/forest/components/<org>/<name>`”. - Keep a manifest with component list, versions, kind, file hashes, and binary platforms. 2. **Add `StdlibProvider`** - Internal API equivalent to: - `list_versions(org, name)` - `get_component_version(org, name, version)` - `get_manifest(org, name, version)` - `materialize_files(org, name, version)` - `materialize_binary(org, name, version, os, arch)` - Hook it into `ComponentsService::sync_components`, `ensure_versioned_dep_cached`, `forest add`, and `forest update`. 3. **Materialize bundled components into existing cache** - Reuse `~/.cache/forest/components/<org>/<name>/<version>/`. - For binary components, write: - `.forest/component/meta.json` - `forest.component.cue` marker - content-addressed binary under existing binary cache layout - This keeps `release prepare`, `forest run`, Deno, templates, and command discovery mostly unchanged. 4. **Handle CUE stdlib imports offline** - Before spawning `cue`, ensure bundled stdlib CUE modules are vendored into: - `cue.mod/pkg/forest.sh/forest/sdk@v0/` - `cue.mod/pkg/forest.sh/forest/deployment@v0/` - Apply to project parsing, publish, generate, global user config, and release prepare. - Keep `CUE_REGISTRY` fallback for non-stdlib imports. 5. **Update packaging** - Current `.woodpecker/release-build.yaml` builds only `forest` and tars only the binary. - Either: - package `components/stdlib` beside the binary and install to a known data dir, or - embed stdlib files into the binary and materialize on demand. - For first cut, packaged data dir is simpler and safer for binary components. - Update `scripts/install.sh` and `forest self update` to install/copy stdlib assets. 6. **Lockfile/source tracking** - Extend `forest.lock` with a stdlib source, e.g. ```text forest/sdk@0.7.0 stdlib:forest-0.2.4 sha256:... ``` - This makes provenance clear and avoids pretending bundled components came from registry. ## Acceptance criteria - Fresh install can parse a project importing `forest.sh/forest/sdk@v0` without a Forest registry. - `forest add forest-contrib/build-rust` resolves bundled latest without registry. - `forest run build` works from a clean machine after install, using bundled `build-rust`. - `forest update` resolves stdlib versions from local catalog before registry. - Multiple stdlib versions can coexist in the bundled catalog. - Non-stdlib components still resolve from registry. - Release tarball/install/self-update preserve stdlib assets. - Tests cover offline CUE import, stdlib materialization, binary command discovery, and registry fallback. ## Main risks - CUE imports and Forest dependencies are separate systems; both need stdlib handling. - Binary stdlib components need per-platform packaging, not just CUE files. - Current normal `forest run` does not appear to use `forest.lock` for partial/latest resolution; this should be fixed or avoided for stdlib specs. - Registry search/detail have latest-oriented assumptions; not blocking local stdlib, but relevant if stdlib is surfaced in UI later.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
kjuulh/gitea-fork
feature/forest-stdlib-bundle
push-sozzwkwzmsmq
main
push-zyouyvtvpwxx
hollow-phase1
v0.1.14
v0.1.13
v0.1.12
v0.1.11
v0.1.10
v0.0.1+alpha.2
v0.0.1+alpha.1
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
rawpotion/forest#1
No description provided.