rawpotion/forest
1
0
Fork 0
Code Issues 2 Pull requests 1 Projects Releases 7 Packages Wiki Activity Actions

Proposal: Git-backed Forest components via srcball #3

New issue
Open
opened 2026-07-08 13:00:13 +02:00 by kjuulh · 0 comments
kjuulh commented 2026-07-08 13:00:13 +02:00
Owner
Copy link

014 - Git-Backed Forest Components via srcball

Status: Phase 1 — Proposal / design review.
Depends on: Forest component v2 (forest.cue + forest.component.cue), component cache, forest update, forest.lock, srcball source proxy.
Driver: Forest's component registry currently owns source upload, file bundles, binary distribution, search metadata, auth, and CUE module publishing. The stronger long-term model is to make Git repositories the package authority and use srcball as a pull-through Git/source cache, so teams publish components by tagging source instead of uploading copies into a Forest-owned registry.

Problem

Forest components are already authored as source packages:

  • forest.cue declares project/component metadata.
  • forest.component.cue declares #Spec, #Commands, #Hooks, and optional tool/runtime facets.
  • Component source lives in Git.
  • Consumers declare dependencies by org/name and configure usage blocks under the same identity.

But distribution still flows through a Forest registry upload path:

  • forest components publish uploads files and binaries.
  • The server stores component files, manifests, per-platform binaries, and CUE OCI modules.
  • forest update and runtime paths pull from Forest registry APIs.

That duplicates what Git already does well: authority, audit, tags, access control, and source history. It also creates a second source of truth for component source bundles.

srcball is a better substrate for the source half of this system: it turns a Git repository/subpath/ref into an immutable cached source artifact, exposes direct package and OCI-style pull endpoints, materializes source trees, and can be backed by a local volume today and object storage later.

Decision

Use srcball to replace Forest's source distribution path, not Forest's component runtime.

Keep Forest's component framework:

  • forest.cue
  • forest.component.cue
  • CUE schemas and codegen
  • forest run
  • release hooks
  • binary/Deno invocation
  • forest.lock
  • local component cache layout

Replace the publish/upload source flow with Git/source resolution:

Git forge = source authority
srcball   = pull-through source cache
Forest    = resolver, cache hydrator, runtime, optional catalog

Do not make srcball a Forest-aware component registry in the first version. Keep it generic: Git locator + optional subpath + ref → cached source artifact.

Goals

  1. Git as package authority: component versions are Git tags/refs, not uploaded source copies.
  2. srcball as source substrate: Forest asks srcball for a repository subpath/ref and receives a source artifact or materialized directory.
  3. Stable Forest identity: consumers still use org/name because usage blocks, commands, and docs depend on that identity.
  4. No runtime rewrite: existing component parsing, command registration, CUE schemas, Deno/binary invocation, and release hooks continue to operate on ~/.cache/forest/components/<org>/<name>/<version>/.
  5. Semver from Git tags: forest update can resolve 0.1, 1, latest, and exact versions from remote tags.
  6. Reproducible locks: forest.lock records the resolved Git commit and source artifact digest.
  7. Private repository path: local mode works with user Git credentials; central mode is explicitly blocked on request identity and authorization.

Non-goals

  • Replacing forest run, component SDK protocols, #Commands, #Hooks, or runtime descriptors.
  • Making srcball own Forest-specific manifests, tool facets, contracts, or search indexes in the first version.
  • Replacing per-platform binary distribution immediately.
  • Running a shared private-repo srcball service without an authorization model.
  • Removing the current Forest registry before source-backed components cover the common paths.

Current Forest shape

Component contract

Component authors define metadata in forest.cue:

project: sdk.#ForestProject & {
    name:         "build-rust"
    organisation: "forest-contrib"
}

forest: component: sdk.#ForestComponent & {
    name:    project.name
    version: "0.1.0"

    upload: {
        source: "./crates/build-rust"
        type:   "rust"
        architectures: {
            linux: { amd64: {}, arm64: {} }
            macos: { amd64: {}, arm64: {} }
        }
    }
}

Component interfaces live in forest.component.cue:

#Spec: sdk.#ForestSpec & { ... }

#Commands: sdk.#ForestCommands & {
    build: {
        description: "Compile the depending component."
        input: {}
        output: {...}
    }
}

#Hooks: sdk.#ForestHooks & { ... }

Consumer projects declare dependencies and usage separately:

dependencies: sdk.#ForestDependencies & {
    "forest-contrib/kubernetes-service": version: "0.1"
    "forest-contrib/terraform-service": path: "../my-local-component"
}

"forest-contrib": "kubernetes-service": sdk.#ForestComponentUsage & {
    env: {
        dev: {
            destinations: [{destination: "k8s-dev", type: "forest/kubernetes@1"}]
            config: replicas: 1
        }
    }
    config: k8s.#Spec & {
        name: "my-service"
    }
}

This means org/name is a stable logical identity. srcball's raw Git locator cannot replace it directly; Forest needs a mapping layer from org/name to Git source.

Registry/API responsibilities today

Forest registry currently provides:

  • GetComponent, GetComponentVersion, ListComponentVersions
  • BeginUpload, UploadFile, CommitUpload, AbortUpload
  • GetComponentFiles
  • UploadBinary, DownloadBinary
  • PublishManifest, GetComponentManifest
  • search/detail/public variants
  • ListOrgTools
  • auth and visibility checks
  • server-side CUE OCI module publishing

Those responsibilities should be split. Source distribution can move to srcball/Git; runtime metadata, cataloging, binary strategy, and auth need separate treatment.

Proposed dependency syntax

Extend #ForestDependency with a source object while keeping existing version and path forms.

dependencies: sdk.#ForestDependencies & {
    "forest-contrib/build-rust": {
        version: "0.1.0"
        source: {
            type:    "git"
            locator: "github.com/rawpotion/forest//apps/forest/components/forest-contrib/build-rust"
            ref:     "forest-contrib/build-rust/v0.1.0"
        }
    }
}

For semver ranges, omit ref and provide a tag prefix:

dependencies: sdk.#ForestDependencies & {
    "forest-contrib/build-rust": {
        version: "0.1"
        source: {
            type:       "git"
            locator:    "github.com/rawpotion/forest//apps/forest/components/forest-contrib/build-rust"
            tag_prefix: "forest-contrib/build-rust/v"
        }
    }
}

Rules:

  • path remains local development.
  • version remains the semantic version requirement.
  • source.git.locator identifies the Git repository and optional component subdirectory using srcball's repo//sub/path convention.
  • ref pins an exact Git ref/tag.
  • tag_prefix supports monorepo component tags.
  • The resolved component's forest.component.version must match the selected semantic version.

forest add behavior

New accepted form:

forest add github.com/rawpotion/forest//apps/forest/components/forest-contrib/build-rust@forest-contrib/build-rust/v0.1.0

Flow:

  1. Parse Git locator/ref.
  2. Fetch or materialize via srcball.
  3. Read forest.cue from the materialized source.
  4. Extract project.organisation, forest.component.name, and forest.component.version.
  5. Insert the canonical dependency key as org/name.
  6. Store the Git source locator/ref alongside the semantic version.
  7. Update cue.mod/module.cue for forest.sh/{org}/{name}@v0 when CUE imports are needed.

The user provides a source. Forest records the component identity declared by the source.

Cache hydration

Forest should keep its current local cache layout:

~/.cache/forest/components/<org>/<name>/<version>/

For a Git-backed dependency, Forest should ask srcball for the source artifact, then materialize the selected subpath into the cache directory:

~/.cache/forest/components/forest-contrib/build-rust/0.1.0/
  forest.cue
  forest.component.cue
  cue.mod/
  crates/build-rust/
  templates/

Existing code paths can then continue to work:

  • ComponentParser
  • ProjectParser
  • forest run
  • forest generate
  • release hooks
  • Deno component resolver
  • template discovery

Forest should not depend on srcball's internal storage paths. Treat srcball as a fetch/materialize API, not as a cache directory to walk directly.

Version resolution

For Git-backed components, forest update should resolve versions from Git tags instead of registry ListComponentVersions.

Suggested monorepo tag convention:

forest-contrib/build-rust/v0.1.0
forest-contrib/build-rust/v0.1.1
forest-contrib/build-go/v0.1.0
forest/deployment/v0.7.0

Single-component repositories may use normal tags:

v0.1.0
v0.1.1

Resolution flow:

  1. List remote tags through srcball/Git.
  2. Filter by tag_prefix when present.
  3. Parse semantic versions.
  4. Apply existing VersionSpec behavior (exact, minor, major, latest, *).
  5. Fetch the selected ref through srcball.
  6. Verify forest.component.version matches the selected version.
  7. Lock resolved commit and artifact digest.

Lockfile extension

Current lockfile entries should grow a source form.

Example:

forest-contrib/build-rust@0.1.3 source:git locator:github.com/rawpotion/forest//apps/forest/components/forest-contrib/build-rust ref:forest-contrib/build-rust/v0.1.3 commit:4764ded5b5fc474ec11c8e821b0cddef1f28de68 digest:sha256:...

For binary components, keep platform-specific binary hashes as separate lock entries or attach them to the same logical dependency:

forest-contrib/build-rust@0.1.3 linux/amd64 sha256:<binary-hash>

Important invariants:

  • Git tag movement is detected by commit mismatch.
  • Source artifact mutation is detected by digest mismatch.
  • Re-resolving a locked dependency should not silently change commit.

CUE module compatibility

Forest currently makes component CUE imports work by publishing CUE files as OCI modules under:

forest.sh/{org}/{name}@v0

srcball's current source OCI artifact is not the same as a CUE module registry artifact. CUE expects module metadata/zip semantics; srcball currently returns source OCI layout artifacts.

Use a staged approach.

Phase A: Forest-side vendoring

After srcball materializes a component source tree, Forest vendors .cue files into the project's cue.mod/pkg/forest.sh/{org}/{name}@v<major>/ directory, matching existing behavior.

This proves source-backed components without changing srcball's media formats.

Phase B: CUE adapter

Add either a Forest-side or srcball-side CUE module adapter:

GET /cue/<module>/@v/list
GET /cue/<module>/@v/<version>.info
GET /cue/<module>/@v/<version>.mod
GET /cue/<module>/@v/<version>.zip

or OCI-compatible CUE module endpoints backed by srcball source artifacts.

Do this after Phase A works.

Binary component strategy

srcball currently distributes source, not per-platform executables. Forest binary/global-tool behavior needs a deliberate strategy.

Option 1: Build on install

srcball source -> Forest cache -> local build -> binary cache

Pros:

  • No binary registry needed.
  • Works with private Git credentials.
  • Keeps Git as the source authority.

Cons:

  • Slow.
  • Requires local toolchains.
  • Poorer global-tool UX.

Good first path for internal components.

Option 2: GitHub Releases / external assets

Declare release assets in component metadata or manifest:

forest: component: {
    external: {
        platforms: [{
            os:     "linux"
            arch:   "amd64"
            url:    "https://github.com/org/repo/releases/download/v0.1.0/tool-linux-amd64"
            sha256: "..."
        }]
    }
}

Pros:

  • GitHub remains source and artifact authority.
  • Good for public/global tools.
  • Avoids rebuilding on every install.

Cons:

  • Private release asset auth needs design.
  • Requires asset naming and checksum discipline.

Preferred long-term path for global tools.

Avoid initially: srcball as binary registry

If srcball starts storing build outputs, it becomes another Forest registry. Defer until a concrete need proves this is worth the complexity.

Auth and private repositories

Local mode is safe:

Forest CLI -> local srcball -> user's Git credentials -> user's cache

Central shared mode is unsafe unless authorized:

User A can access private repo.
srcball caches it.
User B asks for same package/ref.
If cache hit bypasses auth, private source leaks.

Before running central srcball for private repositories, require:

  • request identity,
  • per-request GitHub/Git forge authorization,
  • cache hit authorization checks,
  • cache partitioning or ACL metadata,
  • public/private source separation,
  • revocation behavior.

Until then, shared srcball should serve only public sources or single-tenant/private deployments.

Catalog and discovery

A searchable registry is still useful, but it should be derived, not authoritative.

Possible catalog indexer:

  1. Scan configured GitHub orgs/repos.
  2. Find forest.cue + forest.component.cue.
  3. Parse component metadata, README, methods, tool facet, contracts, tags.
  4. Store searchable rows for UI and forest search.
  5. Do not store source bundles as canonical artifacts.

The catalog can replace registry search/detail while Git/srcball remain the source path.

Migration plan

Milestone 1 — Source-backed local cache hydration

Implement Git-source dependency parsing and materialization through srcball.

Acceptance:

  • A component currently consumed via path: can be consumed via Git source.
  • ~/.cache/forest/components/<org>/<name>/<version>/ is populated from srcball.
  • ComponentParser reads the materialized component.
  • Existing forest run/forest generate paths operate on the cache.

Milestone 2 — Git tag semver resolution

Implement tag listing and version selection for Git-backed deps.

Acceptance:

  • version: "0.1" resolves to highest matching Git tag.
  • latest resolves to the highest semver tag, not mutable HEAD.
  • Lockfile records ref, commit, and digest.
  • Tag movement is detected.

Milestone 3 — CUE import support

Vendor CUE files from materialized source into cue.mod/pkg or add a CUE module adapter.

Acceptance:

  • import "forest.sh/{org}/{name}@v0" works for Git-backed components.
  • cue export works in a clean checkout after forest update.

Milestone 4 — Publish becomes validation/tagging

For Git-backed components, forest publish validates component metadata and release readiness instead of uploading files.

Acceptance:

  • It verifies the Git tag/ref exists.
  • It verifies forest.component.version matches the tag/version.
  • It optionally updates a derived catalog.
  • It does not call UploadFile for source-only components.

Milestone 5 — Binary strategy

Choose local build or external release assets for binary/global-tool components.

Acceptance:

  • Binary components can be installed/run without Forest registry binary storage for at least one supported path.
  • Lockfile records binary SHA when binaries are used.

Milestone 6 — Registry reduction

Delete or deprecate source-file upload paths once Git-backed source components are stable.

Candidate removals/deprecations:

  • source BeginUpload/UploadFile/CommitUpload for Git-backed components,
  • object-store component file bundles,
  • server-side CUE module generation from uploaded source files.

Keep registry/catalog APIs only where they provide metadata/search/auth not covered by Git/srcball.

srcball additions needed

  1. Materialize API

    Forest needs a stable way to hydrate a destination directory without depending on srcball internals.

    SrcballClient::materialize(&PackageSpec, destination: &Path)

    or an HTTP endpoint that streams normalized files.

  2. Reference listing

    Needed for semver tag resolution.

    GET /packages/refs?package=<repo>[//subpath]

    Response should include tags, heads, and commit IDs.

  3. Artifact digest in metadata

    ArtifactMetadata already includes package, version, source ref, commit, and time. Forest also needs the source artifact digest for lockfile integrity.

  4. CUE adapter later

    Add CUE module/proxy endpoints only after Forest-side vendoring proves the model.

  5. Auth model before shared private use

    Central srcball must authorize cache hits, not just Git fetches.

Forest changes needed

  1. Extend #ForestDependency with source.
  2. Add a ComponentSourceResolver abstraction for local path, current registry, and srcball Git sources.
  3. Update forest add to inspect Git-backed component sources and write canonical org/name dependencies.
  4. Update forest update to resolve Git tags and hydrate the component cache from srcball.
  5. Extend forest.lock with source locator/ref/commit/digest.
  6. Keep current cache layout and runtime paths stable.
  7. Add CUE vendoring for Git-backed components.
  8. Split forest publish into source validation/tagging vs legacy registry upload.

Risks

High — private cache authorization

A central srcball can leak private repository content if cache hits are not authorized per requester.

Mitigation: local-only first; central private mode requires identity, GitHub/Git forge authorization, cache ACLs, and revocation behavior.

High — binary distribution

Source fetch does not replace per-platform executable distribution.

Mitigation: build locally first for internal components; use GitHub Releases/external manifests for global tools; defer srcball binary storage.

High — CUE module semantics

CUE module OCI artifacts are not the same as srcball source OCI artifacts.

Mitigation: Forest-side CUE vendoring first, CUE adapter later.

Medium — monorepo version tags

Monorepos need tag naming conventions and prefix filtering.

Mitigation: require explicit tag_prefix; validate component version against selected tag.

Medium — search/catalog regression

A Git-only model may lose registry discovery UX.

Mitigation: build a derived catalog index over Git repositories.

Medium — reproducibility

Git tags can move.

Mitigation: lock commit and artifact digest; fail when a locked tag resolves differently.

Recommended first spike

Pick one existing local path component, for example forest-contrib/build-rust, and consume it from Git through srcball.

Target dependency:

dependencies: {
    "forest-contrib/build-rust": {
        version: "0.1.0"
        source: {
            type:    "git"
            locator: "github.com/rawpotion/forest//apps/forest/components/forest-contrib/build-rust"
            ref:     "forest-contrib/build-rust/v0.1.0"
        }
    }
}

Spike acceptance:

  1. forest update fetches the source through srcball.
  2. Forest materializes the source into ~/.cache/forest/components/forest-contrib/build-rust/0.1.0.
  3. forest.component.cue and forest.cue parse from the cache.
  4. Existing command/runtime discovery sees the same component surface as a local path dependency.
  5. forest.lock records source locator, ref, commit, and digest.
  6. No Forest registry source upload is involved.

Bottom line

The strongest version of this idea is not "rebuild the Forest registry on top of srcball".

It is:

Forest components are Git packages.
srcball is the pull-through source proxy.
Forest remains the component runtime and resolver.
GitHub/Git remains the publisher, ACL boundary, audit log, and release history.

That cuts away duplicated source storage while preserving the valuable parts of Forest: typed CUE contracts, composable components, release hooks, and forest run.

# 014 - Git-Backed Forest Components via srcball **Status:** Phase 1 — Proposal / design review. **Depends on:** Forest component v2 (`forest.cue` + `forest.component.cue`), component cache, `forest update`, `forest.lock`, srcball source proxy. **Driver:** Forest's component registry currently owns source upload, file bundles, binary distribution, search metadata, auth, and CUE module publishing. The stronger long-term model is to make Git repositories the package authority and use srcball as a pull-through Git/source cache, so teams publish components by tagging source instead of uploading copies into a Forest-owned registry. --- ## Problem Forest components are already authored as source packages: - `forest.cue` declares project/component metadata. - `forest.component.cue` declares `#Spec`, `#Commands`, `#Hooks`, and optional tool/runtime facets. - Component source lives in Git. - Consumers declare dependencies by `org/name` and configure usage blocks under the same identity. But distribution still flows through a Forest registry upload path: - `forest components publish` uploads files and binaries. - The server stores component files, manifests, per-platform binaries, and CUE OCI modules. - `forest update` and runtime paths pull from Forest registry APIs. That duplicates what Git already does well: authority, audit, tags, access control, and source history. It also creates a second source of truth for component source bundles. srcball is a better substrate for the source half of this system: it turns a Git repository/subpath/ref into an immutable cached source artifact, exposes direct package and OCI-style pull endpoints, materializes source trees, and can be backed by a local volume today and object storage later. --- ## Decision Use srcball to replace Forest's **source distribution path**, not Forest's **component runtime**. Keep Forest's component framework: - `forest.cue` - `forest.component.cue` - CUE schemas and codegen - `forest run` - release hooks - binary/Deno invocation - `forest.lock` - local component cache layout Replace the publish/upload source flow with Git/source resolution: ```text Git forge = source authority srcball = pull-through source cache Forest = resolver, cache hydrator, runtime, optional catalog ``` Do not make srcball a Forest-aware component registry in the first version. Keep it generic: Git locator + optional subpath + ref → cached source artifact. --- ## Goals 1. **Git as package authority**: component versions are Git tags/refs, not uploaded source copies. 2. **srcball as source substrate**: Forest asks srcball for a repository subpath/ref and receives a source artifact or materialized directory. 3. **Stable Forest identity**: consumers still use `org/name` because usage blocks, commands, and docs depend on that identity. 4. **No runtime rewrite**: existing component parsing, command registration, CUE schemas, Deno/binary invocation, and release hooks continue to operate on `~/.cache/forest/components/<org>/<name>/<version>/`. 5. **Semver from Git tags**: `forest update` can resolve `0.1`, `1`, `latest`, and exact versions from remote tags. 6. **Reproducible locks**: `forest.lock` records the resolved Git commit and source artifact digest. 7. **Private repository path**: local mode works with user Git credentials; central mode is explicitly blocked on request identity and authorization. --- ## Non-goals - Replacing `forest run`, component SDK protocols, `#Commands`, `#Hooks`, or runtime descriptors. - Making srcball own Forest-specific manifests, tool facets, contracts, or search indexes in the first version. - Replacing per-platform binary distribution immediately. - Running a shared private-repo srcball service without an authorization model. - Removing the current Forest registry before source-backed components cover the common paths. --- ## Current Forest shape ### Component contract Component authors define metadata in `forest.cue`: ```cue project: sdk.#ForestProject & { name: "build-rust" organisation: "forest-contrib" } forest: component: sdk.#ForestComponent & { name: project.name version: "0.1.0" upload: { source: "./crates/build-rust" type: "rust" architectures: { linux: { amd64: {}, arm64: {} } macos: { amd64: {}, arm64: {} } } } } ``` Component interfaces live in `forest.component.cue`: ```cue #Spec: sdk.#ForestSpec & { ... } #Commands: sdk.#ForestCommands & { build: { description: "Compile the depending component." input: {} output: {...} } } #Hooks: sdk.#ForestHooks & { ... } ``` Consumer projects declare dependencies and usage separately: ```cue dependencies: sdk.#ForestDependencies & { "forest-contrib/kubernetes-service": version: "0.1" "forest-contrib/terraform-service": path: "../my-local-component" } "forest-contrib": "kubernetes-service": sdk.#ForestComponentUsage & { env: { dev: { destinations: [{destination: "k8s-dev", type: "forest/kubernetes@1"}] config: replicas: 1 } } config: k8s.#Spec & { name: "my-service" } } ``` This means `org/name` is a stable logical identity. srcball's raw Git locator cannot replace it directly; Forest needs a mapping layer from `org/name` to Git source. ### Registry/API responsibilities today Forest registry currently provides: - `GetComponent`, `GetComponentVersion`, `ListComponentVersions` - `BeginUpload`, `UploadFile`, `CommitUpload`, `AbortUpload` - `GetComponentFiles` - `UploadBinary`, `DownloadBinary` - `PublishManifest`, `GetComponentManifest` - search/detail/public variants - `ListOrgTools` - auth and visibility checks - server-side CUE OCI module publishing Those responsibilities should be split. Source distribution can move to srcball/Git; runtime metadata, cataloging, binary strategy, and auth need separate treatment. --- ## Proposed dependency syntax Extend `#ForestDependency` with a source object while keeping existing `version` and `path` forms. ```cue dependencies: sdk.#ForestDependencies & { "forest-contrib/build-rust": { version: "0.1.0" source: { type: "git" locator: "github.com/rawpotion/forest//apps/forest/components/forest-contrib/build-rust" ref: "forest-contrib/build-rust/v0.1.0" } } } ``` For semver ranges, omit `ref` and provide a tag prefix: ```cue dependencies: sdk.#ForestDependencies & { "forest-contrib/build-rust": { version: "0.1" source: { type: "git" locator: "github.com/rawpotion/forest//apps/forest/components/forest-contrib/build-rust" tag_prefix: "forest-contrib/build-rust/v" } } } ``` Rules: - `path` remains local development. - `version` remains the semantic version requirement. - `source.git.locator` identifies the Git repository and optional component subdirectory using srcball's `repo//sub/path` convention. - `ref` pins an exact Git ref/tag. - `tag_prefix` supports monorepo component tags. - The resolved component's `forest.component.version` must match the selected semantic version. --- ## `forest add` behavior New accepted form: ```bash forest add github.com/rawpotion/forest//apps/forest/components/forest-contrib/build-rust@forest-contrib/build-rust/v0.1.0 ``` Flow: 1. Parse Git locator/ref. 2. Fetch or materialize via srcball. 3. Read `forest.cue` from the materialized source. 4. Extract `project.organisation`, `forest.component.name`, and `forest.component.version`. 5. Insert the canonical dependency key as `org/name`. 6. Store the Git source locator/ref alongside the semantic version. 7. Update `cue.mod/module.cue` for `forest.sh/{org}/{name}@v0` when CUE imports are needed. The user provides a source. Forest records the component identity declared by the source. --- ## Cache hydration Forest should keep its current local cache layout: ```text ~/.cache/forest/components/<org>/<name>/<version>/ ``` For a Git-backed dependency, Forest should ask srcball for the source artifact, then materialize the selected subpath into the cache directory: ```text ~/.cache/forest/components/forest-contrib/build-rust/0.1.0/ forest.cue forest.component.cue cue.mod/ crates/build-rust/ templates/ ``` Existing code paths can then continue to work: - `ComponentParser` - `ProjectParser` - `forest run` - `forest generate` - release hooks - Deno component resolver - template discovery Forest should not depend on srcball's internal storage paths. Treat srcball as a fetch/materialize API, not as a cache directory to walk directly. --- ## Version resolution For Git-backed components, `forest update` should resolve versions from Git tags instead of registry `ListComponentVersions`. Suggested monorepo tag convention: ```text forest-contrib/build-rust/v0.1.0 forest-contrib/build-rust/v0.1.1 forest-contrib/build-go/v0.1.0 forest/deployment/v0.7.0 ``` Single-component repositories may use normal tags: ```text v0.1.0 v0.1.1 ``` Resolution flow: 1. List remote tags through srcball/Git. 2. Filter by `tag_prefix` when present. 3. Parse semantic versions. 4. Apply existing `VersionSpec` behavior (`exact`, `minor`, `major`, `latest`, `*`). 5. Fetch the selected ref through srcball. 6. Verify `forest.component.version` matches the selected version. 7. Lock resolved commit and artifact digest. --- ## Lockfile extension Current lockfile entries should grow a source form. Example: ```text forest-contrib/build-rust@0.1.3 source:git locator:github.com/rawpotion/forest//apps/forest/components/forest-contrib/build-rust ref:forest-contrib/build-rust/v0.1.3 commit:4764ded5b5fc474ec11c8e821b0cddef1f28de68 digest:sha256:... ``` For binary components, keep platform-specific binary hashes as separate lock entries or attach them to the same logical dependency: ```text forest-contrib/build-rust@0.1.3 linux/amd64 sha256:<binary-hash> ``` Important invariants: - Git tag movement is detected by commit mismatch. - Source artifact mutation is detected by digest mismatch. - Re-resolving a locked dependency should not silently change commit. --- ## CUE module compatibility Forest currently makes component CUE imports work by publishing CUE files as OCI modules under: ```text forest.sh/{org}/{name}@v0 ``` srcball's current source OCI artifact is not the same as a CUE module registry artifact. CUE expects module metadata/zip semantics; srcball currently returns source OCI layout artifacts. Use a staged approach. ### Phase A: Forest-side vendoring After srcball materializes a component source tree, Forest vendors `.cue` files into the project's `cue.mod/pkg/forest.sh/{org}/{name}@v<major>/` directory, matching existing behavior. This proves source-backed components without changing srcball's media formats. ### Phase B: CUE adapter Add either a Forest-side or srcball-side CUE module adapter: ```text GET /cue/<module>/@v/list GET /cue/<module>/@v/<version>.info GET /cue/<module>/@v/<version>.mod GET /cue/<module>/@v/<version>.zip ``` or OCI-compatible CUE module endpoints backed by srcball source artifacts. Do this after Phase A works. --- ## Binary component strategy srcball currently distributes source, not per-platform executables. Forest binary/global-tool behavior needs a deliberate strategy. ### Option 1: Build on install ```text srcball source -> Forest cache -> local build -> binary cache ``` Pros: - No binary registry needed. - Works with private Git credentials. - Keeps Git as the source authority. Cons: - Slow. - Requires local toolchains. - Poorer global-tool UX. Good first path for internal components. ### Option 2: GitHub Releases / external assets Declare release assets in component metadata or manifest: ```cue forest: component: { external: { platforms: [{ os: "linux" arch: "amd64" url: "https://github.com/org/repo/releases/download/v0.1.0/tool-linux-amd64" sha256: "..." }] } } ``` Pros: - GitHub remains source and artifact authority. - Good for public/global tools. - Avoids rebuilding on every install. Cons: - Private release asset auth needs design. - Requires asset naming and checksum discipline. Preferred long-term path for global tools. ### Avoid initially: srcball as binary registry If srcball starts storing build outputs, it becomes another Forest registry. Defer until a concrete need proves this is worth the complexity. --- ## Auth and private repositories Local mode is safe: ```text Forest CLI -> local srcball -> user's Git credentials -> user's cache ``` Central shared mode is unsafe unless authorized: ```text User A can access private repo. srcball caches it. User B asks for same package/ref. If cache hit bypasses auth, private source leaks. ``` Before running central srcball for private repositories, require: - request identity, - per-request GitHub/Git forge authorization, - cache hit authorization checks, - cache partitioning or ACL metadata, - public/private source separation, - revocation behavior. Until then, shared srcball should serve only public sources or single-tenant/private deployments. --- ## Catalog and discovery A searchable registry is still useful, but it should be derived, not authoritative. Possible catalog indexer: 1. Scan configured GitHub orgs/repos. 2. Find `forest.cue` + `forest.component.cue`. 3. Parse component metadata, README, methods, tool facet, contracts, tags. 4. Store searchable rows for UI and `forest search`. 5. Do not store source bundles as canonical artifacts. The catalog can replace registry search/detail while Git/srcball remain the source path. --- ## Migration plan ### Milestone 1 — Source-backed local cache hydration Implement Git-source dependency parsing and materialization through srcball. Acceptance: - A component currently consumed via `path:` can be consumed via Git source. - `~/.cache/forest/components/<org>/<name>/<version>/` is populated from srcball. - `ComponentParser` reads the materialized component. - Existing `forest run`/`forest generate` paths operate on the cache. ### Milestone 2 — Git tag semver resolution Implement tag listing and version selection for Git-backed deps. Acceptance: - `version: "0.1"` resolves to highest matching Git tag. - `latest` resolves to the highest semver tag, not mutable `HEAD`. - Lockfile records ref, commit, and digest. - Tag movement is detected. ### Milestone 3 — CUE import support Vendor CUE files from materialized source into `cue.mod/pkg` or add a CUE module adapter. Acceptance: - `import "forest.sh/{org}/{name}@v0"` works for Git-backed components. - `cue export` works in a clean checkout after `forest update`. ### Milestone 4 — Publish becomes validation/tagging For Git-backed components, `forest publish` validates component metadata and release readiness instead of uploading files. Acceptance: - It verifies the Git tag/ref exists. - It verifies `forest.component.version` matches the tag/version. - It optionally updates a derived catalog. - It does not call `UploadFile` for source-only components. ### Milestone 5 — Binary strategy Choose local build or external release assets for binary/global-tool components. Acceptance: - Binary components can be installed/run without Forest registry binary storage for at least one supported path. - Lockfile records binary SHA when binaries are used. ### Milestone 6 — Registry reduction Delete or deprecate source-file upload paths once Git-backed source components are stable. Candidate removals/deprecations: - source `BeginUpload`/`UploadFile`/`CommitUpload` for Git-backed components, - object-store component file bundles, - server-side CUE module generation from uploaded source files. Keep registry/catalog APIs only where they provide metadata/search/auth not covered by Git/srcball. --- ## srcball additions needed 1. **Materialize API** Forest needs a stable way to hydrate a destination directory without depending on srcball internals. ```rust SrcballClient::materialize(&PackageSpec, destination: &Path) ``` or an HTTP endpoint that streams normalized files. 2. **Reference listing** Needed for semver tag resolution. ```text GET /packages/refs?package=<repo>[//subpath] ``` Response should include tags, heads, and commit IDs. 3. **Artifact digest in metadata** `ArtifactMetadata` already includes package, version, source ref, commit, and time. Forest also needs the source artifact digest for lockfile integrity. 4. **CUE adapter later** Add CUE module/proxy endpoints only after Forest-side vendoring proves the model. 5. **Auth model before shared private use** Central srcball must authorize cache hits, not just Git fetches. --- ## Forest changes needed 1. Extend `#ForestDependency` with `source`. 2. Add a `ComponentSourceResolver` abstraction for local path, current registry, and srcball Git sources. 3. Update `forest add` to inspect Git-backed component sources and write canonical `org/name` dependencies. 4. Update `forest update` to resolve Git tags and hydrate the component cache from srcball. 5. Extend `forest.lock` with source locator/ref/commit/digest. 6. Keep current cache layout and runtime paths stable. 7. Add CUE vendoring for Git-backed components. 8. Split `forest publish` into source validation/tagging vs legacy registry upload. --- ## Risks ### High — private cache authorization A central srcball can leak private repository content if cache hits are not authorized per requester. Mitigation: local-only first; central private mode requires identity, GitHub/Git forge authorization, cache ACLs, and revocation behavior. ### High — binary distribution Source fetch does not replace per-platform executable distribution. Mitigation: build locally first for internal components; use GitHub Releases/external manifests for global tools; defer srcball binary storage. ### High — CUE module semantics CUE module OCI artifacts are not the same as srcball source OCI artifacts. Mitigation: Forest-side CUE vendoring first, CUE adapter later. ### Medium — monorepo version tags Monorepos need tag naming conventions and prefix filtering. Mitigation: require explicit `tag_prefix`; validate component version against selected tag. ### Medium — search/catalog regression A Git-only model may lose registry discovery UX. Mitigation: build a derived catalog index over Git repositories. ### Medium — reproducibility Git tags can move. Mitigation: lock commit and artifact digest; fail when a locked tag resolves differently. --- ## Recommended first spike Pick one existing local path component, for example `forest-contrib/build-rust`, and consume it from Git through srcball. Target dependency: ```cue dependencies: { "forest-contrib/build-rust": { version: "0.1.0" source: { type: "git" locator: "github.com/rawpotion/forest//apps/forest/components/forest-contrib/build-rust" ref: "forest-contrib/build-rust/v0.1.0" } } } ``` Spike acceptance: 1. `forest update` fetches the source through srcball. 2. Forest materializes the source into `~/.cache/forest/components/forest-contrib/build-rust/0.1.0`. 3. `forest.component.cue` and `forest.cue` parse from the cache. 4. Existing command/runtime discovery sees the same component surface as a local path dependency. 5. `forest.lock` records source locator, ref, commit, and digest. 6. No Forest registry source upload is involved. --- ## Bottom line The strongest version of this idea is not "rebuild the Forest registry on top of srcball". It is: ```text Forest components are Git packages. srcball is the pull-through source proxy. Forest remains the component runtime and resolver. GitHub/Git remains the publisher, ACL boundary, audit log, and release history. ``` That cuts away duplicated source storage while preserving the valuable parts of Forest: typed CUE contracts, composable components, release hooks, and `forest run`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
kjuulh/gitea-fork
feature/forest-stdlib-bundle
push-sozzwkwzmsmq
main
push-zyouyvtvpwxx
hollow-phase1
v0.1.14
v0.1.13
v0.1.12
v0.1.11
v0.1.10
v0.0.1+alpha.2
v0.0.1+alpha.1
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
rawpotion/forest#3
No description provided.