# Artifact Contract Canonical byte digests use Jason's number representation and uppercase hexadecimal control escapes. Preserve original canonical bytes when a reader's decoded number type loses `1.0`, negative zero or large integers. JavaScript re-encoding alone cannot establish the same digest. The packaged `priv/contracts/canonical-json-v1.json` contains shared byte/hash examples. DocShell's JSON artifacts are a public API. Renderers in other repositories consume them and release independently. This notebook builds a small set of artifacts so you can inspect each file and its fields. The examples write to a disposable temp directory. They do not modify `priv/doc_shell/`. ## What is public API The contract includes: - the artifact tree - the envelope around every JSON payload - the `doc-shell/v2` schema version - the shape of `navigation.json`, `search-index.json`, `content.json`, `modules.json`, `guides.json`, `livebooks.json`, `changelog.json`, `openapi.json`, and `manifest.json` - the in-memory presentation shape accepted from graph-backed hosts Changing one of those shapes is a breaking change unless it is strictly backward-compatible. ## Setup Run this notebook from Livebook's default standalone runtime. The setup cell installs the matching DocShell release from Hex. Release tooling updates this version together with the package and the notebook links. ```elixir Mix.install([ {:doc_shell, "== 2.0.0"} ]) Application.ensure_all_started(:doc_shell) DocShell.schema_version() ``` ## Build a small artifact tree The fixture has one module, one Markdown guide, one Livebook notebook, and the default empty OpenAPI document. ````elixir workspace = Path.join( System.tmp_dir!(), "doc_shell_artifact_contract_#{System.unique_integer([:positive])}" ) File.rm_rf!(workspace) File.mkdir_p!(workspace) workspace = File.cd!(workspace, &File.cwd!/0) guide_dir = Path.join(workspace, "guides") livebook_dir = Path.join(workspace, "notebooks") public_dir = Path.join(workspace, "public") private_dir = Path.join(workspace, "private") Enum.each([guide_dir, livebook_dir, public_dir, private_dir], &File.mkdir_p!/1) File.write!(Path.join(guide_dir, "contract-guide.md"), """ --- id: contract-guide title: Contract Guide audience: developers locale: en --- # Contract Guide The renderer reads this guide through `content.json`. """) File.write!(Path.join(livebook_dir, "operator-runbook.livemd"), """ # Operator Runbook Operational notebooks are indexed as documentation, but DocShell does not run their code. ```elixir :ok ``` """) build_opts = [ modules: [DocShell.Config], guide_bases: [guide_dir], livebook_base: livebook_dir, public_dir: public_dir, private_dir: private_dir ] {:ok, result} = DocShell.Build.run(build_opts) %{ public_dir: public_dir, private_dir: private_dir, presentation_keys: Map.keys(result.presentation) |> Enum.sort() } ```` ## Site-ready collections Pass a `DocShell.Generate.Collection` descriptor when a host will publish this build alongside other documentation. The descriptor binds the artifact tree to an exact source checkout. The caller supplies and verifies the revision and tree digest; DocShell records them without running Git or making network requests. Each document has a unique ID and matching provenance. Several releases from one changelog can share a source path, but each has its own document ID. The synthetic API document reserves the ID `openapi` in collection mode. Loading validates source-index entries and exact provenance coverage before qualifying IDs. Unknown source kinds follow the generic source-index contract and retain their kind and metadata. Other extension artifacts remain available as opaque decoded payloads. `DocShell.Json.Canonical.encode/1` and `digest/1` reject duplicate encoded object keys and return tagged errors. `Collection.digest/1` is a convenience for encodable input and raises on invalid values. Portable canonical byte/digest fixtures ship in `priv/contracts/`. The collection checks also cover content and serialization: - Canonical bytes distinguish integers from floats. Projection compatibility uses the same strict distinction. - Embedded index ASTs must agree with content. The synthetic OpenAPI source has no page AST. - Native JSON hashing avoids a redundant encode/decode pass. Other values use the checked protocol path. - Artifact writes reject invalid protocol output and host encoder exceptions before replacement. Source extension fields must be native JSON in ordinary builds too. The collection integrity section of `docs/specs/DSH.01-documentation-sites.md` defines the required behavior; `docs/plans/documentation-sites.md` records the implementation's acceptance status. A custom projector may not discard or change nonempty source ASTs in collection mode. Such a build must fail before publishing; the library must not restore excluded public content implicitly. ```elixir collection = %{ id: "doc_shell", title: "DocShell", version: "0.3.0", revision: String.duplicate("a", 40), tree_digest: "sha256:" <> String.duplicate("b", 64), artifact_dir: public_dir, source_url: "https://example.invalid/doc_shell", edit_base_url: "https://example.invalid/doc_shell/edit/revision" } {:ok, _result} = File.cd!(workspace, fn -> DocShell.Build.run(Keyword.put(build_opts, :collection, collection)) end) {:ok, collection_payload} = DocShell.Artifact.read(Path.join(public_dir, "collection.json")) Map.take(collection_payload, ["schema_version", "descriptor", "content_digest"]) ``` `collection.json` is an additive public artifact. Its source records and artifact digests let an importer validate the complete generation, and `DocShell.Generate.Collection.load/1` qualifies document IDs as `doc_shell:document_id` without starting the host application. Import uses finite limits: | Limit | Default | | --- | --- | | One file | 32 MiB | | Total bytes per corpus, including the manifest | 2 GiB | | Artifacts per corpus | 256 | | Sources per corpus | 100,000 | | JSON nesting depth | 64 | | Collections per `load_many/2` call | 256 | Override limits with positive keywords, for example `Collection.load(descriptor, max_file_bytes: 8 * 1_024 * 1_024)`. The directory must remain caller-owned and stable during import: - Symlink roots (even with `/` or `/.` suffixes), untrusted symlink components, and `..` traversal are rejected. - Ancestors of the existing working/system temporary directories are trusted platform aliases. - Duplicate JSON keys fail rather than being overwritten. ## The artifact tree DocShell writes public renderer artifacts under `:public_dir` and a separate manifest under `:private_dir`. ```elixir Path.wildcard(Path.join(workspace, "**/*.json")) |> Enum.map(&Path.relative_to(&1, workspace)) |> Enum.sort() ``` The three files most renderers read directly are: - `navigation.json` - `search-index.json` - `content.json` The source indexes and `openapi.json` are still public artifacts. They are useful for ingestion, search, coverage reporting, and API reference tooling. ## The envelope Every artifact file has the same outer object. `DocShell.Artifact.read/1` returns only `"data"`; use `read_envelope/1` when the envelope itself matters. ```elixir {:ok, navigation_envelope} = DocShell.Artifact.read_envelope(Path.join(public_dir, "navigation.json")) Map.take(navigation_envelope, ["schema_version", "generated_at", "generation_id"]) ``` The fields are: | Field | Meaning | | --- | --- | | `schema_version` | The public contract version, currently `doc-shell/v2` | | `generated_at` | ISO 8601 UTC timestamp for the build | | `generation_id` | Opaque id shared by every artifact in one build | | `data` | The payload for that artifact | `generation_id` is only for equality checks. Do not sort by it, decode meaning from it, or reuse it between builds. The file itself is compact canonical JSON: keys sorted by UTF-8 bytes and no insignificant whitespace. Build with `pretty: true` to read it by eye; the decoded value is the same. `doc-shell/v2` replaced `doc-shell/v1`. Version 2 moved module member documentation out of `navigation.json`, added member `doc_state`, limited member metadata to an allow-list by default, and switched files to compact canonical JSON. A v2 reader rejects v1 files. ## One generation per tree Every public artifact and the public manifest from one build share a generation id. The runtime cache uses that to reject mixed snapshots. ```elixir Path.wildcard(Path.join(public_dir, "*.json")) |> Map.new(fn path -> {:ok, envelope} = DocShell.Artifact.read_envelope(path) {Path.basename(path), envelope["generation_id"]} end) ``` The value should be the same for every file in that map. ## manifest.json The manifest describes exactly the artifacts beside it. It is written last and marks a committed generation. ```elixir {:ok, public_manifest} = DocShell.Artifact.read(Path.join(public_dir, "manifest.json")) public_manifest["artifacts"] |> Enum.sort() ``` The private directory has its own manifest. Today the default build writes no private artifacts, so the list is empty. ```elixir DocShell.Artifact.read(Path.join(private_dir, "manifest.json")) ``` Each manifest describes its own directory; public and private directories need separate manifests. ## navigation.json `navigation.json` is a list of navigation items. On disk, structs have encoded to maps with string keys. ```elixir {:ok, navigation} = DocShell.Artifact.read(Path.join(public_dir, "navigation.json")) navigation |> Enum.map(&Map.take(&1, ["id", "title", "path", "kind", "children", "meta"])) ``` A module's item lists its documented members as references, never their documentation. Members marked `@doc false` are left out; `anchor` follows ExDoc's fragment convention so a renderer can link into the module page. ```elixir navigation |> Enum.filter(&(&1["kind"] == "module")) |> Enum.map(&{&1["id"], Enum.take(&1["meta"]["members"], 3)}) ``` The default `DocShell.Presentation.StaticGenerator` sorts entries by kind then title and leaves `children` empty. The host supplies any hierarchy, such as grouping modules by namespace or guides by product area. ## search-index.json `search-index.json` has document identity, route path, flattened text, optional tokens, and scoping fields. ```elixir {:ok, search} = DocShell.Artifact.read(Path.join(public_dir, "search-index.json")) search |> Enum.find(&(&1["id"] == "contract-guide")) |> Map.take(["id", "title", "path", "kind", "audience", "locale", "content", "tokens"]) ``` `audience` and `locale` are present as `null` when unset. For guides, they come from frontmatter. For modules and Livebooks they are usually `null`. Tokens are present but empty by default because they duplicate data already in `content`. Enable them only when a host search backend wants a pre-split field. ```elixir {:ok, token_result} = DocShell.Build.run(Keyword.put(build_opts, :search_tokens, true)) token_result.presentation.search |> Enum.find(&(&1.id == "contract-guide")) |> Map.take([:id, :tokens]) ``` ## content.json `content.json` maps each entry id to its parsed Markdown AST. This is where page bodies live. ```elixir {:ok, content} = DocShell.Artifact.read(Path.join(public_dir, "content.json")) Map.keys(content) |> Enum.sort() ``` A content node is recursive. Text nodes are plain strings. Element nodes always carry the same four keys: `tag`, `attrs`, `content`, and `meta`. ```elixir content["contract-guide"] |> List.first() ``` That uniform shape is why a renderer can use one walker for module docs, guides, and notebooks. ## modules.json, guides.json, livebooks.json, and changelog.json The per-source indexes carry identity and metadata for every extracted entry. They intentionally do not carry `"ast"`; the body already lives once in `content.json`. ```elixir {:ok, module_index} = DocShell.Artifact.read(Path.join(public_dir, "modules.json")) {:ok, guide_index} = DocShell.Artifact.read(Path.join(public_dir, "guides.json")) {:ok, livebook_index} = DocShell.Artifact.read(Path.join(public_dir, "livebooks.json")) {:ok, changelog_index} = DocShell.Artifact.read(Path.join(public_dir, "changelog.json")) %{ modules: Enum.map(module_index, &Map.take(&1, ["id", "title", "kind", "meta"])), guides: Enum.map(guide_index, &Map.take(&1, ["id", "title", "kind", "meta"])), livebooks: Enum.map(livebook_index, &Map.take(&1, ["id", "title", "kind", "meta"])), changelog: Enum.map(changelog_index, &Map.take(&1, ["id", "title", "kind", "meta"])) } ``` The source indexes are unfiltered. If `skip_empty` removes an undocumented module from presentation, the module still appears in `modules.json`, which makes the file useful as a coverage report. `modules.json` is the only artifact that carries member documentation. Each member has `kind`, `name`, `arity`, `signatures`, `doc`, `doc_state` (`"present"`, `"hidden"` or `"none"`) and `metadata`. Metadata is limited to the keys in `DocShell.Generate.ExDoc.default_member_metadata/0` unless the build sets `member_metadata`. ```elixir module_index |> Enum.flat_map(& &1["meta"]["members"]) |> Enum.take(2) ``` The `doc-shell/v2` source catalogue is additive. Producers may add a new per-source index to `manifest.json`, and entry `kind` values are open strings. Generic renderers ignore unknown source files and kinds; selective consumers may continue reading only their supported artifacts. Removing a known artifact or removing, renaming, or retyping a known field requires a schema-version change. ```elixir module_index |> List.first() |> Map.has_key?("ast") ``` ## openapi.json `openapi.json` contains the OpenAPI document returned by the configured adapter. With no adapter, DocShell writes a valid empty OpenAPI 3.1 document. ```elixir {:ok, openapi} = DocShell.Artifact.read(Path.join(public_dir, "openapi.json")) Map.take(openapi, ["openapi", "info", "paths"]) ``` Because the artifact is enveloped, standard OpenAPI tooling should not be pointed at `priv/doc_shell/public/openapi.json`. Set `:openapi_spec_path` when a tool needs the bare OpenAPI document. ## In-memory presentation vs. disk JSON Before encoding, presentation data uses structs and atom keys. ```elixir result.presentation.navigation |> List.first() ``` After reading from disk, the same artifact is JSON data with string keys. ```elixir navigation |> List.first() ``` Application code uses typed structs; artifacts contain plain JSON. ## Graph-backed presentation Graph-backed hosts can provide their own presentation data through `DocShell.Presentation.GraphProjector`. The required fields are schema version, navigation, search, and content. `backlinks` are optional. ```elixir presentation = %{ schema_version: DocShell.schema_version(), navigation: [], search: [], content: %{}, backlinks: %{ "contract-guide" => [ %DocShell.Presentation.Backlink{ id: "operator-runbook", title: "Operator Runbook", path: "/docs/livebook/operator-runbook" } ] } } DocShell.Presentation.GraphProjector.validate(presentation) ``` The validator catches malformed projector output before a renderer receives it. ## Contract change checklist Before changing any `doc-shell/v2` shape, answer these questions: | Question | Why it matters | | --- | --- | | Does a renderer already read this field? | Removing or retyping it is breaking | | Can the change be additive and optional? | Optional additions are usually safe | | Does the value stay JSON-native? | Artifacts must not leak Elixir-only terms | | Does the schema version need to change? | Breaking changes require coordination | | Are README, usage rules, and tutorials updated? | The contract docs are part of the API | If a renderer could observe the change, treat it as public API work. ## Rendering untrusted content The AST preserves raw HTML and URL schemes. Renderers must allow-list tags and attributes, reject unsafe URL schemes, and escape text for their output context. Parsing Markdown is not sanitization. ## Recursive content validation Host projectors and changelog sources must provide complete recursive AST nodes and JSON metadata with string keys. Invalid nested content fails validation before output is written. `DocShell.Ast.valid?/1` checks node lists. ## Concurrent artifact writers Individual artifact writes use exclusively created random temporary files in the destination directory, so independent BEAM instances cannot share a temporary file. A rename publishes each complete file. ## Search text Search content preserves adjacent inline text, including words split by formatting. Block elements and line breaks add separators; image alt text is searchable. Token generation uses this same text. ## Document paths Each document path is calculated once and reused by navigation and search. Default paths percent-encode kind and ID as individual URL segments. Use a custom `path_builder` when IDs intentionally represent a path hierarchy. ## Legacy envelope compatibility `Artifact.read/1` and `read_envelope/1` accept current-schema envelopes without `generation_id`. A present ID must be a nonempty string. Previous schema versions, including `doc-shell/v1`, are rejected and must be rebuilt. Runtime caches require an ID on every artifact and manifest to verify one generation. ## Presentation structs and backlinks Presentation structs serialize through `Jason.Encoder` into the documented string-keyed objects. `DocShell.Json.stringify/1` instead converts arbitrary structs into text and must not be used to serialize presentation structs. Projector backlinks are validated and available in the in-memory build result; the build does not write a backlinks artifact. ## Bound nesting before decoding DocShell 2.0.0 exposes `DocShell.Json.Depth.check/2` for integrations that need a positive nesting bound on original JSON bytes. The scanner skips quoted braces and escapes without allocating the decoded tree. Passing the check does not establish valid syntax or unique keys; `DocShell.Json.decode/1` still owns those checks. Collection imports keep their existing `collection_limit` errors. This utility is included in DocShell 2.0.0.