Spec Coverage Matrix
Which requirements have automated audits — and which remain uncovered. Generated by anc generate coverage-matrix from the agentnative principle registry.
Summary
59 total requirements
56 covered
3 uncovered
| Level | Total | Covered | Uncovered |
|---|---|---|---|
| MUST | 28 | 28 | 0 |
| SHOULD | 21 | 18 | 3 |
| MAY | 10 | 10 | 0 |
P1: Non-Interactive by Default
| Level | Requirement | Applicability | Verified by |
|---|---|---|---|
| MUST | Every flag settable via environment variable (falsey-value parser for booleans). | Universal | p1-env-hints p1-env-flags-source |
| MUST | When stdin is not a TTY or `--no-interactive` is set, every blocking-input surface (prompt libraries, read-line, TUI init) resolves from defaults/stdin or exits with an actionable error. | Universal | p1-non-interactive p1-flag-existence p1-non-interactive-source |
| MUST | Headless authentication path (`--no-browser` / OAuth Device Authorization Grant). | CLI authenticates against a remote service | p1-headless-auth |
| MUST | Sensitive inputs are readable via stdin or a `--*-file` flag; flag-value and env-var inputs MAY exist for convenience but MUST NOT be the only path. | CLI accepts secret material (tokens, passwords, keys) as input | p1-secret-non-leaky-path |
| SHOULD | Auto-detect non-interactive context via TTY detection; suppress prompts when stderr is not a terminal. | Universal | p1-tty-detection-source |
| SHOULD | Document default values for prompted inputs in `--help` output. | Universal | p1-defaults-in-help |
| MAY | Rich interactive experiences (spinners, progress bars, menus) when TTY is detected and `--no-interactive` is not set. | Universal | p1-rich-tui |
P2: Structured, Parseable Output
| Level | Requirement | Applicability | Verified by |
|---|---|---|---|
| MUST | `--output` flag selects format with `json` and `jsonl` as canonical machine-readable values; `text` is the default human-facing form. | Universal | p2-json-output p2-structured-output |
| MUST | Data goes to stdout; diagnostics/progress/warnings go to stderr, never interleaved. | Universal | p2-output-module |
| MUST | Exit codes are structured and documented (0 success, 1 general, 2 usage, 77 auth, 78 config). | Universal | p2-structured-exit-codes |
| MUST | When `--output json` is active, errors are emitted as JSON (to stderr) with at least `error`, `kind`, and `message` fields. | Universal | p2-json-errors |
| MUST | CLIs that emit structured output expose the output schema via a `schema` subcommand or `--schema` flag: runtime-discoverable, with a documented format identifier. | Conditional | p2-schema-print |
| SHOULD | JSON output uses a consistent envelope (a top-level object with predictable keys) across every command. | Universal | p2-consistent-envelope |
| SHOULD | Output schemas are also exported to a stable file path (e.g., `schema/<command>.json`) so CI/static-analysis consumers pin without invoking the tool. | Conditional | p2-schema-file |
| SHOULD | `--json` and `--jsonl` are accepted as aliases for `--output json` and `--output jsonl`; the short forms work alongside the canonical enum. | Universal | p2-json-aliases |
| MAY | Additional output formats (CSV, TSV, YAML) beyond the core three. | Universal | p2-more-formats |
| MAY | `--raw` flag for unformatted output suitable for piping to other tools. | Universal | p2-raw-flag |
P3: Progressive Help Discovery
| Level | Requirement | Applicability | Verified by |
|---|---|---|---|
| MUST | Every subcommand ships at least one concrete invocation example (`after_help` in clap). | CLI uses subcommands | p3-subcommand-examples |
| MUST | The top-level command ships 2–3 examples covering the primary use cases. | Universal | p3-help |
| MUST | Top-level `--version` prints a non-empty version line and exits 0. | Universal | p3-version |
| SHOULD | A short version alias (`-V`, `-v`, or `-version`) accompanies `--version` for fast version probes. | Universal | p3-version |
| SHOULD | Examples show human and agent invocations side by side (text then `--output json` equivalent). | Universal | p3-paired-examples |
| SHOULD | Short `about` for command-list summaries; `long_about` reserved for detailed descriptions visible with `--help`. | Universal | p3-about-long-about |
| MAY | Dedicated `examples` subcommand or `--examples` flag for curated usage patterns. | Universal | p3-examples-subcommand |
P4: Fail-Fast, Actionable Errors
| Level | Requirement | Applicability | Verified by |
|---|---|---|---|
| MUST | Parse arguments with `try_parse()` instead of `parse()` so `--output json` can emit JSON parse errors. | Universal | p4-try-parse |
| MUST | Error types map to distinct exit codes (0, 1, 2, 77, 78). | Universal | p4-bad-args p4-exit-codes |
| MUST | Every error message names the failure, the cause, and a concrete remediation (a command or a value, not a hint to consult docs). | Universal | p4-actionable-errors |
| SHOULD | Error types use a structured enum (via `thiserror` in Rust) with variant-to-kind mapping for JSON serialization. | Universal | p4-error-module p4-error-types |
| SHOULD | Config and auth validation happen before any network call, failing at the earliest possible point. | CLI makes network calls | UNCOVERED |
| SHOULD | Error output respects `--output json`: JSON-formatted errors go to stderr when JSON output is selected. | Universal | p4-json-error-output |
| SHOULD | When rejecting input against an enum or fixed-allowed-values set, the error message includes the valid set. | CLI rejects input against a closed set | p4-enumerate-valid-set p4-enumerate-valid-set |
P5: Safe Retries & Mutation Boundaries
| Level | Requirement | Applicability | Verified by |
|---|---|---|---|
| MUST | Destructive operations (delete, overwrite, bulk modify) require an explicit `--force` or `--yes` flag. | CLI has destructive operations | p5-force-yes |
| MUST | The distinction between read and write commands is clear from the command name and help text alone. | CLI has both read and write operations | p5-read-write-distinction |
| MUST | A `--dry-run` flag is present on every write command; dry-run output respects `--output json`. | CLI has write operations | p5-dry-run |
| SHOULD | Write operations are idempotent where the domain allows it: running the same command twice produces the same result. | CLI has write operations | UNCOVERED |
P6: Composable, Predictable Command Structure
| Level | Requirement | Applicability | Verified by |
|---|---|---|---|
| MUST | SIGPIPE is handled so piping to `head`/`tail` does not crash the process (Rust example below; Python/Go/Node have language-specific equivalents). | Universal | p6-sigpipe |
| MUST | Long-running operations handle SIGTERM gracefully: flush or roll back partial writes, release locks, exit non-zero within a bounded window. Next invocation succeeds without manual cleanup. | CLI has long-running operations | p6-sigterm p6-sigterm |
| MUST | TTY detection plus support for `NO_COLOR` and `TERM=dumb`: color codes suppressed when stdout/stderr is not a terminal. | Universal | p6-no-color-behavioral p6-no-color p6-no-color |
| MUST | Shell completions available via a `completions` subcommand (Tier 1 meta-command, needs no config/auth/network). | Universal | p6-completions |
| MUST | Network CLIs ship a `--timeout` flag with a sensible default (e.g., 30 seconds). | CLI makes network calls | p6-timeout |
| MUST | If the CLI uses a pager (`less`, `more`, `$PAGER`), it supports `--no-pager` or respects `PAGER=""`. | CLI invokes a pager for output | p6-no-pager-behavioral p6-no-pager |
| MUST | Agentic flags (`--output`, `--quiet`, `--no-interactive`, `--timeout`) propagate to every subcommand (e.g., `global = true` in clap). | CLI uses subcommands | p6-global-flags |
| SHOULD | Commands that accept input read from stdin when no file argument is provided. | CLI has commands that accept input data | p6-stdin-input |
| SHOULD | Subcommand naming follows a consistent `noun verb` or `verb noun` convention throughout the tool. | CLI uses subcommands | p6-consistent-naming |
| SHOULD | Three-tier dependency gating: Tier 1 (meta) needs nothing, Tier 2 (local) needs config, Tier 3 (network) needs config + auth. | Universal | UNCOVERED |
| SHOULD | Operations are modeled as subcommands, not flags (`tool search "q"`, not `tool --search "q"`). | CLI performs multiple distinct operations | p6-subcommand-operations |
| MAY | `--color auto|always|never` flag for explicit color control beyond TTY auto-detection. | Universal | p6-color-flag |
| MAY | Subcommand verbs MAY follow community-standard names (`get`/`list`/`create`/`update`/`delete`); flag spellings MAY follow widely-used canonical forms (`--force`, `--yes`, `--limit`, `--quiet`, `--verbose`). | CLI uses subcommands | p6-standard-names |
P7: Bounded, High-Signal Responses
| Level | Requirement | Applicability | Verified by |
|---|---|---|---|
| MUST | A `--quiet` flag suppresses non-essential output; only requested data and errors appear. | Universal | p7-quiet |
| MUST | List operations clamp to a documented default maximum; when truncated, indicate it (`"truncated": true` in JSON, stderr note in text). | CLI has list-style commands | p7-output-clamping |
| SHOULD | A `--verbose` flag (or `-v` / `-vv`) escalates diagnostic detail when agents need to debug failures. | Universal | p7-verbose |
| SHOULD | A `--limit` or `--max-results` flag lets callers request exactly the number of items they want. | CLI has list-style commands | p7-limit |
| SHOULD | A `--timeout` flag bounds execution time so agents are not blocked indefinitely. | Universal | p7-timeout-behavioral |
| MAY | Cursor-based pagination flags (`--after`, `--before`) for efficient traversal of large result sets. | CLI returns paginated results | p7-cursor-pagination |
| MAY | Automatic verbosity reduction in non-TTY contexts (same behavior `--quiet` explicitly requests). | Universal | p7-auto-verbosity |
P8: Discoverable Through Agent Skill Bundles
| Level | Requirement | Applicability | Verified by |
|---|---|---|---|
| MUST | When a skill bundle exists, the CLI provides an install path (`tool skill install [<host>]`) that registers the bundle with installed agent runtimes. | Conditional | p8-bundle-install |
| SHOULD | CLIs ship a top-level agent-discoverable markdown bundle (`AGENTS.md`, `SKILL.md`, or equivalent) with YAML frontmatter naming the tool and capability summary. | Universal | p6-agents-md p8-bundle-exists |
| MAY | An `--all` mode auto-detects installed runtimes (Claude Code, Cursor, Codex, OpenCode, etc.) and installs across all. | Conditional | p8-install-all |
| MAY | An update/upgrade subcommand (`tool skill update`) pulls the latest bundle version. | Conditional | p8-bundle-update |
About this matrix
This matrix is generated from the agentnative principle registry — a flat list of every MUST, SHOULD, and MAY across the eight principles, each mapped to the audit IDs that verify it. Requirements with no verifiers are marked UNCOVERED: the spec requires them but anc does not yet test for them.
"Covered" means an audit exists that targets this requirement. "Verified" (on scorecard pages) means the audit actually ran for a specific tool. A requirement can be covered in the matrix but unverified for a particular tool if the audit was skipped (e.g., source audits on a binary-only audit).
