mirror of
https://github.com/mastra-ai/mastra.git
synced 2026-11-17 08:44:46 +00:00
## Summary Adds the **builder-smoke-test** skill under `.claude/skills/` — an internal, agent-readable runbook for end-to-end smoke testing of the Agent Builder feature stack. ## Scope Tooling only. Zero product code, zero runtime dependency on `@mastra/*` packages. - `SKILL.md` — execution flow + 15-section coverage matrix (auth-off / auth-on admin / auth-on member) - `references/` — per-domain expectation tables and curl recipes (auth, permissions, RBAC, stored agents/skills/workspaces, channels, registry, model policy, infrastructure, reconciliation, UI parity, stars) - `scripts/`: - `scaffold.sh` scaffolds a hermetic Mastra project linked back to the current worktree via pnpm overrides - `wait-for-server.sh` polls health while `mastra dev` boots - `preflight.sh` checks env/binaries before a run - `seed-multi-user.sh` writes multi-author rows directly via the `sqlite3` CLI for non-owner / Library Copy flows - `assets/template/` — minimal project template the scaffold renders ## Why a separate PR The skill is internal tooling — it describes the builder API surface and probes it via curl + Studio UI. It does not import or call any product code, so it can ship orthogonally from the agent-builder / RBAC / stars stack itself. ## Compatibility with main The skill targets endpoints/behaviors that do not exist on `main` yet (e.g. `/stored/*`, `/editor/builder/*`, channels, role impersonation, the cookie-leak smoke route in the scaffold template). Running the skill against `main` today will fail — that's expected. It's a planned runbook for the stack landing across the queued split PRs (#16576, #16578, and follow-ups). ## Test plan - N/A — no executable product code in this PR. - The skill itself has been exercised across three full smoke-test runs (auth-off, auth-on admin, auth-on member) on the rollup feature branch and uncovered ~10 product bugs that have already been fixed there. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## ELI5 This adds an internal QA runbook and tooling that scaffolds a small local Mastra project and runs end-to-end smoke tests for the Agent Builder feature. It's documentation, templates, and scripts only (no production runtime code shipped) and is intended to be run against the feature branch (it will fail against main because some endpoints/features are not present). --- ## Overview Adds a hermetic, agent-readable smoke-test skill at .claude/skills/builder-smoke-test/ to validate the Agent Builder feature stack. Includes a detailed runbook (SKILL.md), per-domain reference checklists, a minimal scaffolded project template, and helper scripts to create, validate, and run smoke tests. Tooling-only: no exported/public code changes to the repository and no runtime dependency on @mastra/* in main. The runbook was exercised on the feature rollup branch in three configurations (auth-off, auth-on admin, auth-on member) and helped surface ~10 product fixes applied on that branch. --- ## Key Additions Documentation - SKILL.md: end-to-end workflow, 15-section coverage matrix (auth-off / auth-on admin / auth-on member), execution flow rules, CLI surface (--test, --scope, --auth, --role, --clean, --skip-browser, --dir, --reuse, --openai-key, WorkOS flags), partial-run semantics, preflight/error expectations, cookie-extraction gating (SMOKE_TEST_COOKIE_LEAK), seeding guidance, and reporting template. Per-domain reference docs (curl/jq + UI checks) - Added: agents.md, auth.md, channels.md, defaults.md, infrastructure.md, model-policy.md, permissions.md, picker-allowlist.md, reconciliation.md, registry.md, setup.md, skills.md, stars.md, ui.md, workspace.md — each with step-by-step commands, expected schemas/behaviors, and checklists covering stored agents/skills/workspaces, RBAC, model policy, registry/library-copy, channels, builder defaults, reconciliation, UI parity, and stars. Scaffolded project template (assets/template/) - package.json: ES module template with mastra scripts and pnpm.overrides linking local worktree packages. - src/mastra/index.ts: builder/editor config (LibSQL store, workspace, builder flags, model/tool/agent allowlists), observability, Stagehand browser hooks, Slack config, and a gated /smoke-test/cookie endpoint when SMOKE_TEST_COOKIE_LEAK=1. - src/mastra/auth.ts: WorkOS auth + RBAC provider conditional initialization (AUTH_PROVIDER-driven). - src/mastra/agents/index.ts: example weatherAgent using openai('gpt-4o-mini'). - src/mastra/tools/index.ts: weatherInfo tool with zod validation and static response. - src/mastra/workflows/index.ts: greetWorkflow example. - tsconfig.json: strict bundler configuration. Helper scripts (scripts/) - scaffold.sh: create/reuse a hermetic builder-smoke-test project, substitute __WORKTREE_ROOT__ with the absolute worktree path, write .env (including deterministic WORKOS_COOKIE_PASSWORD when applicable), set AUTH_MODE based on provided WorkOS creds, and run pnpm install unless --reuse. Well-defined exit codes. - preflight.sh: validate project dir, node_modules/@mastra/core presence, required .env keys (OPENAI_API_KEY), and expected auth mode (auto-extends forwarded scaffold args from WORKOS_* when needed); prints PROJECT_DIR/AUTH_MODE and aggregates failures. - wait-for-server.sh: poll http://localhost:${PORT}/api/agents for HTTP 200 with configurable budget/port and alternate-port checks (4112–4114); clear exit codes for success/timeout/alternate-port/invalid args. - seed-multi-user.sh: idempotently seed two published skills owned by a non-primary user directly into the scaffold’s libsql DB via sqlite3 (disables foreign keys during insert) and prints seeded IDs on success. Other edits - Minor doc/snippet fixes and clarifications (cookie header usage for authenticated GETs, infra gate annotation correction, UI route clarifications, create-skill payload requirements, avatar upload endpoint, strengthened preflight parsing and error codes, runnable snippet improvements). --- ## Compatibility & Test Plan Targets endpoints/behaviors not yet present on main (stored endpoints, editor/builder routes, channels, role-impersonation UI hooks, cookie-leak debug route). Running against main is expected to fail; this skill is a runbook to be used as the feature code lands across related PRs (#16576, #16578, and follow-ups). No production code changes in this repo. The runbook was used on the feature rollup branch across three full smoke runs and surfaced ~10 product bugs that were fixed there. --- ## Design Notes / Safety - Hermetic scaffold links to the current worktree via pnpm.overrides for local testing. - WorkOS auth is opt-in via environment/flags; preflight enforces expected auth mode and required WorkOS keys. - Non-owner visibility tests require seeded DB rows (seed-multi-user.sh). - /smoke-test/cookie route is gated by SMOKE_TEST_COOKIE_LEAK and intended only for scaffolded debug cookie extraction. - Template rendering substitutes __WORKTREE_ROOT__ with an absolute path during scaffold. <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/mastra-ai/mastra/pull/16579) <!-- review_stack_entry_end --> <!-- review_stack_entry_start --> [](https://app.coderabbit.ai/change-stack/mastra-ai/mastra/pull/16579) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Mastra Code (anthropic/claude-opus-4-7) <noreply@mastra.ai> |
History
|
|
|---|---|---|
| .. | ||
| seed-fga.mjs | ||
| seed-fga.ts | ||