0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
mastra/examples/agent/scripts
NikAiyer f0f522400d
chore(skills): add builder-smoke-test skill for Agent Builder (#16579)
## Summary

Adds the **builder-smoke-test** skill under `.claude/skills/` — an
internal, agent-readable runbook for end-to-end smoke testing of the
Agent Builder feature stack.

## Scope

Tooling only. Zero product code, zero runtime dependency on `@mastra/*`
packages.

- `SKILL.md` — execution flow + 15-section coverage matrix (auth-off /
auth-on admin / auth-on member)
- `references/` — per-domain expectation tables and curl recipes (auth,
permissions, RBAC, stored agents/skills/workspaces, channels, registry,
model policy, infrastructure, reconciliation, UI parity, stars)
- `scripts/`:
- `scaffold.sh` scaffolds a hermetic Mastra project linked back to the
current worktree via pnpm overrides
  - `wait-for-server.sh` polls health while `mastra dev` boots
  - `preflight.sh` checks env/binaries before a run
- `seed-multi-user.sh` writes multi-author rows directly via the
`sqlite3` CLI for non-owner / Library Copy flows
- `assets/template/` — minimal project template the scaffold renders

## Why a separate PR

The skill is internal tooling — it describes the builder API surface and
probes it via curl + Studio UI. It does not import or call any product
code, so it can ship orthogonally from the agent-builder / RBAC / stars
stack itself.

## Compatibility with main

The skill targets endpoints/behaviors that do not exist on `main` yet
(e.g. `/stored/*`, `/editor/builder/*`, channels, role impersonation,
the cookie-leak smoke route in the scaffold template). Running the skill
against `main` today will fail — that's expected. It's a planned runbook
for the stack landing across the queued split PRs (#16576, #16578, and
follow-ups).

## Test plan

- N/A — no executable product code in this PR.
- The skill itself has been exercised across three full smoke-test runs
(auth-off, auth-on admin, auth-on member) on the rollup feature branch
and uncovered ~10 product bugs that have already been fixed there.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5

This adds an internal QA runbook and tooling that scaffolds a small
local Mastra project and runs end-to-end smoke tests for the Agent
Builder feature. It's documentation, templates, and scripts only (no
production runtime code shipped) and is intended to be run against the
feature branch (it will fail against main because some
endpoints/features are not present).

---

## Overview

Adds a hermetic, agent-readable smoke-test skill at
.claude/skills/builder-smoke-test/ to validate the Agent Builder feature
stack. Includes a detailed runbook (SKILL.md), per-domain reference
checklists, a minimal scaffolded project template, and helper scripts to
create, validate, and run smoke tests. Tooling-only: no exported/public
code changes to the repository and no runtime dependency on @mastra/* in
main. The runbook was exercised on the feature rollup branch in three
configurations (auth-off, auth-on admin, auth-on member) and helped
surface ~10 product fixes applied on that branch.

---

## Key Additions

Documentation
- SKILL.md: end-to-end workflow, 15-section coverage matrix (auth-off /
auth-on admin / auth-on member), execution flow rules, CLI surface
(--test, --scope, --auth, --role, --clean, --skip-browser, --dir,
--reuse, --openai-key, WorkOS flags), partial-run semantics,
preflight/error expectations, cookie-extraction gating
(SMOKE_TEST_COOKIE_LEAK), seeding guidance, and reporting template.

Per-domain reference docs (curl/jq + UI checks)
- Added: agents.md, auth.md, channels.md, defaults.md,
infrastructure.md, model-policy.md, permissions.md, picker-allowlist.md,
reconciliation.md, registry.md, setup.md, skills.md, stars.md, ui.md,
workspace.md — each with step-by-step commands, expected
schemas/behaviors, and checklists covering stored
agents/skills/workspaces, RBAC, model policy, registry/library-copy,
channels, builder defaults, reconciliation, UI parity, and stars.

Scaffolded project template (assets/template/)
- package.json: ES module template with mastra scripts and
pnpm.overrides linking local worktree packages.
- src/mastra/index.ts: builder/editor config (LibSQL store, workspace,
builder flags, model/tool/agent allowlists), observability, Stagehand
browser hooks, Slack config, and a gated /smoke-test/cookie endpoint
when SMOKE_TEST_COOKIE_LEAK=1.
- src/mastra/auth.ts: WorkOS auth + RBAC provider conditional
initialization (AUTH_PROVIDER-driven).
- src/mastra/agents/index.ts: example weatherAgent using
openai('gpt-4o-mini').
- src/mastra/tools/index.ts: weatherInfo tool with zod validation and
static response.
- src/mastra/workflows/index.ts: greetWorkflow example.
- tsconfig.json: strict bundler configuration.

Helper scripts (scripts/)
- scaffold.sh: create/reuse a hermetic builder-smoke-test project,
substitute __WORKTREE_ROOT__ with the absolute worktree path, write .env
(including deterministic WORKOS_COOKIE_PASSWORD when applicable), set
AUTH_MODE based on provided WorkOS creds, and run pnpm install unless
--reuse. Well-defined exit codes.
- preflight.sh: validate project dir, node_modules/@mastra/core
presence, required .env keys (OPENAI_API_KEY), and expected auth mode
(auto-extends forwarded scaffold args from WORKOS_* when needed); prints
PROJECT_DIR/AUTH_MODE and aggregates failures.
- wait-for-server.sh: poll http://localhost:${PORT}/api/agents for HTTP
200 with configurable budget/port and alternate-port checks (4112–4114);
clear exit codes for success/timeout/alternate-port/invalid args.
- seed-multi-user.sh: idempotently seed two published skills owned by a
non-primary user directly into the scaffold’s libsql DB via sqlite3
(disables foreign keys during insert) and prints seeded IDs on success.

Other edits
- Minor doc/snippet fixes and clarifications (cookie header usage for
authenticated GETs, infra gate annotation correction, UI route
clarifications, create-skill payload requirements, avatar upload
endpoint, strengthened preflight parsing and error codes, runnable
snippet improvements).

---

## Compatibility & Test Plan

Targets endpoints/behaviors not yet present on main (stored endpoints,
editor/builder routes, channels, role-impersonation UI hooks,
cookie-leak debug route). Running against main is expected to fail; this
skill is a runbook to be used as the feature code lands across related
PRs (#16576, #16578, and follow-ups). No production code changes in this
repo. The runbook was used on the feature rollup branch across three
full smoke runs and surfaced ~10 product bugs that were fixed there.

---

## Design Notes / Safety

- Hermetic scaffold links to the current worktree via pnpm.overrides for
local testing.
- WorkOS auth is opt-in via environment/flags; preflight enforces
expected auth mode and required WorkOS keys.
- Non-owner visibility tests require seeded DB rows
(seed-multi-user.sh).
- /smoke-test/cookie route is gated by SMOKE_TEST_COOKIE_LEAK and
intended only for scaffolded debug cookie extraction.
- Template rendering substitutes __WORKTREE_ROOT__ with an absolute path
during scaffold.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/mastra-ai/mastra/pull/16579)

<!-- review_stack_entry_end -->

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/mastra-ai/mastra/pull/16579)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Mastra Code (anthropic/claude-opus-4-7) <noreply@mastra.ai>
History 2026-05-14 12:21:02 -05:00
..
seed-fga.mjs chore(skills): add builder-smoke-test skill for Agent Builder (#16579) 2026-05-14 12:21:02 -05:00
seed-fga.ts chore(skills): add builder-smoke-test skill for Agent Builder (#16579) 2026-05-14 12:21:02 -05:00