0
Fork 0
mirror of https://github.com/mastra-ai/mastra.git synced 2026-11-17 08:44:46 +00:00
Commit graph

2 commits

Author SHA1 Message Date
NikAiyer
f0f522400d
chore(skills): add builder-smoke-test skill for Agent Builder (#16579)
## Summary

Adds the **builder-smoke-test** skill under `.claude/skills/` — an
internal, agent-readable runbook for end-to-end smoke testing of the
Agent Builder feature stack.

## Scope

Tooling only. Zero product code, zero runtime dependency on `@mastra/*`
packages.

- `SKILL.md` — execution flow + 15-section coverage matrix (auth-off /
auth-on admin / auth-on member)
- `references/` — per-domain expectation tables and curl recipes (auth,
permissions, RBAC, stored agents/skills/workspaces, channels, registry,
model policy, infrastructure, reconciliation, UI parity, stars)
- `scripts/`:
- `scaffold.sh` scaffolds a hermetic Mastra project linked back to the
current worktree via pnpm overrides
  - `wait-for-server.sh` polls health while `mastra dev` boots
  - `preflight.sh` checks env/binaries before a run
- `seed-multi-user.sh` writes multi-author rows directly via the
`sqlite3` CLI for non-owner / Library Copy flows
- `assets/template/` — minimal project template the scaffold renders

## Why a separate PR

The skill is internal tooling — it describes the builder API surface and
probes it via curl + Studio UI. It does not import or call any product
code, so it can ship orthogonally from the agent-builder / RBAC / stars
stack itself.

## Compatibility with main

The skill targets endpoints/behaviors that do not exist on `main` yet
(e.g. `/stored/*`, `/editor/builder/*`, channels, role impersonation,
the cookie-leak smoke route in the scaffold template). Running the skill
against `main` today will fail — that's expected. It's a planned runbook
for the stack landing across the queued split PRs (#16576, #16578, and
follow-ups).

## Test plan

- N/A — no executable product code in this PR.
- The skill itself has been exercised across three full smoke-test runs
(auth-off, auth-on admin, auth-on member) on the rollup feature branch
and uncovered ~10 product bugs that have already been fixed there.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## ELI5

This adds an internal QA runbook and tooling that scaffolds a small
local Mastra project and runs end-to-end smoke tests for the Agent
Builder feature. It's documentation, templates, and scripts only (no
production runtime code shipped) and is intended to be run against the
feature branch (it will fail against main because some
endpoints/features are not present).

---

## Overview

Adds a hermetic, agent-readable smoke-test skill at
.claude/skills/builder-smoke-test/ to validate the Agent Builder feature
stack. Includes a detailed runbook (SKILL.md), per-domain reference
checklists, a minimal scaffolded project template, and helper scripts to
create, validate, and run smoke tests. Tooling-only: no exported/public
code changes to the repository and no runtime dependency on @mastra/* in
main. The runbook was exercised on the feature rollup branch in three
configurations (auth-off, auth-on admin, auth-on member) and helped
surface ~10 product fixes applied on that branch.

---

## Key Additions

Documentation
- SKILL.md: end-to-end workflow, 15-section coverage matrix (auth-off /
auth-on admin / auth-on member), execution flow rules, CLI surface
(--test, --scope, --auth, --role, --clean, --skip-browser, --dir,
--reuse, --openai-key, WorkOS flags), partial-run semantics,
preflight/error expectations, cookie-extraction gating
(SMOKE_TEST_COOKIE_LEAK), seeding guidance, and reporting template.

Per-domain reference docs (curl/jq + UI checks)
- Added: agents.md, auth.md, channels.md, defaults.md,
infrastructure.md, model-policy.md, permissions.md, picker-allowlist.md,
reconciliation.md, registry.md, setup.md, skills.md, stars.md, ui.md,
workspace.md — each with step-by-step commands, expected
schemas/behaviors, and checklists covering stored
agents/skills/workspaces, RBAC, model policy, registry/library-copy,
channels, builder defaults, reconciliation, UI parity, and stars.

Scaffolded project template (assets/template/)
- package.json: ES module template with mastra scripts and
pnpm.overrides linking local worktree packages.
- src/mastra/index.ts: builder/editor config (LibSQL store, workspace,
builder flags, model/tool/agent allowlists), observability, Stagehand
browser hooks, Slack config, and a gated /smoke-test/cookie endpoint
when SMOKE_TEST_COOKIE_LEAK=1.
- src/mastra/auth.ts: WorkOS auth + RBAC provider conditional
initialization (AUTH_PROVIDER-driven).
- src/mastra/agents/index.ts: example weatherAgent using
openai('gpt-4o-mini').
- src/mastra/tools/index.ts: weatherInfo tool with zod validation and
static response.
- src/mastra/workflows/index.ts: greetWorkflow example.
- tsconfig.json: strict bundler configuration.

Helper scripts (scripts/)
- scaffold.sh: create/reuse a hermetic builder-smoke-test project,
substitute __WORKTREE_ROOT__ with the absolute worktree path, write .env
(including deterministic WORKOS_COOKIE_PASSWORD when applicable), set
AUTH_MODE based on provided WorkOS creds, and run pnpm install unless
--reuse. Well-defined exit codes.
- preflight.sh: validate project dir, node_modules/@mastra/core
presence, required .env keys (OPENAI_API_KEY), and expected auth mode
(auto-extends forwarded scaffold args from WORKOS_* when needed); prints
PROJECT_DIR/AUTH_MODE and aggregates failures.
- wait-for-server.sh: poll http://localhost:${PORT}/api/agents for HTTP
200 with configurable budget/port and alternate-port checks (4112–4114);
clear exit codes for success/timeout/alternate-port/invalid args.
- seed-multi-user.sh: idempotently seed two published skills owned by a
non-primary user directly into the scaffold’s libsql DB via sqlite3
(disables foreign keys during insert) and prints seeded IDs on success.

Other edits
- Minor doc/snippet fixes and clarifications (cookie header usage for
authenticated GETs, infra gate annotation correction, UI route
clarifications, create-skill payload requirements, avatar upload
endpoint, strengthened preflight parsing and error codes, runnable
snippet improvements).

---

## Compatibility & Test Plan

Targets endpoints/behaviors not yet present on main (stored endpoints,
editor/builder routes, channels, role-impersonation UI hooks,
cookie-leak debug route). Running against main is expected to fail; this
skill is a runbook to be used as the feature code lands across related
PRs (#16576, #16578, and follow-ups). No production code changes in this
repo. The runbook was used on the feature rollup branch across three
full smoke runs and surfaced ~10 product bugs that were fixed there.

---

## Design Notes / Safety

- Hermetic scaffold links to the current worktree via pnpm.overrides for
local testing.
- WorkOS auth is opt-in via environment/flags; preflight enforces
expected auth mode and required WorkOS keys.
- Non-owner visibility tests require seeded DB rows
(seed-multi-user.sh).
- /smoke-test/cookie route is gated by SMOKE_TEST_COOKIE_LEAK and
intended only for scaffolded debug cookie extraction.
- Template rendering substitutes __WORKTREE_ROOT__ with an absolute path
during scaffold.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/mastra-ai/mastra/pull/16579)

<!-- review_stack_entry_end -->

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/mastra-ai/mastra/pull/16579)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Mastra Code (anthropic/claude-opus-4-7) <noreply@mastra.ai>
2026-05-14 12:21:02 -05:00
Grayson Hicks
86c0298e64
feat(auth): add Fine-Grained Authorization (FGA) with WorkOS provider - resource-level enforcement across agents, workflows, tools, memory, and MCP (#15410)
## Description

Adds Fine-Grained Authorization (FGA) support to Mastra for
relationship-based, resource-level access control. The branch introduces
the core FGA provider/manager interfaces, typed `MastraFGAPermissions`,
`checkFGA()`/`FGADeniedError`, `server.fga` config, and a capabilities
flag. FGA checks are backward compatible and no-op when no provider is
configured.

Runtime enforcement is wired through agent `generate()`/`stream()`,
workflow execution, tool execution, MCP tool listing/execution, server
route handlers, and memory/thread operations. List and search paths now
filter agents, workflows, tools, MCP tools, and threads to only the
resources the user can access, with thread checks carrying the owning
`resourceId` and request context for tenant-aware providers.

Adds `MastraFGAWorkos` in `@mastra/auth-workos` backed by the WorkOS
Authorization API. It supports resource and permission mapping,
organization membership resolution and caching, organization scoping,
service-token/JWT claim context, resource management, role assignment
management, and accessible-resource filtering.

Updates the built-in server adapters (Express, Fastify, Hono, Koa) to
run route-level FGA on built-in and custom routes, including
request-aware resource ID resolvers and path/query/body params. The
branch also preserves auth refresh headers in adapter auth flows and
fixes MCP tool list output schema metadata while adding FGA coverage.

Docs, tests, and changesets are included for `@mastra/core`,
`@mastra/server`, `@mastra/auth-workos`, `@mastra/mcp`, and the server
adapter packages.

## Related Issue(s)

Fixes #14089

## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Code refactoring
- [ ] Performance improvement
- [x] Test update

## Checklist
- [x] I have made corresponding changes to the documentation (if
applicable)
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have addressed all Coderabbit comments on this PR

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Ryan Hansen <rphansen91@gmail.com>
2026-05-04 10:49:08 -05:00