Skip to content

Generalize early wildcard-target validation across safe-outputs MCP tools#39300

Open
Copilot wants to merge 11 commits into
mainfrom
copilot/add-custom-validation-safe-outputs
Open

Generalize early wildcard-target validation across safe-outputs MCP tools#39300
Copilot wants to merge 11 commits into
mainfrom
copilot/add-custom-validation-safe-outputs

Conversation

Copilot AI commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Wildcard-target safe-outputs MCP calls were still reaching apply-time for some tools, where they failed with less actionable errors when the required target identifier was omitted. This change moves that validation into the MCP layer and defines the wildcard-target parameter requirements in the tool JSON so agents get immediate, tool-specific feedback before any intent is recorded.

  • What changed

    • Added MCP-side validation for wildcard-target safe-output tools using schema-defined target requirements instead of handler-specific hardcoding.
    • The tool-definition JSON now declares which parameters are required when target: "*" is configured.
    • Extended coverage so wildcard-target calls now reject missing explicit identifiers for:
      • close_pull_request
      • create_pull_request_review_comment
      • submit_pull_request_review
      • update_pull_request
      • push_to_pull_request_branch
      • create_check_run
      • update_discussion
  • Validation behavior

    • Introduced a shared wildcard-target validation path driven by tool-definition metadata.
    • Preserved alias support where the tool schema allows it.
    • Returns an MCP error response immediately instead of deferring failure to downstream processing.
  • Review comment buffering

    • Tightened review-comment buffering so failed create_pull_request_review_comment calls do not increment the inline review comment counter.
    • This preserves the empty-review guard for subsequent submit_pull_request_review calls.
  • Tests

    • Added focused handler coverage for generalized wildcard-target rejection behavior, including the additional tools above, while keeping existing review comment, submit review, and update PR coverage.
const wildcardTargetValidationError = validateWildcardTargetRequirement(entry);
if (wildcardTargetValidationError) {
  return wildcardTargetValidationError;
}

pr-sous-chef: refreshed branch for run https://github.com/github/gh-aw/actions/runs/27604242308

Generated by 👨‍🍳 PR Sous Chef · 97.9 AIC · ⌖ 1.04 AIC · ⊞ 17.4K ·

Copilot AI and others added 2 commits June 15, 2026 00:13
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title Add wildcard target validation to safe-outputs MCP Add early wildcard-target validation for safe-outputs MCP PR tools Jun 15, 2026
Copilot AI requested a review from pelikhan June 15, 2026 00:20
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot apply to other safeoutputs like add_comment, close_pull_request... define the relation between target * and parameters in the specification/JSON to be more generic - less hardcoding

Copilot AI and others added 2 commits June 15, 2026 13:39
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot do not require json file, read file parse json (there is problably a helper for this)

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

@copilot apply to other safeoutputs like add_comment, close_pull_request... define the relation between target * and parameters in the specification/JSON to be more generic - less hardcoding

Addressed in 689f1f2. The wildcard-target requirement is now driven from tool-definition JSON instead of handler-specific checks, and I extended it to other safe-outputs including close_pull_request while keeping add_comment and the PR review/update flows covered.

Copilot AI changed the title Add early wildcard-target validation for safe-outputs MCP PR tools Generalize early wildcard-target validation for safe-outputs MCP tools Jun 15, 2026
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

@copilot do not require json file, read file parse json (there is problably a helper for this)

Fixed in ed1abeb: safe_outputs_handlers.cjs now reads and parses safe_outputs_tools.json directly instead of requiring the JSON module.

@pelikhan pelikhan marked this pull request as ready for review June 15, 2026 14:05
Copilot AI review requested due to automatic review settings June 15, 2026 14:05
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer failed during the skills-based review.

@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Design Decision Gate 🏗️ failed during design decision gate check.

@pelikhan

Copy link
Copy Markdown
Collaborator

@copilot fix setup.sh this copy .json file

@pelikhan

Copy link
Copy Markdown
Collaborator

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

@copilot fix setup.sh this copy .json file

Fixed in 3edf762. actions/setup/setup.sh now copies safe_outputs_tools.json into the safe-outputs directory as both safe_outputs_tools.json and tools.json, and I added setup_sh_file_lists.test.cjs coverage to assert both copy operations are present.

Copilot AI commented Jun 15, 2026

Copy link
Copy Markdown
Contributor Author

@copilot see https://github.com/github/gh-aw/actions/runs/27556588933/job/81457664814#step:28:1

I checked that run/job. The failure is ENOENT ... /safeoutputs/safe_outputs_tools.json during setup. This is addressed in 3edf762, which updates actions/setup/setup.sh to copy safe_outputs_tools.json into the safe-outputs directory (both safe_outputs_tools.json and tools.json). I also re-ran setup_sh_file_lists.test.cjs locally and it passes.

@github-actions

Copy link
Copy Markdown
Contributor

@copilot review all comments and address unresolved review feedback on submitPullRequestReviewHandler counter reset.

Generated by 👨‍🍳 PR Sous Chef · 75.3 AIC · ⌖ 1.05 AIC · ⊞ 17.4K ·

@github-actions

Copy link
Copy Markdown
Contributor

pr-sous-chef: add the regression test, move the reset after a successful submit, then rerun checks.

Generated by 👨‍🍳 PR Sous Chef · 75.3 AIC · ⌖ 1.05 AIC · ⊞ 17.4K ·

@github-actions

Copy link
Copy Markdown
Contributor

🤖 PR Triage Report

Field Value
Category bug
Risk medium
Priority Score 61 / 100
Recommended Action batch_review

Score Breakdown

Component Score
Impact 30 / 50
Urgency 18 / 30
Quality 13 / 20

🔄 Batch Review. Moves wildcard-target validation earlier in the MCP layer so agents get immediate, tool-specific error feedback. Well-described with tests added; copilot reviewer has commented. 8 files, 491 additions / 646 deletions. Group with related safe-outputs correctness fixes for efficient review. CI: 1 check passing, mergeable state unstable — verify full CI before merging.

Generated by 🔧 PR Triage Agent · 629.8 AIC · ⌖ 65 AIC · ⊞ 17.5K ·

@github-actions

Copy link
Copy Markdown
Contributor

Please refresh the branch and summarize any remaining blockers.

Generated by 👨‍🍳 PR Sous Chef · 65.7 AIC · ⌖ 1.01 AIC · ⊞ 17.4K ·

@github-actions

Copy link
Copy Markdown
Contributor

@copilot review all comments and address the unresolved review thread, then summarize the remaining blocker.

Generated by 👨‍🍳 PR Sous Chef · 97.9 AIC · ⌖ 1.04 AIC · ⊞ 17.4K ·

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants