Skip to content

Expose public submission failure diagnostics - #524

Merged
msaroufim merged 1 commit into
mainfrom
agent/expose-submission-diagnostics
Jul 29, 2026
Merged

Expose public submission failure diagnostics#524
msaroufim merged 1 commit into
mainfrom
agent/expose-submission-diagnostics

Conversation

@msaroufim

Copy link
Copy Markdown
Member

What changed

  • Return allowlisted runtime diagnostics (stdout, stderr, success, exit_code, and duration) for the authenticated owner's public test and benchmark runs.
  • Include stored compilation diagnostics for those same public runs.
  • Keep all secret-run, leaderboard-run, and profile-run diagnostics redacted.
  • Add regression coverage for both public diagnostic visibility and secret-data redaction.

Why

KernelBot already persists compiler output and runtime tracebacks in runs.compilation and runs.meta, but /user/submissions/{id} only returned the structured evaluator result. When a kernel raised during benchmark warm-up, the actionable traceback existed in the database but could not reach Popcorn CLI or an autonomous agent.

Impact

This API addition lets owner-authenticated clients diagnose existing and future public test/benchmark failures without rerunning submissions. It is additive and preserves the existing secret-validation boundary.

Validation

  • uv run --with pytest pytest tests/test_admin_api.py -q — 62 passed
  • Focused public/secret diagnostic regression test passed
  • ruff check --ignore E501 passed on touched files
  • Full non-Docker test attempt reached 129 passed and 3 skipped; Docker-backed tests could not start because Docker Desktop was not running

@github-actions

Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  src/libkernelbot
  utils.py
Project Total  

This report was generated by python-coverage-comment-action

@msaroufim
msaroufim marked this pull request as ready for review July 29, 2026 15:57
@msaroufim
msaroufim merged commit 727212c into main Jul 29, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant