feat: add verified industry reference examples - #224
Open
svonava wants to merge 6 commits into
Open
Conversation
Contributor
Author
|
@coderabbitai review |
Contributor
Author
|
Review update:\n\n- Removed the endpoint-specific Docling identifier translation.\n- All new runs now use the canonical SIE model ID, docling, on local and Cloud endpoints.\n- Preserved the acquisition-time configured identifier in each verified manifest and documented why it differs from the canonical model actually served.\n- Reran 42 locked offline tests plus Ruff lint and format across all three packages.\n\nThis keeps the examples compatible with PR #222 and avoids client-side model naming rules. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThree runnable SIE examples add end-to-end review pipelines for financial filings, NTSB bearing-failure reconstruction, and CMS prior authorization, with configurations, CLI evaluators, evidence artifacts, documentation, and tests. ChangesFinancial Filing Review
Maintenance Triage Review
Prior Authorization Review
Estimated code review effort: 4 (Complex) | ~60 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
Contributor
Author
|
@coderabbitai full review |
✅ Action performedFull review finished. Your included review limit is currently reached under our Fair Usage Limits Policy. Your recent PR review activity is in the 95th percentile or higher among CodeRabbit users, so adaptive limits apply. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 49 minutes. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (2)
examples/maintenance-triage-agent/tests/test_review.py (1)
214-221: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a failure test for the chunk-identity gate.
_require_ranked_evidencerejects evidence rows missingchunk_idortext(review.py lines 181-182), and_row_for_locationrejects ambiguous location matches (review.py lines 227-228). Neither branch is exercised, yet both are provenance guarantees this example advertises.🧪 Suggested addition
def test_review_fails_closed_on_evidence_without_chunk_identity() -> None: evidence = ranked_evidence() del evidence[0]["chunk_id"] try: build_review(structured_data(), evidence) except RuntimeError as exc: assert "chunk identity" in str(exc) else: raise AssertionError("build_review accepted evidence without a chunk id")As per path instructions, "Check that commands match the implementation and that tests cover failure paths".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/maintenance-triage-agent/tests/test_review.py` around lines 214 - 221, Add failure-path coverage in test_review.py for the chunk-identity gate used by build_review: create valid ranked evidence, remove a row’s chunk_id (or text), and assert RuntimeError with the expected “chunk identity” message; fail explicitly if build_review accepts it. Also add coverage for _row_for_location rejecting ambiguous location matches, preserving the advertised provenance guarantees.Source: Path instructions
examples/prior-authorization-review-agent/tests/test_review.py (1)
1-360: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd failure-path tests for the remaining fail-closed gates.
Coverage exists for
build_review's field-mismatch gates, but not for_source_scope(no matching fragment),_require_ranked_evidence(missing/malformed reranked rows),_chunks(no usable chunks), or the "not overdue" branch inbuild_review. These are part of the same deterministic fail-closed contract the README highlights ("Missing, conflicting, or untraceable model output stops the run").As per path instructions, "Check that commands match the implementation and that tests cover failure paths."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/prior-authorization-review-agent/tests/test_review.py` around lines 1 - 360, Add failure-path tests covering the remaining fail-closed gates: make _source_scope reject evidence with no matching fragment, make _require_ranked_evidence reject missing or malformed reranked rows, and make _chunks reject input with no usable chunks. Add a build_review test for the not-overdue encounter case, asserting the documented failure behavior. Reuse existing fixtures/helpers and assert the expected RuntimeError messages.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/financial-filing-agent/financial_filing/review.py`:
- Around line 136-143: Update _require_entity_evidence so required_spans also
includes the restated diluted EPS source span "$1.34", preserving the existing
normalization, missing-span detection, and error behavior.
In `@examples/maintenance-triage-agent/maintenance_triage/__init__.py`:
- Line 1: Update the module docstring in maintenance_triage to describe that the
example performs read-only maintenance triage using the published NTSB SPC-24-06
page spread, removing the inaccurate reference to synthetic plant records.
In `@examples/maintenance-triage-agent/maintenance_triage/review.py`:
- Around line 478-480: Update the SIEClient construction in the maintenance flow
to pass the already-read timeout variable from
config["cluster"]["provision_timeout_s"] instead of the hardcoded 900, while
preserving the existing timeout forwarding for subsequent calls.
- Around line 452-462: Update the mapped-field and triage calculation flow to
derive both asserted values: add and validate east_palestine_time via
_source_match in _map_exact_source_fields, then compute
salem_to_east_palestine_minutes from the mapped times like
sebring_to_salem_minutes; also derive derailment.total_cars from the validated
derailment_statement as 1 hopper car plus 37 others, replacing the literal 38.
In `@examples/maintenance-triage-agent/README.md`:
- Around line 62-77: Update the evidence-bundle artifact list in the README to
include the per-stage GLiNER outputs written by run():
raw/entities-detector-{index}.json and raw/entities-outcome.json. Keep the
existing artifact entries unchanged and ensure the documented paths match the
implementation.
In `@examples/prior-authorization-review-agent/prior_authorization/review.py`:
- Around line 389-415: Update review.py lines 389-415 to populate
missing_documentation, review_conclusion, and payment_action from the validated
outcome fields rather than literals; update README.md lines 26-32 to accurately
describe verbatim versus canonical fields; update evaluate.py lines 50-60 to use
token/substring assertions matching _require_tokens instead of fixed-string
equality.
---
Nitpick comments:
In `@examples/maintenance-triage-agent/tests/test_review.py`:
- Around line 214-221: Add failure-path coverage in test_review.py for the
chunk-identity gate used by build_review: create valid ranked evidence, remove a
row’s chunk_id (or text), and assert RuntimeError with the expected “chunk
identity” message; fail explicitly if build_review accepts it. Also add coverage
for _row_for_location rejecting ambiguous location matches, preserving the
advertised provenance guarantees.
In `@examples/prior-authorization-review-agent/tests/test_review.py`:
- Around line 1-360: Add failure-path tests covering the remaining fail-closed
gates: make _source_scope reject evidence with no matching fragment, make
_require_ranked_evidence reject missing or malformed reranked rows, and make
_chunks reject input with no usable chunks. Add a build_review test for the
not-overdue encounter case, asserting the documented failure behavior. Reuse
existing fixtures/helpers and assert the expected RuntimeError messages.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 45314b75-564f-486d-8637-593b0866fab3
⛔ Files ignored due to path filters (66)
.coderabbit.yamlis excluded by none and included by noneexamples/financial-filing-agent/fixtures/SOURCES.mdis excluded by!examples/**/fixtures/**and included byexamples/**,examples/**/fixtures/SOURCES.mdexamples/financial-filing-agent/fixtures/pathward-filing-packet.htmlis excluded by!examples/**/fixtures/**and included byexamples/**examples/financial-filing-agent/fixtures/pathward-q3-fy2023.jsonis excluded by!examples/**/fixtures/**and included byexamples/**examples/financial-filing-agent/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/financial-filing-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/parsed.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities-original-10q.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities-restated-10ka.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities-status.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/gliner2-original-10q.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/gliner2-restated-10ka.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/gliner2-status.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/mapped.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/rerank.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/retrieve.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/fixtures/SOURCES.mdis excluded by!examples/**/fixtures/**and included byexamples/**,examples/**/fixtures/SOURCES.mdexamples/maintenance-triage-agent/fixtures/STRUCTURED_OUTPUT_PROBE.mdis excluded by!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/fixtures/east-palestine-bearing-spread.pdfis excluded by!**/*.pdf,!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/fixtures/output-schema-probe-request.jsonis excluded by!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/fixtures/output-schema-probe-response.jsonis excluded by!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/maintenance-triage-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/parsed.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-detector-0.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-detector-1.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-detector-2.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-outcome.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-cause.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-derailment.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-east-palestine.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-engineer.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-salem.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-sebring.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/mapped.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/rerank.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/retrieve.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/fixtures/SOURCES.mdis excluded by!examples/**/fixtures/**and included byexamples/**,examples/**/fixtures/SOURCES.mdexamples/prior-authorization-review-agent/fixtures/cms-l1851-insufficient-documentation.htmlis excluded by!examples/**/fixtures/**and included byexamples/**examples/prior-authorization-review-agent/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/prior-authorization-review-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/parsed.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-0.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-1.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-2.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-3.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-4.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-requirement-0.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/gliner2-outcome.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/gliner2-requirements.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/gliner2-submission.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/mapped.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/rerank.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/retrieve.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**
📒 Files selected for processing (28)
examples/README.mdexamples/financial-filing-agent/.env.exampleexamples/financial-filing-agent/.gitignoreexamples/financial-filing-agent/README.mdexamples/financial-filing-agent/config.yamlexamples/financial-filing-agent/financial_filing/__init__.pyexamples/financial-filing-agent/financial_filing/evaluate.pyexamples/financial-filing-agent/financial_filing/review.pyexamples/financial-filing-agent/pyproject.tomlexamples/financial-filing-agent/tests/test_review.pyexamples/maintenance-triage-agent/.env.exampleexamples/maintenance-triage-agent/.gitignoreexamples/maintenance-triage-agent/README.mdexamples/maintenance-triage-agent/config.yamlexamples/maintenance-triage-agent/maintenance_triage/__init__.pyexamples/maintenance-triage-agent/maintenance_triage/evaluate.pyexamples/maintenance-triage-agent/maintenance_triage/review.pyexamples/maintenance-triage-agent/pyproject.tomlexamples/maintenance-triage-agent/tests/test_review.pyexamples/prior-authorization-review-agent/.env.exampleexamples/prior-authorization-review-agent/.gitignoreexamples/prior-authorization-review-agent/README.mdexamples/prior-authorization-review-agent/config.yamlexamples/prior-authorization-review-agent/prior_authorization/__init__.pyexamples/prior-authorization-review-agent/prior_authorization/evaluate.pyexamples/prior-authorization-review-agent/prior_authorization/review.pyexamples/prior-authorization-review-agent/pyproject.tomlexamples/prior-authorization-review-agent/tests/test_review.py
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (4)
examples/prior-authorization-review-agent/prior_authorization/review.py (1)
455-457: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueClient timeout duplicates the configured value as a literal.
timeoutis read fromprovision_timeout_son Line 455, then Line 457 hardcodestimeout_s=900, so editingconfig.yamlsilently leaves the socket timeout behind.♻️ Proposed refactor
- with SIEClient(config["cluster"]["url"], api_key=config["cluster"]["api_key"] or None, timeout_s=900) as client: + with SIEClient(config["cluster"]["url"], api_key=config["cluster"]["api_key"] or None, timeout_s=timeout) as client:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/prior-authorization-review-agent/prior_authorization/review.py` around lines 455 - 457, Use the existing provision timeout value from config["cluster"]["provision_timeout_s"] when constructing SIEClient in the review flow, replacing the hardcoded timeout_s=900 while preserving the surrounding client setup.examples/prior-authorization-review-agent/README.md (1)
70-82: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueEvidence bundle listing omits the per-row GLiNER artifacts.
review.pyLine 598 writesraw/entities-requirement-<n>.jsonandraw/entities-case-<n>.jsonper ranked row, which aren't documented here. Readers comparing the run directory against this table will see unlisted files.As per path instructions, "Review runnable source, tests, configuration, and documentation. Check that commands match the implementation".
📝 Proposed doc addition
runs/<run-id>/raw/entities.json combined GLiNER entity spans +runs/<run-id>/raw/entities-requirement-*.json per-passage GLiNER requirement spans +runs/<run-id>/raw/entities-case-*.json per-passage GLiNER case spans🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/prior-authorization-review-agent/README.md` around lines 70 - 82, Update the evidence bundle listing in the README to include the per-ranked-row artifacts written by review.py, specifically raw/entities-requirement-<n>.json and raw/entities-case-<n>.json. Place them with the other raw GLiNER/entity outputs and describe their per-row purpose accurately.Source: Path instructions
examples/prior-authorization-review-agent/tests/test_review.py (1)
278-305: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winNo test exercises the evaluator's failing path.
Every assertion here drives
evaluate_review/evaluate_runtoTrue; nothing covers a review that fails a check, so thefailrendering and theSystemExit(1)inmainare untested.As per path instructions, "Check that commands match the implementation and that tests cover failure paths".
💚 Proposed test
def test_evaluator_fails_on_a_mutated_review(tmp_path: Path) -> None: review = build_valid_review() review["overdue_by_months"] = 0 (tmp_path / "review.json").write_text(json.dumps(review), encoding="utf-8") assert evaluate_run(tmp_path) is False evaluation = json.loads((tmp_path / "evaluation.json").read_text(encoding="utf-8")) assert evaluation["passed"] is False assert any(check["name"] == "one-month-gap" and not check["passed"] for check in evaluation["checks"])🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/prior-authorization-review-agent/tests/test_review.py` around lines 278 - 305, Add a failure-path test alongside the existing evaluator tests, using a mutated review with overdue_by_months set to 0. Assert evaluate_run returns False, evaluation.json is written with passed set to False, and the one-month-gap check is recorded as failed; also cover main’s SystemExit(1) behavior if that entry point is exposed.Source: Path instructions
examples/prior-authorization-review-agent/prior_authorization/evaluate.py (1)
105-106: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUnhelpful traceback when the run directory is wrong.
eval-paon a missing or incomplete run dir raises a bareFileNotFoundError. A guarded message keeps the example approachable.♻️ Proposed refactor
def evaluate_run(run_dir: Path) -> bool: - review = json.loads((run_dir / "review.json").read_text(encoding="utf-8")) + review_path = run_dir / "review.json" + if not review_path.exists(): + raise SystemExit(f"No review.json in {run_dir}. Run `uv run review-pa` first.") + review = json.loads(review_path.read_text(encoding="utf-8"))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/prior-authorization-review-agent/prior_authorization/evaluate.py` around lines 105 - 106, Update evaluate_run to guard loading review.json and raise a clear, user-facing error when the run directory is missing or incomplete, instead of exposing a bare FileNotFoundError. Preserve normal JSON loading for valid run directories and include the expected review.json location in the message.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/financial-filing-agent/README.md`:
- Around line 60-75: Update the “Evidence bundle” artifact list in the README to
document the per-stage files written by review.py:
raw/entities-original-10q.json, raw/entities-restated-10ka.json, and
raw/entities-status.json, alongside the existing combined raw/entities.json
entry.
In `@examples/prior-authorization-review-agent/prior_authorization/review.py`:
- Around line 555-563: Update the evidence terms passed to _select_evidence_rows
in the outcome_rows selection so the face-to-face documentation phrase uses an
apostrophe-free substring that matches both straight and typographic
apostrophes. Leave the other outcome terms and selection behavior unchanged.
- Around line 448-457: Update run to handle FileExistsError from raw_dir.mkdir
when the fixed run_id already exists, and provide a clear user-facing message
instructing the user to choose a new run ID or delete the existing run
directory. Preserve the current behavior for new run IDs and keep the directory
non-overwriting.
---
Nitpick comments:
In `@examples/prior-authorization-review-agent/prior_authorization/evaluate.py`:
- Around line 105-106: Update evaluate_run to guard loading review.json and
raise a clear, user-facing error when the run directory is missing or
incomplete, instead of exposing a bare FileNotFoundError. Preserve normal JSON
loading for valid run directories and include the expected review.json location
in the message.
In `@examples/prior-authorization-review-agent/prior_authorization/review.py`:
- Around line 455-457: Use the existing provision timeout value from
config["cluster"]["provision_timeout_s"] when constructing SIEClient in the
review flow, replacing the hardcoded timeout_s=900 while preserving the
surrounding client setup.
In `@examples/prior-authorization-review-agent/README.md`:
- Around line 70-82: Update the evidence bundle listing in the README to include
the per-ranked-row artifacts written by review.py, specifically
raw/entities-requirement-<n>.json and raw/entities-case-<n>.json. Place them
with the other raw GLiNER/entity outputs and describe their per-row purpose
accurately.
In `@examples/prior-authorization-review-agent/tests/test_review.py`:
- Around line 278-305: Add a failure-path test alongside the existing evaluator
tests, using a mutated review with overdue_by_months set to 0. Assert
evaluate_run returns False, evaluation.json is written with passed set to False,
and the one-month-gap check is recorded as failed; also cover main’s
SystemExit(1) behavior if that entry point is exposed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 998e61b4-151c-4ea8-ab11-12532d8edcef
⛔ Files ignored due to path filters (66)
.coderabbit.yamlis excluded by none and included by noneexamples/financial-filing-agent/fixtures/SOURCES.mdis excluded by!examples/**/fixtures/**and included byexamples/**,examples/**/fixtures/SOURCES.mdexamples/financial-filing-agent/fixtures/pathward-filing-packet.htmlis excluded by!examples/**/fixtures/**and included byexamples/**examples/financial-filing-agent/fixtures/pathward-q3-fy2023.jsonis excluded by!examples/**/fixtures/**and included byexamples/**examples/financial-filing-agent/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/financial-filing-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/parsed.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities-original-10q.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities-restated-10ka.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities-status.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/entities.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/gliner2-original-10q.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/gliner2-restated-10ka.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/gliner2-status.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/mapped.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/rerank.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/raw/retrieve.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/financial-filing-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/fixtures/SOURCES.mdis excluded by!examples/**/fixtures/**and included byexamples/**,examples/**/fixtures/SOURCES.mdexamples/maintenance-triage-agent/fixtures/STRUCTURED_OUTPUT_PROBE.mdis excluded by!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/fixtures/east-palestine-bearing-spread.pdfis excluded by!**/*.pdf,!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/fixtures/output-schema-probe-request.jsonis excluded by!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/fixtures/output-schema-probe-response.jsonis excluded by!examples/**/fixtures/**and included byexamples/**examples/maintenance-triage-agent/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/maintenance-triage-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/parsed.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-detector-0.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-detector-1.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-detector-2.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities-outcome.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/entities.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-cause.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-derailment.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-east-palestine.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-engineer.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-salem.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/gliner2-sebring.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/mapped.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/rerank.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/retrieve.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/fixtures/SOURCES.mdis excluded by!examples/**/fixtures/**and included byexamples/**,examples/**/fixtures/SOURCES.mdexamples/prior-authorization-review-agent/fixtures/cms-l1851-insufficient-documentation.htmlis excluded by!examples/**/fixtures/**and included byexamples/**examples/prior-authorization-review-agent/uv.lockis excluded by!**/*.lock,!examples/**/uv.lockand included byexamples/**examples/prior-authorization-review-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/parsed.mdis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-0.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-1.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-2.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-3.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-case-4.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities-requirement-0.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/entities.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/gliner2-outcome.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/gliner2-requirements.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/gliner2-submission.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/mapped.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/parse.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/rerank.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/raw/retrieve.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**
📒 Files selected for processing (28)
examples/README.mdexamples/financial-filing-agent/.env.exampleexamples/financial-filing-agent/.gitignoreexamples/financial-filing-agent/README.mdexamples/financial-filing-agent/config.yamlexamples/financial-filing-agent/financial_filing/__init__.pyexamples/financial-filing-agent/financial_filing/evaluate.pyexamples/financial-filing-agent/financial_filing/review.pyexamples/financial-filing-agent/pyproject.tomlexamples/financial-filing-agent/tests/test_review.pyexamples/maintenance-triage-agent/.env.exampleexamples/maintenance-triage-agent/.gitignoreexamples/maintenance-triage-agent/README.mdexamples/maintenance-triage-agent/config.yamlexamples/maintenance-triage-agent/maintenance_triage/__init__.pyexamples/maintenance-triage-agent/maintenance_triage/evaluate.pyexamples/maintenance-triage-agent/maintenance_triage/review.pyexamples/maintenance-triage-agent/pyproject.tomlexamples/maintenance-triage-agent/tests/test_review.pyexamples/prior-authorization-review-agent/.env.exampleexamples/prior-authorization-review-agent/.gitignoreexamples/prior-authorization-review-agent/README.mdexamples/prior-authorization-review-agent/config.yamlexamples/prior-authorization-review-agent/prior_authorization/__init__.pyexamples/prior-authorization-review-agent/prior_authorization/evaluate.pyexamples/prior-authorization-review-agent/prior_authorization/review.pyexamples/prior-authorization-review-agent/pyproject.tomlexamples/prior-authorization-review-agent/tests/test_review.py
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
examples/maintenance-triage-agent/tests/test_review.py (1)
124-128: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExercise the new mapping and validation paths end to end.
The mapping test preserves
"8:52 p.m .", but the review reconstruction test uses normalized"8:52 p.m."instead of passing_map_exact_source_fields()output intobuild_review(). Add that integration case, plus fail-closed tests for an alteredeast_palestine_timeand an incorrect/malformed derailment count.As per path instructions, “Review runnable source, tests, configuration, and documentation” and “Check that commands match the implementation and that tests cover failure paths.”
Also applies to: 227-251
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/maintenance-triage-agent/tests/test_review.py` around lines 124 - 128, The existing mapping test does not exercise the mapped values through review reconstruction or failure validation. Extend the tests around test_maps_only_exact_ranked_source_text_after_model_gates and build_review to pass _map_exact_source_fields() output into build_review(), then add fail-closed cases for altered east_palestine_time and incorrect or malformed derailment counts.Source: Path instructions
examples/prior-authorization-review-agent/prior_authorization/review.py (1)
459-469: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a regression test for configured timeout propagation.
The pipeline now uses the configured timeout for both
SIEClient(timeout_s=...)and each operation’sprovision_timeout_s, but the new tests do not assert this configuration contract. Mock the client and verify the configured value reaches both the constructor and SIE calls.As per path instructions, review runnable source, tests, configuration, and documentation; verify commands match implementation and tests cover failure paths.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/prior-authorization-review-agent/prior_authorization/review.py` around lines 459 - 469, Extend the regression tests for the extraction pipeline around SIEClient and client.extract to mock the client and assert the configured cluster provision_timeout_s is passed both as the SIEClient timeout_s constructor argument and as each operation’s provision_timeout_s argument. Cover the configured value through the relevant SIE calls without changing production behavior.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/maintenance-triage-agent/maintenance_triage/review.py`:
- Line 482: Update the engineer_action entry in the review output to reuse the
mapped evidence’s exact sentence rather than the paraphrased text, or explicitly
label the paraphrase as a derived summary. Ensure synthetic results are
distinguishable from source evidence and preserve the existing evidence mapping.
---
Nitpick comments:
In `@examples/maintenance-triage-agent/tests/test_review.py`:
- Around line 124-128: The existing mapping test does not exercise the mapped
values through review reconstruction or failure validation. Extend the tests
around test_maps_only_exact_ranked_source_text_after_model_gates and
build_review to pass _map_exact_source_fields() output into build_review(), then
add fail-closed cases for altered east_palestine_time and incorrect or malformed
derailment counts.
In `@examples/prior-authorization-review-agent/prior_authorization/review.py`:
- Around line 459-469: Extend the regression tests for the extraction pipeline
around SIEClient and client.extract to mock the client and assert the configured
cluster provision_timeout_s is passed both as the SIEClient timeout_s
constructor argument and as each operation’s provision_timeout_s argument. Cover
the configured value through the relevant SIE calls without changing production
behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: e41e42ba-7f07-4f6b-bde7-d1a81b25bdda
⛔ Files ignored due to path filters (5)
examples/maintenance-triage-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/maintenance-triage-agent/verified-run/raw/mapped.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/evaluation.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/manifest.jsonis excluded by!examples/**/verified-run/**and included byexamples/**examples/prior-authorization-review-agent/verified-run/review.jsonis excluded by!examples/**/verified-run/**and included byexamples/**
📒 Files selected for processing (10)
examples/financial-filing-agent/README.mdexamples/financial-filing-agent/financial_filing/review.pyexamples/maintenance-triage-agent/README.mdexamples/maintenance-triage-agent/maintenance_triage/__init__.pyexamples/maintenance-triage-agent/maintenance_triage/review.pyexamples/maintenance-triage-agent/tests/test_review.pyexamples/prior-authorization-review-agent/README.mdexamples/prior-authorization-review-agent/prior_authorization/evaluate.pyexamples/prior-authorization-review-agent/prior_authorization/review.pyexamples/prior-authorization-review-agent/tests/test_review.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
4b5ea9cEvidence and provenance
docling-project/docling. Manifests show the internaldoclingID used by the direct local server during verification.Verification
Live NVIDIA L4 runs:
2026-07-25T02:28:01.848776+00:002026-07-25T02:35:39.667778+00:002026-07-25T02:35:45.228957+00:00Local package checks:
Summary by CodeRabbit
evaluation.jsonand manifest-updated evidence artifacts..env.exampleand configuration files; added packaging metadata and CLI entry points.