Skip to content

fix(server): stabilize flaky delete telemetry unit test - #2521

Merged
derekwaynecarr merged 1 commit into
NVIDIA:mainfrom
derekwaynecarr:decarr/fix-sandbox-telemetry-flake
Jul 28, 2026
Merged

fix(server): stabilize flaky delete telemetry unit test#2521
derekwaynecarr merged 1 commit into
NVIDIA:mainfrom
derekwaynecarr:decarr/fix-sandbox-telemetry-flake

Conversation

@derekwaynecarr

Copy link
Copy Markdown
Collaborator

Summary

  • Fix flaky unit test delete_handler_ends_telemetry_for_the_resolved_sandbox_id by replacing yield_now() spin-loop with sleep(10ms) polling and increasing the timeout from 1s to 5s

Related Issue

Pre-existing flake observed on main — the test intermittently times out under CI load.

Changes

The test holds the global sandbox_sync_guard and spawns a delete handler, then polls for delete_gate_entry_count() to become non-zero. The spawned task must complete two SQLite queries (via spawn_blocking) before acquiring the delete gate. In the single-threaded #[tokio::test] runtime, yield_now() creates a CPU-bound spin-loop that starves the blocking thread pool, causing the 1-second timeout to expire on loaded CI machines.

  • Replace tokio::task::yield_now() with tokio::time::sleep(10ms) to release CPU between polls
  • Increase timeout from 1s to 5s, consistent with the similar create_sandbox_with_providers_waits_for_sandbox_sync_guard test

Testing

  • Test passes 10/10 consecutive runs locally
  • mise run pre-commit passes

Checklist

Replace yield_now() spin-loop with sleep(10ms) polling in
delete_handler_ends_telemetry_for_the_resolved_sandbox_id. The
single-threaded tokio runtime starves the spawn_blocking threads
used by SQLite when yield_now() burns 100% CPU waiting for the
delete gate entry count. Increase the timeout from 1s to 5s for
consistency with similar guard tests.

Signed-off-by: Derek Carr <decarr@redhat.com>
@derekwaynecarr
derekwaynecarr requested review from a team, maxamillion and mrunalp as code owners July 28, 2026 15:45
@copy-pr-bot

copy-pr-bot Bot commented Jul 28, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@derekwaynecarr

Copy link
Copy Markdown
Collaborator Author

/ok to test c496a5b

@derekwaynecarr
derekwaynecarr added this pull request to the merge queue Jul 28, 2026
Merged via the queue into NVIDIA:main with commit b78c861 Jul 28, 2026
30 checks passed
@derekwaynecarr
derekwaynecarr deleted the decarr/fix-sandbox-telemetry-flake branch July 28, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants