Stop in-flight forks from making the source name ambiguous - #315
Draft
sjmiller609 wants to merge 1 commit into
Draft
Stop in-flight forks from making the source name ambiguous#315sjmiller609 wants to merge 1 commit into
sjmiller609 wants to merge 1 commit into
Conversation
A fork clones the source's instance directory before writing its own metadata, so between the clone and that write the fork's directory holds a verbatim copy of the source's metadata, including its id and name. Any concurrent lookup of the source by name sees two instances with that name and fails with ErrAmbiguousName, so fan-out that forks by source name loses requests: 19 of 50 concurrent forks returned 409 ambiguous. Ignore metadata whose id does not match the directory it was loaded from, and skip metadata.json in the fork clone since the fork's own metadata is written immediately after. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Forking by source name fails under concurrency. Reproduced with 25 concurrent forks of one standby source: 19 of 50 fork calls returned
409 {"code":"ambiguous","message":"multiple resources match, use full ID"}. The same run with the source addressed by ID succeeded 50/50.Mechanism
forkclones the source's instance directory (copyForkSourceGuestDirectory→cloneGuestDirectoryForFork) and only afterwards writes the fork's own metadata (forkMeta.Id/forkMeta.Name). In between, the fork's directory holds a verbatim copy of the source'smetadata.json, including the source'sIdandName.findInstanceMetadataByNameOrIDPrefixwalks every metadata file and counts name matches, so during that window it sees two instances named<source>and returnsErrAmbiguousName. Every concurrent fork widens the window, so the wider you fan out, the more lookups of the source fail.Fix
query.go: ignore metadata whoseIddoesn't match the directory it was loaded from. A directory that describes another instance isn't that instance, and this also covers any other path that clones a guest directory.snapshot_alias_lock.go: skipmetadata.jsonin the fork clone. The fork's metadata is written fromforkMetamoments later, so copying the source's was both the cause of the window and wasted I/O.+11 non-test lines. No API change, and no behavior change for callers that address instances by ID.
Tests
lib/instances/resolve_test.go: a directory holding another instance's metadata must not make that instance's name ambiguous, and the fork's own name still resolves once its metadata is written. Passes.go vetandgofmtclean; the resolver / list / admission / delete unit tests inlib/instancespass.TestForkCloudHypervisorFromRunning*) fail in my environment with "Image should be ready after 60 seconds" — they need the root/KVM test environment. I verified they fail identically on a clean checkout, so this change isn't responsible; CI should be the judge.🤖 Generated with Claude Code