benchmarks: Add codebase-memory-mcp, cs, and ck to benchmarks - #237
Open
Pringled wants to merge 15 commits into
Open
benchmarks: Add codebase-memory-mcp, cs, and ck to benchmarks#237Pringled wants to merge 15 commits into
Pringled wants to merge 15 commits into
Conversation
New retrieval baselines: cs (Code Spelunker), codebase-memory-mcp (search_graph, fast mode), added to the main quality/speed tables and by-language breakdown. GitNexus and ck were evaluated but excluded from the main comparison (documented in "Excluded methods"): GitNexus lacks tree-sitter grammars for 6/19 benchmark languages, and ck's release binary can't complete its embedding-model download behind a TLS-intercepting proxy. Also fixes speed_benchmark.py, which had bit-rotted against the current SembleIndex API (from_path(model=...) and search(mode=...) were both removed) and refreshes semble/BM25/ColGREP/ripgrep speed numbers and plots accordingly. CodeRankEmbed numbers are left as last-known-good; its benchmark script has the same API-drift bug and is being tracked separately.
codegraph (colbymchenry/codegraph, 62k stars) scores 0.539 NDCG@10 across all 63 repos but doesn't support Haskell/Bash/Elixir/Zig (fails gracefully, unlike GitNexus's hard failure) so it's excluded from the main table on the same grounds as codanna/GitNexus. Adds a "Common-language subset" table restricting every tool to the 13 languages GitNexus and codegraph both support, computed from already-published per-language numbers (no new benchmark runs for tools already in the main table).
ck.py had a real bug independent of the earlier TLS issue: --json --quiet produces JSONL (one object per line), not a wrapped JSON array, so every query silently returned zero results. Fixed the parser and re-verified against the real embedding-search output. The TLS block itself needed a local rebuild: BeaconBay/ck's published binary links fastembed with the hf-hub-rustls-tls feature (bundled CA store), which can't complete the model download behind a TLS-intercepting proxy even though curl/pip on the same network work fine. Swapping to hf-hub-native-tls in a from-source build (documented in benchmarks/README.md) fixes it — one Cargo.toml line, no logic changes. Full run: 0.646 NDCG@10 across 61/63 repos (abseil-cpp and zig exceeded the 600s per-repo indexing timeout; both languages remain covered by their other repos). Added to Main results, By-language, and the common-language-subset table. Also fixes coderankembed.py, which silently indexed with the wrong model (SembleIndex.from_path no longer accepts a raw model object). Rebuilt the index via the lower-level create_index_from_path + SembleIndex constructor, which duck-types fine at runtime despite the StaticModel type hint. Verified on flask (0.914 hybrid, 0.890 semantic) but not run across all 63 repos — CodeRankEmbed inference is slow on CPU (~21s/repo just to index), left for a deliberate follow-up run.
…im README verbosity BM25's index_ms was a passthrough of semble's full hybrid-index build time (dense embeddings + BM25), not a real standalone BM25 measurement - confirmed by profiling that BM25's own add_document cost is ~3-4% of that number. Now builds and times a real BM25-only index from semble's already-chunked content: 45ms vs the previous 586ms. token_efficiency.py had the same SembleIndex.from_path(model=...) and missing file_walker exports bit-rot as the other benchmark scripts. Fixed and re-ran: chunk_size dropped 1500->750 chars back in June, which this benchmark was never rerun against - expected tokens per query is 348 (was 566), recall improves across the board. Propagated the refreshed numbers into the top-level README too. Also, per feedback: dropped the common-language-subset table (not worth a whole section for two excluded tools), trimmed the verbose GitNexus/codegraph exclusion writeups to match the terse style of the existing codanna/claude-context entries, and cut the ck TLS-rebuild troubleshooting note (not something users need).
abseil-cpp completed within a longer indexing timeout (0.644 NDCG@10); zig's stdlib repo is still being retried separately with a larger budget before finalizing the README/plots.
…pos) Bulk `ck --index` on zig's stdlib never completes, even at a 1-hour timeout. Indexed it file-by-file with `ck --add` instead (each call persists immediately, so it's resumable and doesn't lose progress on a timeout) - all 550 files indexed successfully. ck now covers the full 63/63 repos: 0.642 NDCG@10.
benchmarks/baselines/probe.py calls `probe search`, which expects boolean/keyword syntax. Translating natural-language queries into that syntax is `probe agent`'s job (LLM-backed), which the harness never invokes. Scoring `probe search` against raw NL queries tests an entry point it wasn't designed for, so it's misleading to keep in the main comparison. Moved to Excluded methods, dropped from the by-language table and the speed-vs-quality plots.
probe search runs free and local, same category as ripgrep/cs, which we keep despite the same NL-query disadvantage. Excluding it wasn't consistent with that precedent. Its NL-capable probe agent mode needs a paid LLM API key and isn't tested - noted in the Methods entry instead of a full exclusion.
The git checkout used to bring probe back also reverted the earlier GitNexus/codegraph trim (that edit was never committed). Re-applying it: no scores, no explanatory asides, matching codanna/claude-context.
…k drift - Delete benchmarks/baselines/gitnexus.py and codegraph.py plus their stale results JSON: both tools are excluded from the comparison and the scripts were never run, so keep only the explanatory bullets in Excluded methods. - Drop the matching "Running the benchmarks" instructions for them. - Restore CodeRankEmbed to speed_benchmark.py's main() loop; it was silently dropped as a side effect of an unrelated API-drift fix, not a decision. - Remove leftover dead/broken code (old SembleIndex API) that was left behind when CodeRankEmbed was dropped, and add a missing mypy type-ignore. - Fix a rounding typo in the main results table (ripgrep p50: 15ms -> 14ms).
- Re-measure semble's index/query speed (518.2ms avg index, 0.91ms p50), with real per-repo data backing it, saved as speed-b468bbf0a1a5.json. Other tools' numbers untouched (not rerun). - Update benchmarks/README.md main table and plot.py to the new semble numbers; regenerate both speed-vs-quality plots. - Round the speed-comparison prose (README.md and benchmarks/README.md) to ~100x indexing / ~20x query / ~500ms, since single-run numbers fall within normal cold-start variance and shouldn't imply false precision. - Fix stale claims in README.md's intro paragraph left over from an older measurement (was "98% fewer tokens" / "200x faster indexing" / "10x faster queries", inconsistent with the rest of the doc and never updated to match the real benchmark numbers).
Codecov Report✅ All modified and coverable lines are covered by tests. 🚀 New features to boost your workflow:
|
Confidence Score: 4/5The PR is not yet safe to merge because its published BM25 query latency excludes the ordering needed to produce ranked results. The CodeRankEmbed prompt-selection issue and the prior wrong-index BM25 path are fixed, but the new standalone BM25 timer stops at an unordered partition and therefore reports an incomplete query cost. Files Needing Attention: benchmarks/speed_benchmark.py Reviews (2): Last reviewed commit: "Fix CodeRankEmbed encoding bug and BM25 ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…fected benchmarks - Fix _AsymmetricWrapper.encode() in coderankembed.py and speed_benchmark.py: query vs. document mode was detected from batch size (len==1), so single-chunk files were encoded with the query prompt during indexing instead of the document prompt, violating CodeRankEmbed's asymmetric encoding contract. Now detected via presence of the use_multiprocessing kwarg, which matches semble's actual query-time vs. index-time call sites (semble.search never passes it, semble.index.dense.embed_chunks always does). - Fix speed_benchmark.py's _bench_bm25: it built a standalone BM25 index but then measured query latency via SembleIndex.search(alpha=0.0), which still runs dense encoding, ANN search, and reranking regardless of alpha. Now queries the standalone BM25 index directly. BM25 query p50: 0.77ms -> 0.16ms. - Rerun baselines/coderankembed.py (63 repos) and the CodeRankEmbed portion of speed_benchmark.py under the fix. The previous committed CodeRankEmbed numbers also predated an unrelated chunker rewrite (chonkie -> semble's custom chunker), so the comparison was stale on more than just the encoding bug. NDCG@10: semantic 0.765 -> 0.839, hybrid 0.862 -> 0.861. Index time roughly doubled (57s -> 116s) since the new chunker produces substantially more chunks to embed. - Update benchmarks/README.md (main table, by-language table and its sort order, speed multiplier prose) and README.md accordingly; regenerate both speed-vs-quality plots.
Comment on lines
+175
to
+177
| if scores.size: | ||
| k = min(_TOP_K, scores.size) | ||
| np.argpartition(-scores, kth=k - 1)[:k] |
There was a problem hiding this comment.
When _bench_bm25 processes a non-empty score array, it times only np.argpartition and discards the unordered candidates, while the actual BM25 search path also sorts those candidates into ranked results. This understates the published BM25 query latency by omitting required ranking work.
Suggested change
| if scores.size: | |
| k = min(_TOP_K, scores.size) | |
| np.argpartition(-scores, kth=k - 1)[:k] | |
| if scores.size: | |
| k = min(_TOP_K, scores.size) | |
| partitioned = np.argpartition(-scores, kth=k - 1)[:k] | |
| np.argsort(-scores[partitioned]) |
CodeRankEmbed and CodeRankEmbed Hybrid now score and query close enough that their two-line "CodeRankEmbed\nHybrid" label collided with the plain "CodeRankEmbed" label. Single-line it so both labels sit cleanly next to their own marker.
_bench_bm25 timed np.argpartition but discarded its output without sorting the selected candidates. The real BM25 search path (semble.search._sort_top_k) always sorts the partitioned top-k into ranked order, so the benchmark was still missing real work despite the earlier fix to query the standalone index. Rerun: BM25 index 44ms -> 47ms, query p50 0.16ms -> 0.17ms (small since k=50 sort is cheap, but real).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds three more methods to our benchmarks:
It also fixes a few outdated things in our benchmarks, most notably, the Semble indexing time is slightly higher (500ms) but the token reduction vs ripgrep is much higher (since we decreased the default chunk size).