fix: CI trustworthiness + cpp-FuSa/RELAY version currency - #27
Open
SoundMatt wants to merge 1 commit into
Open
Conversation
Bundles the audit findings that all touch the same two CI workflow files (and the docs/version-strings kept in sync with what CI actually pins/checks) -- splitting them into separate PRs would have meant repeatedly conflicting edits to the same job definitions. cpp-FuSa (re-checked against `gh release list` at time of writing -- v0.17.1 is current latest): - Bump the pinned cpp-FuSa ref from v0.15.0 to v0.17.1 in both the fusa-asil-b and sarif jobs (closes #21). Verified the newer tool version's real behavior against this repo rather than just bumping blindly: - `cpfusa init ... --force` was running before the traceability step and silently wiping the real, checked-in `.fusa-reqs.json` (hundreds of REQ-* entries) with an empty scaffold, so live traceability coverage was always computed against zero requirements. Dropped `--force` (unneeded in CI -- the files are already checked in) and the now-unnecessary `|| true` (closes #19). - The ISO 26262 / IEC 61508 gap-analysis steps were wrapped in `|| true`, and separately ran too early in the job (before boundary/tara/fmea/safety-case/sas/sci had produced anything), so most of their own evidence was invisible to them. Reordered those steps to run last, added the missing `cpfusa release` step (sbom.json/provenance.json/artifact-manifest.json were already listed in the evidence-upload step but never actually generated by any step), and added a real CHANGELOG.md (closes #23 for real, not just by deleting the dead link -- and happens to also satisfy the ISO 26262 "change management" objective). With the full evidence set in place and in the right order, gaps drop from 16/20 to 11/20 (ISO 26262 ASIL-B) and to 10/18 (IEC 61508 SIL-2). - `cpfusa iso26262`/`iec61508` exit 1 whenever any objective is "gap" -- that's real tool behavior, not a bug, so blanket `|| true` was hiding genuine regressions along with a genuine cpfusa v0.17.1 limitation: 11 of the 20 ASIL-B objectives (and 10 of 18 SIL-2 objectives) have no evidence-detection logic at all in this tool version and report "gap" unconditionally regardless of real project evidence (filed upstream as SoundMatt/cpp-FuSa#57). Replaced `|| true` with an explicit gate that fails only if the gap count regresses past that documented, verified floor (closes #20). - Did NOT add `cpfusa verify` to the job: its ctest-output regex breaks on any multi-word test name (this repo's own convention) and silently drops 170 of 171 real passing tests from `.fusa-evidence.json`, which would be actively misleading safety evidence rather than merely incomplete (filed upstream as SoundMatt/cpp-FuSa#58). The `6-7.1`/`6-7.2`/`3-7.4`/`3-7.5` objectives that depend on it remain honest gaps for now. RELAY (re-checked against `git tag` in SoundMatt/RELAY at time of writing): - `README.md`/`requirements/requirements.json`/`.fusa-reqs.json` claimed spec v1.11; CI floated `@latest`. v2.0.0 is RELAY's current tagged release, but its go.mod doesn't declare the `/v2` module path suffix Go's semantic import versioning requires for a v2+ major version, so `go install .../relay@v2.0.0` fails outright (filed upstream as SoundMatt/RELAY#68) -- confirmed locally before picking a target. v1.14.0 is therefore the actual current installable latest. Pinned CI to `@v1.14.0` and bumped every `kSpecVersion`/`spec_version` string and doc citation to match (verified `relay conform --strict` / `relay interop --protocol LIN` both still PASS against v1.14.0 -- none of v1.12-v1.14 touch anything cpp-LIN implements) (closes #22). Docs: - ROADMAP.md's v0.1.0 checklist still said the master schedule runner used `std::stop_token`; the shipped API has used `std::atomic<bool>&` since commit fc1c046 (C++17 compat). Corrected (closes #24). - release.yml's macOS binary build has failed on every tagged release (v0.1.0-v0.4.0): its Configure step had no explicit generator (defaulting to Xcode's multi-config generator on macOS, which doesn't match a bare `ctest` invocation with no --config), and separately its Build step only built the `cpp-lin-cli` target, never `cpplin_tests` at all, on any OS. Added `-G Ninja` (matching ci.yml's own working pattern) and build all targets; fixed the now-config-subdirectory-free Windows artifact path; set `fail-fast: false` on the release matrix so one OS failing doesn't cancel the other two before they get a chance to run (closes #13). Closes #13, #19, #20, #21, #22, #23, #24 Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bundles the audit findings that all touch the same two CI workflow files
(and the docs/version strings that need to stay in sync with what CI
actually pins) — splitting these into separate PRs would have meant
repeatedly conflicting edits to the same job definitions.
cpp-FuSa (re-checked
gh release list --repo SoundMatt/cpp-FuSaat timeof writing — v0.17.1 is current latest):
fusa-asil-bandsarifjobs (closes CI pins cpp-FuSa to v0.15.0, missing conformance fixes through v0.17.1 that affect this repo's own generated evidence #21).cpfusa init ... --forcewas running before the traceability step andsilently wiping the real, checked-in
.fusa-reqs.json(hundreds of realREQ-*entries) with an empty scaffold every run — traceability coveragewas always computed against zero requirements. Dropped
--force(filesare already checked in) and the now-unneeded
|| true(closes fusa-asil-b CI job'scpfusa init --forcewipes the real .fusa-reqs.json before traceability runs, so coverage is always reported as 0% #19).|| true,and separately ran too early (before
boundary/tara/fmea/safety-case/sas/scihad produced anything), so most of their ownevidence was invisible to them. Reordered those steps to run last, added
the missing
cpfusa releasestep (sbom.json/provenance.json/artifact-manifest.jsonwere already listed in the evidence-upload stepbut never actually generated by anything), and added a real
CHANGELOG.md(closes release.yml templates a link to a nonexistent CHANGELOG.md (unreachable while the release job never completes) #23 for real, not just by deleting the dead link —and happens to also satisfy the ISO 26262 "change management" objective).
With the full evidence set in place and in the right order, gaps drop
from 16/20 → 11/20 (ISO 26262 ASIL-B) and to 10/18 (IEC 61508 SIL-2).
cpfusa iso26262/iec61508exit 1 whenever any objective is "gap" —that's real tool behavior, not a bug — so blanket
|| truewas hidinggenuine regressions along with a genuine cpfusa v0.17.1 limitation: 11 of
20 ASIL-B objectives (10 of 18 SIL-2) have no evidence-detection logic
at all in this tool version and report "gap" unconditionally regardless
of real project evidence (filed upstream: iso26262/iec61508 detect_status() has no evidence-detection case for ~40% of objectives — always 'gap' regardless of real evidence cpp-FuSa#57).
Replaced
|| truewith an explicit gate that fails only if the gap countregresses past that documented, verified floor (closes ISO 26262 / IEC 61508 gap-analysis steps in fusa-asil-b CI are wrapped in
|| true, so most unaddressed clauses never fail the build #20).cpfusa verifyto the job: its ctest-outputregex breaks on any multi-word test name (this repo's own convention) and
silently drops 170 of 171 real passing tests from
.fusa-evidence.json—actively misleading safety evidence, not merely incomplete (filed
upstream: verify: ctest-output regex breaks on multi-word test names — silently drops almost all results instead of failing cpp-FuSa#58). The objectives that depend on it remain
honest gaps for now.
RELAY (re-checked
git tagin SoundMatt/RELAY at time of writing):requirements.json/.fusa-reqs.jsonclaimed spec v1.11; CIfloated
@latest. v2.0.0 is RELAY's current tagged release, but itsgo.moddoesn't declare the/v2module path suffix Go's semantic importversioning requires for v2+, so
go install .../relay@v2.0.0failsoutright (filed upstream: v2.0.0 tag is unusable as a Go module: go.mod still says module github.com/SoundMatt/RELAY (needs /v2 suffix) RELAY#68 — confirmed locally before
picking a target). v1.14.0 is therefore the actual current installable
latest. Pinned CI to
@v1.14.0and bumped everykSpecVersion/spec_versionstring and doc citation to match (verifiedrelay conform --strict/relay interop --protocol LINboth still PASS againstv1.14.0 — none of v1.12–v1.14 touch anything cpp-LIN implements)
(closes README and requirements.json claim RELAY spec v1.11 conformance; actual current spec is v2.0, and CI floats @latest #22).
Docs:
ROADMAP.md's v0.1.0 checklist still said the master schedule runnerused
std::stop_token; shipped code has usedstd::atomic<bool>&sincecommit fc1c046 (C++17 compat). Corrected (closes ROADMAP.md still claims master schedule runner uses std::stop_token; shipped code uses std::atomic<bool>& #24).
release.yml's macOS binary build has failed on every tagged release(v0.1.0–v0.4.0): no explicit generator (defaulting to Xcode's
multi-config generator on macOS, mismatched with a bare
ctestinvocation), and separately its Build step only ever built the
cpp-lin-clitarget, nevercpplin_tests, on any OS. Added-G Ninja(matchingci.yml's own working pattern) and build alltargets; fixed the now-config-subdirectory-free Windows artifact path;
set
fail-fast: falseon the release matrix so one OS failing doesn'tcancel the other two before they get a chance to run (closes [P1] Release workflow's macOS binary build has failed on every tagged release (v0.1.0-v0.4.0) — ctest finds no built test executable #13).
Closes #13, #19, #20, #21, #22, #23, #24
Test plan
ctest— 171/171 passrelay conform --strict/relay interop --protocol LINagainstcpfusa-independent build — both PASS at RELAY v1.14.0fusa-asil-bjob sequence run locally against realcpfusav0.17.1 binary, in a fresh checkout, in the new step order — every
step exits 0;
iso26262/iec61508gates printgaps=11 <= 11/gaps=10 <= 10and passsbom.json/provenance.json/artifact-manifest.json(previouslynever generated) now exist after the added
cpfusa releasestep