Skip to content

fix: CI trustworthiness + cpp-FuSa/RELAY version currency - #27

Open
SoundMatt wants to merge 1 commit into
mainfrom
ci/fusa-version-and-conformance-currency
Open

fix: CI trustworthiness + cpp-FuSa/RELAY version currency#27
SoundMatt wants to merge 1 commit into
mainfrom
ci/fusa-version-and-conformance-currency

Conversation

@SoundMatt

Copy link
Copy Markdown
Owner

Summary

Bundles the audit findings that all touch the same two CI workflow files
(and the docs/version strings that need to stay in sync with what CI
actually pins) — splitting these into separate PRs would have meant
repeatedly conflicting edits to the same job definitions.

cpp-FuSa (re-checked gh release list --repo SoundMatt/cpp-FuSa at time
of writing — v0.17.1 is current latest):

RELAY (re-checked git tag in SoundMatt/RELAY at time of writing):

Docs:

Closes #13, #19, #20, #21, #22, #23, #24

Test plan

  • ctest — 171/171 pass
  • relay conform --strict / relay interop --protocol LIN against
    cpfusa-independent build — both PASS at RELAY v1.14.0
  • Full fusa-asil-b job sequence run locally against real cpfusa
    v0.17.1 binary, in a fresh checkout, in the new step order — every
    step exits 0; iso26262/iec61508 gates print gaps=11 <= 11 /
    gaps=10 <= 10 and pass
  • sbom.json/provenance.json/artifact-manifest.json (previously
    never generated) now exist after the added cpfusa release step
  • YAML-validated both changed workflow files

Bundles the audit findings that all touch the same two CI workflow
files (and the docs/version-strings kept in sync with what CI
actually pins/checks) -- splitting them into separate PRs would have
meant repeatedly conflicting edits to the same job definitions.

cpp-FuSa (re-checked against `gh release list` at time of writing --
v0.17.1 is current latest):
- Bump the pinned cpp-FuSa ref from v0.15.0 to v0.17.1 in both the
  fusa-asil-b and sarif jobs (closes #21). Verified the newer tool
  version's real behavior against this repo rather than just bumping
  blindly:
  - `cpfusa init ... --force` was running before the traceability
    step and silently wiping the real, checked-in `.fusa-reqs.json`
    (hundreds of REQ-* entries) with an empty scaffold, so live
    traceability coverage was always computed against zero
    requirements. Dropped `--force` (unneeded in CI -- the files are
    already checked in) and the now-unnecessary `|| true` (closes
    #19).
  - The ISO 26262 / IEC 61508 gap-analysis steps were wrapped in
    `|| true`, and separately ran too early in the job (before
    boundary/tara/fmea/safety-case/sas/sci had produced anything),
    so most of their own evidence was invisible to them. Reordered
    those steps to run last, added the missing `cpfusa release` step
    (sbom.json/provenance.json/artifact-manifest.json were already
    listed in the evidence-upload step but never actually generated
    by any step), and added a real CHANGELOG.md (closes #23 for
    real, not just by deleting the dead link -- and happens to also
    satisfy the ISO 26262 "change management" objective). With the
    full evidence set in place and in the right order, gaps drop
    from 16/20 to 11/20 (ISO 26262 ASIL-B) and to 10/18 (IEC 61508
    SIL-2).
  - `cpfusa iso26262`/`iec61508` exit 1 whenever any objective is
    "gap" -- that's real tool behavior, not a bug, so blanket
    `|| true` was hiding genuine regressions along with a genuine
    cpfusa v0.17.1 limitation: 11 of the 20 ASIL-B objectives (and
    10 of 18 SIL-2 objectives) have no evidence-detection logic at
    all in this tool version and report "gap" unconditionally
    regardless of real project evidence (filed upstream as
    SoundMatt/cpp-FuSa#57). Replaced `|| true` with an explicit gate
    that fails only if the gap count regresses past that documented,
    verified floor (closes #20).
  - Did NOT add `cpfusa verify` to the job: its ctest-output regex
    breaks on any multi-word test name (this repo's own convention)
    and silently drops 170 of 171 real passing tests from
    `.fusa-evidence.json`, which would be actively misleading
    safety evidence rather than merely incomplete (filed upstream as
    SoundMatt/cpp-FuSa#58). The `6-7.1`/`6-7.2`/`3-7.4`/`3-7.5`
    objectives that depend on it remain honest gaps for now.

RELAY (re-checked against `git tag` in SoundMatt/RELAY at time of
writing):
- `README.md`/`requirements/requirements.json`/`.fusa-reqs.json`
  claimed spec v1.11; CI floated `@latest`. v2.0.0 is RELAY's current
  tagged release, but its go.mod doesn't declare the `/v2` module
  path suffix Go's semantic import versioning requires for a v2+
  major version, so `go install .../relay@v2.0.0` fails outright
  (filed upstream as SoundMatt/RELAY#68) -- confirmed locally before
  picking a target. v1.14.0 is therefore the actual current
  installable latest. Pinned CI to `@v1.14.0` and bumped every
  `kSpecVersion`/`spec_version` string and doc citation to match
  (verified `relay conform --strict` / `relay interop --protocol LIN`
  both still PASS against v1.14.0 -- none of v1.12-v1.14 touch
  anything cpp-LIN implements) (closes #22).

Docs:
- ROADMAP.md's v0.1.0 checklist still said the master schedule runner
  used `std::stop_token`; the shipped API has used
  `std::atomic<bool>&` since commit fc1c046 (C++17 compat). Corrected
  (closes #24).
- release.yml's macOS binary build has failed on every tagged release
  (v0.1.0-v0.4.0): its Configure step had no explicit generator
  (defaulting to Xcode's multi-config generator on macOS, which
  doesn't match a bare `ctest` invocation with no --config), and
  separately its Build step only built the `cpp-lin-cli` target, never
  `cpplin_tests` at all, on any OS. Added `-G Ninja` (matching
  ci.yml's own working pattern) and build all targets; fixed the
  now-config-subdirectory-free Windows artifact path; set
  `fail-fast: false` on the release matrix so one OS failing doesn't
  cancel the other two before they get a chance to run (closes #13).

Closes #13, #19, #20, #21, #22, #23, #24

Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment