case-studyautomotiveCI/CD

Integrating Timing Verification Tools into Automotive CI: A Vector/RocqStat Case Study

mmytest

2026-02-18

9 min read

Practical walkthrough: how an automotive team integrated RocqStat into VectorCAST CI to enforce WCET and halve regression time.

Hook: When timing faults stop the pipeline

Flaky tests, late-stage timing failures, and week-long regression runs are the symptoms. For automotive teams delivering safety-critical ECUs, the real problem is predictable timing: you need to prove worst-case execution time (WCET) early and continuously. This case study walks through how an automotive team integrated RocqStat into VectorCAST and their CI server to enforce timing constraints, speed regression runs, and make timing verification a first-class, automated gate in 2026.

Executive summary (most important first)

In late 2025–early 2026, Vector Informatik's acquisition of StatInf's RocqStat accelerated integrated timing analysis inside the Vector testing ecosystem. Our customer-engineering walkthrough shows how an automotive software team:

Integrated RocqStat into VectorCAST projects and CI pipelines;
Automated WCET measurement and statistical timing verification as a CI gate;
Cut regression wall time by ~40–60% through delta-WCET analysis, test selection, and caching;
Detected two high-risk timing regressions before integration testing, avoiding costly HIL rework.

Why this matters in 2026

Software-defined vehicles and rising ASIL requirements put timing safety squarely on the critical path. Since Vector's 2026 acquisition of RocqStat, unified toolchains that combine MC/DC/unit testing with timing-aware WCET estimation are now production-ready. Teams that put timing analysis into CI early reduce risk and regulatory friction, and they lower the expensive feedback loop caused by late detection of timing regressions.

"Timing safety is becoming a critical ..." — Eric Barton, Senior VP, Code Testing Tools, Vector

Team & project context (customer-engineering snapshot)

The team: 12 embedded software engineers, 3 test engineers, and 1 DevOps engineer. The product: a mid-range domain controller ECU with mixed-critical tasks (control loops at 1 ms, diagnostics, and communications). Legacy toolchain used VectorCAST for unit and integration tests, but timing verification was ad-hoc: instrumentation-based measurements on target hardware and slow HIL runs.

Objectives:

Make WCET estimation repeatable and automated;
Fail CI on significant timing regressions before HIL;
Reduce wall-clock regression time while keeping tight coverage and timing guarantees.

Prerequisites & versions

Before integration, the team standardized the environment. Key prerequisites:

VectorCAST 11.5+ with the VectorCAST Automation CLI installed;
RocqStat toolchain (post-acquisition distribution provided by Vector) with CLI and plugin support;
Cross-toolchain for target (GCC for embedded target or vendor toolchain);
CI server (GitLab Runner 16.x or Jenkins LTS 2.400+);
Artifact storage (Nexus/Artifactory) for caching test binaries and timing profiles; see notes on artifact storage and cost trade-offs when planning retention.
Hardware-in-the-Loop (HIL) lab for final validation (optional but recommended).

Step 1 — Create a timing-aware VectorCAST project

The first engineering task was to extend VectorCAST projects to emit timing metadata consumable by RocqStat. The team added a build step that compiles with timing instrumentation where necessary and ensures symbol and map files are preserved.

Key actions:

Enable Map file generation in the linker flags (e.g., -Wl,-Map=app.map).
Keep symbol information for correlation (-g, but limited on production builds).
Build a lightweight instrumentation mode: full instrumentation only for stress test runs; sampling instrumentation for CI unit tests.

Example build flags (GCC-style):

// local build wrapper
CFLAGS="-O2 -g -fno-omit-frame-pointer"
LDFLAGS="-Wl,-Map=app.map"
make clean all CFLAGS="$CFLAGS" LDFLAGS="$LDFLAGS"

Step 2 — Add RocqStat analysis into VectorCAST runs

VectorCAST runs produce test execution traces and coverage artifacts. RocqStat requires either binary-level traces or execution traces plus binary metadata to run statistical WCET estimation. The team chose a two-mode approach:

CI mode: fast, conservative statistical WCET estimation using trace samples and previously-collected execution profiles;
Nightly/full mode: exhaustive RocqStat analysis using extended tracing on target hardware.

Integration pattern:

VectorCAST build & run (unit/integration tests) → produce coverage and lightweight traces;
Invoke RocqStat CLI to merge traces and compute a delta-WCET relative to baseline;
Publish RocqStat report (JSON + HTML) as a CI artifact and fail pipeline on policy breaches.

Sample CLI workflow

# Example (simplified) - run inside CI job
# 1. Run VectorCAST tests
vectorcast-cli run --project my_vcast_proj --automation

# 2. Export execution traces
vectorcast-cli export-traces --project my_vcast_proj --out traces/

# 3. Run RocqStat analysis
rocqstat analyze --binaries build/bin/app.elf --traces traces/ --out rocq_reports/

# 4. Emit JSON summary
rocqstat summary --in rocq_reports/ --format json --out rocq_summary.json

Step 3 — CI pipeline integration (GitLab example)

The team chose GitLab CI for branch-level gating. Their pipeline had three key stages: build, unit tests + timing analysis, and merge gates. The critical innovation was a delta-WCET check that compares the new WCET to the accepted baseline and fails if the increase exceeds thresholds.

stages:
  - build
  - test
  - timing

build:
  stage: build
  script:
    - ./build.sh
  artifacts:
    paths:
      - build/bin/app.elf

unit_tests:
  stage: test
  script:
    - vectorcast-cli run --project my_vcast_proj --automation
  artifacts:
    paths:
      - traces/

timing_analysis:
  stage: timing
  script:
    - rocqstat analyze --binaries build/bin/app.elf --traces traces/ --out rocq_reports/
    - rocqstat compare --new rocq_reports/summary.json --baseline artifacts/baseline_wcets.json --threshold 5 --exit-on-violation
  dependencies:
    - unit_tests
  artifacts:
    paths:
      - rocq_reports/

Key CI behaviors implemented:

Fail early if delta-WCET > threshold (e.g., 5% or an absolute microsecond bound);
Publish human-readable WCET HTML + JSON artifacts for traceability;
Record per-function WCET deltas so developers see precise change sources.

Step 4 — Gating policy and actionable reporting

Automating timing does not mean noisy failures. The team designed gating policy to be deterministic and useful:

Hard gate: absolute violation of a safety-critical path (pre-defined by code review) always fails CI;
Soft gate: small delta increases (e.g., <5%) trigger warnings and create a ticket but do not block merges;
Delta review: merges that change WCET for more than N functions require explicit engineering approval;
Nightly full-run: more aggressive checks on nightly builds where full RocqStat analysis runs on target hardware.

Reports included:

Per-function WCET (mean, p95, p99)
Call-stack annotated hotspots
Source-linked diffs highlighting new code paths that increased WCET

Step 5 — Speeding up regressions: delta analysis, caching, and selection

Running full timing analysis on every commit would be expensive. The team used three complementary strategies to reduce wall time while maintaining safety:

Delta-WCET — only analyze functions and modules touched by the commit and their callers. RocqStat's merging of traces allowed focused analysis with statistical confidence.
Binary caching — store compiled artifacts and instrumentation profiles in Artifactory; reuse when compile inputs are unchanged. See our notes on artifact storage and edge cost trade-offs when sizing caches.
Regression selection — prioritize tests that exercise modified code paths (VectorCAST test-to-source mapping enabled this).

Result: the median CI timing-analysis stage dropped from ~45 minutes to ~12–18 minutes for typical feature branches.

Step 6 — Handling uncertainty and statistical WCET

RocqStat provides statistical WCET estimation, which is inherently probabilistic. The team used a mixed approach:

Use statistical WCET in CI for fast feedback and to catch regressions;
Require deterministic WCET assurance on release branches using full target instrumentation and worst-case path analysis (e.g., path-sensitive static analysis) when necessary for ASIL D artifacts;
Document confidence levels and assumptions in the test report artifacts for traceability to ISO 26262 evidence.

Results — measurable business and engineering outcomes

After three months of integration and tuning, the team observed:

Regression time cut by ~50% (from 6 hours nightly to 3 hours), with branch CI timing-analysis stage typically under 20 minutes;
Early detection of two timing regressions caused by new error-handling logic that would have failed late on HIL; fixing them in unit-level CI saved an estimated 3–4 days per bug in schedule;
Improved confidence in WCET budgets, reducing conservative overprovisioning and saving CPU budget on the ECU task scheduler;
Clear traceability: all WCET artifacts tied to commits for audit and cert evidence.

Common pitfalls and how we mitigated them

Noisy instrumentation: excessive instrumentation slowed tests. Mitigation: switched to selective sampling instrumentation for CI.
False positives from statistical variance: short trace sets produced spurious spikes. Mitigation: require repeated CI samples or run quick randomized micro-benchmarks to increase confidence.
Toolchain drift: differing compiler flags between CI and target distorted WCET. Mitigation: enforce build matrix parity and store toolchain container images.
Trace size & storage: large trace artifacts filled storage quickly. Mitigation: compress traces and store only summaries for short-lived branches; keep full traces for release branches. (Also consider cache-testing practices from cache testing when tuning retention/eviction.)

Advanced strategies & 2026 trends to adopt

Looking forward in 2026, teams should consider:

Multi-core timing analysis: as domain controllers use multi-core designs, combine RocqStat's probabilistic models with scheduler-aware analysis tools;
AI-assisted test selection: use machine learning to predict the tests that most affect WCET and prioritize them in CI; see practical guides on automating selection with AI.
Hardware-in-loop (HIL) hybrid runs: combine fast ROCQ/VectorCAST CI checks with weekly HIL full-WCET verification for final assurance;
Policy as code: encode WCET thresholds, ASIL ties, and per-function safety levels into policy modules that CI enforces automatically; governance patterns discussed in the governance playbook are applicable here.

Sample Jenkinsfile snippet for teams using Jenkins

pipeline {
  agent any
  stages {
    stage('Build') {
      steps { sh './build.sh' }
    }
    stage('Unit & VectorCAST') {
      steps { sh 'vectorcast-cli run --project my_vcast_proj --automation' }
    }
    stage('RocqStat Analyse') {
      steps {
        sh 'rocqstat analyze --binaries build/bin/app.elf --traces traces/ --out rocq_reports/'
        sh 'rocqstat compare --new rocq_reports/summary.json --baseline baseline_wcets.json --threshold 5 --exit-on-violation'
      }
    }
  }
}

Lessons learned & best practices

Start with a small scope: integrate RocqStat on a single module to validate the CI flow before scaling. See guidance on starting small in the hybrid micro-studio playbook.
Keep CI and release builds parity: mismatched flags are the leading cause of timing surprises.
Use delta analysis aggressively to keep CI fast and actionable.
Invest in clear, developer-friendly reports. Per-function diffs accelerate fixes.
Maintain traceability for compliance: store baseline WCETs, thresholds, and approvals alongside artifacts.

Actionable takeaways

Automate WCET checks in CI with a delta-WCET gate to catch regressions early.
Use two modes: fast statistical checks in CI and deeper target-based analyses on release branches.
Prioritize tests that exercise modified code using VectorCAST mappings to reduce analysis time.
Define clear gating policies (hard vs soft) and document them for audits and teams.
Leverage caching & artifacts to reduce unnecessary rebuilds and accelerate CI.

Final thoughts & future-proofing

In 2026, integrating timing verification into CI is not optional — it's expected for teams building safety-critical automotive software. The Vector/RocqStat integration enables practical, automated timing checks that scale with modern CI practices. The customer-engineering approach in this case study demonstrates that a pragmatic combination of statistical WCET in CI and deterministic analysis on releases yields both speed and safety.

Call to action

If your team faces flaky timing, slow regressions, or certification pressure, start small: pick a single module, set a baseline WCET, and add RocqStat checks to your VectorCAST CI job. Need help implementing the pipeline, tuning thresholds, or preparing artifacts for ISO 26262? Contact our customer-engineering team for a prescriptive workshop and a tailored integration plan to get timing verification working in your CI in 4–6 weeks.

mytest

Contributor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.