KENSAI Product Update: Nightly Test Evidence Replaces Comfort Metrics

April 26 started with the kind of failure that useful systems should preserve instead of smoothing over: the nightly freemium test suite did not pass. The win is that the failure is now precise enough to fix.

What the nightly run proved

The April 26 cron receipt recorded the full KENSAI freemium suite from the platform repository. The result was not a vague red build. It was a concrete failure map: pnpm test exited with code 1 after 636 tests, with 310 failed, 243 passed, and 83 skipped.

The dominant failure class was API-backed tests expecting a service on 127.0.0.1:4000 while the API was not available, producing connection refusals and status-zero assertions. That is not a product mystery. It is a missing preflight and service orchestration defect.

What was missing

Two more gaps were recorded instead of hidden. The root package did not define pnpm test:e2e, so the runner had no stable e2e entry point. The coverage command failed immediately because @vitest/coverage-v8 was missing. A direct Playwright fallback did start, but broad auth and admin failures plus one-minute validation timeouts made it noisy enough to kill rather than misread as useful signal.

Why this is a product update

A failing test run is not automatically progress. A failing test run with counts, dominant error classes, missing tools, and the next repair path is progress. The quality system now knows the difference between application behavior that is failing and infrastructure that was never started.

That matters for KENSAI because customer-facing security claims depend on boring operational truth. If the freemium path needs the API, the runner must start it or verify it before tests begin. If coverage is a required gate, the provider must be installed. If e2e is part of the release story, it needs a real script instead of an implied convention.

Next repair path

The repair sequence is now clear: add or restore a root test:e2e script, install the coverage provider, and make the freemium runner start and health-check the API on port 4000 before API-dependent tests execute. Anything else would be theatre.

This is the standard KENSAI should keep: preserve the uncomfortable measurement, name the broken preconditions, and turn a red run into a short fix list.

Bottom line

The useful standard is simple: claims become real when the prerequisite, artifact, and route all line up. Today’s work keeps that standard visible.