Introduction: the wrong place to start
The instinct when automating in a customer's closed environment is to port the regression suite. It is the biggest asset, it represents the most work, and it is what the team is proud of. It is also the worst possible first delivery.
A regression suite in an unfamiliar environment produces dozens of failures on the first run, almost none of which are the application's fault: a certificate the browser does not trust, a clock that drifts, 64 MB of shared memory, an integration that does not exist here, a service account that expired last week. The customer sees a wall of red and concludes that the automation does not work.
Sanity checks are the right first delivery, and not as a consolation prize. They are the smallest artefact that proves the entire chain end to end, and in a closed environment proving the chain is the hard part.
This piece belongs to a series on test automation in closed environments. The overview gives the environment smoke test one paragraph. This is the long version: a precondition suite you can copy, the trust-store trap it exists to catch, and the arithmetic that keeps it small.
1The chain, link by link
Between "we have tests" and "someone acts on a result" there are six links. Every one of them can break on its own, in an environment you cannot see:
| Link | What has to be true | How it breaks behind an air gap |
|---|---|---|
| Binary | The runner and its browsers are on the machine, at the version you shipped | The browser cache never arrived, or does not match the installed Playwright (installing offline, keeping it current) |
| Environment | Libraries, fonts, shared memory, clock, disk, trust store | 64 MB of /dev/shm, a private CA, a clock minutes off (the full catalogue) |
| Configuration | Application address, credentials, feature flags | A base URL from another environment, an expired service account |
| Execution | Something starts a run without you | A cron entry on your laptop, a terminal session someone closed |
| Evidence | A report that survives the machine and opens offline | Written inside a container that no longer exists, or fetching its fonts from a CDN |
| Delivery | A person receives the result, or can reach it | Nobody subscribed, the mail relay is not permitted, the report sits on a host nobody can log into |
The regression suite exercises exactly one of these, execution, and only after the other five already work. A sanity set exercises all six with a handful of assertions, which is why it belongs first.
2What belongs in the set
Small, fast, and boring by design. If it takes longer than five minutes, or fails for reasons that need a meeting, it is not a sanity check any more.
Environment preconditions. Before anything touches the application: is the address answering, does the browser trust its certificate, is the clock within tolerance of the application host, is there shared memory for a browser and disk for a report.
One authentication path. Whatever identity mechanism this environment uses, exercised once. It catches an expired service account earlier than anything else, and an expired service account in a closed environment is a ticket that takes days.
One critical business path, end to end. Not a comprehensive one. The single flow whose failure would mean the deployment is not usable. If it has to write data, it writes records it can recognise and remove, and the customer knows they exist.
One report artefact. Produced, written to disk, and openable on a machine with no network. A run that passes and leaves nothing behind has not proven the chain.
That is four groups and perhaps a dozen assertions. Resist the urge to grow it: the value is in its stability, and every addition is one more way for it to fail for reasons the customer will read as noise. Section 7 puts a number on that.
3Preconditions as a project, not a describe block

The difference between a sanity run that helps and one that adds confusion is whether the environment assertions run first and stop everything else when they fail. A run that reports 48 failures is noise. A run that reports one failed precondition and nothing else is an answer.
A test.describe('preconditions') at the top of a file does not do that: the other tests run anyway and time out one by one. Playwright's project dependencies do. A project that depends on another starts only when every test in it has passed:
// playwright.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
use: { baseURL: process.env.APP_URL },
reporter: [['html', { open: 'never' }], ['junit', { outputFile: 'results.xml' }]],
projects: [
// Runs first. If any of it fails, nothing below starts.
{ name: 'preconditions', testMatch: /preconditions\.spec\.ts/ },
// One sign-in, saved for every sanity test after it.
{ name: 'login', testMatch: /login\.setup\.ts/, dependencies: ['preconditions'] },
{
name: 'sanity',
testMatch: /sanity\.spec\.ts/,
dependencies: ['login'],
use: { storageState: '.auth/state.json' },
},
],
});The preconditions themselves are short. Each assertion carries a message, because the message is what the person reading the report will see:
// preconditions.spec.ts
import { test, expect } from '@playwright/test';
import { statfsSync } from 'node:fs';
const freeMb = (path: string) => {
const fs = statfsSync(path);
return Math.floor((fs.bavail * fs.bsize) / 2 ** 20);
};
test('the application answers', async ({ request }) => {
const res = await request.get('/health', { timeout: 10_000 });
expect(res.status(), 'GET /health').toBe(200);
});
test('the browser trusts the certificate', async ({ page }) => {
// Not the same check as the one above: the API request and the browser
// read different trust stores, and only the browser runs the suite.
await page.goto('/');
});
test('the clock agrees with the application host', async ({ request }) => {
const res = await request.get('/health');
const skew = Math.abs(Date.now() - Date.parse(res.headers()['date'])) / 1000;
expect(Math.round(skew), 'seconds between this clock and the application host')
.toBeLessThan(60);
});
test('the browser has enough shared memory', () => {
test.skip(process.platform !== 'linux', '/dev/shm is a Linux mount');
expect(freeMb('/dev/shm'), 'MB free in /dev/shm').toBeGreaterThanOrEqual(1024);
});
test('there is room for the report', () => {
expect(freeMb('.'), 'MB free where the report is written')
.toBeGreaterThanOrEqual(2048);
});Run it with npx playwright test --project=sanity. Playwright runs the projects it depends on first, so nobody has to remember the order.
We ran this set in the official Playwright 1.59 image, with the network switched off and the test application's clock moved 214 seconds ahead. The result was one failed test, the clock, reported as seconds between this clock and the application host with Received: 214 underneath, and the login and sanity projects listed as did not run. In the JUnit file those two appear as skipped, so a CI dashboard shows one failure rather than a page of timeouts.
Three details that matter in practice:
- The
Dateheader has one-second resolution and comes from whatever terminates HTTP. If a reverse proxy sits in front of the application, you are measuring the proxy's clock. Say so in the message, so nobody chases the wrong host. - 1 GB of
/dev/shmis a floor, not a recommendation. A container gets 64 MB unless told otherwise, andshm_sizeoripc: hostfixes it. The catalogue of offline failures explains why its absence looks like a race condition rather than a resource limit. - The free-disk check looks where the report is written, not at
/. A full data volume on a host with an empty root partition is a common way to lose the evidence of a run that otherwise passed.
Thirty seconds of checks that turn a category of confusing failures into one readable sentence. It is the highest-value code in the entire delivery, and it is usually the part nobody writes.
4The trust store the suite actually uses
The second precondition deserves a section of its own, because it is the one most teams get wrong in the same way.
In a closed environment the application's certificate is almost always issued by the customer's own CA. Someone adds that CA to the runner, the API calls start working, and everyone moves on. Then every browser test fails with net::ERR_CERT_AUTHORITY_INVALID.
The API request and the browser do not read the same trust store. We tried three ways of installing a private CA in the official Playwright image (Ubuntu 22.04, the Chromium that ships with Playwright 1.59), against a test application whose certificate that CA had signed:
| CA installed in | request.get() | page.goto() |
|---|---|---|
NODE_EXTRA_CA_CERTS only | trusted | ERR_CERT_AUTHORITY_INVALID |
The system store only, via update-ca-certificates | rejected, with a hint to use --use-system-ca | ERR_CERT_AUTHORITY_INVALID |
The NSS database, plus NODE_EXTRA_CA_CERTS | trusted | trusted |
On Linux, Chromium takes locally added CAs from the NSS database in the home directory of the user it runs as, not from the system bundle. Node carries its own list and reads neither unless told to. A setup that works for both:
# certutil comes with libnss3-tools. Run it as the user the tests run as.
mkdir -p "$HOME/.pki/nssdb"
certutil -d "sql:$HOME/.pki/nssdb" -N --empty-password
certutil -d "sql:$HOME/.pki/nssdb" -A -t "C,," -n "internal-ca" -i internal-ca.pem
# And for the API requests in the same suite:
export NODE_EXTRA_CA_CERTS=/etc/pki/internal-ca.pemTwo consequences. The precondition has to go through page, not request, or it passes on exactly the machine where the suite will fail. And the tempting fix, ignoreHTTPSErrors: true, makes the precondition pass by switching off the thing it checks. In a customer environment it is also a finding waiting for the security review.
5A failure the customer can act on
The point of a readable precondition is that its failure has an owner, and the owner is not you. Write the runbook as a table the customer can follow without calling anyone:
| The report says | Whose it is | What they do |
|---|---|---|
GET /health is not 200 | Whoever deployed the application | The deployment is not up; there is nothing to test yet |
ERR_CERT_AUTHORITY_INVALID | Platform team | Add the CA to the browser's store on the runner, as above |
| Seconds between this clock and the application host | Infrastructure | Fix time sync on whichever of the two hosts drifted |
MB free in /dev/shm | Whoever runs the container | shm_size or ipc: host in the runner's definition |
| MB free where the report is written | Operations | Retention, or a bigger volume |
| The service account cannot sign in | Identity team | Expired password, locked account, or a missing group |
| The critical path fails, preconditions passed | Application team | This one is a real result |
Only the last row is about the application, and that is the point. By the time the critical path runs, everything else has been ruled out, and a red result means what it says.
6Why this is the right first delivery
It fails in ways the customer can act on. Every failure points at something in their environment that they own and can fix, rather than at your test code.
It proves the chain, not the coverage. Once a sanity run is scheduled, executes, reports and reaches a person, adding the regression suite is a change of scope rather than a new integration.
It survives the feedback loop. A small, stable set is something the customer will actually run at each deployment. A large brittle one gets skipped exactly when it matters.
It sets the expectation correctly. The first automation the customer sees behaves predictably. That is worth more, politically, than demonstrating breadth.
7A feedback loop measured in days, and what it does to suite size
In a closed delivery model the loop between a change and its result is commit, package, transfer, maintenance window, run on site, report sent back, analysis. Three days if it goes well. That changes what a red run costs: each one is a loop, not a re-run.
A rough model makes the consequence concrete. Say every test, independently, fails for a reason that is not the application once in a hundred runs: a slow page, a lock held by another team's job, a test data collision. The chance that a whole run comes back green falls with every test you add, and the number of runs you wait, on average, for a green one rises:
| Tests | Whole run green | Runs until a green one | At three days a loop |
|---|---|---|---|
| 12 | 89% | 1.1 | 3.4 days |
| 40 | 67% | 1.5 | 4.5 days |
| 300 | 5% | 20 | about two months |
The numbers are illustrative. Real failures cluster rather than arrive independently, and one in a hundred is generous for a suite that has never run in this environment. The shape is the point: at the customer's site the size of a suite is a cost, not an achievement. The sanity set stays at a dozen assertions, and the regression suite runs upstream, on staging, where a red run costs minutes. The overview's distribution of effort puts only smoke and acceptance tests in the customer row for exactly this reason.
8Where it runs, and who can start it
Three moments, in increasing order of value:
After every deployment, automatically, before anyone is told the deployment succeeded. This is the one that changes behaviour, because it turns "the deployment finished" into "the deployment works".
On a schedule, nightly or hourly depending on how much changes underneath. It catches expired certificates, filled disks and drifting clocks before a user does.
On demand, by someone who is not you. If running the checks requires your presence, you have built a demo rather than a capability.
That last one is the real test of the delivery. A set that only the vendor can start is not automation from the customer's point of view. Handing it over means one command or one button, credentials supplied by the environment rather than typed in by whoever runs it, the report in a known place, and the table from section 5 next to it.
9Growing from here
The order that works, once the sanity set is stable and running unattended:
- Widen the critical path into a small acceptance set, still under ten minutes.
- Add the regression suite, but run it on a schedule rather than per deployment.
- Add cross-run history, because by now there is enough data for trends to mean something.
- Only then consider the long tail: performance, compatibility, matrix runs.
Each step happens on a chain that already works. That is the whole point of starting small: none of these steps is an integration project, because the integration was done in week one with a dozen assertions.
10Where a runner fits, and where vallus does
Everything above can be done with the framework alone and a scheduler. What a runner adds in a closed environment is the parts around the test, and those are the parts this series has been about: a suite that installs offline, takes its browsers from an internal mirror, updates deliberately, vendors its dependencies, reports without a cloud and survives a security review. Sanity checks are where all of that becomes visible to the customer for the first time.
vallus is the runner we built for that case. For a sanity set specifically:
- A workflow runs its steps in order and, by default, stops at the first one that fails. The project chain above runs as a single step; a separate preconditions step in front of the suite works too.
- Three ways to start it, none of them yours. A schedule for the hourly or nightly run, a token-authenticated webhook the deployment pipeline calls when it finishes, and a Run button for anyone with the executor role and permission for that workflow.
- Credentials the person pressing Run never sees. A workflow variable is a value, a secret encrypted with the instance key, or, from the Team tier up, a reference to OpenBao or HashiCorp Vault resolved when the run starts.
- Delivery is part of the workflow. Notifications are chosen per workflow, on failure by default, and go to Slack, Teams, any webhook or e-mail, so an internal chat server or mail relay is enough.
- The runner checks its own side. A strip on the home screen reports the Playwright version the project pins, missing browsers, the mirror, free disk and the licence. Grey means it could not tell, never that all is well.
- Evidence stays on the customer's disk. Playwright HTML or Allure reports, traces and run history, with flaky tests and newly failing specs picked out across runs.
What it does not do: it does not write your preconditions, and it knows nothing about the application's clock, certificate or health beyond what your suite asserts. It is not an uptime monitor either: between scheduled runs, nothing is watching.
To see a sanity set running behind an air gap, book a demo and ask for the 21-day evaluation key, which has every feature. The Free licence (two users, one project, one run at a time) is enough for exactly one sanity set, with scheduling, containerised runs, reports and history included.
11Conclusion
Starting with the regression suite optimises for what you already built. Starting with sanity checks optimises for what has to be proven first, which in a closed environment is not coverage but the chain itself.
- A dozen assertions that exercise all six links beat a thousand that exercise one.
- Preconditions belong in the suite, as a project the rest depends on, checked through the same browser and trust store the suite uses, and failing with a sentence someone else can act on.
- Someone other than you must be able to run it. Until that is true, there is a demo rather than a delivery.
Download the offline copyOne HTML file with everything inside - opens with no internet at all.
