Back vallusby ExpandIt The password is in the trace
An emergency exit gate at an elevated railway station

Closed environments · test artefacts

The password is in the trace

What test artefacts carry when they leave a closed environment, and an export that takes only what may cross

Test artefacts · data protection · exports

Woodhaven Bl Jamaica Av 15 - BMT Jamaica.jpg - Tdorante10, CC BY-SA 4.0 (Wikimedia Commons)

Introduction: everyone watches the wrong door

Most of the effort around closed environments points inwards. How to get browsers in. How to get packages in. How to get an approved build in without a route to the internet. Those are the problems people write about, including me, and they are real.

They are also the cheap half. Getting an unreviewed binary into an isolated network costs that network an unknown artefact. Getting test data out of it can be a data protection incident or a breach of contract, depending on the sector and on what exactly moved.

The outbound direction gets far less attention because inbound failures are loud. A missing browser stops the pipeline and someone raises a ticket. Outbound movement succeeds quietly. Somebody attaches a trace to a defect, mails a report to a manager or copies a results folder to a laptop, and the file arrives exactly as intended. Nobody looks at it again until an audit asks where it went.

This piece belongs to a series on test automation in closed environments. The overview covers what a testing tool sends out on its own, and how to prove it sends nothing. This text is about the files people move by hand. I planted six values in a test run and then searched every file the run left behind, so what follows is a measurement and not a list of worries.


1What actually leaves

"Data left the environment" sounds like someone copying a database. In practice it is almost never that, and the mundane version is what makes it hard to see.

What movesWhy somebody moved it
The HTML reportA manager or a customer wants to see the result
A trace fileDebugging something that only fails there
Screenshots and videoAttached to a defect in the tracker
JUnit XML or JSON resultsImported into test management or a dashboard
The run's logSent to someone who can read it
A saved login stateCopied so a colleague can reproduce a failure

None of it looks like data at the moment it moves. It looks like work product. The question is what each of those files contains, and the person moving one is usually the last to know.


2Six planted values, one failing test

The setup was small on purpose. A tiny web application, the official Playwright 1.59.1 image and no network. The application holds six synthetic values, each tagged so it can be found again:

One test signs in, opens the account page and fails on purpose, because failures are when a run keeps its artefacts. The configuration is the one most projects start from: trace: 'retain-on-failure', screenshot: 'only-on-failure', video: 'retain-on-failure', and the HTML, JSON and JUnit reporters. After the run, a script searched every file it left, inside zip archives and inside the report's embedded data as well.

ArtefactWhat turned up in it
Trace (trace.zip)All six
HTML reportAll six: it embeds the trace
Screenshot on failureName and IBAN, on the image
VideoName and IBAN, frame by frame
error-context.mdName and IBAN
JSON reportIBAN, in the error message
JUnit XMLIBAN, in the failure message

The screenshot and the video are images, so no text search finds anything in them. I found the values there by looking. That is worth remembering when a tool advertises that it scans artefacts for secrets.


3The trace is a recording, not a picture

A trace is easy to think of as a richer screenshot. It is a recording of everything the browser did, and in this run it held the password in five separate places:

The token was in the Authorization header of the request that carried it. The cookie was on every request after login. The unshown field was in the stored response body: never on screen, and still in the file. The IBAN appeared in the page snapshots and seven times in the call log of the failing assertion, which prints what it found on every retry.

The trace options can shrink this, but not make it safe. With trace: { mode: 'retain-on-failure', snapshots: false, screenshots: false, sources: false } the network log went too, so the token, the cookie and the unshown field were gone. The password was still there three times (the action, the step title and the call log), and the IBAN was still in the assertion's call log. A smaller trace is still a trace.


4The report inherits everything

The HTML report is the artefact most often sent to someone outside, because it opens in any browser. It is also the one that collects the rest.

It embeds the trace and the screenshots of every failed test, so one file carries all six values.

It lists the steps of every test, including those that passed. In a second run, the login moved into a setup project with tracing switched off and its test passed. The report still showed Fill "…" getByLabel('Password'), with the real password, in that test's step list. Turning the trace off does not touch the step titles.

The JSON and JUnit results carry the error message, and an assertion's message includes what it received. The JUnit file is what gets imported into test management, so the IBAN travels on into the tracker with it.

error-context.md is written next to each failure: a text snapshot of the page, convenient for reading a failure without opening the trace. Here it held the name and the IBAN.

Two files people rarely think of complete the picture:


5What can legitimately leave

More than teams assume, which is why the informal route so often turns out to have been unnecessary.

Counts and durations. Pass and fail totals, durations, flake rates. Numbers about tests are rarely sensitive. The sensitive part is what the tests touched. A dashboard fed by counts satisfies most management reporting without a single artefact leaving.

Test titles, after reading them. The title of the test above was account shows the right IBAN. That is harmless; titles that name a customer, a product nobody has announced yet or an internal system are not. Read them before publishing them to a shared tool.

Findings, described rather than attached. "The account page shows the wrong IBAN after a currency change" travels fine. The trace that proves it does not.

Artefacts of a run on synthetic data. If the environment runs on generated data with a documented seed, most objections to moving artefacts disappear. This is the strongest argument for investing in synthetic data in the first place.

The practical test is not "is this sensitive" but "does anyone outside this environment have a documented right to hold it". The overview's section on the legal and contractual dimension covers why personal data, contract clauses and sector rules all end up asking that question.


6Keep it out of the artefacts in the first place

Masking that happens before sending depends on somebody remembering, and eventually somebody is in a hurry. What can be done when the artefact is written is worth doing there. Three changes covered most of what the experiment found.

Sign in through the API, in a project with no trace. The password then never passes through a fill, so it has no step title, no snapshot and no call log:

typescript
// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  use: {
    baseURL: process.env.APP_URL,
    trace: 'retain-on-failure',
    // The automatic screenshot cannot mask anything: see afterEach below.
    screenshot: 'off',
    video: 'off',
  },
  projects: [
    // Signs in once. Nothing in this project is traced.
    { name: 'login', testMatch: /login\.setup\.ts/, use: { trace: 'off' } },
    {
      name: 'tests',
      testMatch: /.*\.spec\.ts/,
      dependencies: ['login'],
      use: { storageState: '.auth/state.json' },
    },
  ],
});
typescript
// tests/login.setup.ts
import { test as setup } from '@playwright/test';

setup('sign in', async ({ request }) => {
  // A form post rather than a typed field: no step title, no input snapshot.
  await request.post('/login', {
    form: { user: process.env.APP_USER!, password: process.env.APP_PASSWORD! },
    maxRedirects: 0,
  });
  await request.storageState({ path: '.auth/state.json' });
});

Take the failure screenshot yourself, masked. In Playwright 1.59 the automatic screenshot option accepts only fullPage and omitBackground. There is no mask. An explicit page.screenshot() has one, which paints the given elements over:

typescript
test.afterEach(async ({ page }, testInfo) => {
  if (testInfo.status !== testInfo.expectedStatus) {
    await testInfo.attach('failure', {
      body: await page.screenshot({
        mask: [page.getByTestId('name'), page.getByTestId('iban')],
      }),
      contentType: 'image/png',
    });
  }
});

Video has no mask at all, which is why it is off.

With exactly this code, the same planted values and the same failing test, I searched everything the run left again. The password was in no file at all. The report had one screenshot, with the name and the IBAN painted over. What remained:

ArtefactStill in itWhy
Tracetoken, cookie, name, IBAN, unshown fieldIts network log and snapshots
JSON and HTML reportIBANThe assertion quotes it
error-context.mdname, IBANA snapshot of the page
.auth/state.jsoncookieIt is a login

That table is the honest result. Configuration removes what the tests themselves put into the artefacts. It cannot remove what the application shows and sends, because recording that is the point of a trace. So the rule that follows is not "make the trace safe". It is traces, screenshots, error-context.md and the login state stay inside, and only something built for the purpose leaves. Keep .auth/ out of version control as well.


7An export that carries only what may cross

A form, a stamp and a named authority. The export log is the plain version of the same record: what left, when
A form, a stamp and a named authority. The export log is the plain version of the same record: what left, when, who sent it and a checksum to prove it.Bureaucracy Illustration Czech.jpg - Petr Novák, CC BY-SA 2.5 (Wikimedia Commons)

The smallest useful version is a script that reads Playwright's JSON report and writes counts. It never copies error messages, attachments or output, so nobody under deadline pressure can include them by accident:

javascript
// export-summary.mjs - node export-summary.mjs report.json export/
// Builds the one file that may leave: counts from Playwright's JSON report,
// never its error messages, attachments or stdout. Test titles only when the
// agreed list says they may cross.
import { createHash } from 'node:crypto';
import { appendFileSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs';
import { userInfo } from 'node:os';

const [reportPath, outDir] = process.argv.slice(2);
const report = JSON.parse(readFileSync(reportPath, 'utf8'));
const titles = process.env.EXPORT_TITLES === 'yes';

const failed = [];
const walk = (suite, path) => {
  for (const spec of suite.specs ?? [])
    for (const test of spec.tests)
      if (test.status === 'unexpected')
        failed.push([...path, spec.title].join(' > '));
  for (const child of suite.suites ?? []) walk(child, [...path, child.title]);
};
for (const suite of report.suites) walk(suite, [suite.title]);

const { startTime, duration, expected, unexpected, flaky, skipped } = report.stats;
const summary = {
  startTime, durationSeconds: Math.round(duration / 1000),
  passed: expected, failed: unexpected, flaky, skipped,
  ...(titles ? { failedTests: failed } : {}),
};

mkdirSync(outDir, { recursive: true });
const body = JSON.stringify(summary, null, 2) + '\n';
writeFileSync(`${outDir}/summary.json`, body);
const sha256 = createHash('sha256').update(body).digest('hex');
// The record of what crossed: one line per export, kept on the inside.
const row = [new Date().toISOString(), userInfo().username, 'summary.json', sha256];
appendFileSync('exports.log', row.join(' ') + '\n');
console.log(`summary.json ${sha256}`);

Run against the report from section 2, the one that held all six values, it wrote this:

json
{
  "startTime": "2026-10-19T08:10:45.367Z",
  "durationSeconds": 4,
  "passed": 0,
  "failed": 1,
  "flaky": 0,
  "skipped": 0
}

The same search found none of the six values in it. With EXPORT_TITLES=yes it adds the titles of the failed tests, which is the switch to flip only once someone has agreed that titles may cross.

The log line matters more than it looks. exports.log stays inside and records the date, the user and the SHA-256 of every file that left. It turns "somebody sent the results out" into a list that can be checked against what the receiving side holds. It protects the people doing the work at least as much as it protects the data.


8When you are asked to send more

Sometimes the agreed path exists and the request is still for the trace, because the person asking wants to debug it themselves. What helps is short:


9A checklist

Know what you have

saved login state

than what it usually contains

Keep it out at the source

Make the path real


10Where vallus fits, and where it does not

vallus is the self-hosted runner we build for closed environments. Everything a run produces (Playwright HTML or Allure reports, traces, screenshots, run history) stays on the instance's own disk, and vallus sends no telemetry. What leaves is whatever a workflow is configured to send, and each of those is narrower than the artefacts above:

What it does not do: it does not mask screenshots, strip traces or scan reports, and the log redaction covers workflow secrets only, not values that came from a .env file. Section 6 is still your suite's job. And it cannot decide what may leave. That list belongs to whoever owns the data.

To see where a run's artefacts live on an instance behind an air gap, book a demo and ask for the 21-day evaluation key, which has every feature. The Free licence (two users, one project, one run at a time) keeps reports, traces and history on your own disk as well.


11Conclusion

The inbound problem is an engineering inconvenience. The outbound problem is a liability, and it lands on whoever moved the file rather than whoever asked for it.

  1. A trace is a recording. In one failing test it held a password, a token, a session cookie and a field that was never on screen. The HTML report embeds it, and lists typed values as step titles even for tests that passed.
  2. Configuration removes what the tests put in, not what the application shows. Sign in through the API, mask the screenshots you take yourself, and treat traces as data that does not leave.
  3. The fix is a path, not a policy. A script that writes counts, a list of what may cross and a log of what did is a day's work, and it is what makes the informal route unnecessary.