Introduction: the half of the problem that is not technical
Every article about closed environments describes the procedure as it should be. A package assembled deliberately, checksummed, transferred through whatever channel the organisation sanctions, verified on arrival. All of that is correct, and all of it assumes the sanctioned channel answers in time.
When it does not, something else happens instead, and it happens in almost every organisation that has this kind of control. It is not written down anywhere, nobody is measured on it, and it is the reason a great deal of carefully designed security produces no security at all.
This piece closes a series on test automation in closed environments. The earlier parts describe how browsers, packages and updates should cross into a closed network, and what should and should not come out of it. This one is about what happens when the official route cannot keep up, which is the reason those procedures matter at all. There is no procedure in it and nothing to configure.
1What actually happens
What I meet in practice is something else. In closed environments the people who know how to get work done are frequently the people who know how to get around the controls, and that knowledge circulates. Not as a secret and not with any sense of wrongdoing, but as practical advice, passed from somebody who has been there to somebody who has just arrived. Nobody in those conversations is trying to fix the problem permanently. They are showing where the gap is.
One case stands for many. The browsers and the Playwright binaries for a specific version were supplied by the delivery team itself, outside the official review path. Not because anybody was careless about security, and not because the rules were unclear. Because the sanctioned route had no usable response time, and the work had a date on it. The same applied to the npm packages a proof of concept needed: things were installed on machines outside the domain and carried in, and everyone involved understood that this was not how it was supposed to work.
2What "outside the review path" means in practice
It is worth being precise about what that looks like in practice, because the phrase "supplied outside the review path" hides the actual work. Every browser binary for every Playwright version is located by hand, on a machine outside the domain, copied, and added to a package. The package is then carried into an environment described, in every document about it, as verified and controlled. Nobody opens it. Nobody asks for a checksum, a provenance statement or a list of what is inside.
That is not an oversight by one careless person. It is what the incentives produce. The reporting on a programme like that measures delivered scope, test pass rates and dates: the tables and the charts that go up the chain every fortnight. Nothing in that reporting asks where the binaries came from, so nothing in the day-to-day behaviour treats it as a question. An environment can be genuinely isolated at the network level and still have an open door, if the only thing anyone is measured on is whether the numbers arrive on Thursday.
3The part that does not appear in any process description
There is a part of this that does not show up in any process description, and it is the reason the section is here at all. The people doing the carrying know what it risks. They wonder each time whether this will be the day somebody notices, whether a question is coming, whether one more fix gets into the test framework without anyone asking how it got there. They do it because the alternative is to stop delivering, and because the expectation from above is steady: more, faster, regardless of what the constraints make possible.
Ask whether it worries them and the answer is usually no. It is usually not true either, and the colleagues asking generally know it is not true. Some of them say plainly that this is not how the work should be done, and they are right. What nobody in the conversation can answer is the part underneath: the deliverable cannot be produced without the tooling, and there is no sanctioned way to get the tooling in. The objection is correct in principle and has nowhere to land in practice, which is the most demoralising shape a disagreement can take.
Ask around and it is rarely one team's habit. Different teams, different tasks, the same skipping of the sanctioned route to hit a date. A handful of people are scrupulous about it; the majority are not, and have stopped thinking of it as a decision at all. It has become the way the work gets done.
4Two things follow, and both are uncomfortable
Two things follow from that, and both are uncomfortable.
The first is that the control is not functioning as a control anywhere in an organisation like that, which is a different and much larger problem than one team taking a shortcut. A rule that a small minority breaks is a rule with an enforcement problem. A rule that nearly everyone routes around is a rule that has been quietly replaced by something else, and the something else is undocumented. Security teams have a name for the general pattern, shadow IT. In a closed environment it takes a sharper form, because the thing being worked around is the perimeter itself.
The second is about direction of travel, and it matters more than anything above. Getting tooling into a closed environment is the visible half of this. The half that actually carries risk is what moves out: test data, code, artefacts and reports leaving an isolated network because a deliverable is expected somewhere it cannot legitimately go. The inbound version costs you an unreviewed binary. The outbound version can be a data protection incident, a contract breach, or worse. What a trace or a report actually carries is a measurement of its own. The point here is narrower: outbound is where informal permission from a manager is worth precisely nothing. "We were told it was fine" is not a defence anyone has ever successfully used, because the person who said it was fine usually had no authority to say so.
That is the situation worth recognising, because it repeats. The people raising the objection are right. The person working around the control is not wrong about the constraint either. Both positions are accurate, and neither party has the authority to fix what is actually broken, which sits one or two levels above both of them: a process with no workable path, and a schedule written as though the path were instant.
I want to be careful about what this is for. It is not a complaint, and it is not an excuse. It is the part organisations do not see when they leave a control in place that has no workable path through it. The risk does not disappear. It gets pushed down to the person with the least authority to change the process and the most exposure if it goes wrong: someone junior enough to be told what the deadline is, and senior enough to be held responsible for how the software got onto the machine. They carry it privately, every day, and the organisation carries it too without knowing.
The pattern is usually the same. The people above the engineer know it is happening. Permission is given on calls and only on calls, never in writing, which means it can be extended and disowned at the same time. The official permissions carry on in parallel with a life of their own: requested, tracked, eventually granted, on a timeline that has nothing to do with the work they were meant to authorise. The part worth sitting with is not that an engineer breaks a rule. It is that the rule is visibly not working, in full view of the people who could fix it, and nothing in the reporting ever asks.
5What it costs, and what it is worth writing down
I am not recommending it. It is worth writing down for a different reason.
A control with no tolerable turnaround does not stop the traffic. It moves it somewhere nobody is looking. An organisation like that does not end up with fewer unreviewed binaries inside its perimeter. It ends up with the same binaries, arriving through a path that leaves no record, no checksum and no list of what was installed, while its documentation continues to describe the environment as controlled. That is strictly worse than an imperfect official channel, because the imperfect channel at least knows what went through it.
The second lesson is about what gets measured. A security control that nobody is accountable for enforcing is a control in name only, and it stays that way for as long as the reporting rewards delivery and is silent about provenance. If the status report has a column for scope and a column for pass rate, and no column for how software entered the environment, then the honest reading is that provenance is not actually a requirement, whatever the policy document says.
6Where you sit decides what you can do about it
None of this is fixable from one seat, but each seat has something it can do that the others cannot.
If you are on the delivery side, the answer is to make the official route survivable. A package that is complete, verifiable and boring - the browsers for an exact Playwright version, a checksum for every file, a list of what is inside - gives the people running the review something they can approve quickly. A request that arrives as "we need internet access on the test machine" does not.
If you are on the receiving side, and your engineers are quietly carrying software in on their own, the useful response is not tighter enforcement. It is to look at how long the sanctioned path takes, because that number is what produced the behaviour. A framework that ships every month against a transfer that takes weeks is arithmetic that does not close on its own. An internal mirror exists mainly to make the fast path and the approved path the same path.
And if you are the person doing the carrying, the thing worth knowing is that this is a structural problem wearing your name badge. Write down what you brought in and when, even when nobody asks: it is the only thing that turns an undocumented act into a record somebody can review later. Then put the turnaround time in front of whoever owns the schedule, as a number rather than a complaint. Neither of those makes the pressure go away. Both of them move the decision to the level where it actually belongs.
7Where vallus fits, and where it does not
vallus is the self-hosted Playwright runner we build for closed environments, and it cannot fix any of the above. A review queue is an organisational fact, not a technical one. What a tool can do is shrink what has to cross, and make what does cross easy to approve:
- One package per Playwright version, browsers inside. Each release ships the runner without browsers and, for each Playwright version customers are on, a package with the whole Playwright browser set inside (Chromium, Firefox, WebKit and ffmpeg), for amd64 and arm64. That is one archive to review instead of a browser directory located by hand on a machine outside the domain.
- A checksum for every file, and versions that do not change. Each published version comes with a
SHA256SUMSfile, and a version is never replaced once it is published, so the checksum a reviewer approved is still the one on the download page a month later. - Nothing to open on the way out. No telemetry and no licence server: the licence is a file verified locally, so approving the tool does not mean approving a firewall exception.
What it does not do: it does not shorten anyone's review, it does not bring your suite's npm dependencies with it, and it keeps no record of software someone carried in by hand. That list, what came in and when, is still yours to write down.
To see what a release package contains before it goes into a review, book a demo and ask for the 21-day evaluation key, which has every feature. The Free licence (two users, one project, one run at a time) is the other way to try it on your own disk.
8Conclusion
The procedures in the rest of this series only work if somebody can follow them in the time the work allows. Where nobody can, the procedure stays on paper and the work goes around it.
- A control with no workable path does not stop the traffic. It moves it to a route with no record, no checksum and no list of what crossed, while the documents keep describing the environment as controlled.
- The risk lands on whoever has the least authority. Permission given only on calls can be extended and disowned at the same time. It protects nobody who acted on it, least of all when something goes out rather than in.
- The fix is a turnaround time, not tighter enforcement. A complete, checksummed package on one side, a mirror and a measured review time on the other, and a written record of what crossed in between make the approved path the fast one.