A blocked connection proves nothing

KubernetesRKE2air gapnetwork securitynetwork policyCalicosupply chain securitysigned imagesKyvernotestingGo

Run curl from a pod, watch it time out, and call the cluster isolated. That test also passes with a typo in the address, a dead listener or a broken pod.

So my rule for this lab was that a block counts only if the same attempt succeeds once the layer blocking it is removed.

The setup

The lab is one RKE2 node in a VM on a NUC, on an isolated network. It installs from an offline bundle, pulls only from a read-only internal registry, and admits only images signed with the bundle's key. Two independent layers stop traffic leaving:

  • In the cluster: a default-deny network policy that also covers new namespaces.
  • At the VM: a black-hole default route, plus a host firewall.

The test

A Go program on the host opens and closes each layer, then tries to reach a canary listener from a pod. The canary lives on temporary host addresses, so opening the VM path never opens a route to the internet.

                  policy on    policy off
VM path closed    blocked      blocked
VM path open      blocked      REACHED

Only the bottom-right cell connects, and the canary's own log has to confirm it. That one cell shows the other three were blocked by the layers, not by a broken test.

make rebuild destroys the VM, rebuilds it and runs everything: 185 checks. It took 225 seconds to build and 572 seconds to test, with downloads cached.

What the controls caught

A control that did nothing. To show that cluster DNS can't leak a name, the test temporarily adds a forwarder to CoreDNS and expects a random label to reach the canary. Nothing arrived. The Corefile said cache 30 with two spaces, so my edit never matched and no forwarder was added. Without the control, the result would have read as "zero leaks".

A check that accepted a crash loop. An older test checked for the Running phase, which a crash-looping pod still reports. Its probe pod was in fact exiting on a bad argument. It now has to be ready.

A cost I hadn't planned for. With the registry down, running pods carry on and containers restart from cache, but new pods are refused, even signed ones, because the verifier needs the registry to check the signature. That fails closed, and a pod rescheduled during an outage won't start.

What this doesn't prove

  • The canary is on the host, not a remote network.
  • Public IP and IPv6 targets are blocked, but have no open-path control, because the test never opens the internet.
  • The gateway address stayed unreachable even with the policy off, so its block isn't shown to come from the policy.
  • Only TCP and DNS over UDP were tested, not ICMP.
  • It is one node in a lab, designed for air-gapped environments and not certified for anything.

Next comes what runs inside the gap: retrieval, then a gateway for AI that can act on real systems only after a person confirms.