Deterministic network fault model

Add one explicitly configured virtual network to SimFerret and prove that a seeded, recorded network outage can be reproduced exactly. The increment should expand the RFD 18 process-restart scenario across a QEMU-supported network replay boundary without adding a custom network stack, privileged host networking, multiple virtual machines, or uncontrolled Internet access.

The result establishes whether QEMU’s replay filter and an unprivileged network backend are a viable base for later partitions, packet faults, multiple nodes, and state-space exploration.

RFD 18 proved exact record and replay of one diskless, one-vCPU guest controlled through a replay-aware serial port. Its acceptance run completed without semantic divergence, detected artifact and VM-identity tampering, and passed in GitHub Actions. It also remained limited to guest loopback communication and a process failure.

The final RFD 18 evidence recommends a deterministic network fault model as the next increment. A network replay spike is narrower than general container packaging and gives the project evidence about the QEMU device boundary before it commits to multi-node workloads or snapshot branching.

RFD 18 recorded approximately 64 seconds for the normal scenario and 70 seconds for each replay, with an 8.5 MB replay log. The per-byte acknowledged ISA serial protocol contributes operational overhead. This RFD records the additional cost of a NIC and network traffic but does not combine that measurement with an unrelated control-channel redesign.

QEMU record/replay supports network input through one filter-replay object per network backend. In record mode the filter writes packets entering QEMU from the backend to the replay log. In replay mode it disables backend interaction and injects those packets from the log. Outbound packets and guest device execution remain deterministic consequences of the initial state and recorded inputs.

The initial spike selected the emulated RTL8139 NIC used by QEMU’s documented record/replay example. Record mode uses an unprivileged, restricted user-mode network backend and one replay filter:

Replay still requires a netdev object with the recorded identifier. The initial supported profile retains the same restricted user backend and replay filter but removes all fixture files before replay:

Using one backend type avoids claiming that a listening socket is isolated from other local processes. The fixture identity is a semantic input, while its runtime directory is reconstructed for each launch and is empty in replay mode. The replay filter must prevent backend packet interaction; the empty directory also ensures that an accidental live TFTP request cannot reproduce the recorded response.

The record backend is restricted from reaching arbitrary external networks. It exposes only the minimum fixture service or forwarding path required by the scenario. TAP, host bridges, raw sockets, and host firewall configuration are excluded because they require privileges and would make the GitHub-hosted CI environment materially different from local execution.

The pinned kernel provides RTL8139 support as compressed modules rather than built-in code. The reproducible initramfs includes mii and 8139cp, decompresses them at build time, and verifies that 8139cp binds the NIC. The source and decompressed module digests and load order are part of the initial state identity. The NIC option ROM is disabled with romfile="" because this direct-kernel workload does not perform network boot.

The acceptance scenario retains one guest and the small auditable fixtures from RFD 18. Traffic under test must traverse the emulated NIC; guest loopback traffic does not satisfy this RFD.

The initial outage topology is:

The guest uses a fixed MAC address and static network configuration so DHCP and DNS do not add inputs unrelated to the test. The first fixture is immutable content served by QEMU’s restricted built-in TFTP service. The controller materializes and digests that content in a private run directory. The backend has no route to an external service.

Each request identifier maps to one canonical relative TFTP filename. The file contains the expected request identifier and payload in a bounded canonical encoding. The guest reports values parsed from the received file rather than manufacturing a successful response from its command. Intentional corruption changes the served bytes while preserving the expected values.

The guest agent continues to emit structured semantic events over the existing serial channel. Network traffic is an application input, not a replacement control plane. In replay mode the fixture content is absent and the restricted backend has no input to serve; QEMU must supply all incoming packets from the recorded replay log. This passive replay condition is an acceptance requirement rather than an implementation detail.

The first fault is a bounded outbound outage for the configured fixture peer. At a seed-derived choice point, the controller sends a recorded fault command to the guest agent. The agent installs a prohibit route for that peer address, confirms the transition as a structured event, and later removes the route at another explicit choice point. A disposable spike proved immediate failure and recovery with this mechanism through an e1000 NIC; the implementation repeats the result with the selected RTL8139 identity.

The route targets only the fixture peer and may not alter unrelated guest traffic. A pinned static BusyBox ip implementation applies the route without requiring nftables, iptables, qdisc, or their module dependency closures. The tool digest, arguments, route representation, and resulting event are part of the initial-state and scenario identities.

The fault model is event-driven. Host sleeps, host packet timing, and task wake order do not determine when the outage begins or ends. Infrastructure deadlines may terminate a hung run, but they are not semantic fault choices.

The recorded scenario performs this sequence:

The route produces an immediate administrative rejection: the local network operation must report EACCES. Installing or removing the wrong route, a missing fixture file, a malformed response, or an infrastructure timeout cannot satisfy the outage property.

Silent packet loss, port-specific filtering, probabilistic loss, latency, duplication, corruption, reordering, bandwidth limits, and asymmetric partitions are later extensions. The pinned kernel exposes nftables, iptables, and netem as modules, but no corresponding userspace tool is currently pinned in the guest. Kernel configuration alone is not evidence that those later faults are usable.

The scenario seed materializes the request plan, outage activation point, and restoration point into the choice plan before those choices affect the guest. A seed alone remains insufficient for exact replay.

The RFD 18 replay identity is extended with at least:

Runtime-only socket paths and host-assigned process identifiers are diagnostic, not semantic, but their endpoint type and safety behavior remain part of the launch contract. Replay rejects missing, extra, or changed semantic network identity fields and changed order-sensitive device or filter sequences before starting QEMU. Serialization order of typed object fields is not semantic.

Incoming packets are already represented in QEMU’s replay log and are not duplicated into SimFerret’s semantic event stream. Semantic events identify the request, response, fault transition, and property result. Optional packet diagnostics must be bounded and must not become a second source of replay truth.

The scenario retains RFD 18's response-integrity safety property and adds typed network fault events. It passes only when:

The checker uses structured guest and controller events rather than parsing human logs or inferring a partition solely from elapsed host time. The outage bound is measured from request-attempt event to typed result event; the recovery bound is measured from restoration-confirmed event to matching response event. A missing event or host watchdog expiration is an infrastructure failure, not a semantic unavailable result.

Run directories retain the RFD 18 manifest, scenario, choices, replay log, normalized events, assertions, and diagnostics. The manifest includes the extended network identity and digests every additional semantic artifact.

Network diagnostics should make first failures actionable. They include a sanitized QEMU network configuration with runtime paths replaced by typed placeholders, backend mode, fault transitions, bounded packet counters when available, and QEMU errors. Secrets, ambient environment, temporary socket paths, and unrestricted packet payload captures are not added to normal run artifacts.

QEMU does not reliably diagnose replay identity errors itself. In the completed spikes, omitting the replay filter or changing the MAC address caused a hang until the watchdog expired. SimFerret must validate the complete network identity and mandatory filter before launch rather than relying on QEMU to fail fast.

The acceptance scenario runs as an ordinary process on a GitHub-hosted x86-64 Linux runner. It uses QEMU TCG and unprivileged user-mode networking, so it does not require nested virtualization, /dev/kvm, TAP devices, or persistent host network configuration.

The same CLI invocation is supported on an x86-64 Linux self-hosted runner. The GitHub runner remains outside the deterministic guest: placing the Actions agent inside the guest would introduce GitHub’s network and timing behavior into the simulation boundary.

CI uploads a retained, bounded failure bundle distinct from atomically published replayable runs. Launch and watchdog failures retain enough sanitized network, QEMU, and guest diagnostics to identify the failed boundary. A later thin GitHub Action may wrap the stable CLI, but Action-specific code is not part of this increment.

This increment is complete when:

This proposal does not include:

Completion produces evidence about NIC replay fidelity, passive replay, network-log growth, fault control, CI portability, and runtime cost. That evidence informs the next RFD.

If the boundary is reliable, reproducible OCI workload packaging is the recommended next product-facing increment so users can apply the network model to their own software. Checkpoint branching should follow once a real workload and network traffic provide representative snapshot and exploration costs. Broader packet faults or a second guest move earlier only if the evidence shows that the initial outage is too narrow to validate the model.

Reproducible OCI workload packaging would improve user onboarding immediately, but it would preserve only the process-restart fault proven by RFD 18. The final RFD 18 evidence explicitly recommends validating QEMU’s network boundary first.

Checkpoint branching would begin the exploration engine, but its throughput and snapshot costs are not representative while the guest has no external device traffic and only a small built-in fixture.

TAP networking offers familiar host integration but requires privileges and host configuration that are unavailable or undesirable on GitHub-hosted runners. QEMU’s user-mode backend is narrower and supported by its documented replay filter.

A custom deterministic switch or TCP/IP stack would offer precise packet scheduling, but it is not required to answer whether QEMU can record and replay a controlled network boundary. Such a component requires separate evidence and an RFD after a concrete limitation is observed.

Guest nftables or qdisc would support narrower and richer packet faults. The pinned kernel has the relevant modules, but the current guest contains neither their dependency closures nor pinned control tools. A peer-specific route is the smallest mechanism already proven to answer this RFD’s outage question.

Two guests would model a realistic partition, but coordinating two independent QEMU record/replay timelines introduces a new synchronization problem. The single-guest scenario isolates the network replay question first.

Delivery phases are tracked in RFD 19 implementation checklist, with preliminary results in the adjacent evidence record.