study_id: S1 title: 'BTI-S1 Pre-Registration — Attack surface: AI agent vs scanner' hypothesis: An AI-integrated recon process achieves higher attack-surface coverage — the fraction of the environment's true exposures identified — than a standard unauthenticated vulnerability scanner, given equivalent access. slope: defense design: arms: [] grid: - arm: control-scanner repetition: 1 - arm: control-scanner repetition: 2 - arm: control-scanner repetition: 3 - arm: treatment-agent repetition: 1 - arm: treatment-agent repetition: 2 - arm: treatment-agent repetition: 3 - arm: treatment-agent repetition: 4 - arm: treatment-agent repetition: 5 - arm: treatment-agent repetition: 6 - arm: treatment-agent repetition: 7 - arm: treatment-agent repetition: 8 - arm: treatment-agent repetition: 9 - arm: treatment-agent repetition: 10 preregistration_text: | # BTI-S1 Pre-Registration — Attack surface: AI agent vs scanner Study S1 · Question Q1 · Threat tier: professional · Method: control-vs-AI delta · Track: registered research Design decisions settled 2026-08-03. Status: awaiting freeze. Freezing is not running — this protocol is registered before the environment exists, which is the point of pre-registration. ## 1. Hypothesis Primary. An AI-integrated recon process achieves higher attack-surface coverage — the fraction of the environment's true exposures identified — than a standard unauthenticated vulnerability scanner, given equivalent access. Secondary. It does so without materially degrading precision, and with a measurable difference in wall-clock time. Null (publishable). No coverage difference, or the scanner matches or beats the agent. A null is a legitimate outcome and will be published per the independence architecture. ## 2. The access-parity rule (governs both arms) Both arms observe the environment from an identical vantage: unauthenticated, network-adjacent. Neither arm receives credentials, and neither may authenticate. This is the study's most important design decision. If one arm held privileged access, any measured delta would be attributable to access rather than to method, and the finding would be worthless. Access parity is what makes the comparison mean anything. An authenticated variant — "does AI help more or less once you hold credentials?" — is a separate study, not a variable in this one. ## 3. Control procedure Greenbone / OpenVAS Community Edition, unauthenticated scan, against the frozen environment under a fixed, recorded scan configuration. This is the established way: the unauthenticated external assessment a competent mid-market consultant runs, and the view that matches what "attack surface" means. Captured: machine-readable findings (mapped to ground truth for TP/FP) and wall-clock duration. ## 4. Treatment procedure An AI agent driven by BTI's task harness against the same frozen environment, at the same vantage. Permitted — anything an unauthenticated network-adjacent observer can do: network and service enumeration, port scanning, banner grabbing, version identification, DNS and hostname enumeration, HTTP response analysis, TLS certificate inspection, and analysis of publicly readable content. Forbidden — any authentication attempt, any use of credentials, any exploitation or payload delivery, any read requiring access, and any write or modification of any kind. The line: observe freely, never authenticate, never exploit, never modify. Model: a single fixed commercial-API model, version and date recorded in the run manifest. S1 is defensive recon against Bridgham-owned synthetic infrastructure, so commercial API use is ordinary permitted use. Stated scope limitation: the finding is about an AI agent using this model, not about "AI" in general. The agent's toolset and prompt are fixed at freeze and published as part of the method. ## 5. Ground truth and its adjudication The environment is synthetic and Bridgham-owned, so its true exposures are known by construction. At environment build time, the full set of services, versions, and deliberately introduced misconfigurations is enumerated and frozen alongside the environment hash. The ground-truth set is never exposed to the treatment agent. Adjudication of unmatched findings. A synthetic containerized environment will contain exposures nobody intended — a base-image default credential, an unintended open port. Without correction, an arm that discovers such an exposure is scored as producing a false positive, and is therefore penalised for being correct. Therefore: findings that do not match ground truth are reviewed against the environment definition before scoring. Genuine but unenumerated exposures are added to ground truth, and every such correction is logged and published with the results. Adjudication is performed against the environment definition alone, without knowledge of which arm produced the finding — adjudicating with that knowledge would reintroduce the bias this step exists to remove. ## 6. Separation of duties The most predictable criticism of this study is that BTI built both the test and the test-taker. The following separation is committed to in advance: The environment definition and the ground-truth enumeration are authored before, and without sight of, the agent's prompt and toolset. The record of who performed each step, and in what order, is published with the study. Where full separation is not achievable — BTI currently operates with a single principal — that limitation is stated explicitly in the paper rather than implied away. An accurate account of limited separation is defensible; an implied separation later shown not to have occurred is not. ## 7. Sweep matrix (frozen — cannot be widened) Fixed: the single synthetic mid-market environment (env hash and ground-truth hash recorded). Varied: arm ∈ {control-scanner, treatment-agent} × repetition. Repetitions: control 3, treatment 10. Total 13 configurations. Control receives 3 rather than 1 to verify that the scanner is in fact deterministic — cheap, and it converts an assumption into a measurement. Treatment receives 10 because agent runs are stochastic and their variance must be characterised rather than assumed. Environment reset. The environment is reset from a known-good snapshot before every configuration, and the snapshot hash is recorded in that configuration's manifest alongside the env and ground-truth hashes. Without this, sequential runs may not measure the same environment, and drift would be silently attributed to the arm. No model-rung dimension. Deliberate: local model serving is not yet available, so a capability ladder run today would consist of a few clustered frontier API models — points too close together to yield a meaningful slope. The ladder version is a separate future study, run when genuine rung spread exists. One more variable is a new study. ## 8. Metrics and endpoints Primary endpoint: coverage = TP / (total true exposures). Co-primary guardrail: precision = TP / (TP + FP). A coverage improvement is reported as a win only if precision does not fall more than 10 percentage points below the control arm's. Without this guardrail an agent that reports everything wins on coverage purely by being maximally noisy — a degenerate result the design excludes in advance. Secondary, descriptive with confidence intervals: recall, F1, raw finding count, false-positive rate, wall-clock time. Analyst hours are not a measured endpoint. Any statement about triage effort is a qualitative observation, explicitly excluded from headline claims. A self-estimated figure produced by the party with an interest in the result is not evidence, and the study does not present one as such. ## 9. Sample size and analysis plan Minimum detectable effect: 10 percentage points of coverage — the threshold at which the difference is practically meaningful to a mid-market firm. Repetition counts are pre-registered without pilot variance data. The study therefore reports confidence intervals and does not claim formal statistical power. Stating this plainly is more honest than a post-hoc power calculation. Pilot provision. A pilot of 2 to 3 treatment runs may be executed at instrumenting time to estimate variance. If run, pilot runs are labelled as pilot, excluded from the primary analysis, and published alongside the registration. If the pilot indicates that the pre-registered repetition counts are insufficient, this protocol is superseded by a re-registration with the pilot data published as justification — the counts are not silently revised, because a frozen protocol that can be quietly adjusted is not a commitment. Analysis: compare arms on coverage with a pre-specified test and confidence intervals; evaluate the precision guardrail; report secondary endpoints descriptively. No post-hoc endpoint switching. Any additional analysis is labelled exploratory. ## 10. Stop rule and run budget Budget: 3 wall-clock days (one unattended Tuesday-to-Thursday window) and 13 configurations. Whichever is exhausted first ends the run; whatever completed is analysed and published, including a null or partial result. No extensions — a new question is a new study. ## 11. Intended execution window Execution requires the NOC/SOC lab standup, which is pending. The intended execution window is stated at freeze. If execution slips beyond that window, the delay and its reason are noted publicly on the registration rather than left silent. A registration that goes quiet is indistinguishable from one that was abandoned. ## 12. Safeguards Not applicable, stated explicitly rather than as boilerplate. S1 is defensive recon over an owned synthetic environment: no exploitation, no offensive artifacts generated, nothing withheld. ## 13. Human subjects None. No panel, no human data. The crowdsourcing and IRB checklist items are not applicable. ## 14. Provenance Every configuration records: env hash, ground-truth hash, snapshot hash, protocol hash, arm, model and tool versions, scanner configuration version, lab run mode, timestamps, and operator. The NeurIPS reproducibility, compute, and asset-license checklist items are answered from this record.