Inside a Cyber Drill: How We Test Defenses Under Realistic Attack Conditions
Introduction
Organizations invest heavily in their security stack. EDR, SIEM, backup infrastructure, incident response plans. But most of them never validate whether those defenses actually work under realistic attack conditions. Tabletop exercises test processes on paper. Penetration tests look for ways in. Neither of them answers the question of what actually happens when an advanced threat actor operates inside your environment, whether your security tools catch them and whether your team responds effectively. It does not tell you whether your infrastructure actually recovers.
That is what a cyber drill is for. We build an isolated replica of the client’s production environment, deploy the same security controls with the same policies, and run real attack scenarios against it. Depending on the engagement, that could mean detonating ransomware, exfiltrating data, compromising domain infrastructure, or simulating supply chain attacks. The isolation means we can go further than any production-based test allows, and measure exactly what the security stack catches and what it misses.
We have been running these engagements for clients across different sectors, and the results consistently surface gaps that other assessment types miss.
This post covers how we approach these engagements and walks through results from recent ones.
What Is a Cyber Drill
A cyber drill is a controlled simulation of attack scenarios executed against an isolated replica of the target organization’s environment. The replica mirrors production: same Active Directory structure, same EDR policies, same backup architecture, same business applications. The isolation means we can execute scenarios that would be impossible against production systems.
This makes it different from other assessment types.
Tabletop exercises are discussion-based. Teams walk through a hypothetical scenario and discuss responses. Useful for testing communication and decision-making, but no payloads are executed and no technical controls are validated.
Penetration testing focuses on identifying vulnerabilities and proving exploitability. The scope is generally “can an attacker get in, and how far.” A cyber drill starts from the assumption that the attacker is already inside and focuses on detection, response, and recovery.
Red team engagements operate against production with stealth constraints. This means certain attack scenarios like ransomware detonation, backup destruction, domain controller compromise and large-scale data exfiltration cannot be executed without risking business operations. A cyber drill removes that limitation. The environment is isolated, so we can execute full attack chains through to their conclusion and measure the actual outcome without business risk.
Our Methodology
We follow a seven-stage process for every engagement. The stages are sequential but the process adapts to each organization’s threat profile and objectives.
1. Planning and Objective Definition: Define scope, select threat scenarios, agree on success criteria and rules of engagement, and establish kill switch mechanisms.
2. Environment Preparation: Build an isolated replica mirroring production, these include domain services, endpoint protection, backup infrastructure, business applications and whatever is in scope. We use Infrastructure as Code to make replicas redeployable and validate builds in our internal labs before client deployment. If the replica does not accurately reflect production, the results are not reliable.
3. Scenario Development: Develop threat scenarios with both off-the-shelf simulation tools and custom payloads at varying sophistication levels. Each scenario has defined expected outcomes and success criteria.
4. Drill Execution: Execute scenarios systematically with full logging and event capture. Kill switches remain active throughout.
5. Detection and Response Assessment: Evaluate what the security stack detected, what it missed, how fast alerts fired, and whether SIEM correlation rules and SOC workflows would have caught the attack in progress.
6. Recovery and Continuity Testing: After destructive scenarios, assess whether backup and recovery mechanisms actually work and whether recovery time objectives hold up under real attack conditions.
7. Analysis and Reporting: Consolidate findings into prioritized recommendations with risk scoring, mapped directly to the scenarios that exposed each gap.
Our Approach to Attack Simulation
The payload development approach is one of the things that determines whether a drill produces useful results or just a pass/fail outcome. If you only test with one type of payload, you get a binary result. It either gets caught or it does not, and you learn very little about where your defensive threshold actually sits.
We develop custom attack tooling tailored to each engagement’s threat focus. For ransomware-focused drills, which is what we will walk through in the example below, the threat space is not homogeneous. A commodity ransomware operator buying network access from initial access brokers is not using the same techniques as a RaaS affiliate running a mature operation with dedicated development teams. And neither of them operates like an APT group building custom payloads from scratch with specific targets in mind. For this type of engagement, we developed a tiered payload framework to reflect that reality:
Commodity level: Ransomware with standard encryption, typical pre-encryption behavior (shadow copy deletion, service termination), and no evasion. This is the baseline. It represents the opportunistic attacker and tests whether the EDR is deployed and configured correctly.
RaaS level: Payloads that incorporate techniques seen in modern ransomware operations. Telemetry blinding to reduce what the EDR can observe, unhooking of security monitoring functions, and active efforts to neutralize defensive tools before they can respond. This represents organized criminal groups running ransomware-as-a-service with professional tooling.
APT level: Payloads with minimal forensic footprint and kernel-level defense evasion. The execution flows are custom-built for the target environment. These operate below the detection threshold of most behavioral analysis engines and represent the most sophisticated adversaries.
Isolation payloads: Stripped-down variants that test specific defensive layers on their own. For example, a payload that performs encryption with none of the typical pre-encryption activity so we can isolate whether the EDR detects the encryption behavior independently or relies on the noisier signals that normally precede it which tells you whether your detection is catching the ransomware itself or just the setup activity around it.
We combine this with established open-source breach and attack simulation platforms for coverage of known attack patterns and common lateral movement techniques. This gives us both ends of the spectrum: known threats that a properly configured EDR should catch, and custom threats that test the limits of detection capabilities.
All of this sits within an internal toolkit we actively develop and maintain specifically for cyber drill engagements. It includes the tiered payload framework, credential harvesting modules that operate in memory without leaving disk artifacts, lateral movement capabilities, and a centralized dashboard that aggregates session data and events across all scenarios in real time. The toolkit supports multiple attack chains and scenario types beyond what we cover in this post, and we are actively considering releasing an open-source version of it in the future.
Results From Recent Engagements
In multiple recent engagements we tested ransomware resilience for organizations with a reasonably mature security posture. Most had endpoint detection and response deployed, backup infrastructure with multiple repository types, and security policies across their environment.
The isolated replicas include domain services with Active Directory, EDR across all systems, backup infrastructure with both standard and hardened repositories, departmental file servers, and application servers running web applications and databases. Most engagements run approximately two-three weeks from environment preparation through final reporting.
We operate under an assumed breach model. The starting position is that the attacker had already gained internal network access and obtained valid domain credentials.
Scenarios Using Known Threat Patterns
We start with scenarios using known attack patterns and established simulation tools. The EDRs usually handle these well.
In one engagement we deployed autonomous malware with command-and-control communications, credential harvesting, and lateral movement capabilities. Despite the malware incorporating multiple evasion techniques, the EDR’s behavioral analysis engine flagged the autonomous network navigation and credential access patterns. The EDR blocked it before it could propagate to additional hosts.
We also ran fileless attack scenarios that operated entirely in memory using known exploitation and lateral movement techniques. The behavioral analysis engine detected these and shut them down.
This validated that the baseline security investment was operational and correctly configured. The EDR was detecting known threats as expected meaning that the deployment is sound and the policies are tuned correctly.
Scenarios Using Custom Payloads
We always see different results with custom-built payloads that have no existing signatures or known behavioral fingerprints.
In one scenario, we simulated a social engineering attack where a domain administrator was led to execute a malicious payload disguised as a routine system update. The security stack did not flag it. The execution context was a trusted administrator account performing what appeared to be normal maintenance, and that implicit trust was enough to bypass the security controls entirely. The domain controller was encrypted. Active Directory authentication went down and Group Policy stopped propagating. DNS resolution failed shortly after. From a business continuity standpoint, the domain was non-functional.
In another scenario in a specific engagement, we deployed our custom zero-day payloads against backup infrastructure, departmental file servers, and business application servers. The outcome was consistent: no detection and no alerts. Full encryption across every target. File shares containing data from multiple departments were rendered inaccessible. The business application server, which hosted web applications and databases, was completely disrupted, the web server returned errors, the database crashed, and the application files were encrypted. Backup repositories running on standard infrastructure were encrypted alongside the production systems they were supposed to recover.
One detail stood out on the backup testing. Hardened, immutable backup repositories were not affected by the ransomware. The payloads could not modify or encrypt those backup files. Standard backup repositories, however, were encrypted just like everything else, which highlights the value of immutable backup configurations.
The SIEM did ingest relevant events. High-volume file modifications, service disruptions, and other signals were logged. But there were no correlation rules to flag these patterns as ransomware activity, so no alerts were generated from them. The data was there but the detection logic was not.
We also tested a custom payload against a different endpoint protection product in an isolated lab environment. The result was the same, the payload executed without detection.
| Scenario Type | Payload Approach | Detection |
| Known threat patterns | Established tools, known behaviors | Detected and blocked |
| Custom zero-day payloads | Purpose-built, no existing signatures | Not detected / Full compromise |
Simulation Toolkit in Action
The following walkthrough shows what configuring and executing a ransomware simulation scenario looks like using our purpose-built toolkit.
Figure 1: Configuring options and capabilities (1)
Figure 2: Configuring options and capabilities (2)
Figure 3: Configuring safety options
Figure 4: Getting a callback after executing on the target machine
Figure 5: Simulation tracking logs (1)
Figure 6: Simulation tracking logs (2)
Figure 7: Full data recovery on the infected machine
Lessons Learned
Known-threat detection is necessary but not sufficient: EDRs reliably catch known attack patterns. But custom payloads built to evade a specific product will get past it. A single detection layer is not enough regardless of which product it is.
Security resilience requires live testing to validate: Tabletop exercises test response plans. Vulnerability scans test patch hygiene. Neither tests whether the security stack actually detects and stops an attack during execution. The only way to know is to run the scenario under realistic conditions.
Privileged access remains a high-value attack vector: The domain controller compromise did not use an exploit or a vulnerability. A trusted administrator executed a malicious payload in a trusted context, and the security stack allowed it because it trusts administrators. Code signing validation for software updates, strict change management, and continuous monitoring of privileged actions are the controls that address this gap.
Testing at multiple sophistication levels is more useful than a single pass/fail test: Knowing that defenses hold at the commodity level but fail at the RaaS level tells you exactly where to focus improvement efforts and budget. A single payload gives one data point. A tiered approach maps the defensive threshold across realistic adversary profiles.
Wrapping Up
Every organization has gaps in its defenses. The point of a cyber drill is to find them in a controlled setting before a real attacker does, and to turn the findings into a concrete, prioritized improvement plan based on what actually happened rather than theoretical risk assessments.
In the engagements discussed above, the organizations walked away with clear evidence of what worked, what did not, and what to prioritize. The security team had specific, scenario-backed recommendations to take to leadership, and leadership had measurable data to inform budget decisions.
We run these engagements across different sectors and threat scenarios. If validating your organization’s security resilience under realistic conditions is something your team is considering, reach out and we can discuss what a drill would look like for your environment.
Thanks for reading.