The last few years have seen the emergence of a plethora of cloud security tools designed to provide increased visibility and automated guardrails for cloud environments. Solutions like cloud-native application protection platforms (CNAPP), cloud security posture management (CSPM), and cloud infrastructure entitlement management (CIEM) can detect misconfigurations, prevent identity breaches, and even auto-remediate certain issues. (You can read more on the specific goals and scope of each here.)
However, while these tools identify an impressive breadth of risks, companies still struggle to operationalize remediation at scale.
The average organization sees up to 50 misconfigurations per day*, with security teams often bogged down triaging each alert manually. At this pace, critical cloud vulnerabilities persist for weeks or months, leading to data exposures and outages down the line. This persistence of vulnerability is known as dwell time, or the period between the introduction of a given misconfiguration and its remediation. We can represent this as Time to Detect (TTD) + Time to Remediate (TTR). More so than any other, this is the metric that definitively says how long an organization was vulnerable.
It’s clear that despite powerful security tooling, there are missing elements in the remediation process. Teams lack context, orchestration, owner identification and thoughtful prioritization to turn insights into resolved risks. The remediation lifecycle itself remains manual, inconsistent, and painfully slow.
*A finding in The State of Cloud Security 2021 report – 45% of organizations experience between 1 and 50 cloud misconfigurations per day.
Two Extremes: Neither Sufficient
There are many deployment models for cloud. For example, using Infrastructure as Code (IAC) may create opportunities to remediate misconfigurations by changing the source code. This document is focused on the process while acknowledging diversity in specific remediation techniques.
Today’s landscape offers two extremes for remediation, both insufficient on its own.
Manual remediation leverages human expertise to carefully contextualize and fix issues, but can’t handle the overwhelming volume of cloud misconfigurations.
➤This approach offers two models: one (1) in which DevOps teams resolve misconfigured resources, and one (2) in which dedicated cloud security engineers or SOC analysts own remediation.
Pure automation provides efficiency yet often causes incidents by blindly enforcing changes without considering unique environmental context and deployment methods.
The table below summarizes each approach and the pros and cons of each.
| What it involves | Benefits | Drawbacks | |
|---|---|---|---|
| Manual remediation by experts – remediation by DevOps | In a DevOps remediation model, the team that owns the misconfigured resource is responsible for fixing it. | Teams can more easily prioritize alerts for their workload. | DevOps teams often have little insight into how the inherited environment affects the misconfiguration impact from a security perspective. |
| Manual remediation by experts – remediation by cloud security experts or SOC analysts | Cloud Security experts, or SOC analysts reviewing security alerts, making prioritization decisions, and manually overseeing the remediation process. | Cloud Security experts are responsible for prioritization, and help focus developers on the immediate prioritiesThis model is thorough, ensuring that security risks are addressed holistically | Not scalable – slow and inconsistent workflow leaves critical risks unremediated. No workload context, and lack of permissions needed to evaluate that the production process is not going to be affected. |
| Pure automation | Automation-based remediation uses ticket creation and API-based flows to execute simple what-if based actions – classifying, prioritizing, and automatically driving the remediation process | Efficient, enables teams to rapidly work through security alert backlogs and remediate at scale | Risks unexpected breaking changes due to lacking environment and process context |
The limitations of each approach on its own become clear when we look at the data.
The average security team spends over 75% of its time triaging alerts – meaning that no team of experts alone can handle the deluge of incoming cloud security alerts generated by CNAPP, CSPM, and other cloud security tooling.
And stepping through specific, familiar use cases will convince even the most automation-happy security teams that pure automation-powered approaches can’t get the job done either. See the inset (“Pitfalls of a purely automated approach: RDS encryption”) for an overview of the complexities involved in a seemingly straightforward remediation task.
The 4 Pillars of Cloud Remediation
In working with our customers to systematically strengthen their cloud security posture, we’ve observed that the single biggest “failure mode” is treating the remediation process itself as an afterthought.
What does this mean? Traditional cloud remediation experiences four primary challenges. First, the disjointed nature of security tools and compliance standards may duplicate security alerts in a non-obvious way. For example, one compliance standard might issue an alert on an AWS security group open to all ports, protocols, and IP addresses, while another issues an alert because the security group allows access on a database port. They are separate alerts, but alerting about the same security misconfiguration. The inflated volume of alerts creates additional noise for remediation teams to sift through.
Second, traditional cloud remediation typically resources from teams that may be reprioritized, resulting in increased dwell time. Few organizations raise cloud security remediation to a priority that keeps those resources dedicated to the mission.
Third, recruiting and retaining cloud and cybersecurity talent continues to be a challenge for most organizations. Dwell time increases as teams take on unfamiliar challenges. High impact remediation efforts may be deprioritized because teams lack the confidence to deploy without operational impact.
Last, the process for cloud security remediation differs from traditional remediation. At a high level, the processes are similar, but the details about how the cloud is deployed and managed interrupt traditional remediation processes.
We believe that an effective cloud remediation workflow rests on four pillars. We call this Tamnoon’s TARP process for the full remediation lifecycle.
Tamnoon’s TARP process for the remediation lifecycle
| T –––––> | A –––––> | R –––––> | P | |
| What it is | Triage and prioritization | Impact analysis | Resolution | Prevention |
| What it involves | Enrich alerts with business and security context to understand priority based on risk levels | Model different remediation options to choose the path optimizing for security, operations, and cost | Apply the remediation safely through customized playbooks integrated with CloudOps and Security tooling | Learn from remediation events to prevent future occurrence of similar risks |
Delivering on all four pillars requires tight humans-in-the-loop guidance coupled with automation to create a coordinated, scalable process. At each stage, it’s important for security leaders to keep in mind the following questions:
- What does the stage involve? Why does it matter?
- What are traditional approaches to each stage, and where do they fall short?
- How should organizations think about objectives, process, and technology for the stage.
The Bottom Line
In summary, cloud remediation is a “team sport” that requires not just the right technology – but the right processes as well. Differing cloud deployment approaches and the complexity of cloud environments make remediating in the cloud a distinct challenge for security experts, SOC analysts and DevOps alike. Modern cloud environments require a remediation approach that balances both security and production, and coordinates insights from tools with human guidance to drive outcomes.
