Automating AWS Compliance: From Detection to Remediation

Most teams running AWS at any scale end up with Security Hub enabled: a few hundred buckets, instances, roles, and databases spread across several regions, continuously graded against a benchmark like CIS AWS Foundations. When a control fails, Security Hub says so immediately. That part has always worked.

What did not work was everything after that. A control would go red, someone would open the console, find the offending resource, fix it by hand, and watch the score go green again. A week later, it would go red again because whatever produced the finding was still happening. That was our position for months: fixing symptoms one resource at a time, and paying for it on repeat.

So we built a small remediation layer that closes the loop. When a resource appears without the configuration a control requires, we apply the fix within seconds, before anyone sees a finding. This is the story of why and how.

The Problem: Detection Was Never the Bottleneck

Take one control as an example: S3.5, AWS S3 buckets should require requests to use SSL. The fix is a single deny statement on the bucket policy, conditioned on aws:SecureTransport being false. It takes about ninety seconds to apply by hand.

Ninety seconds is not the problem. Doing it forever is. Four things made the manual loop expensive:

It fired on human behavior, not on deployments. Our buckets were being created by engineers in the console, by five different people over the audit window, not by Terraform. A policy in our IaC(Infrastructure as Code) modules would have covered none of them.

Fixing it did not stop it. Every fix addressed one bucket. The next one created went out non-compliant exactly like the last, and the score began sliding the moment we stopped watching.

The score only showed part of the estate. Security Hub findings come from AWS Config rules, and Config records per region. Ours ran in one region while our buckets lived in three. Of seven genuinely non-compliant buckets, only two ever appeared as findings. The other five were not compliant. They were invisible, which is a very different thing.

Nothing accumulated. The same fix, applied by whoever was on hand, learned nothing and left nothing behind. Nothing improved; only people got tired.

The bottleneck was never detection. Security Hub had been telling us for months. It was that correction ran at human speed and human frequency, and resource creation did not.

The Reframe: Fix It Where It Is Created

The instinct with a failing control is to work the finding list: sort by severity, fix what is red, get the score up. But the finding list is a symptom. Working it faster just means running the same manual loop with more urgency.

We reframed the goal. Rather than closing findings, we wanted findings never to open: the moment a bucket is created without the required policy, something applies it in seconds, unattended.

That set our constraints. The automation had to catch resources at creation rather than only on change, apply the fix without ever damaging configuration that was already there, be safe to run repeatedly against the same resource, cover the resources that already existed as well as the ones still to come, and above all, never trade a compliance finding for an outage.

What We Considered

We built more for this problem than we expected, and custom code should be the last option, not the first.

AWS Config remediation with SSM Automation was the obvious route. Config already evaluates the rule; attaching a remediation configuration to it needs no new event plumbing. AWS also publishes a large family of managed runbooks (the AWSConfigRemediation-* documents) that cover many common controls outright. We ruled it out for our case for two specific reasons: no managed runbook existed for this remediation, so we would have been authoring a custom automation document anyway, and Config remediation only reaches regions where a recorder is running, which would have meant paying for recorders in two more regions to cover five buckets.

Preventive controls, meaning service control policies and now resource control policies, genuinely stop insecure access at the organization boundary, and where you can use them they are the better answer. But they didn't solve our problem: the control reads the bucket policy, so an org-level deny leaves S3.5 failing even when the bucket is, in practice, secure. A guardrail and a green control are not the same thing, and it is worth checking what a rule actually evaluates before assuming one implies the other.

Policy-as-code tools were the other serious option. Cloud Custodian in particular expresses exactly this kind of rule as YAML filters plus actions, runs on a schedule or from events, and would have covered several controls without a code deployment each. For a broader program it is the right shape. For a single control, we chose not to take on another framework to operate.

Shift-left scanning with checkov, tfsec or OPA in the pipeline is the cheapest fix of all when it applies, because nothing non-compliant is ever created. It applied to none of our buckets, for the reason above: they were not created through a pipeline.

We chose EventBridge on the CloudTrail event, routed to a Lambda function. It fires within seconds of the resource appearing, needs no Config recorder in regions we hadn't enabled, and passes the resource identifier directly in the event.

Design Principles

Four principles shaped the build, and they map onto the order the function actually runs.

Trigger on creation, not just on change

Reviewing the automation that already existed in the account, we found three EventBridge rules and two Lambda functions from earlier attempts at this same control. Each one watched PutBucketPolicy and DeleteBucketPolicy. None watched CreateBucket.

That is the entire gap. Those rules re-applied the policy when somebody removed it, which is a real regression path and worth keeping, but they never fired for a bucket created without a policy in the first place, and that was the case generating all the manual work. The rules looked like coverage and provided none.

The general form of this: for any control, ask what full set of events can produce a non-compliant resource. Creation is almost always one of them, and almost always the one that gets left out.

Merge, never overwrite

This is the single most important decision in the whole design.

A bucket policy is not scratch space. It carries cross-account replication grants, service access, log delivery, CDN origin access: configuration that other systems depend on. A remediation that writes its own policy over the top of that trades a compliance finding for an unattended outage, at whatever hour the bucket happened to be created.

So the function reads the current policy, starts from an empty document only if there genuinely is not one, checks whether the control is already satisfied and stops if it is, removes any earlier copy of its own statement by statement ID, appends the one statement the control requires, checks the result against the 20 KB policy limit, and writes the whole document back. Removing its own previous statement before re-adding it keeps repeated runs idempotent instead of stacking near-duplicates until the policy no longer fits.

Mirror the control exactly; never exceed it

When we wrote the "is this already compliant?" check, we instinctively required the strictest version of the statement: a deny covering both the bucket ARN and the object ARN.

The Config rule behind the control is more permissive. s3-bucket-ssl-requests-only accepts a deny scoped to the object ARN alone, and several AWS service-managed buckets in our account are written exactly that way and graded compliant. Had we shipped the stricter check, the function would have "fixed" every already-passing bucket, on every event, forever, rewriting AWS-managed policies to close findings that did not exist.

The control's definition is the specification. Be permissive on read, strict on write: accept anything the control accepts, and when you do act, write the better version.

Degrade safely, never break what you are protecting

The remediation must not become a new source of incidents. Failed API calls still emit AWS CloudTrail events, so the function exits on any event carrying an error code rather than chasing a bucket that was never created. Create calls are not reliably read-your-writes consistent, so a read immediately afterward retries with backoff, but only on genuinely transient codes, because a retry loop that swallows a permissions error turns a loud failure into a silent one. And because the fix is itself a policy write, it matches its own trigger; the compliance check breaks that loop on the second pass, and a filter on the calling principal stops the extra invocation.

Architecture: An Event Rule, a Function, and a Policy Merge

The system is deliberately small. No new service runs: one EventBridge rule, one Lambda function, and one AWS IAM role.

Event-driven remediation only fixes the future, so it shipped alongside a one-off sweep for existing buckets, report-only by default, with an explicit flag to write. The important detail is that the sweep imports the same compliance check and the same fix as the live function. Two independent definitions of "compliant" drift, and you end up with a backfill that disagrees with your remediation about which resources need fixing.

One Skeleton, Several Controls

Almost none of that design is specific to TLS. The skeleton is the same for any control worth automating: decide which API calls can produce a non-compliant resource, read the current state, exit if the control is already satisfied, apply the smallest change that satisfies it, and leave everything else alone. Changing the control means changing three things, the read call, the compliance test and the write call, and nothing else.

Here is how the same shape lands on the other controls we picked up, chosen because they generated repeat findings.

S3 buckets should block public access. Structurally the easiest of the set, because there is nothing to merge. Public access block is its own sub-resource rather than part of the bucket policy: four booleans, and the remediation writes all four as true. The triggers are CreateBucket, plus PutBucketPublicAccessBlock and DeletePublicAccessBlock to cover somebody turning it off later. The trap is not technical. Some buckets are public on purpose, whether that is a static site origin, a published dataset, or a bucket fronting a CDN without origin access control, and a remediation that does not know the difference takes them offline. So this one needs an exception path before it can run unattended: a tag the bucket owner sets, read by the function before it writes anything, so the exception is visible in the console rather than buried in code.

KMS key rotation should be enabled. A single EnableKeyRotation call, triggered on CreateKey and on DisableKeyRotation. It is about as safe as a remediation gets, because rotation is transparent to callers: AWS retains the previous backing keys, so data encrypted under an older version still decrypts normally. The subtlety is eligibility. Automatic rotation applies only to customer-managed symmetric encryption keys whose key material AWS generated. Asymmetric keys, HMAC keys, imported material, keys in a custom key store, AWS-managed keys and anything pending deletion cannot be rotated at all. The function has to read the key's metadata and skip those cleanly, because a call that can never succeed should not look like a transient failure.

AWS IAM access keys should be rotated every 90 days. This one reads like the others. A credential is out of policy, and the fix is to replace it. It does not behave like the others at all, and it is the reason for the next section.

Knowing What Not to Automate

Not every failing control should be handed to a machine, and the finding text is no guide at all, because the dangerous ones read exactly like the safe ones.

The distinction that has held up for us is between a control whose fix puts a resource into a configuration, and one whose fix destroys or replaces something in use.

The first three controls above sit firmly in the left column, which is why they can run unattended. Access key rotation sits in the right one, and looks deceptively similar: "rotate keys older than 90 days" reads like configuration drift, but it is a controlled outage. The old credential stops working, and whatever still holds it stops working too. For those, we automate everything up to the edge: create the replacement, stage it, notify with the exact commands needed to finish, and stop there. Not because the code could not take the last step, but because the decision to break something live belongs to somebody who can watch it.

Results: The Finding That Never Opens

The change is easiest to see in what stopped happening.

Before: a bucket is created in the console. Config evaluates it on its next cycle. A finding appears in Security Hub. It sits in the queue until someone triages it, opens the console, and applies the policy by hand, typically hours to days later, and only if the region was one we could see at all.

After: the same bucket is created. EventBridge fires on the CloudTrail event within seconds, the function reads the empty policy, writes the deny statement, and the bucket is compliant before Config has evaluated it. There is no finding to triage, because there is nothing non-compliant left to find.
What that bought us:

The recurring work disappeared rather than getting faster. These controls no longer generate manual tickets, because the state they complain about no longer persists long enough to be found.

Regression is covered too. Because the same function fires on policy writes and deletes, a bucket whose policy is replaced by hand, a common way a resource silently regresses after a console edit, is corrected the same way.

The score reflects the estate more honestly. Not because the number improved, but because we wrote down what stayed out of scope. We deliberately deployed to one region and accepted that four buckets elsewhere remain non-compliant and invisible. That gap is now a recorded decision rather than something nobody had noticed.

Post-Implementation Review

The gap is almost always the creation event. Every prior attempt in the account watched for the configuration being removed, and none watched for the resource arriving without it. If you inherit compliance automation, read what it actually triggers on before adding more. A rule that fires on the wrong event is indistinguishable from coverage until you check.

Merging, not overwriting, is what makes it safe to leave alone. Everything else in the design is a detail beside this one. A remediation with write access to resource policies is one blind put away from an outage, and it will run at three in the morning with nobody watching.

A check stricter than the control is a bug, not extra safety. It does not make you more compliant. It makes you a generator of unnecessary writes against resources you may not own. Verify the real threshold empirically: find a resource the control already grades compliant, and read what it actually has.

A compliance score is scoped to wherever Config is recording. Everywhere else is not compliant. It is invisible. We only found five of our seven non-compliant buckets by sweeping every region with the SDK, and the dashboard had never mentioned them.

The boring failure is the likely one. Our first live test failed with no policy applied. The cause was not the event rule, the IAM role or the logic, all of which were correct: the Lambda console pre-fills the handler name as lambda_function.lambda_handler, and our entry point was called something else. Every invocation died at import. Design the first live test so that class of bug is visible: create a throwaway resource, confirm the fix landed, and read the invocation logs rather than assuming.

Conclusion: From Score to System

Security Hub will always tell you that a control is failing. The gap has always been the distance between that red control and a resource that stays fixed. Closing it by hand, one resource at a time, was costing us real engineering hours for work that produced nothing durable.
The fix was not a better dashboard or a stricter review process. It was to move correction to the moment of creation, merge rather than overwrite, and be honest about which controls a machine should be allowed to close on its own.
A compliance score is not a project you finish; it is a level you hold. Holding it by hand means repeating yourself forever. Holding it with a system means the finding never opens in the first place.