Governance, Security & Lifecycle Protection with Crossplane

This article is part of a five-blog series where we share a real client use case — how we reimagined their cloud infrastructure strategy with Crossplane, GitOps, and a hybrid approach with Terraform.

Missed earlier posts? Read Blog 1 – Why We Looked Beyond Terraform, Blog 2 – Designing a Hybrid Crossplane Architecture, and Blog 3 – Onboarding & Importing AWS Resources.

Bridging from Blog 3

Blog 3 was about safe adoption. Blog 4 is about keeping that adoption safe at scale. In Blog 3, we showed how we onboarded AWS resources with Crossplane — installing the provider, creating our first AWS RDS, and safely importing existing infrastructure using:

• external-name → attach to existing resources.
• Orphan → prevent accidental deletion.
• ObserveOnly → watch without modifying.

That solved the adoption problem. But it immediately raised a bigger question:

“How do we make sure developers don’t accidentally create unsafe infrastructure — or worse, delete production resources?”

This is where governance and lifecycle protection became critical.

Takeaway: Adoption made Crossplane usable. Governance made it safe.

Why We Needed Governance in GitOps

Crossplane gave us declarative infrastructure, but it didn’t guarantee that developers would declare it safely. Without governance, a single YAML mistake could put production at risk.

In early trials, we saw common risks:

• Setting publiclyAccessible: true on RDS by accident.
• Forgetting deletionPolicy: Orphan, risking accidental deletion of production DBs.
• Skipping security basics — like encryption, backups, or tagging.

In a GitOps workflow, every commit flows straight to the cluster. That’s the magic — but also the danger. Without guardrails, one wrong YAML = direct production impact.

Takeaway: GitOps makes delivery fast. Governance makes it safe.

Why We Chose Kyverno

We needed a policy engine that could:

• Validate manifests before they reached AWS.
• Run inside Kubernetes (no external dependencies).
• Be friendly for developers (no new DSL to learn).

There were several options — including OPA/Gatekeeper — but Kyverno stood out because:

• Kubernetes-native → policies are CRDs, no foreign language.
• YAML-first → easy for Kubernetes teams, no Rego needed.
• Validation + Mutation → block unsafe configs or auto-fix them.
• GitOps-ready → unsafe manifests fail in ArgoCD before reaching AWS.

Takeaway: Kyverno matched Crossplane’s philosophy — declarative, Kubernetes-native, YAML-first.

Kyverno vs. OPA/Gatekeeper: Quick Comparison

Narrative Note:

OPA is more flexible for multi-domain policies, but for Kubernetes-native infra governance, Kyverno’s YAML-first approach aligned better with GitOps and developer workflows.

Takeaway: “Kyverno gave us safety rails without slowing teams. OPA stayed in our toolkit for future, complex cross-domain policies.”

Step 1: Installing Kyverno

Installing Kyverno was our first governance milestone. Just like Crossplane, we kept it simple at first — install, verify, and build trust.

Kyverno installation (from official docs):

Verify: 

Step 2: Audit vs Enforce Mode

Before enforcing policies, we started small — focusing on visibility first. We didn’t enforce policies on day one. First, we started in Audit Mode → warnings only (visible in ArgoCD logs).

Once teams were comfortable and had fixed their manifests, we switched to Enforce Mode → strict blocking of unsafe configs.

Takeaway: Warn first, then block — smooth adoption without slowing developers.

Step 3: Enforcing Lifecycle Policies

By Blog 3, one rule had become sacred: never break prod.

That meant no accidental deletions, no unsafe imports, and no public databases slipping through.

We baked this golden rule into Kyverno policies:

• deletionPolicy: Orphan → external infra never deleted.
• managementPolicy: ObserveOnly → imports stay safe until promoted.

Example: Disallow Public RDS

This Kyverno policy blocks any AWS RDS that tries to go public or skips deletion protection:

These policies ensured that even if developers made mistakes, production resources remained safe.

With this in place, unsafe DBs never made it past ArgoCD — GitOps caught them before AWS ever saw them.

Takeaway: Governance isn’t about slowing teams down — it’s about making the safe path the default.

Step 4: Mutation Policies for Safe Defaults

Validation blocks unsafe configurations — but mutation auto-fixes them before they reach production.
Instead of just warning developers, Kyverno automatically adds missing safety defaults, such as deletionPolicy: Orphan.

Even if developers forgot, Kyverno ensured infrastructure stayed safe — protecting production by default.

Takeaway: Mutation policies auto-heal YAMLs — safety without extra steps.

Step 5: RBAC Guardrails + IRSA

Governance isn’t only about YAML policies — it’s also about who can touch what.

To prevent accidental production changes, we introduced strict access boundaries.

Developers got view-only access. This ensured every Crossplane action was authorized and auditable. Infrastructure changes flowed through GitOps pipelines, never through random kubectl commands.

Crossplane itself assumed AWS roles dynamically via IRSA (AWS IAM Roles for Service Accounts) — no more long-lived access keys sitting in Kubernetes Secrets.

Example: RBAC + IRSA Setup

How It Works

• ClusterRole → defines which Crossplane resources (e.g., AWS RDSInstance) can be accessed.
• ClusterRoleBinding → ties that role to the Crossplane ServiceAccount.
• ServiceAccount + IRSA → securely maps the ServiceAccount to an AWS IAM Role, giving temporary credentials to perform AWS API actions.

This design enforces least-privilege access while keeping every action traceable through both Kubernetes audit logs and AWS CloudTrail.

Takeaway:

• RBAC protects the cluster, IRSA secures AWS access — together they make GitOps truly safe for production.
• RBAC restricted infra access, while IRSA secured AWS credentials without static keys.

Step 6: Compliance & Auditability

Once guardrails were in place, compliance came naturally.

GitOps and Kyverno turned reviews from a checklist into a continuous system:

• Every manifest — version-controlled and traceable in Git.
• Every violation — logged in ArgoCD and Kyverno audit reports.
• Every policy — mapped directly to enterprise frameworks.

Compliance-as-code replaced manual governance meetings and Excel trackers.

Mapped to real-world standards:

What used to be manual checklists became automated GitOps rules.

• CIS Benchmarks → encryption, tagging, and no public AWS S3 buckets.
• PCI-DSS → deletion protection and backups for all databases.
• SOC 2 → automated evidence collection via Git history and logs.

Takeaway: “With GitOps + policy automation, compliance stopped being a burden — it became a byproduct of how infra was managed.”

Step 7: Multi-Tenancy Guardrails

In a shared Crossplane platform, multiple teams worked within the same cluster — but with strict boundaries. Governance wasn’t just about safety; it was about containment.

We used namespace-level guardrails to prevent blast radius incidents:

• Team A → full access to AWS S3 resources only.
• Team B → limited to managing AWS RDS instances.
• Shared VPCs → read-only access across teams.

Takeaway: Shared doesn’t mean risky — guardrails made multi-tenancy safe by design.\

A Real-World Example

Right after Blog 3’s onboarding, a developer accidentally pushed an RDS manifest with publiclyAccessible: true.

ArgoCD Sync → Failed

Kyverno’s webhook intercepted the manifest and blocked the change before it could ever reach AWS.

The developer fixed the values.yaml, re-synced, and ✅ the deployment succeeded.

It proved our policies worked — safety was built-in, not bolted on.

Takeaway: GitOps + Kyverno prevented unsafe infrastructure from reaching production — automation became the new safety net.

Step 8: Governance is a Living Framework

Governance isn’t a checkbox — it’s a heartbeat that keeps evolving with your platform.

New AWS services mean new policies. New compliance rules bring new guardrails. Security teams raise new requirements.

Instead of a one-time setup, we treated governance as a living system — continuously adapting, reviewing, and improving through GitOps.

Each GitOps sync became an opportunity to refine and improve our rules.

Takeaway: “Good governance doesn’t slow change — it grows with it.”

Key Learnings

Blog 3 was about adoption — Blog 4 was about safe adoption at scale.

Here’s what we learned along the way:

• Kyverno was the right fit — Kubernetes-native, YAML-first, and GitOps-friendly.
• Audit → Enforce rollout made policy adoption smooth and non-disruptive.
• Mutation policies provided invisible yet effective guardrails.
• RBAC + IRSA secured infra access and eliminated the use of static AWS credentials.
• Compliance checks shifted from manual reviews to automated validation.
• Governance became a continuous journey — not a one-time checkbox.

Conclusion

Governance transformed Crossplane from an infra engine into a trusted enterprise platform.

• Kyverno validated and mutated manifests before they ever reached AWS.
• RBAC enforced least-privilege access for developers.
• IRSA secured AWS credentials dynamically — no long-lived keys.
• Lifecycle safeguards (Orphan, ObserveOnly) protected production infra.
• Compliance and auditability became automated through GitOps pipelines.

With these guardrails in place, leadership gained the confidence to scale Crossplane adoption safely across teams, environments, and services.

Next: Blog 5 – GitOps Integration and the Road Ahead

We’ll show how we fully integrated Crossplane with ArgoCD workflows — including sync waves, drift correction, and ephemeral environments per pull request — to achieve true GitOps-native infrastructure delivery.