8
8
Table of Contents

This article is part of a five-blog series where we share a real client use case — how we reimagined their cloud infrastructure strategy with Crossplane, GitOps, and a hybrid approach with Terraform.

Missed earlier posts? Start with Blog 1 – Why We Looked Beyond Terraform and Blog 2 – Designing a Hybrid Crossplane Architecture.

Bridging from Blog 2

In Blog 1, we shared why Terraform started showing cracks at scale — state chaos, drift, and slow ephemeral environments.

In Blog 2, we designed a hybrid model where Terraform built the stable runway and Crossplane, guided by GitOps, flew agile on top.

But design on paper is one thing. Running it in production was another challenge altogether.
The client’s biggest fear was clear: “Don’t break prod.

With 50+ cloud microservices already serving millions of requests, we couldn’t afford downtime. So we approached onboarding step by step: install Crossplane carefully, create new infra safely, and adopt existing AWS resources without disruption.

Takeaway: Blogs 1 & 2 built the case and the blueprint. Blog 3 is about making it real — without breaking production.

Crossplane in a Nutshell

Before diving into installation, let’s recap how Crossplane actually works in production.

Unlike Terraform’s fire-and-forget model, Crossplane doesn’t just build infra and stop. It continuously babysits resources — watching for drift, fixing mismatches, and making sure infra always matches what’s declared in Git.

For a platform team nervous about importing live AWS resources, this was reassuring: Crossplane wouldn’t “recreate” or “overwrite” unless we explicitly told it to. Instead, it would adopt existing resources and keep them in sync.

Crossplane in a nutshell
Crossplane in action: YAML goes into the friendly Crossplane plane , which talks to AWS APIs, and the reconciliation loop keeps everything in sync.

In simple terms:

  • You define AWS resources as Kubernetes YAMLs.
  • Crossplane controllers communicate with AWS APIs.
  • A reconciliation loop continuously keeps infra in sync with the declared spec.

Takeaway: Crossplane does for cloud infra what Kubernetes does for pods — always on, always in sync.

Core Building Blocks

To understand how Crossplane talks to AWS, you need to know its building blocks:

  • Provider → Plugin that connects Crossplane to AWS (provider-aws).
  • ProviderConfig → Tells the provider how to authenticate (IRSA or Secret).
  • Managed Resources (MRs) → CRDs representing AWS resources (e.g., RDSInstance, S3Bucket).
  • XRDs (Composite Resources) → Optional higher-level abstractions bundling MRs.
    We skipped these in early phases to keep things transparent.

Core Building Blocks of XRDs

Takeaway: Providers connect, MRs define, XRDs abstract. Start raw, then layer abstractions as needed.

Bridging Back to Blog 1 (Hybrid Setup in Action)

As shared in Blog 1, our hybrid model was 90% Crossplane and 10% Terraform.

Terraform handled the foundational setup — including provisioning the AWS EKS cluster with Amazon EKS Provisioned Control Plane where Crossplane runs — while Crossplane took charge of everything above it: Virtual Private Cloud, Amazon Relational Database Service, and other cloud resources.

You can learn about Amazon RDS vs Amazon Aurora here

This balance allowed us to keep Terraform’s stability for the base layer while leveraging Crossplane’s Kubernetes-native reconciliation for everything dynamic.

Terraform, EKS and Crossplane

Prerequisites

Before onboarding resources into Crossplane, the platform team set up some basics to ensure a smooth rollout:

  • A running Amazon EKS cluster.
    Terraform provisions the base EKS cluster (as discussed in Blog 1 & 2).

A running EKS cluster

  • Tools installed:
    Make sure these CLI tools are configured on your local or CI/CD environment:

Kubectl install

kubectl terminal
  • Verify Amazon EKS connectivity before proceeding:
    Before creating the Crossplane namespace, check that kubectl is connected to the correct Amazon Elastic Kubernetes Service cluster:

AWS EKS connectivity verification

If you see an error like:

error

It means your context isn’t set. Reconfigure your Amazon EKS context:

AWS EKS context

  • Namespace:
    Once connectivity is confirmed, create the namespace for Crossplane:

Namespace creation

Namespace terminal log
  • AWS credentials:
    a) Option A (recommended) → AWS IAM Role for Service Account (IRSA).
    b) Option B → Kubernetes Secret with AWS credentials.

For early testing, the team went with a Kubernetes Secret (Option B):

Kubernetes secret (option b)

Create the secret in your cluster:

Secret creation in cluster

secret creation in terminal

Takeaway: Whether IRSA or Secrets, Crossplane needs a secure bridge into AWS before it can manage infra.

Installing Crossplane

Step one in earning trust was to install Crossplane — but cautiously.

We didn’t jump straight into abstractions (XRDs). Instead, we started with just the bare essentials:

  • Install Crossplane via Helm.
  • Add the AWS provider.
  • Configure ProviderConfig with credentials.

This way, Crossplane could connect to AWS, but nothing risky was created yet.

Takeaway: Helm + Provider + Config = a safe, minimal starting point.

The Installation Journey

From Helm install → pods running → AWS provider added → ProviderConfig ready.

Crossplane installation journey
The Crossplane installation journey: from Helm install to pods running, AWS provider added, and ProviderConfig ready.

1. Install Crossplane with Helm

Install Crossplane with Helm

Install Crossplane with Helm

Verify installation:

kubectl get pods -n crossplane-system

Installation verification

You should see crossplane pods running.

2. Install AWS Provider

In this step, we’ll install the AWS provider for Crossplane. You can choose between three options depending on your needs — Community (Free), Official Upbound (Free), or Enterprise (Paid).

Option 1 – Community Provider (crossplane-contrib)

  • Free & Open Source
  • Maintained by the Crossplane community.
  • deal for learning, testing, or non-critical workloads.

provider.yaml

provider.yaml

provider.yaml

Apply it:

kubectl apply -f provider.yaml

kubectl apply terminal

kubectl terminal

Option 2 – Official Upbound Provider (upbound)

  • Free & Open Source (with optional paid support)
  • Maintained by Upbound, the creators of Crossplane.
  • Provides signed builds, enhanced QA, and long-term maintenance.
  • Recommended for production or enterprise use.

api version crossplane

Note: In this demo, we are using the Community Provider (crossplane-contrib).

3. Create ProviderConfig

providerconfig.yaml

Create ProviderConfig

Apply it:

kubectl apply -f providerconfig.yaml

kubectl apply terminal

At this point, Crossplane could talk to AWS — safely, with nothing created yet.

Takeaway: Start small. Helm + Provider + Config = ready to manage AWS, without surprises.

How Crossplane Works (Step by Step)

Think of Crossplane as Kubernetes for cloud infra — always watching, always correcting. Every YAML you apply kicks off a loop, not a one-time script. Here’s how the reconciliation cycle works step by step.

How Crossplane Works (Step by Step)

Here’s what happens under the hood:

  • Define YAML (CRD created) → You declare a resource (e.g., RDSInstance) in Kubernetes YAML.
  • Controller reacts → The Crossplane controller detects the new CRD.
  • Authenticate → It reads AWS credentials from the ProviderConfig.
  • Call AWS APIs → Crossplane provisions or updates the actual AWS resource.
  • Update status → The CRD in Kubernetes is updated (e.g., Ready: True).
  • Continuous reconciliation → Crossplane keeps watching for drift and re-syncs if anything changes.

Takeaway: Terraform builds once. Crossplane builds and babysits forever.

First Managed Resource

With the building blocks in place, it was time to test Crossplane in action. Instead of jumping straight into production, we started small — by creating a simple PostgreSQL database.

Docs

First Managed Resource Docs

First Managed Resource

Apply it:

kubectl apply -f rds.yaml

AWS dashboard

Key Points:

  • Secure credentials -> DB password stored in a Kubernetes secret (db-pass).
  • Safe deletion -> deletionPolicy: Orphan ensured the DB wouldn’t be destroyed if YAML was removed.
  • Status feedback → You could check resource health with:

Check status:

rds status check

rds status check terminal

Takeaway: Start with a non-critical demo resource (like a test DB). Build trust in Crossplane before touching production.

The demo DB proved Crossplane worked — but that was a clean, greenfield resource. The real world is messier. Most enterprises don’t start fresh; they already have dozens of RDS databases, Amazon EKS clusters, and AWS Simple Storage Service buckets in production. The next challenge was to bring these under Crossplane’s management — without breaking them.

Importing Existing Resources

The three shields of safety when importing existing resources: external-name (adopt, don’t create), deletionPolicy: Orphan (don’t delete prod), and managementPolicy: ObserveOnly (watch, don’t touch).

Importing Existing Resources
Crossplane adoption in action: three shields (external-name, Orphan, ObserveOnly) + side-by-side create vs adopt workflow.

This was the make-or-break moment — we weren’t just creating new infra anymore; we were pulling existing production Amazon RDS and Amazon EKS under Crossplane’s control. One wrong step could mean downtime.

Crossplane gave us three critical safety levers to make this possible:

a) external-name → Attach, don’t create

  • By default, Crossplane assumes it needs to create a resource.
  • With external-name, we tell Crossplane: “This AWS resource already exists — just manage this ID/ARN.

Example: 

external name

Real-world meaning: Instead of spinning up a duplicate DB or cluster, Crossplane “adopts” the resource you already have.

b) deletionPolicy: Orphan → Never delete prod by mistake

  • Normally, deleting a CRD in Kubernetes deletes the corresponding cloud resource.
  • With Orphan, we override that behavior: the AWS resource stays alive even if the YAML is removed.

Example:

orphan deletion policy

  • Real-world meaning: If someone accidentally deletes a YAML in Git (or rolls back a commit), production infra won’t vanish. This setting alone has saved many teams from catastrophic outages.

c) managementPolicy: ObserveOnly → Watch before touching

  • In ObserveOnly mode, Crossplane will connect to the resource but will not modify anything.
  • It simply reports the current state back into Kubernetes.

Example:

management policy

  • Real-world meaning: Teams get visibility inside Kubernetes without the risk of Crossplane changing production settings. Once comfortable, you can move step by step toward full management.

Example: Import Existing AWS Subnets and EKS NodeGroup (Community Provider)

This example uses the community provider (crossplane-contrib/provider-aws) to demonstrate how to connect existing networking components to AWS EKS.

Import Existing AWS Subnets

Import Existing Subnets:-

Each subnet is linked using its real AWS Subnet ID through the crossplane.io/external-name annotation.

This allows Crossplane to recognize and manage (but not recreate) the subnets.

Import Existing Subnets script

Crossplane script 2

Import Existing AWS EKS NodeGroup

This brings your AWS EKS NodeGroup under Crossplane management
while referencing the existing cluster and imported subnets.

Import Existing AWS EKS NodeGroup

Import Existing AWS EKS NodeGroup 2

Takeaway

  • external-name – Tells Crossplane to adopt an existing AWS resource instead of creating one.
  • deletionPolicy: Orphan – Ensures your actual AWS resources stay intact even if their YAML definitions are removed.
  • managementPolicy: ObserveOnly – Provides watch-only import mode, but is available only in the Upbound provider.
  • Even with the free community provider, you can safely import networking and EKS resources using these guardrails—no downtime, no duplication, and no risk to production infra.

Note

This demo uses the Community Provider (crossplane-contrib/provider-aws),
which fully supports safe adoption through external-name and Orphan.

However, the managementPolicy: ObserveOnly feature—used for “read-only” imports—is available only in the Upbound Official Provider (provider-family-aws).

If you plan to adopt existing AWS RDS, AWS IAM, or AWS EKS resources in production with complete “observe-only” capability, consider switching to the Upbound provider for full support and long-term stability.

Observability & Troubleshooting

The first imports didn’t always work. Sometimes AWS creds were wrong, sometimes IAM permissions were missing.”

→ Suggest:

“The first imports didn’t always succeed — sometimes AWS creds were wrong, sometimes IAM permissions were missing. Instead of guessing, kubectl describe gave instant visibility into Crossplane’s view of the world.

“Importing is scarier than creating — but with the three shields and kubectl describe, you can adopt production infra without downtime.”

A Real-World Lesson

When we first tested importing a production AWS RDS, someone accidentally skipped ObserveOnly. Crossplane immediately tried to reconcile the settings — which 
caused a brief scare — but thanks to Orphan, nothing was deleted.
From that day on, we enforced a strict rule:

“Always start with ObserveOnly when adopting existing infrastructure”.

Key Learnings

  1. Install Crossplane + AWS Provider
  2. Configure ProviderConfig (IRSA or Secret)
  3. Start with a new resource (e.g. , demo DB)
  4. Import prod infra with external-name + Orphan + ObserveOnly
  5. Always Verify with ‘kubectl describe

Conclusion

By starting small, adopting carefully, and enforcing safety levers, we turned fear (‘don’t break prod’) into trust. With this foundation, the team felt confident expanding Crossplane to additional services such as S3, SQS, and IAM.

Next: Blog 4 – Governance, Security & Lifecycle Protection, where we show how guardrails like Kyverno, RBAC, and IRSA kept infra safe while scaling Crossplane across teams.

12
Let's discuss your cloud challenges and see how CloudKeeper can solve them all!
Meet the Author
  • Neetesh Yadav
    Senior Devops Engineer

    Neetesh specializes in designing, automating, and managing scalable DevOps pipelines across cloud-native infrastructures.

No Comments Yet
Leave a Comment

Speak with our advisors to learn how you can take control of your Cloud Cost