How to Design the Hybrid Crossplane Architecture?

This article is part of a five-blog series where we share a real client use case — how we reimagined their cloud infrastructure strategy with Crossplane, GitOps, and a hybrid approach with Terraform.

📖 Missed the first part? Read Blog 1 – Why We Looked Beyond Terraform before continuing.

Bridging from Blog 1

In Blog 1, we shared why we looked beyond Terraform. It worked well for provisioning static infrastructure, but as our platform grew to 50+ microservices with global ambitions, cracks began to show.

• Ephemeral environments: Developers needed to spin up infra for every pull request, but Terraform’s state locks made this slow and painful.
• Drift management: If AWS resources drifted, Terraform wouldn’t act until the next manual apply — a risky delay at scale.

Crossplane solved these gaps with its continuous reconciliation model. Just like Kubernetes keeps pods aligned with their declared spec, Crossplane keeps infra in sync with Git. That made it the better fit for our growing platform.

Takeaway: Terraform provisioned well, but Crossplane managed better.

In this post, we’ll dive into the hybrid architecture we designed — balancing Terraform’s stability with Crossplane’s agility, all running inside a single Kubernetes cluster.

A Real-World Tradeoff

One of the first debates we faced was simple but critical:

“Go all-in on Crossplane or keep Terraform for the base?”

• App teams wanted Crossplane for its GitOps-native workflows and self-service ephemeral environments.
• Platform engineers were cautious. Base infra (Virtual Private Cloud, networking, Amazon EKS Provisioned Control Plane) was too sensitive to risk in an early migration experiment.

The compromise:

• Terraform → Foundation: VPC, subnets, AWS EKS control plane (stable, rarely changes, already trusted).
• Crossplane → Dynamic layer: AWS RDS, AWS S3, AWS SQS (developer-facing, high-churn, agility critical).

Takeaway: Terraform gave us stability, Crossplane gave us speed. Together, they gave us both.

The Hybrid Design: Terraform + Crossplane in One Cluster

We landed on a two-layer hybrid design — both inside the same Kubernetes cluster:

Terraform (Foundation Layer)

• Provisioned VPCs, subnets, and the AWS EKS control plane.
• Best for stable, less-frequently changing infra.

Crossplane (Dynamic Layer)

• Managed AWS RDS, AWS S3, AWS SQS directly from Kubernetes.
• Continuously reconciled with AWS → no drift.
• Integrated with ArgoCD → fully GitOps-native.

Takeaway: Terraform laid the runway. Crossplane flew above it. GitOps was the control tower.

Terraform builds the base. Crossplane manages the edge. GitOps keeps them in sync.

"This hybrid design let us use each tool where it was strongest — Terraform for the stable foundation (VPCs, networking, AWS EKS control plane), and Crossplane for dynamic, developer-facing resources like AWS RDS, AWS S3, and AWS SQS. By combining them in a single cluster, we gained both stability and agility, all fully reconciled through GitOps with ArgoCD."

Why This Approach Worked

The hybrid model worked because it played to the strengths of each tool:

• Terraform gave stability → Base infra like VPCs and control planes rarely change, and Terraform was already trusted here.
• Crossplane unlocked agility → Perfect for high-churn resources such as databases, queues, and storage.
• A single cluster simplified ops → Fewer moving parts, less overhead.
• Scalability for teams → Developers self-served infra through Git, while platform teams enforced guardrails.

Takeaway: Hybrid wasn’t a compromise — it was optimization.

Standardized Developer Workflow

Once the hybrid foundation was in place, the next challenge was consistency.
With 50+ microservices, consistency mattered. We standardized everything with Helm + GitOps:

Each microservice had two Helm charts:

• Application chart → deployments, configs, autoscaling.
• Infrastructure chart →Crossplane-managed resources.

Developers only touched values.yaml — e.g.:

Guardrails (versioning, encryption, deletionPolicy) were hardcoded in templates, so developers got freedom with safety.

Takeaway: Developers declared intent. Guardrails enforced compliance.

Modular Repository Structure

After standardizing workflows, the next challenge was scaling infra code across teams. We solved this by modularizing Helm charts.

We modularized infra into reusable Helm charts:

• terraform-base/ → VPCs, AWS EKS control plane.
• crossplane-eks/ → node groups, cluster add-ons.
• rds-guard/ → AWS RDS with encryption & backups.
• s3-secure/ → AWS S3 buckets with versioning & policies.

Takeaway: Modular charts = reusability, consistency, and built-in security guardrails.

Separate Repositories for Terraform & Crossplane

Even with modularity, one problem remained: Terraform and Crossplane in the same repo created friction. The solution was clear — separate them.

Terraform Repository

• Handled base, less-frequently changing infrastructure
• Examples: VPCs, Subnets, Core Networking, AWS EKS Control Plane

Crossplane Repository

• Ran inside the Kubernetes cluster for dynamic, developer-facing resources
• Examples: AWS RDS Instances, S3 Buckets, SQS Queues, Provider configs & guardrails

This separation gave platform teams control of the foundation with Terraform, while enabling developers to move fast with Crossplane without worrying about Terraform state files.

Takeaway: Separation gave platform teams control and developers freedom.

Infra Workflow in Action

End-to-end flow in the hybrid model:

1. Terraform provisions the base (VPC, subnets, AWS EKS control plane).
2. Crossplane is deployed in the cluster to manage AWS RDS, AWS S3, and AWS SQS.
3. Developers update values.yaml in Git.
4. ArgoCD syncs changes.
5. Crossplane reconciles infra in AWS.

Takeaway: Infra requests became Git commits. No tickets. No manual drift fixes.

Conclusion

Terraform gave us stability. Crossplane gave us speed. GitOps kept it aligned. Together, they created a balanced model that scaled without slowing teams down.

Takeaway: “Hybrid Infra isn’t compromise — it’s the sweet spot”.

Next: Blog 3 – Onboarding AWS Resources & Importing Existing Infra, where we show how we safely onboarded VPCs, EKS, and RDS with Crossplane and imported existing AWS resources without downtime.