How We Automated 79% of Our DevOps Ticket Workload, Without Giving Anyone the Keys

1. The Ticket Treadmill

Every DevOps team knows this queue. "Please add an environment variable to the staging config." "Can you bump this value in prod?" "I need access to this database." Each request is small, safe in isolation, and completely blocking for the developer who raised it. Each one is also a ticket, a context switch, and fifteen minutes of an engineer's attention spent on work a machine should do.

When we measured our own queue over the last two quarters, the shape was striking. 41% of all tickets reaching the platform team were environment-variable updates. Another 38% were access-provisioning requests: databases, developer tooling, CI. Two repetitive, rule-based categories that together made up 79% of our total ticket workload.

Today, that 79% no longer exists as tickets. Developers serve themselves through the Self-Service Portal for Developers we built for exactly this class of work. When a human is needed at all, it is a Team Lead or platform admin tapping Approve on a Slack message. Nobody on DevOps runs a command.

2. The Constraint That Shaped Everything

The easy version of this project is a web form that calls AWS with a powerful service role. That version gets vetoed in the first security review, and it should be. A portal that holds broad credentials is a privilege-escalation bridge: whoever compromises the portal, or finds one missing if statement in it, inherits everything the portal can do.

So we set a constraint before writing any code: the portal must be safer than the manual process it replaces, not just faster. A DevOps engineer handling a ticket by hand applies judgment about scope. Our system had to encode that judgment into boundaries that hold even if the application code is wrong.

That constraint produced three pillars:

1. Scope by construction. Cloud credentials are minted on request, mathematically narrowed to the single resource path the request pertains to. There is no code path that yields a broad credential.
2. Approval-gated writes. Sensitive changes do not exist in AWS until a human approves them. A rejected request leaves no residue.
3. Two-layer audit. Every action is recorded twice, once at the application layer with full business context, and once by AWS CloudTrail, with a join key linking the two.

The rest of this post walks through each pillar in the order a request experiences them.

3. Architecture at 10,000 Feet

The portal is a FastAPI application living in a dedicated operations AWS account, deliberately separated from the accounts that hold the actual configuration. It is fronted by a private, VPC-internal elastic load balancer, reachable only over the corporate VPN. Sign-in runs over SAML through CloudKeeper Prism, the centralized single sign-on product we’ve built for teams looking to simplify this task.  And it runs at CloudKeeper, the same login our engineers already use for every internal tool, so the portal never handles a password. There is no public ingress at all.

Three design choices in this diagram carry most of the security weight:

• Multi-account isolation is the outermost wall. The portal's own account contains no workload configuration. Every read or write of a real parameter requires an sts: AssumeRole into a different account, across a trust policy we control.
• State and secrets never share a store. DynamoDB holds application state (users, team policies, pending approvals, audit records). The parameters themselves live only in SSM Parameter Store in the workload accounts. The portal's catalog table stores parameter names and counts, never values, so a compromise of the portal's database leaks no secrets.
• The Git repository stays the source of truth. Every created parameter is also committed to the deployment repository through an automated merge request, so infrastructure-as-code and Parameter Store can never silently drift.

4. The Real Security Boundary: Per-Request Session Policies

This is the idea the whole system rests on.

The cross-account AWS IAM roles the portal assumes are intentionally coarse. The non-prod read role, for example, grants read on parameter/cln/*, the entire configuration namespace. If the boundary stopped there, the portal would effectively hold the keys to everything.

It does not stop there, because AWS lets you attach an inline session policy at AssumeRole time, and the effective permissions of the session are the intersection of the role's policy and the session policy. The session can only ever be narrower than the role. So on every single request, the portal computes a one-off policy scoped to exactly the environment and service the user is operating on:

The credentials returned by STS for this session cannot read /cln/dev/payments-api/*or anything else, no matter what the application code requests afterward. If a bug (or an attacker) tricks the portal into requesting the wrong path, AWS itself returns an AccessDenied response. The enforcement lives in the cloud provider, below our code, which is precisely where you want your last line of defense.

For production writes, we narrow even further. An approved prod change is applied with a session policy scoped to one exact parameter key, not a wildcard:

• "Resource": "arn:aws:ssm:us-east-1:333333333333:parameter/cln/prod/billing-api/DB_POOL_SIZE"

A few more properties of every assumed session:

Just as important is the role split. We use four distinct roles instead of one:

The prod apply lane physically cannot delete a parameter; the capability is absent from the role itself. And because bypass actions run under a different role ARN, they produce a visibly distinct signal in AWS CloudTrail. Anyone reviewing logs can separate "the normal approval pipeline did this" from "an admin overrode the process" without parsing application logs.

5. Defense in Depth: The Layers Above IAM

The session policy is the last wall, not the only one. A request that reaches STS has already survived five checks, each one enforced server-side:

• Identity. Login is SAML through CloudKeeper Prism only; there are no portal-local passwords. Sessions are short-lived signed cookies (HttpOnly, Secure, SameSite), so a stolen cookie has a small window and scripts cannot read it.
• Team RBAC. A policy table maps each team to the services it may read, write, or create. Ownership is granted by the platform team, never self-declared. The check runs on every request, on the server.
• Module gating. Entire capability areas (parameter management, access provisioning, and so on) are switched per team. Crucially, the gate is enforced at the API layer, not by hiding buttons.
• Path shape. Every parameter path must match a strict /cln/{env}/{service}/{key} regex, and paths are built from validated components that may not contain separators. There is no string a user can submit that reaches AWS as a traversal into someone else's namespace.
• Sensitive-value redaction. Values whose key names match secret-like patterns (passwords, tokens, credentials) are redacted when viewed by anyone outside the owning team, so read access does not become a cross-team secret leak.

The philosophy: the UI is never the control. Every layer that matters answers at the API, and the layer that matters most answers inside AWS itself.

6. Approvals Without a Queue: Slack as the Control Plane

Boundaries alone don't remove tickets; they make it safe to remove tickets. The second half of the story is an approval flow fast enough that nobody misses the old queue.

Routine work flows freely; anything with a blast radius gets exactly one human, and that human is the person accountable for the service, not a central DevOps queue.

The mechanics matter, so here is the full lifecycle of a prod change:

Four deliberate decisions live in that diagram:

• The held-value pattern. The requested value sits in the approvals table, and only there, until the moment of approval. Approve reads the held value and writes it. Reject deletes the pending merge request and never touches SSM. There are no "ghost parameters" that existed for a while and were rolled back; a rejected change simply never happened in AWS.
• Approving is not a permission you get from Slack. Channel membership grants nothing. When a button is clicked, the worker resolves the Slack user back to a portal identity and re-checks authorization against the database: an admin must actually hold the admin role, and a Team Lead may only approve requests belonging to their own team, stamped on the request at submission time so later membership changes cannot shift approvers.
• The Slack integration is outbound-only. The worker uses Slack's Socket Mode, an outbound WebSocket, so the portal needs no public endpoint to receive interaction callbacks. This one choice is what lets the entire system live behind a private, VPN-only load balancer.
• Overrides are loud, not forbidden. Admins can bypass the flow when operationally necessary, but bypasses run on the dedicated bypass role and are labeled distinctly in the audit trail. The escape hatch exists, and using it is unmissable.

The result, from the approver's chair: a card appears in Slack with who, what, where, and old versus new value. They tap Approve, confirm, and are done. That is the entire remaining DevOps involvement in what used to be a ticket.

7. SOC2-Grade Audit: Two Layers, One Join Key

SOC2 evidence questions reduce to: who did what, to which resource, when, and what was the outcome? We answer at two independent layers.

Layer 1: an append-only application audit table. Every action writes a record to a DynamoDB table that the application only ever inserts into; there is no update or delete path in code. A single record carries the full business context:

Details worth stealing:

• Denials are logged, not just successes. A request that fails RBAC writes a record with rbac_decision: "deny" and a reason. For an auditor, proof that the control rejects is as important as proof that it permits; for us, denial records are the early-warning feed for misconfigured teams and probing behavior.
• Every actor in a decision is captured. An approved change produces records for the request, the approval, and the apply, tied together by approval_id. Rejections carry the rejector and their stated reason. Admin overrides carry an explicit admin-bypass-* action label.
• Secondary indexes make investigations cheap. The table is queryable by user, by environment-plus-service, and by approval, so "everything this person touched this quarter" is one query, not a log-trawling exercise.

Layer 2: CloudTrail, with a join key. The application table says what the portal intended; CloudTrail records what actually happened inside AWS, written by the platform rather than by our code. The two layers meet at the RoleSessionName. Because every assumed session is named ck-{action}-{user}-{epoch} (or ck-approval-{id} for approval applies), every PutParameter event in the workload account's CloudTrail carries the acting portal user or approval in its identity block. An investigator can start from an AWS CloudTrail event and land on the exact portal request, with its approver and diff, in seconds, or start from an audit row and confirm AWS saw exactly that call and nothing more.

Neither layer trusts the other, and that is the point. Application logs alone can be doubted ("the app could write anything"); CloudTrail alone lacks business context ("a role wrote a parameter, but on whose behalf, and who approved?"). Together, with a shared key, they close the loop.

8. What Changed

The numbers moved the way we hoped. Environment-variable changes, 41% of the platform team's ticket workload over the last two quarters, went fully self-service on day one. Access provisioning for databases, developer tooling, and CI, another 38% of the workload, flowed through the same approval-gated architecture. Together, that is 79% of our total ticket workload, automated end to end.

The remaining 21% is the roadmap, not the residue. The next targets are service-setup automation, so that standing up a new service becomes a portal flow rather than a project, and AI-powered debugging summaries for deployment failures, so that even the tickets we cannot eliminate arrive pre-diagnosed rather than raw.

Just as telling is what DevOps involvement looks like now. For the majority of requests: nothing. For the rest: a Slack card and one click by the person who owns the affected service. The platform team's role shifted from executing changes to defining policy: which team owns which service, which modules each team can use, and where approval is required. The system executes.

And because every boundary is structural, the usual anxiety of self-service, "what if someone finds a way around it?", has a concrete answer: around it to where? The credentials in play at any moment cannot reach past a single service path; prod writes cannot happen without an approval row and cannot delete anything, and both AWS and the portal are independently writing down everything that occurs.

9. Takeaways

Three principles from this build that transfer to any internal platform touching cloud resources:

1. Make session policies the boundary, not roles. Roles define the ceiling of what your platform could ever do; a per-request inline session policy defines what this request can do. The intersection is enforced by AWS, below your code, immune to your bugs.
2. An approval is a held value, never a pre-written one. Nothing should exist in the target system until approval. Reject must be a no-op on the world, not a rollback.
3. Log the denials. Allowed actions tell auditors your system works; denied actions prove your controls do. They are also your best security telemetry.

None of this required exotic technology: STS, DynamoDB, Slack, and a strict habit of asking "what enforces this if the code is wrong?" The 79% was the reward for answering that question at every layer.