From Failing Cloud Controls to Buyer-Ready Evidence: A Worked Remediation Example

Most writing about controls stops at the finding: "MFA isn't enforced," "audit logs are off," "the production branch isn't protected." The hard part — and the part a buyer's security team actually cares about — is everything after the finding: the change you made, the proof that it took, and the artifact you can hand over without leaking anything you shouldn't.

This is that part, worked end to end on a single fictional company. It follows one review from the email that triggered it to the evidence pack that closed it. The stack is Google Cloud plus GitHub because that's where the concrete configuration is easiest to show; the same shape applies on AWS or Azure, and real engagements routinely cover those.

Fictional data, and not executed as written. "Northwind Signals" is not a real company. Project IDs, account names, numbers, and dates are invented to show the shape of the work. The Terraform and CLI snippets are written against current GCP and GitHub provider documentation to illustrate the approach — they have not been applied to a live project in this form. Resource names, role choices, and scopes change; validate anything here in a disposable project, and confirm the control mapping with your auditor, before relying on it.

The trigger

Northwind Signals is a 14-person B2B SaaS selling an analytics product. A mid-market prospect — call them the buyer — sent a security questionnaire as a gate on a $90k/year contract. Three line items came back as hard blockers after the buyer's first pass:

1. "Confirm audit logging is enabled across your production cloud and retained for at least one year."
2. "Confirm least-privilege access; provide evidence no individual holds project-wide administrative access."
3. "Confirm your production branch enforces code review and status checks before merge."

Northwind runs everything in one GCP project (nw-prod-8842) and one GitHub org. There's a Vanta trial that had already flagged all three, plus a handful of others, but nobody had done the engineering to clear them.

The findings

Before touching anything, the current state gets captured — both because you can't prove a fix without a before, and because the "before" is the first thing that tells you how bad it is.

Confirming the "before" with commands, not the dashboard, matters: the export is the first evidence artifact, and it's the thing you diff against at the end.

# Finding 1 — audit config and retention, before
gcloud projects get-iam-policy nw-prod-8842 --format="json(auditConfigs)"
# -> {}   (no Data Access logging configured)
gcloud logging buckets describe _Default \
  --location=global --format="value(retentionDays)"
# -> 30
 
# Finding 2 — who holds broad roles, before
gcloud projects get-iam-policy nw-prod-8842 \
  --flatten="bindings[].members" \
  --filter="bindings.role:(roles/owner OR roles/editor)" \
  --format="table(bindings.role, bindings.members)"
 
# Finding 3 — branch protection, before
gh api repos/northwind-signals/api/branches/main/protection
# -> 404 Not Found

Prioritization

All three are High and all three are deal-blocking, so the ordering isn't by severity — it's by blast radius and reversibility. The rule: do the changes that can't break production first, and the ones that can, last, behind a maintenance window.

1. Audit logging + retention — additive, in the sense that it removes no permission and blocks no deploy. It is not free of consequences, though: see the rollout risks under Fix 1 before you enable DATA_READ everywhere. Do it first; it also starts the clock on collecting real log evidence.
2. Branch protection — low blast radius, but it can wedge open PRs the moment required status checks turn on. Do it second, after a heads-up to the team, with the exact check names confirmed.
3. Least-privilege IAM — highest blast radius. Removing roles/editor from the CI service account will break the deploy if any pipeline step needed a permission the replacement roles don't grant. Do it last, staged, with a tested rollback.

Fix 1 — audit logging and one-year retention

Everything goes through Terraform, not the console, so the fix survives the next person clicking around and so the change itself is reviewable.

# audit.tf — enable Data Access logging for all services
resource "google_project_iam_audit_config" "all_services" {
  project = "nw-prod-8842"
  service = "allServices"
 
  audit_log_config {
    log_type = "ADMIN_READ"
  }
  audit_log_config {
    log_type = "DATA_READ"
  }
  audit_log_config {
    log_type = "DATA_WRITE"
  }
}
 
# Retain the _Default log bucket for 400 days (>1 year, with headroom)
resource "google_logging_project_bucket_config" "default_retention" {
  project        = "nw-prod-8842"
  location       = "global"
  bucket_id      = "_Default"
  retention_days = 400
  locked         = false # leave unlocked until retention is confirmed correct
}

Rollout risks — read before copying this

Admin Activity logs are always on and free. The change above additionally enables Data Access logs, and DATA_READ on allServices is the setting people regret. Google's own Data Access audit logs guidance is explicit that it can:

• Generate very large log volume, with matching ingestion cost. On a quiet 14-person project this might be a rounding error; on a project doing high-QPS reads against Cloud Storage, BigQuery, or Firestore it can dwarf the rest of your logging bill. Nobody can tell you the number in advance — enable it on one service, watch the volume for a day, then widen.
• Affect authenticated Cloud Storage browser downloads, which is the surprise that generates the support ticket.

So the honest sequence is: enable ADMIN_READ broadly, turn on DATA_READ / DATA_WRITE per service rather than allServices first, add exclusion filters for the noisy high-volume readers you don't need sampled, and confirm the bill before calling it done. Enabling everything at once is how a control meant to please a buyer turns into a cost incident.

Verify

gcloud projects get-iam-policy nw-prod-8842 --format="json(auditConfigs)"
# -> auditConfigs now lists allServices with ADMIN_READ/DATA_READ/DATA_WRITE
gcloud logging buckets describe _Default \
  --location=global --format="value(retentionDays)"
# -> 400

Then the part people skip: generate a real log line and prove it's queryable, so the evidence shows the pipeline works, not just that a setting is flipped.

gcloud logging read \
  'logName:"cloudaudit.googleapis.com%2Fdata_access"' \
  --limit=1 --freshness=1d --project=nw-prod-8842

Fix 2 — branch protection as a ruleset

The buyer asked about main specifically, so both production repos get an identical ruleset: required review, required status checks, no force-push, no deletion. Using a ruleset (not the legacy branch-protection API) because it's what GitHub now recommends and it's cleaner to express in Terraform.

# github.tf
resource "github_repository_ruleset" "protect_main" {
  for_each    = toset(["api", "web"])
  name        = "protect-main"
  repository  = each.value
  target      = "branch"
  enforcement = "active"
 
  conditions {
    ref_name {
      include = ["~DEFAULT_BRANCH"]
      exclude = []
    }
  }
 
  rules {
    deletion         = true # block branch deletion
    non_fast_forward = true # block force-push
 
    pull_request {
      required_approving_review_count   = 1
      dismiss_stale_reviews_on_push     = true
      require_code_owner_review         = true
      require_last_push_approval        = true
    }
 
    required_status_checks {
      required_check {
        context = "ci/test"
      }
      strict_required_status_checks_policy = true # require branch up to date
    }
  }
}

The one that bites: context = "ci/test" has to match the check name your CI actually reports, exactly. Get it wrong and every PR blocks forever on a check that never arrives. Confirm the live name before applying:

gh api repos/northwind-signals/api/commits/main/check-runs \
  --jq '.check_runs[].name'
# -> "ci/test"   (matches the ruleset context above)

Verify

Two different kinds of evidence here, and it's worth not conflating them.

Configuration evidence — the ruleset exists and says what we think it says:

gh api repos/northwind-signals/api/rulesets --jq '.[] | {name, enforcement}'
# -> {"name":"protect-main","enforcement":"active"}
gh api repos/northwind-signals/api/rulesets/<id> --jq '.rules[].type'
# -> pull_request, required_status_checks, non_fast_forward, deletion

Functional evidence — the ruleset actually rejects a write. This needs care: running a bare git push origin main with nothing to push just returns Everything up-to-date, which exercises nothing and proves nothing. You have to push a real commit at the protected ref, and you don't want to do that experiment with anything you'd mind landing:

# Throwaway commit on a scratch clone — never in someone's working tree
git clone --depth=1 git@github.com:northwind-signals/api.git /tmp/rs-check
cd /tmp/rs-check
git commit --allow-empty -m "ruleset check (should be rejected)"
git push origin HEAD:main
# -> remote: error: GH013: Repository rule violations found for refs/heads/main
# -> remote: - Changes must be made through a pull request
# ! [remote rejected] HEAD -> main (push declined due to repository rule violations)
rm -rf /tmp/rs-check   # nothing landed; the commit only ever existed locally

The rejected push is the stronger artifact — it shows the control does something rather than merely is configured. But label the two honestly in the pack: a ruleset export is configuration evidence, and only the rejection transcript is functional proof.

Fix 3 — least privilege, staged

This is the one that can break the deploy, so it's staged: figure out what the CI service account actually uses, grant exactly those roles, then remove roles/editor, with the old binding one command away from being restored.

Northwind's CI only builds a container, pushes it to one Artifact Registry repository, and deploys one Cloud Run service. The important part is not just "predefined roles instead of editor" — it's binding each role at the narrowest resource that works, not at the project. A project-wide roles/artifactregistry.writer still lets CI write to every repository you ever create; a project-wide roles/iam.serviceAccountUser lets it impersonate every service account in the project. Both are a smaller blast radius than editor and still wider than this pipeline needs.

# iam.tf — replace roles/editor on the CI service account with roles bound at
# the specific repository / service account / service, not the project.
locals {
  ci_sa      = "serviceAccount:ci-deploy@nw-prod-8842.iam.gserviceaccount.com"
  region     = "us-central1"
  runtime_sa = "api-runtime@nw-prod-8842.iam.gserviceaccount.com"
}
 
# Push images to ONE repository, not every repo in the project.
resource "google_artifact_registry_repository_iam_member" "ci_push" {
  project    = "nw-prod-8842"
  location   = local.region
  repository = "app-images"
  role       = "roles/artifactregistry.writer"
  member     = local.ci_sa
}
 
# Act as ONE runtime service account, not every SA in the project.
resource "google_service_account_iam_member" "ci_act_as_runtime" {
  service_account_id = "projects/nw-prod-8842/serviceAccounts/${local.runtime_sa}"
  role               = "roles/iam.serviceAccountUser"
  member             = local.ci_sa
}
 
# Deploy revisions to ONE existing service. run.developer, not run.admin —
# admin additionally allows IAM changes on the service (e.g. making it public).
resource "google_cloud_run_v2_service_iam_member" "ci_deploy" {
  project  = "nw-prod-8842"
  location = local.region
  name     = "api"
  role     = "roles/run.developer"
  member   = local.ci_sa
}

Two caveats worth stating plainly, because they're where this gets misapplied: service-scoped run.developer covers deploying revisions to an existing service; creating a brand-new service still needs the role at project level, so service creation stays a human/Terraform-apply step here. And run.developer is the narrowest role that worked for this pipeline — check your own deploy against the Cloud Run IAM roles reference rather than copying this.

The human grants — where the ask is genuinely not met yet

The buyer's ask was "provide evidence no individual holds project-wide administrative access." It's tempting to remove the two engineers' roles/owner and call the row closed. It isn't, and this is the part worth being precise about.

Swapping owner for editor would be theatre — editor is a broad primitive role that can still modify almost everything. So the two engineers drop to roles/viewer plus the specific roles their work needs. But somebody still needs a way back in when IAM is misconfigured, and the usual answer — a break-glass account holding standing roles/owner — means the literal answer to the buyer's question is still yes, an individual holds project-wide admin. That account is an individual. Reporting "no individual owner" while breakglass@ sits there with owner is the kind of claim a second reviewer checks and catches.

Two defensible ways forward, and you have to pick one out loud:

1. Keep standing break-glass and report the ask as open, with the compensating controls named: dedicated account, hardware-key MFA, no routine use, alert on every authentication, quarterly attestation. Many buyers accept this. What they don't accept is finding it themselves.
2. Remove standing Owner entirely and grant it just-in-time. GCP's Privileged Access Manager issues a time-bound, approved, audited roles/owner grant on request, so the steady state genuinely has no individual owner and the ask closes cleanly.

Northwind took (1) for the deal deadline and scheduled (2) — which is why the map table below carries Open, not Closed, on that row.

resource "google_project_iam_member" "eng_viewer" {
  for_each = toset(["engineer-a@northwind.example", "engineer-b@northwind.example"])
  project  = "nw-prod-8842"
  role     = "roles/viewer"
  member   = "user:${each.value}"
}
 
# The one engineer who ships gets deploy rights on the service only.
resource "google_cloud_run_v2_service_iam_member" "eng_a_deploy" {
  project  = "nw-prod-8842"
  location = local.region
  name     = "api"
  role     = "roles/run.developer"
  member   = "user:engineer-a@northwind.example"
}

Even this leaves a real finding open: roles/viewer is itself a primitive role and grants read across the project, including some data reads. Replacing it with job-function roles per service is a follow-on piece of work, and the evidence pack says so rather than claiming human IAM is solved. Reporting a control as "closed" when it's "narrowed" is how evidence packs lose credibility with the second reviewer who checks.

The sequence that matters: apply the narrow bindings, run one real deploy to prove nothing broke, and only then remove the broad grant.

# 1. apply the narrow roles (above), then run a real deploy through CI — it passes
# 2. only now remove the broad grant
gcloud projects remove-iam-policy-binding nw-prod-8842 \
  --member="$CI_SA" --role="roles/editor"
# rollback if the next deploy fails:
#   gcloud projects add-iam-policy-binding nw-prod-8842 \
#     --member="$CI_SA" --role="roles/editor"

Verify

Two checks, because the project policy alone no longer tells the whole story — the narrow grants live on the resources, not the project:

# 1. Project policy: who still holds a primitive role?
gcloud projects get-iam-policy nw-prod-8842 \
  --flatten="bindings[].members" \
  --filter="bindings.role:(roles/owner OR roles/editor)" \
  --format="table(bindings.role, bindings.members)"
# -> ci-deploy holds neither; engineer-a/-b no longer hold owner.
# -> breakglass@northwind.example STILL holds roles/owner. That is the open
#    finding, not a passing check — file this output as-is rather than filtering
#    the row out.
 
# 2. Resource-level policies: the narrow grants are where we think they are
gcloud artifacts repositories get-iam-policy app-images \
  --location=us-central1 --project=nw-prod-8842
gcloud iam service-accounts get-iam-policy \
  api-runtime@nw-prod-8842.iam.gserviceaccount.com --project=nw-prod-8842
gcloud run services get-iam-policy api \
  --region=us-central1 --project=nw-prod-8842

The finding → fix → verification → evidence map

This table is the deliverable. It's what turns "we fixed some stuff" into something a buyer's security reviewer can check line by line — including the row that is only partially closed.

The Open row is deliberate, and it's the most important row in the table. A standing break-glass Owner means the buyer's literal ask is not met, and roles/viewer is itself a primitive role, so "least privilege for humans" isn't true either. Shipping the pack with that row marked Open — plus the compensating controls and the PAM plan — costs you one awkward paragraph. Shipping it marked Closed costs you the reviewer's trust in every other row, including the ones that really are closed.

What goes to the buyer vs. what stays internal

Not everything you generate is safe to send. The internal originals contain identifiers you don't want in a prospect's inbox; the buyer gets a sanitized subset that proves the control without oversharing.

Goes to the buyer (sanitized):

• The finding → fix → verification → evidence table above.
• The Terraform for each control, with the project ID replaced by PROJECT_ID and the service-account email redacted to ci-deploy@….
• A statement of the retention setting (400 days) and that Data Access logging is enabled — not the log contents.
• The fact of the rejected direct push, as a redacted transcript.

Stays internal / auditor-only:

• The raw get-iam-policy JSON with real member emails and the numeric project number.
• Actual log entries (they contain resource names, caller IPs, principal emails).
• The break-glass owner account details and its rotation procedure.
• Anything naming the buyer, so one prospect's pack never leaks to the next.

The mechanics of that split — what to redact, how to deliver it, and keeping an internal original next to a sanitized external copy — are their own discipline. The kit's safe evidence sharing guide covers it in full.

Where this leaves Northwind

Two blockers fully cleared and one narrowed but still open, each change in Terraform so it survives the next deploy, each with an artifact filed in the evidence folder and its row in the register marked current. The buyer got a five-page pack: the map table with its honest Open row, the sanitized config, and the verification transcripts. The deal moved to legal — the open human-IAM item didn't block it, because it was disclosed with compensating controls and a PAM plan attached rather than discovered later.

Limitations and honest caveats

• This is one small single-project environment. A real multi-project org, a shared VPC, or an org-policy hierarchy is more work and the IAM story gets harder.
• Three findings were shown. A full questionnaire is usually 30–150 line items; most are faster than these, some are slower (a tested backup-restore process, for instance, is real engineering).
• "Green in Vanta" is not the same as "the auditor accepted it." The evidence here is built to satisfy a buyer's security team; a SOC 2 auditor may want the same controls sampled across an observation window.
• Human IAM is narrowed, not solved — see the Open row above. A standing break-glass Owner still exists, and getting from roles/viewer to per-service job-function roles is its own piece of work.
• Enabling Data Access logs has real volume, cost, and Cloud Storage side effects. The per-service rollout in Fix 1 is the point, not a detail.
• The Terraform and CLI here illustrate the approach and have not been applied to a live project as written. Role names and scopes change; validate in a disposable project first.

If you've got a questionnaire with a few of these staring back at you and no one to do the engineering, that's exactly the Controls Review. The Cloud Controls Evidence Kit has the templates this example files into, and a 15-minute fit call is the fastest way to find out whether it's a fit.