Update Workflow

Update agent script logic

The agent container runs a script with the following flow:

Honor dashboard on/off toggle / pin / approvals

Read current image_tag from ConfigMap dag-release

List available tags from the Hilt release ECR + release channel

Filter tags by UPDATE_POLICY (or converge to a pin)

No eligible update?

Exit 0 (nothing to do). Optional workload drift reconcile only.

Line change (new MAJOR.MINOR)?

Yes → pull dag-delivery-bundle:<NEW_TAG> (infra + schema snapshot), reconcile infra/migrations when enabled (approval-gated). Held approvals skip the image roll. Patches (same line) skip this step — images only.

Verify the candidate tag exists in ALL 6 repositories

Ensures Hilt published a complete release, not a partial push.

Mirror images

# daemonless registry-to-registry copy (crane), not docker pull/tag/push
crane copy 297433793776.dkr.ecr.us-east-1.amazonaws.com/${repo}:${NEW_TAG} \
  ${CUSTOMER_ECR}/${prefix}${repo}:${NEW_TAG}

Update ECS task definitions

Force ECS redeployments

aws ecs update-service --cluster collector \
  --service collector-service --force-new-deployment
aws ecs update-service --cluster dashboard \
  --service dashboard-frontend-service --force-new-deployment
aws ecs update-service --cluster dashboard \
  --service dashboard-backend-service --force-new-deployment
aws ecs update-service --cluster dashboard \
  --service dashboard-query-service --force-new-deployment

Restart EKS deployments

kubectl rollout restart deployment/enrichment-app \
  deployment/dag-classification -n default
# Sites running the GNN also have a calibration deployment. It uses the same
# image as dag-classification and exchanges calibration records with it, so
# the two must always be on the same tag.
kubectl rollout restart deployment/dag-calibration -n default 2>/dev/null || true

Wait for stability

aws ecs wait services-stable --cluster collector --services collector-service
aws ecs wait services-stable --cluster dashboard \
  --services dashboard-frontend-service dashboard-backend-service \
             dashboard-query-service
kubectl rollout status deployment/enrichment-app --timeout=300s
kubectl rollout status deployment/dag-classification --timeout=300s
kubectl rollout status deployment/dag-calibration --timeout=300s 2>/dev/null || true

Health check

curl -sf http://dashboard-backend-service:8080/healthz
curl -sf http://dashboard-query-service:8080/healthz

Update ConfigMap with the new tag

kubectl patch configmap dag-release -p '{"data":{"image_tag":"NEW_TAG"}}'

Send webhook notification

POST $WEBHOOK_URL with JSON payload:

{
  "text": "Hilt auto-update: v2026.04.27 → v2026.04.28 applied successfully",
  "previous_tag": "v2026.04.27",
  "new_tag": "v2026.04.28",
  "services_updated": [
    "collector",
    "enrichment",
    "classification",
    "dashboard-frontend",
    "dashboard-backend",
    "dashboard-query"
  ],
  "status": "success"
}

On failure after mirroring

Send failure webhook.
Do not update the ConfigMap (next run will retry).
Exit 1 (CronJob records failure in Kubernetes events).

Rollback

If an update causes issues, operators can manually revert:

# Point back to the previous tag
kubectl patch configmap dag-release -n default \
  -p '{"data":{"image_tag":"v2026.04.27"}}'

# Force rollout to previous images (already in customer ECR)
./scripts/rollout-services.sh

Do not roll the detection deployments back with kubectl rollout undo.From the release that introduces the detection scale-out, dag-classification runs with its state volume mounted read-only and its per-host state kept in Kafka. An image-tag rollback alone leaves the older image unable to write that volume: it stays up, warns every 300 s, and cold-starts from whatever the volume held at seed time on its next restart.Roll those back the way they were rolled forward — re-apply the previous release’s Terraform, which restores both the image and the volume and environment it expects:

# from the release you are returning to
terraform apply

dag-calibration runs the same image and exchanges calibration records with dag-classification, so it moves with it — never roll one without the other. Deployments with no state volume (enrichment-app, the dashboard services) can still be reverted with the tag patch above.

The auto-update agent will not re-apply the problematic tag on the next run because it only moves forward. If the operator sets the ConfigMap tag back, the agent will see the “new” tag again and attempt the update. To prevent this, set auto_update_enabled = false or auto_update_dry_run = true in terraform.tfvars and re-apply.

Findings

Estate

Settings

Update agent script logic

Rollback