> ## Documentation Index
> Fetch the complete documentation index at: https://hilt.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Update Workflow

> Step-by-step update flow and rollback procedures for the auto-update agent.

## Update agent script logic

The agent container runs a script with the following flow:

**1. Honor dashboard on/off toggle / pin / approvals**

**2. Read current image\_tag from ConfigMap dag-release**

**3. List available tags from the Hilt release ECR + release channel**

**4. Filter tags by UPDATE\_POLICY (or converge to a pin)**

**5. No eligible update?**

Exit 0 (nothing to do). Optional workload drift reconcile only.

**6. Line change (new MAJOR.MINOR)?**

Yes → pull `dag-delivery-bundle:<NEW_TAG>` (infra + schema snapshot),
reconcile infra/migrations when enabled (approval-gated). Held approvals
skip the image roll. Patches (same line) skip this step — **images only**.

**7. Verify the candidate tag exists in ALL 6 repositories**

Ensures Hilt published a complete release, not a partial push.

**8. Mirror images**

```bash theme={null}
# daemonless registry-to-registry copy (crane), not docker pull/tag/push
crane copy 297433793776.dkr.ecr.us-east-1.amazonaws.com/${repo}:${NEW_TAG} \
  ${CUSTOMER_ECR}/${prefix}${repo}:${NEW_TAG}
```

**9. Update ECS task definitions**

**10. Force ECS redeployments**

```bash theme={null}
aws ecs update-service --cluster collector \
  --service collector-service --force-new-deployment
aws ecs update-service --cluster dashboard \
  --service dashboard-frontend-service --force-new-deployment
aws ecs update-service --cluster dashboard \
  --service dashboard-backend-service --force-new-deployment
aws ecs update-service --cluster dashboard \
  --service dashboard-query-service --force-new-deployment
```

**11. Restart EKS deployments**

```bash theme={null}
kubectl rollout restart deployment/enrichment-app \
  deployment/dag-classification -n default
# Sites running the GNN also have a calibration deployment. It uses the same
# image as dag-classification and exchanges calibration records with it, so
# the two must always be on the same tag.
kubectl rollout restart deployment/dag-calibration -n default 2>/dev/null || true
```

**12. Wait for stability**

```bash theme={null}
aws ecs wait services-stable --cluster collector --services collector-service
aws ecs wait services-stable --cluster dashboard \
  --services dashboard-frontend-service dashboard-backend-service \
             dashboard-query-service
kubectl rollout status deployment/enrichment-app --timeout=300s
kubectl rollout status deployment/dag-classification --timeout=300s
kubectl rollout status deployment/dag-calibration --timeout=300s 2>/dev/null || true
```

**13. Health check**

```bash theme={null}
curl -sf http://dashboard-backend-service:8080/healthz
curl -sf http://dashboard-query-service:8080/healthz
```

**14. Update ConfigMap with the new tag**

```bash theme={null}
kubectl patch configmap dag-release -p '{"data":{"image_tag":"NEW_TAG"}}'
```

**15. Send webhook notification**

`POST $WEBHOOK_URL` with JSON payload:

```json theme={null}
{
  "text": "Hilt auto-update: v2026.04.27 → v2026.04.28 applied successfully",
  "previous_tag": "v2026.04.27",
  "new_tag": "v2026.04.28",
  "services_updated": [
    "collector",
    "enrichment",
    "classification",
    "dashboard-frontend",
    "dashboard-backend",
    "dashboard-query"
  ],
  "status": "success"
}
```

**16. On failure after mirroring**

* Send failure webhook.
* Do **not** update the ConfigMap (next run will retry).
* Exit 1 (CronJob records failure in Kubernetes events).

## Rollback

If an update causes issues, operators can manually revert:

```bash theme={null}
# Point back to the previous tag
kubectl patch configmap dag-release -n default \
  -p '{"data":{"image_tag":"v2026.04.27"}}'

# Force rollout to previous images (already in customer ECR)
./scripts/rollout-services.sh
```

> [!WARNING]
>
> **Do not roll the detection deployments back with `kubectl rollout undo`.**
>
> From the release that introduces the detection scale-out, `dag-classification`
> runs with its state volume mounted **read-only** and its per-host state kept in
> Kafka. An image-tag rollback alone leaves the older image unable to write that
> volume: it stays up, warns every 300 s, and cold-starts from whatever the
> volume held at seed time on its next restart.
>
> Roll those back the way they were rolled forward — re-apply the previous
> release's Terraform, which restores both the image and the volume and
> environment it expects:
>
> ```bash theme={null}
> # from the release you are returning to
> terraform apply
> ```
>
> `dag-calibration` runs the same image and exchanges calibration records
> with `dag-classification`, so it moves with it — never roll one without the
> other. Deployments with no state volume (`enrichment-app`, the dashboard
> services) can still be reverted with the tag patch above.

The auto-update agent will not re-apply the problematic tag on the next run because it only moves forward. If the operator sets the ConfigMap tag back, the agent will see the "new" tag again and attempt the update. To prevent this, set `auto_update_enabled = false` or `auto_update_dry_run = true` in `terraform.tfvars` and re-apply.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.