Skip to content
kix /docs
Install the CLI

How-to guide Author packages and clusters

Add a post-deploy health check

Use kix.healthCheck when Kubernetes readiness is necessary but not enough. For example, ready Pods do not prove that an application can reach its database or that a monitoring agent is successfully writing samples.

The helper adds a Job that runs after the package root and every endpoint used by the probe are ready. A non-zero exit fails the deployment, and the deploy output includes the last lines printed by the probe.

This guide assumes your package already has a workload, a Service exposed as self.service, and a list-valued build field.

Add the health-check options

Section titled “Add the health-check options”

Expose the standard options in the package:

how-to/application/web-package.nix
    healthCheck = kix.options.healthCheck;

View source on GitHub ↗

This gives cluster authors two settings:

OptionDefaultPurpose
healthCheck.enabledfalseRender and run the probe
healthCheck.retryFor90Seconds the script may retry before failing

You can extend the option set with package-specific fields by merging another attribute set into kix.options.healthCheck.

Write the probe script

Section titled “Write the probe script”

Keep the script beside the package so it can be reviewed and tested on its own. This shell probe retries an HTTP endpoint for the period Kix supplies in RETRY_FOR:

how-to/application/health-check.sh
set -u


deadline=$(( $(date +%s) + RETRY_FOR ))


while true; do
  if curl --fail --silent --show-error --max-time 10 "$APP_URL" >/dev/null; then
    echo "ok    GET $APP_URL"
    exit 0
  fi


  echo "FAIL  GET $APP_URL"
  if [ "$(date +%s)" -ge "$deadline" ]; then
    echo "gave up after ${RETRY_FOR}s"
    exit 1
  fi


  sleep 5
done

View source on GitHub ↗

Keep one pass through the checks below 90 seconds. Kix reserves that much time after the retry window so the Job can report its final failure before its deadline expires.

Add the probe as a build entry

Section titled “Add the probe as a build entry”

Append kix.healthCheck to the package’s build list:

how-to/application/web-package.nix
    (kix.healthCheck {
      runtime = kix.healthCheck.runtimes.shell;
      env =
        { self, ... }:
        {
          APP_URL = self.service.out.url { };
        };
      script = builtins.readFile ./health-check.sh;
    })

View source on GitHub ↗

The callback value self.service.out.url { } matters. It gives the Job both the address to call and the dependency information attached to that address. Kix uses it to order the Job after the Service and to derive network-policy egress when network policy is enabled.

For a probe with several targets, add each target through env. The helper also provides deps, containing every dependency injected into any build entry in the package.

Use requires only for resources the probe must wait for but does not address. The package root is always included automatically.

Enable and verify the probe

Section titled “Enable and verify the probe”

Enable it on an instance:

how-to/application/cluster.nix
            healthCheck = {
              enabled = true;
              retryFor = 60;
            };

View source on GitHub ↗

Check the rendered cluster before deploying:

kix-examples/ 
❱ kix check how-to-application
 TOOL       RESULT  DETAILS                       
 eval       pass    17 manifests evaluated        
 scorecard  pass    0 errors, 11 warnings, 2 info

The output should include no health-check validation errors. Render the Job to confirm the helper is enabled:

kix-examples/ output excerpt offline capture 
❱ kix build how-to-application -o json
Show outputHide output · 101 lines 
{
  "apiVersion": "batch/v1",
  "kind": "Job",
  "metadata": {
    "annotations": {
      "kix.run/depends-on": "8767b7nzgc1x5bpfa9gv71cpk9z8h0iw,cd3l507fvaf0wg5azfza49x6wjsiiynz,dlzbv1nahmar8f7b9jm6rmyvmqczmzdb,requires:jh345cnnffy1bnj4cz5x0hwq6ibxvkkr",
      "kix.run/identity-hash": "4sa9i19dly4hh06r5alinlr7pf289igd",
      "kix.run/package": "production",
      "kix.run/package-namespace": "how-to-app",
      "kix.run/rerun": "on-change"
    },
    "labels": {
      "app.kubernetes.io/component": "health-check",
      "app.kubernetes.io/instance": "production",
      "app.kubernetes.io/managed-by": "kix",
      "app.kubernetes.io/name": "production-health"
    },
    "name": "production-health",
    "namespace": "how-to-app"
  },
  "spec": {
    "activeDeadlineSeconds": 150,
    "backoffLimit": 0,
    "template": {
      "metadata": {
        "labels": {
          "app.kubernetes.io/component": "health-check",
          "app.kubernetes.io/name": "production-health"
        }
      },
      "spec": {
        "automountServiceAccountToken": false,
        "containers": [
          {
            "command": [
              "/bin/sh",
              "-eu",
              "/probe/health-check.sh"
            ],
            "env": [
              {
                "name": "APP_URL",
                "value": "http://production.how-to-app.svc.cluster.local:80"
              },
              {
                "name": "RETRY_FOR",
                "value": "60"
              }
            ],
            "image": "docker.io/curlimages/curl:8.14.1@sha256:9a1ed35addb45476afa911696297f8e115993df459278ed036182dd2cd22b67b",
            "name": "probe",
            "resources": {
              "limits": {
                "memory": "128Mi"
              },
              "requests": {
                "cpu": "50m",
                "memory": "64Mi"
              }
            },
            "securityContext": {
              "allowPrivilegeEscalation": false,
              "capabilities": {
                "drop": [
                  "ALL"
                ]
              },
              "readOnlyRootFilesystem": true,
              "runAsNonRoot": true
            },
            "terminationMessagePolicy": "FallbackToLogsOnError",
            "volumeMounts": [
              {
                "mountPath": "/probe",
                "name": "production-health-script",
                "readOnly": true
              }
            ]
          }
        ],
        "restartPolicy": "Never",
        "securityContext": {
          "runAsGroup": 65534,
          "runAsNonRoot": true,
          "runAsUser": 65534,
          "seccompProfile": {
            "type": "RuntimeDefault"
          }
        },
        "volumes": [
          {
            "configMap": {
              "name": "production-health-script"
            },
            "name": "production-health-script"
          }
        ]
      }
    }
  }
}

Deploy the cluster:

kix-examples/ live capture 
❱ kix deploy how-to-application -y
Show outputHide output · 78 lines 
Building cluster 'how-to-application'...
Cluster how-to-application: 17 manifests
Connecting to cluster...
No previous activation on cluster. First deploy.

  _cluster
    ~ cluster-level resources (1 changed, 3 added)
  how-to-app
    + preview 1.0.0 (3 resources)
    + production 1.0.0 (5 resources)
  kube-system
    + platform-dns (0 resources)
    + platform-storage (0 resources)

  Plan: cluster-level changes, 4 added
  Resources: 4 real content, 0 dep-affected
  ↻ 1 under the rerun rule (deleted first when live): Job/production-health@how-to-app

Not checked by the dry run (13); their changes come from comparing builds:
  ? 5 its CRD is changed by this deploy, and was checked as it is now:
      Activation/how-to-application-6rhi3wpk9iml
      PackageInstance/platform-dns@kube-system
      PackageInstance/platform-storage@kube-system
      PackageInstance/preview@how-to-app
      PackageInstance/production@how-to-app
  ? 8 its namespace is created by this deploy:
      ConfigMap/preview@how-to-app
      ConfigMap/production-health-script@how-to-app
      ConfigMap/production@how-to-app
      Deployment/preview@how-to-app
      Deployment/production@how-to-app
      and 3 more (-o json lists them)

Not checked by any preview: whether Pods start, pull their images, and pass their probes.

Objects: 17 written, 0 deleted (sha256:0259ef15fef8d74b8e97783aa5ecce1e719dd93ce351b4ef1561e5cd7d9017e3)
plan: 17 nodes
  ~ Namespace/kube-system configured
  + Namespace/how-to-app created
  ✔ Namespace/kube-system ready
  ✔ Namespace/how-to-app ready
  + CustomResourceDefinition/activations.kix.run created
  + CustomResourceDefinition/packageinstances.kix.run created
  + ConfigMap/preview@how-to-app created
  ✔ ConfigMap/preview@how-to-app ready
  + ConfigMap/production-health-script@how-to-app created
  ✔ ConfigMap/production-health-script@how-to-app ready
  + ConfigMap/production@how-to-app created
  ✔ ConfigMap/production@how-to-app ready
  + Deployment/preview@how-to-app created
  + Deployment/production@how-to-app created
  ✔ CustomResourceDefinition/activations.kix.run ready
  ✔ CustomResourceDefinition/packageinstances.kix.run ready
  + PackageInstance/platform-dns@kube-system created
  + PackageInstance/platform-storage@kube-system created
  ✔ PackageInstance/platform-storage@kube-system ready
  … Deployment/preview@how-to-app waiting: Deployment/preview: 0/1 ready, 1/1 updated (idle 10s) (14s of 180s); 3 objects wait for it: Service/preview@how-to-app and 2 more
  … Deployment/production@how-to-app waiting: Deployment/production: 0/2 ready, 2/2 updated (idle 10s) (14s of 180s); 4 objects wait for it: Service/production@how-to-app and 3 more
  … PackageInstance/platform-dns@kube-system waiting: Deployment/coredns: 0/2 ready, 2/2 updated (12s of 180s); 3 objects wait for it: Activation/how-to-application-6rhi3wpk9iml, Job/production-health@how-to-app, and 1 more
  ✔ PackageInstance/platform-dns@kube-system ready
  ✔ Deployment/production@how-to-app ready
  + Service/production@how-to-app created
  ✔ Service/production@how-to-app ready
  + recreate Job/production-health@how-to-app created
  ✔ Deployment/preview@how-to-app ready
  + Service/preview@how-to-app created
  ✔ Service/preview@how-to-app ready
  + PackageInstance/preview@how-to-app created
  ✔ PackageInstance/preview@how-to-app ready
  … Job/production-health@how-to-app waiting: Job/production-health: 0 succeeded (idle 0s) (14s of 180s); 2 objects wait for it: PackageInstance/production@how-to-app and 1 more
  ✔ Job/production-health@how-to-app ready
  + PackageInstance/production@how-to-app created
  ✔ PackageInstance/production@how-to-app ready
  ~ Activation/how-to-application-6rhi3wpk9iml configured
  ✔ Activation/how-to-application-6rhi3wpk9iml ready
  • activation 'how-to-application-6rhi3wpk9iml' → Active

Deploy complete: 15 created, 2 configured, 0 unchanged, 0 failed

The deployment does not become active until the Job succeeds. If the script exits non-zero, Kix reports the Job as failed and prints its final output below the failure.

For example, a probe that reaches the application but rejects its response is reported with the final lines from the script:

kix-examples/ 
❱ kix deploy how-to-application -y
Building cluster 'how-to-application'...
Cluster how-to-application: 17 manifests
Connecting to cluster...
Active activation: how-to-application-6rhi3wpk9iml (6rhi3wpk...)

  how-to-app
    ~ production 1.0.0 (2 changed, 1 dep-affected)

  Plan: 1 updated, 3 unchanged
  Resources: 2 real content, 1 dep-affected
  ↻ 1 under the rerun rule (deleted first when live): Job/production-health@how-to-app

Not checked by any preview: whether Pods start, pull their images, and pass their probes.

Objects: 4 written, 0 deleted (sha256:5037d09e6ec0b2a7880beb73105783d95829454d4915e9243492a7edec7f22f0)
plan: 17 nodes
  ~ ConfigMap/production-health-script@how-to-app configured
  ✔ ConfigMap/production-health-script@how-to-app ready
  + recreate Job/production-health@how-to-app created
  ⚠ Job/production-health@how-to-app JobBackoffExceeded: Job failed 1 times (backoffLimit=0). Kix did not deploy the 2 objects that depend on it.
      ok    GET http://production.how-to-app.svc.cluster.local:80
      the application returned an unexpected response

Deploy of 'how-to-application' failed: 0 created, 1 configured, 13 unchanged, 1 failed, 2 not started

  Failed resources (1):
    - Job/production-health@how-to-app: JobBackoffExceeded: Job failed 1 times (backoffLimit=0)

  Cluster state:
    - 0 created and 1 configured object(s) from this run finished and are ready on the cluster, 1 failed (during apply or after being applied), and 2 were never sent to the cluster.
    - Deployment record how-to-application-fx3j0jasq82s is marked Failed, and how-to-application-6rhi3wpk9iml remains the Active deployment record.
    - The next `kix deploy how-to-application` on a machine with this build will compare against what this failed run applied.

  Next:
    Fix the failed resource(s) above and run `kix deploy how-to-application` again.
(exit code: 1)

The Job carries kix.run/rerun: on-change. When the Job’s identity hash changes, Kix deletes the existing Job in the foreground and recreates it. When the Job is unchanged, kix deploy and kix rollback inspect the live Job: they skip it if its previous run succeeded and recreate it if its previous run failed (the in-cluster operator reports an unchanged failed Job without recreating it so a failing probe does not retry in a loop).