Argo Workflows is a Kubernetes-native workflow engine for orchestrating sequences of containerised steps. Each step in a workflow runs as a Kubernetes Pod, which means the engine gets scheduling, resource isolation, retry semantics, and persistent logging for free by delegating to the kubelet (the per-node agent that manages Pod lifecycle).
It was originally built at Applatix (acquired by Intuit in 2018) and is now a CNCF (Cloud Native Computing Foundation) graduated project.
Why not just use GitLab CI / GitHub Actions?
GitLab CI runs each job on a runner (a VM or container outside the cluster). This creates a fundamental mismatch when the thing being tested is a Kubernetes workload:
- You have to port-forward services to the runner, which is fragile and slow.
- The runner can’t use k8s DNS (
svc.namespace.svc.cluster.local) to address services. - Fan-out (running many jobs in parallel with dependencies) requires either complex YAML or an external tool.
- Artifact passing between jobs requires an external store (S3, GitLab packages).
With Argo Workflows, test steps run inside the cluster as Pods. They reach services directly via k8s DNS. There is no tunnel, no port-forward, no network gymnastics.
GitLab CI still runs the build and submits the workflow (argo submit). The split is: CI orchestrates the build; Argo Workflows orchestrates the tests.
Core concepts
Workflow
The unit of execution. A Workflow is a Kubernetes custom resource (kind: Workflow) that describes a DAG (directed acyclic graph) or sequence of steps. Each Workflow run creates its own set of Pods and is fully isolated. Workflows are ephemeral — once complete, the Pods are gone but the Workflow object (with status) remains queryable via argo get or the UI.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: hello-world-
spec:
entrypoint: say-hello
templates:
- name: say-hello
container:
image: alpine:3.19
command: [echo]
args: ["Hello from Argo Workflows"]generateName makes each submission create a unique Workflow name (e.g. hello-world-xk4f2). The entrypoint field names the template to run first — every Workflow must have exactly one.
WorkflowTemplate
A reusable, versioned Workflow definition stored in the cluster (kind: WorkflowTemplate). Think of it as a named function you can call with arguments. GitLab CI submits a Workflow that references a WorkflowTemplate rather than sending the full definition each time. This keeps CI pipelines thin and moves test logic into the cluster.
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: run-tests
namespace: argo
spec:
entrypoint: test-suite
arguments:
parameters:
- name: image
- name: test-filter
value: "" # optional, defaults to empty (run all)
templates:
- name: test-suite
container:
image: "{{workflow.parameters.image}}"
command: [pytest]
args: ["--filter={{workflow.parameters.test-filter}}", "/tests"]Submit it from CLI or CI:
argo submit --from workflowtemplate/run-tests \
-p image=registry.example.com/tests:abc123 \
-p test-filter="test_payments" \
-n argo --waitTemplate types
| Type | What it does |
|---|---|
container | Runs a single container (the most common type) |
script | Inline script interpreted inside a container |
dag | Declares tasks with dependencies — a directed acyclic graph |
steps | Sequential or parallel sequences of steps |
suspend | Pauses the workflow until a human or external signal resumes it |
resource | Creates/patches/deletes any Kubernetes resource |
DAG workflows
dag is the right choice when tasks have complex dependencies (A runs before B and C, B runs before D, C and D can run in parallel). Each task names which other tasks it depends on; Argo schedules them as soon as all dependencies complete.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: ci-pipeline-
spec:
entrypoint: pipeline
templates:
- name: pipeline
dag:
tasks:
- name: build
template: build-image
- name: unit-tests
template: run-unit
dependencies: [build]
- name: integration-tests
template: run-integration
dependencies: [build]
- name: report
template: publish-report
dependencies: [unit-tests, integration-tests]
- name: build-image
container:
image: docker:24-dind
command: [docker, build, -t, "registry.example.com/app:latest", .]
- name: run-unit
container:
image: registry.example.com/app:latest
command: [pytest, tests/unit/, --junitxml=/tmp/unit.xml]
- name: run-integration
container:
image: registry.example.com/app:latest
command: [pytest, tests/integration/, --junitxml=/tmp/integration.xml]
- name: publish-report
container:
image: alpine:3.19
command: [sh, -c, "echo 'All tests passed'"]The execution order: build runs first. Once it succeeds, unit-tests and integration-tests run in parallel. report waits for both to finish.
Steps workflows
steps is simpler — a list of stages, where each stage is a list of tasks that run in parallel within that stage. Use steps for linear or mildly parallel flows.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: deploy-
spec:
entrypoint: deploy-pipeline
templates:
- name: deploy-pipeline
steps:
# Stage 1: run sequentially (single item)
- - name: lint
template: run-lint
# Stage 2: run in parallel (multiple items)
- - name: test-api
template: run-tests
arguments:
parameters: [{name: suite, value: api}]
- name: test-web
template: run-tests
arguments:
parameters: [{name: suite, value: web}]
# Stage 3: after both tests pass
- - name: deploy
template: run-deploy
- name: run-lint
container:
image: registry.example.com/app:latest
command: [ruff, check, .]
- name: run-tests
inputs:
parameters:
- name: suite
container:
image: registry.example.com/app:latest
command: [pytest, "tests/{{inputs.parameters.suite}}/"]
- name: run-deploy
container:
image: bitnami/kubectl:1.29
command: [kubectl, rollout, restart, deployment/app, -n, production]The double-dash - - syntax: the outer list is stages (sequential), the inner list is tasks within a stage (parallel). Stage 1 runs lint alone, stage 2 runs test-api and test-web simultaneously, stage 3 deploys after both pass.
Artifacts — passing data between steps
Workflows pass data between steps via artifacts. An artifact is a file (or tarball) stored in an S3-compatible bucket. Step A writes to a path; Argo uploads it to the artifact repository. Step B declares the same artifact as an input; Argo downloads it before the Pod starts.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: artifact-passing-
spec:
entrypoint: pipeline
templates:
- name: pipeline
steps:
- - name: generate
template: produce-report
- - name: consume
template: upload-report
arguments:
artifacts:
- name: report
from: "{{steps.generate.outputs.artifacts.report}}"
- name: produce-report
container:
image: registry.example.com/test-runner:latest
command: [pytest, --junitxml=/tmp/results.xml, tests/]
outputs:
artifacts:
- name: report
path: /tmp/results.xml
- name: upload-report
inputs:
artifacts:
- name: report
path: /tmp/results.xml
container:
image: amazon/aws-cli:2.15
command: [aws, s3, cp, /tmp/results.xml, "s3://reports/{{workflow.name}}.xml"]The produce-report template declares an output artifact at /tmp/results.xml. Argo captures this file after the container exits and stores it in the configured artifact repository. The upload-report template declares the same artifact as an input — Argo downloads it to /tmp/results.xml before the container starts. The expression {{steps.generate.outputs.artifacts.report}} references the artifact by step name and artifact name.
Parameters — passing values between steps
Parameters pass small values (strings, numbers) between steps without needing S3 storage. A template declares output parameters by reading a file or using an expression; downstream steps reference them.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: param-passing-
spec:
entrypoint: pipeline
templates:
- name: pipeline
steps:
- - name: get-version
template: read-version
- - name: build
template: build-image
arguments:
parameters:
- name: tag
value: "{{steps.get-version.outputs.parameters.version}}"
- name: read-version
container:
image: alpine:3.19
command: [sh, -c, "cat VERSION > /tmp/version.txt"]
outputs:
parameters:
- name: version
valueFrom:
path: /tmp/version.txt
- name: build-image
inputs:
parameters:
- name: tag
container:
image: docker:24-dind
command: [docker, build, -t, "registry.example.com/app:{{inputs.parameters.tag}}", .]valueFrom.path tells Argo to read the file at that path after the container exits and use its contents (trimmed) as the parameter value. This avoids storing single values as S3 artifacts.
Exit handlers — cleanup on completion
A template that runs after the workflow completes, regardless of success or failure. Used for cleanup (delete a k8s namespace, send a Slack notification, archive logs). Declared as onExit in the Workflow spec.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: with-cleanup-
spec:
entrypoint: run-tests
onExit: cleanup
templates:
- name: run-tests
steps:
- - name: setup-ns
template: create-namespace
- - name: test
template: run-test-suite
- name: cleanup
container:
image: bitnami/kubectl:1.29
command: [kubectl, delete, namespace, "test-{{workflow.name}}", --ignore-not-found]
- name: create-namespace
resource:
action: create
manifest: |
apiVersion: v1
kind: Namespace
metadata:
name: "test-{{workflow.name}}"
- name: run-test-suite
container:
image: registry.example.com/tests:latest
command: [pytest, tests/]
env:
- name: TEST_NAMESPACE
value: "test-{{workflow.name}}"The cleanup template deletes the ephemeral namespace whether tests pass or fail. The resource template type creates the namespace declaratively — Argo applies the manifest using its ServiceAccount’s RBAC permissions.
Exit handler scope
The exit handler runs as a separate Pod. It cannot access artifacts or parameters from the main workflow unless you explicitly pass them via
{{workflow.outputs.parameters.*}}.
Resource templates — managing Kubernetes objects
The resource template type creates, patches, or deletes any Kubernetes resource directly from the workflow. This avoids needing kubectl in a container image.
- name: create-test-db
resource:
action: create
successCondition: status.readyReplicas > 0
failureCondition: status.phase == Failed
manifest: |
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-postgres
namespace: "test-{{workflow.name}}"
spec:
replicas: 1
selector:
matchLabels:
app: test-postgres
template:
metadata:
labels:
app: test-postgres
spec:
containers:
- name: postgres
image: postgres:16
env:
- name: POSTGRES_PASSWORD
value: testonlysuccessCondition and failureCondition tell Argo to poll the created resource’s status. The step succeeds once readyReplicas > 0 and fails if phase == Failed. Without these, Argo considers the step complete as soon as the API server accepts the manifest — which is usually too early.
Retry and timeout policies
Individual templates can declare retry strategies and timeouts to handle transient failures (network blips, flaky tests, slow cold starts).
- name: flaky-integration-test
retryStrategy:
limit: 3
retryPolicy: OnFailure # only retry on non-zero exit, not on errors like OOM
backoff:
duration: "10s"
factor: 2 # 10s, 20s, 40s
maxDuration: "2m"
activeDeadlineSeconds: 600 # kill the Pod after 10 minutes
container:
image: registry.example.com/tests:latest
command: [pytest, tests/integration/, -x]
resources:
requests:
memory: 512Mi
cpu: 500m
limits:
memory: 1GiretryPolicy options: Always (retry on any failure including OOMKilled), OnFailure (only non-zero exit codes), OnError (only Argo system errors like Pod scheduling failures). activeDeadlineSeconds is a hard timeout — the Pod gets a SIGTERM after this duration regardless of retry state.
Artifact storage
Argo Workflows needs an S3-compatible endpoint configured as the artifactRepository in the controller ConfigMap. Each namespace that runs workflows needs access to the same bucket (or separate per-workflow-type buckets).
# Part of the argo-workflows ConfigMap (workflow-controller-configmap)
artifactRepository:
s3:
endpoint: garage.internal:3900
bucket: argo-artifacts
insecure: true # HTTP, not HTTPS (internal network)
accessKeySecret:
name: argo-s3-creds
key: accessKey
secretKeySecret:
name: argo-s3-creds
key: secretKeyIn the TESTIO cluster: Garage (an S3-compatible distributed object store written in Rust) is used as the artifact store. Three buckets:
argo-e2e— Playwright HTML reports and screenshots (30-day retention)argo-tf— Terraform provider acceptance test results (14-day retention)argo-backend— Backend integration test JUnit XML (14-day retention)
How a CI pipeline submits a workflow
# .gitlab-ci.yml
submit-tests:
stage: test
image: argoproj/argocli:v3.5
script:
- |
argo submit --wait \
--from workflowtemplate/terraform-acceptance \
-p image=registry.example.com/tf-tests:${CI_COMMIT_SHA} \
-n argo
- argo get @latest -n argo -o json | jq '.status.phase'The --from workflowtemplate/ form submits a Workflow derived from the named template, passing parameters (like the image tag). --wait blocks until the workflow completes and exits non-zero on failure — so GitLab CI fails the job if any Argo step fails. @latest is a shorthand for the most recently submitted workflow by this client.
Conditional execution
Steps can be skipped based on conditions using when expressions. Conditions can reference the status of previous steps or workflow parameters.
- name: pipeline
steps:
- - name: test
template: run-tests
- - name: deploy-staging
template: deploy
when: "{{steps.test.status}} == Succeeded"
arguments:
parameters: [{name: env, value: staging}]
- - name: notify-failure
template: send-slack
when: "{{steps.test.status}} == Failed"The when field accepts expressions comparing template tags to string values. Common comparisons: step status (Succeeded, Failed, Skipped), parameter values ("{{workflow.parameters.deploy}}" == "true"), and artifact existence.
ServiceAccount and RBAC
Workflow Pods need a Kubernetes ServiceAccount with permissions to create/delete namespaces, services, and pods if they manage test infrastructure. The Argo controller itself needs its own ServiceAccount to manage Workflow objects and Pods.
apiVersion: v1
kind: ServiceAccount
metadata:
name: argo-workflow-sa
namespace: argo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: argo-workflow-role
namespace: argo
rules:
- apiGroups: [""]
resources: [pods, pods/log, namespaces, services]
verbs: [create, get, list, watch, delete]
- apiGroups: [apps]
resources: [deployments]
verbs: [create, get, delete]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: argo-workflow-binding
namespace: argo
subjects:
- kind: ServiceAccount
name: argo-workflow-sa
roleRef:
kind: Role
name: argo-workflow-role
apiGroup: rbac.authorization.k8s.ioReference the ServiceAccount in workflow specs:
spec:
serviceAccountName: argo-workflow-sa
entrypoint: ...Least privilege
Create separate ServiceAccounts per workflow type. E2E tests that create namespaces need broader permissions than unit tests that only read secrets. Don’t grant namespace-delete to a workflow that only runs
pytest.
RBAC is the main operational complexity of Argo — plan it early.
Argo UI
The Argo server exposes a web UI (port 2746 by default) showing:
- All workflow runs with status (running, succeeded, failed)
- Per-step logs and output artifacts
- DAG visualisation
- Ability to retry or resubmit failed workflows
In the TESTIO cluster the UI is exposed via an Ingress, protected by the same OIDC proxy as other internal services.
Comparison with alternatives
| Engine | Architecture | Strengths | Weaknesses |
|---|---|---|---|
| Argo Workflows | k8s-native CRDs, one Pod per step | First-class k8s integration, DAG support, built-in UI, artifact management | Only runs on k8s, RBAC complexity |
| Tekton | k8s-native (Pipeline, Task, Run CRDs) | Supply chain attestation (Tekton Chains), fine-grained task reuse | More CRDs, no built-in UI, steeper learning curve |
| Jenkins | Long-running master + agents | Mature plugin ecosystem, familiar to most teams | Not cloud-native, stateful master, each job is a process not a Pod |
| GitHub Actions / GitLab CI | External runners | Excellent for build/deploy, easy to start | Runners are outside the cluster, can’t use k8s DNS for service communication |
When to use Argo Workflows: Your tests need to talk to services running in the cluster, you want declarative DAG orchestration, and you’re already running Kubernetes.
When to stay with CI runners: Your tests are self-contained (no cluster dependencies), or your team doesn’t operate Kubernetes infrastructure.
Key mental model
Argo Workflows is to test orchestration what Kubernetes is to service deployment: it makes the desired state of a test run declarative, retryable, and observable.
The key shift is from “run a script on a runner” to “declare a graph of containers with dependencies and let the cluster schedule them.”