Install Argo Workflows via Helm and Create a simple workflow

Posted on by Yuki Nakamura
A simple workflow with Argo Workflows

Container-native Workflow engine

Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. It’s been used for many use cases, such as Machine Learning pipelines, Infrastructure automation, and CI/CD. In this article, I will explain how to install it via Helm and create a simple workflow using it.

  1. Preparation
    1. Required
    2. Optional
  2. Add argo’s chart repository
    1. Check available charts’ versions and Argo Workflows’s versions
    2. Run helm install
    3. Access Argo Workflows UI
    4. Check resources of Argo Workflows
      1. Namespaced resources
      2. Non-namespaced resources
      3. CustomResourceDefinitions
  3. Test: Create a simple workflow with Argo Workflows
    1. Create a namespace
    2. Create a WorkflowTemplate
    3. Submit the workflow via Argo Workflows UI
  4. (Optional) Manage Argo Workflows via ArgoCD
    1. Create Application manifest of Argo Workflows
    2. Push the manifests to manifests repository
    3. Check if it has been installed on ArgoCD UI.
  5. Wrap up

Preparation

Please install tools below.

Required

  • kubernetes[1]
  • Helm[1]

[1] Here are my versions of each tool.

$ minikube profile list
|----------|-----------|---------|--------------|------|---------|---------|-------|--------|
| Profile  | VM Driver | Runtime |      IP      | Port | Version | Status  | Nodes | Active |
|----------|-----------|---------|--------------|------|---------|---------|-------|--------|
| minikube | docker    | docker  | 192.168.49.2 | 8443 | v1.26.3 | Running |     1 | *      |
|----------|-----------|---------|--------------|------|---------|---------|-------|--------|

$ helm version 
version.BuildInfo{Version:"v3.12.0", GitCommit:"c9f554d75773799f72ceef38c51210f1842a1dea", GitTreeState:"clean", GoVersion:"go1.20.4"}

Optional

If you’d like to install Argo Events with GitOps way, please prepare below, too.

[2] Here are my ArgoCD version.

$ argocd version --short | grep argocd-server
argocd-server: v2.7.4+a33baa3.dirty

Add argo’s chart repository

ArgoCD’s Helm Charts are maintained on argoproj/argo-helm. Also, they are hosted at https://argoproj.github.io/argo-helm/ . Let’s add it to your helm.

helm repo add argo https://argoproj.github.io/argo-helm

If you’ve already added it, please update it.

helm repo update argo

There are several charts in the repository. You can find argo-events’ chart by the following command.

helm search repo argo | grep argo-workflows

example:

helm search repo argo | grep argo-workflows
argo/argo-workflows             0.32.1          v3.4.9          A Helm chart for Argo Workflows

Check available charts’ versions and Argo Workflows’s versions

You can check available versions via helm search with –versions option.

helm search repo argo/argo-workflows --versions | head -5

example:

NAME                    CHART VERSION   APP VERSION     DESCRIPTION                    
argo/argo-workflows     0.32.1          v3.4.9          A Helm chart for Argo Workflows
argo/argo-workflows     0.32.0          v3.4.9          A Helm chart for Argo Workflows
argo/argo-workflows     0.31.0          v3.4.8          A Helm chart for Argo Workflows
argo/argo-workflows     0.30.0          v3.4.8          A Helm chart for Argo Workflows

This time, I chose the latest Chart version (0.32.1) of Chart that uses v3.4.9 Argo Events App.

Run helm install

Install Argo Events by helm install! We will switch the authentication mode to server so that we can bypass the UI login for now.

helm install my-argo-workflows argo/argo-workflows -n kube-system --version 0.32.1 --set "server.extraArgs={--auth-mode=server}"

Access Argo Workflows UI

Open a port-forward so you can access the UI.

kubectl -n kube-system port-forward deployment/my-argo-workflows-server 2746:2746

Access http://localhost:2746 via browser. You can see Argo Workflows UI!

Argo Workflows UI

Check resources of Argo Workflows

Argo Workflows has many k8s’ objects. Let’s take a look at them.

Namespaced resources

kubectl get -A -l helm.sh/chart=argo-workflows-0.32.1 \
"$(kubectl api-resources --namespaced=true --verbs=list -o name | tr "\n" "," | sed -e 's/,$//')"

NAMESPACE     NAME                                                        DATA   AGE
kube-system   configmap/my-argo-workflows-workflow-controller-configmap   1      104s

NAMESPACE     NAME                                 ENDPOINTS           AGE
kube-system   endpoints/my-argo-workflows-server   10.244.4.187:2746   104s

NAMESPACE     NAME                                                         READY   STATUS    RESTARTS   AGE
kube-system   pod/my-argo-workflows-server-b5f96c488-zqxrk                 1/1     Running   0          104s
kube-system   pod/my-argo-workflows-workflow-controller-57dbf49cbc-cbzbn   1/1     Running   0          104s

NAMESPACE     NAME                                                   SECRETS   AGE
kube-system   serviceaccount/my-argo-workflows-server                0         104s
kube-system   serviceaccount/my-argo-workflows-workflow-controller   0         104s

NAMESPACE     NAME                               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
kube-system   service/my-argo-workflows-server   ClusterIP   10.108.15.21   <none>        2746/TCP   104s

NAMESPACE     NAME                                                    READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/my-argo-workflows-server                1/1     1            1           104s
kube-system   deployment.apps/my-argo-workflows-workflow-controller   1/1     1            1           104s

NAMESPACE     NAME                                                               DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/my-argo-workflows-server-b5f96c488                 1         1         1       104s
kube-system   replicaset.apps/my-argo-workflows-workflow-controller-57dbf49cbc   1         1         1       104s

NAMESPACE     NAME                                                            ADDRESSTYPE   PORTS   ENDPOINTS      AGE
kube-system   endpointslice.discovery.k8s.io/my-argo-workflows-server-l4k5t   IPv4          2746    10.244.4.187   104s

NAMESPACE     NAME                                                               ROLE                              AGE
default       rolebinding.rbac.authorization.k8s.io/my-argo-workflows-workflow   Role/my-argo-workflows-workflow   104s
kube-system   rolebinding.rbac.authorization.k8s.io/my-argo-workflows-workflow   Role/my-argo-workflows-workflow   104s
 
NAMESPACE     NAME                                                        CREATED AT
default       role.rbac.authorization.k8s.io/my-argo-workflows-workflow   2023-08-07T12:57:08Z
kube-system   role.rbac.authorization.k8s.io/my-argo-workflows-workflow   2023-08-07T12:57:08Z

Non-namespaced resources

kubectl get -A -l helm.sh/chart=argo-workflows-0.32.1 \
"$(kubectl api-resources --namespaced=false --verbs=list -o name | tr "\n" "," | sed -e 's/,$//')"

NAME                                                                                                  ROLE                                                                 AGE
clusterrolebinding.rbac.authorization.k8s.io/my-argo-workflows-server                                 ClusterRole/my-argo-workflows-server                                 2m27s
clusterrolebinding.rbac.authorization.k8s.io/my-argo-workflows-server-cluster-template                ClusterRole/my-argo-workflows-server-cluster-template                2m27s
clusterrolebinding.rbac.authorization.k8s.io/my-argo-workflows-workflow-controller                    ClusterRole/my-argo-workflows-workflow-controller                    2m27s
clusterrolebinding.rbac.authorization.k8s.io/my-argo-workflows-workflow-controller-cluster-template   ClusterRole/my-argo-workflows-workflow-controller-cluster-template   2m27s

NAME                                                                                           CREATED AT
clusterrole.rbac.authorization.k8s.io/my-argo-workflows-admin                                  2023-08-07T12:57:08Z
clusterrole.rbac.authorization.k8s.io/my-argo-workflows-edit                                   2023-08-07T12:57:08Z
clusterrole.rbac.authorization.k8s.io/my-argo-workflows-server                                 2023-08-07T12:57:08Z
clusterrole.rbac.authorization.k8s.io/my-argo-workflows-server-cluster-template                2023-08-07T12:57:08Z
clusterrole.rbac.authorization.k8s.io/my-argo-workflows-view                                   2023-08-07T12:57:08Z
clusterrole.rbac.authorization.k8s.io/my-argo-workflows-workflow-controller                    2023-08-07T12:57:08Z
clusterrole.rbac.authorization.k8s.io/my-argo-workflows-workflow-controller-cluster-template   2023-08-07T12:57:08Z

CustomResourceDefinitions

kubectl get customresourcedefinitions --sort-by=.metadata.creationTimestamp \
| (sed -u 1q; tac)

NAME                                   CREATED AT
workflowtemplates.argoproj.io          2023-08-07T12:57:08Z
workflowtasksets.argoproj.io           2023-08-07T12:57:08Z
workflowtaskresults.argoproj.io        2023-08-07T12:57:08Z
workflows.argoproj.io                  2023-08-07T12:57:08Z
workfloweventbindings.argoproj.io      2023-08-07T12:57:08Z
workflowartifactgctasks.argoproj.io    2023-08-07T12:57:08Z
cronworkflows.argoproj.io              2023-08-07T12:57:08Z
clusterworkflowtemplates.argoproj.io   2023-08-07T12:57:08Z
...

Test: Create a simple workflow with Argo Workflows

The simple workflow

Now, we’re ready to use Argo Workflows. So, let’s create a simple workflow with Argo Workflows! You can see all manifests in my GitHub repo.

Create a namespace

Let’s create a namespace named argo-workflows-lab.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: argo-workflows-lab
EOF

Create a WorkflowTemplate

In this test we will use WorkflowTemplates, so that we can submit the workflow from Argo Workflows UI easily.

kubectl apply -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: workflow-template-submittable
  namespace: argo-workflows-lab
spec:
  templates:
  - name: cowsay
    inputs:
      parameters:
      - name: message
    container:
      image: docker/whalesay
      command: [cowsay, "{{inputs.parameters.message}}"]
  - name: diamond
    dag:
      tasks:
      - name: A
        template: cowsay
        arguments:
          parameters: [{name: message, value: A}]
      - name: B
        dependencies: [A]
        template: cowsay
        arguments:
          parameters: [{name: message, value: B}]
      - name: C
        dependencies: [A]
        template: cowsay
        arguments:
          parameters: [{name: message, value: C}]
      - name: D
        dependencies: [B, C]
        template: cowsay
        arguments:
          parameters: [{name: message, value: D}]
EOF

Please check the WorkflowTemplate has been created.

kubectl get workflowtemplates.argoproj.io -n argo-workflows-lab

NAME                            AGE
workflow-template-submittable   107m

Submit the workflow via Argo Workflows UI

Access the “Workflow Templates” page from the left menu.

Access Workflow Templates Page

Click “workflow-template-submittable”.

List of WorkflowTemplates

Click “SUMBIT”.

Submit button

Choose “diamond” in Entrypoint, and click “SUBMIT”.

Submit Workflow

You can see the progress of the workflow on the UI!

The progress of the workflow

You can also check logs of each step on UI.

Access the log of each step
Logs of one step

(Optional) Manage Argo Workflows via ArgoCD

If you’re managing kubernetes add-on using App of Apps pattern of ArgoCD, please do followings.

Create Application manifest of Argo Workflows

Store the following in a file called my-argo-workflows.yaml.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-argo-workflows
spec:
  destination:
    namespace: kube-system
    server: 'https://kubernetes.default.svc'
  source:
    path: ''
    repoURL: 'https://argoproj.github.io/argo-helm'
    targetRevision: 0.32.1
    chart: argo-workflows
    helm:
      parameters:
        - name: server.extraArgs
          value: '{--auth-mode=server}'
  sources: []
  project: default
  syncPolicy:
    automated:
      prune: false
      selfHeal: false

Push the manifests to manifests repository

git add my-argo-workflows.yaml \
&& git commit -m "Add Application manifest of Argo Workflows"  \
&& git push

Here is my commit on GitHub.

Check if it has been installed on ArgoCD UI.

Please sign in your ArgoCD UI, then check if Argo Workflows has been installed.

Argo Workflows

Wrap up

We installed Argo Workflows via Helm. Then, we created the simple workflow. I hope to create a more practical workflow for CI/CD and ETL in the near future.

Thank you for reading! 🐙

Unknown's avatar

Published by Yuki Nakamura