Continuous-Deployment: Air-Gapped with the Armory Operator

Mon, 01 Jan 0001 00:00:00 +0000

Overview

This guide details how you can host the Armory Continuous Deployment Bill of Materials (BOM) and Docker images, as well as the Armory Operator Docker images, in your air-gapped environment. The steps at a high level are:

Clone the spinnaker-kustomize-patches repo, which contains helper scripts as well as Kustomize patches.
Deploy S3-compatible MinIO to store the BOM.
Download the BOM.
Copy the BOM to your MinIO bucket.
Host Armory Continuous Deployment Docker images in your private Docker registry.
Download the Armory Operator.
Host Armory Operator Docker images in your private Docker registry.
Update Armory Operator configuration.
Deploy the Armory Operator in your Kubernetes cluster.

Before you begin

You are familiar with the Armory Operator and configuring Armory Continuous Deployment using Kustomize patches.
You have read the introduction to air-gapped environments.
You have public internet access.
You have administrator access to your Kubernetes cluster.
You have created two namespaces in Kubernetes: spinnaker and spinnaker-operator.
You have access to a private Docker registry with credentials to push images.
You have installed the AWS CLI.
You have installed yq version 4+. This is used by helper scripts.

Clone the `spinnaker-kustomize-patches` repo

Armory maintains the spinnakaker-kustomize-patches repo, which contains common configuration options for Armory Continuous Deployment or Spinnaker as well as helper scripts. The patches in this repo give you a reliable starting point when adding and removing features.

Configuration in this repository is meant for Armory Continuous Delivery. To make it compatible with Spinnaker instead, apply the utilities/switch-to-oss.yml patch.

To start, create your own copy of the spinnaker-kustomize-patches repository by clicking the Use this template button:

If you intend to update your copy from upstream, use Fork instead. See Creating a repository from a template for the difference between Use this template and Fork.

Once created, clone this repository to your local machine.

Deploy MinIO for storage

Now that you have cloned the spinnaker-kustomize-patches repo, you need to create a storage bucket to host the BOM. MinIO is a good choice for the bucket since it’s S3 compatible and runs as a pod in Kubernetes. A ready to use MinIO component can be referenced in spinnaker-kustomize-patches/core/persistence/in-cluster.

# Your kustomization.yml file

resources:
 - path/to/spinnaker-kustomize-patches/core/persistence/in-cluster/minio.yml

When you look at the content of the minio.yml manifest, you see that MinIO needs a secret key called minioAccessKey:

env:
 # MinIO access key and secret key
 - name: MINIO_ACCESS_KEY
 value: "minio"
 - name: MINIO_SECRET_KEY
 valueFrom:
 secretKeyRef:
 name: minio-secret-key
 key: minioAccessKey

minioAccessKey is stored in a Kubernetes secret called minio-secret-key. You create the secret by adding a generator to your kustomization.yml file.

secretGenerator:
 - name: minio-secret-key
 options:
 disableNameSuffixHash: true
 literals:
 - minioAccessKey=MyAccessKeyValue

Consult the kustomize documentation for more ways to configure secrets.

Deploy MinIO with your infrastructure:
```
kubectl apply -k .
```

Download the BOM

Decide which Armory Continuous Deployment version you want to deploy. Check Armory Release Notes for the latest supported versions.

The spinnaker-kustomize-patches/utilities/airgap directory contains helper scripts for air-gapped environments. Use bomdownloader.sh to download the version of the Armory Continuous Deployment BOM that you require.

bomdownloader.sh takes two command line parameters in the following order:

Armory Continuous Deployment version; for example, 2.34.
The name of your Docker registry; for example, my.jfrog.io/myteam/armory.

The script creates a halconfig folder, downloads the necessary files, and updates the BOM to use the Docker registry you specified. To download the BOM:

Switch to the spinnaker-kustomize-patches directory.

Run bomdownloader.sh <armory-version> <docker-registry>. For example, if you want to download the 2.25.0 BOM and your registry is my.jfrog.io/myteam/armory:

./utilities/airgap/bomdownloader.sh 2.25.0 my.jfrog.io/myteam/armory

Output is similar to:

download: s3://halconfig/versions.yml to halconfig/versions.yml
download: s3://halconfig/bom/2.25.0.yml to halconfig/bom/2.25.0.yml
download: s3://halconfig/profiles/clouddriver/2.25.3/clouddriver.yml to halconfig/profiles/clouddriver/2.25.3/clouddriver.yml
...
Version 2.25.0 is ready to be uploaded to your private bucket.
For example, with "aws cp --recursive" or "gsutil cp -m -r ..."

Inspecting your file system, you should see the new halconfig folder. For example, if you specified Armory Continuous Deployment v2.25.0, your file system should be:

halconfig
 ┣ bom
 ┃ ┗ 2.25.0.yml
 ┣ profiles
 ┃ ┣ clouddriver
 ┃ ┣ dinghy
 ┃ ┣ echo
 ┃ ┣ fiat
 ┃ ┣ front50
 ┃ ┣ gate
 ┃ ┣ igor
 ┃ ┣ kayenta
 ┃ ┣ monitoring-daemon
 ┃ ┣ orca
 ┃ ┣ rosco
 ┃ ┗ terraformer
 ┗ versions.yml

Each subdirectory in the profiles directory contains a <service-name>.yml profile file.

If you need to change your Docker registry, you can manually edit the <armory-version>.yml file located under halconfig/bom. Update the value for the key artifactSources.dockerRegistry.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


version: 2.25.0
timestamp: "2021-03-25 09:28:32"
services:
 clouddriver:
 commit: de3aa3f0
 version: 2.25.3
 deck:
 commit: 516bcf0a
 version: 2.25.3
 ...
 terraformer:
 commit: 5dcae243
 version: 2.25.0
dependencies:
 redis:
 version: 2:2.8.4-2
artifactSources:
 dockerRegistry: my.jfrog.io/myteam/armory

Copy the BOM

With the MinIO pod running, copy your local BOM into your MinIO bucket.

Set environment variables:

Update the AWS_SECRET_ACCESS_KEY with the minioAccessKey value you created in the Deploy MinIO to host the BOM section.
```
export AWS_ACCESS_KEY_ID=minio
export AWS_SECRET_ACCESS_KEY=<minioAccessKey>
```

Port forward the MinIO service:

kubectl port-forward svc/minio 9000 -n spinnaker

Copy your local BOM to MinIO:

aws s3 mb s3://halconfig --endpoint=http://localhost:9000
aws s3 cp --recursive halconfig s3://halconfig --endpoint=http://localhost:9000

Host the Armory Continuous Deployment Docker images

There are two options for hosting the Docker images: 1) configure your Docker registry as a proxy for docker.io/armory; or 2) download the images and push them to your private Docker registry.

Proxy to `docker.io/armory`

Configure docker.io/armory as a remote repository within your private Docker registry. If you are using JFrog Artifactory, you can follow the instructions in the Remote Docker Repositories section of the JFrog docs.

Download images

You can use the imagedownloader.sh helper script in the spinnaker-kustomize-patches/utilities/airgap directory to download and push the images to your private Docker registry.

The execution format is:

./utilities/airgap/imagedownloader.sh <armory-version>

<armory-version> is the version you specified in the Download the BOM section.

When you run imagedownloader.sh from the spinnaker-patches-repository directory, the script looks for the downloaded BOM and proceeds to download, tag, and push the images for that particular version to the private Docker registry you specified when you ran bomdownloader.sh.

Download the Armory Operator

Download and unpack the latest Armory Operator release into the spinnaker-kustomize-patches/operator folder. Run the following command from your spinnaker-kustomize-patches/operator directory:

bash -c 'curl -L https://github.com/armory-io/spinnaker-operator/releases/latest/download/manifests.tgz | tar -xz'

You should see the following directory structure in the spinnaker-kustomize-patches/operator folder:

operator
 ┣ deploy
 ┃ ┣ crds
 ┃ ┣ openshift
 ┃ ┣ operator
 ┃ ┗ spinnaker
 ┣ halyard-local.yml
 ┣ kustomization.yml
 ┣ patch-config.yaml
 ┗ patch-validations.yaml

Host the Armory Operator Docker images

You can find the operatorimageupdate.sh script in spinnaker-kustomize-patches/utilities/airgap. The script does the following:

Downloads the Armory Operator Docker images and updates their names.
Pushes the images to the Docker registry you specify in the command line.
Updates the Armory Operator’s kustomization.yml with the new image names.

From the spinnaker-kustomize-patches/operator folder, execute the operatorimageupdate.sh script:

../utilities/airgap/operatorimageupdate.sh <your-docker-registry>

Update Armory Operator configuration

Update MinIO secret access key

You also need to update Armory Operator configuration to include the secret access key for MinIO. Locate spinnaker-kustomize-patches/operator/patch-config.yml and update the AWS_SECRET_ACCESS_KEY value with the minioAccessKey value you created in the Deploy MinIO to host the BOM section.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


apiVersion: apps/v1
kind: Deployment
metadata:
 name: spinnaker-operator
spec:
 template:
 spec:
 containers:
 - name: spinnaker-operator
 env:
 - name: AWS_ACCESS_KEY_ID # you can choose to use a secret for these values
 value: minio
 - name: AWS_SECRET_ACCESS_KEY # you can choose to use a secret for these values
 value: changeme
 volumeMounts:
 - mountPath: /opt/spinnaker/config/halyard.yml
 name: operator-config
 subPath: halyard-local.yml
 - name: halyard
 env:
 - name: AWS_ACCESS_KEY_ID # you can choose to use a secret for these values
 value: minio
 - name: AWS_SECRET_ACCESS_KEY # you can choose to use a secret for these values
 value: changeme
 volumeMounts:
 - mountPath: /opt/spinnaker/config/halyard-local.yml
 name: operator-config
 subPath: halyard-local.yml
 volumes:
 - configMap:
 defaultMode: 420
 name: operator-config
 name: operator-config

Update Halyard configuration

The Armory Operator uses its own Halyard installation to deploy and manage Armory Continuous Deployment. You need to configure the new BOM location in spinnaker-kustomize-patches/operator/halyard-local.yml. Update your halyard-local.yml to match the content of the highlighted lines in the following example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


halyard:
 halconfig:
 directory: /home/spinnaker/.hal
spinnaker:
 config:
 # This section is used in air-gapped environments to specify an alternate location for the Bill Of Materials (BOM).
 input:
 gcs:
 enabled: false # If the BOM is stored in a GCS bucket, switch this to true.
 bucket: halconfig # Name of the bucket where the BOM is located.
 #region: us-west-2 # Bucket region; region does not matter for MinIO.
 enablePathStyleAccess: true # If you are using a platform that does not support PathStyleAccess, such as MinIO, switch this to true (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro).
 endpoint: http://minio.spinnaker:9000
 anonymousAccess: false

Deploy the Armory Operator

Now that you have updated halyard-local.yml and patch-config.yml, you can deploy the Armory Operator using its kustomization.yml file. By default, the Armory Operator deploys to the spinnaker-operator namespace. From the spinnaker-kustomize-patches/operator directory, execute:

kubectl apply -k .

After the Armory Operator pod is running, verify that the S3-compatible MinIO bucket is properly configured and that the bucket contains the BOM.

The following example uses version 2.25.0 and a private Docker registry called my.jfrog.io/myteam/armory.

Run the following to access the Halyard container running in the spinnaker-operator pod:
```
kubectl exec -ti deploy/spinnaker-operator -c halyard -- bash
```

Verify that the bucket is properly configured by running:

aws s3 ls --endpoint=http://minio.spinnaker:9000 s3://halconfig/

Inspect the contents of the BOM for the version you downloaded in Download the BOM:

hal version bom <version-number>

If you run the command passing in 2.25.0, output is similar to:

+ Get BOM for 2.25.0
Success
version: 2.25.0
timestamp: '2021-03-25 09:28:32'
services:
 echo:
 version: 2.25.2
 commit: 3a098acc
 clouddriver:
 version: 2.25.3
 commit: de3aa3f0
...
dependencies:
 redis:
 version: 2:2.8.4-2
artifactSources:
 dockerRegistry: my.jfrog.io/myteam/armory

Help resources

Contact Armory Support or use the Spinnaker Slack #armory channel.

What’s next

Configure and deploy Armory Continuous Deployment using Kustomize patches.

Armory Docs – Install Armory Continuous Deployment for Spinnaker in Air-Gapped Environments