Armory Docs – Tasks

Plugins: Configure Mutual TLS Authentication

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

You need the following to configure mTLS:

A Certificate Authority (CA) certificate in the pem format, used for validating the issuer for mTLS requests.
Clouddriver certificate and corresponding private key.
Agent certificate and corresponding private key.

The Agent only supports PKCS#8 keys. The PKCS#12 key store you set up for Armory CD can still use PKCS#8 keys. So, the Armory Scale Agent should use a PKCS#8 key while other services use PKCS#12 keys.

Agent plugin configuration

Create a Kubernetes secret

Create a secret that contains your Clouddriver certificate and corresponding key.

kubectl create secret tls <clouddriver-secret-name> \
 --cert=<your-clouddriver-cert> --key=<your-clouddriver-key>

Or:

kubectl create secret generic <clouddriver-secret-name> \
 --from-file=<your-clouddriver-cert> --from-file=<your-clouddriver-key>

Mount the secret

Mount the secret in the plugin.

In your agent-plugin/clouddriver-plugin.yaml file, spec.kustomize.clouddriver.deployment.patchesStrategicMerge section, add the following lines to mount the Clouddriver cert from your secret:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


spec:
 kustomize:
 clouddriver:
 deployment:
 patchesStrategicMerge:
 - |
 spec:
 template:
 spec:
 containers:
 - name: clouddriver
 volumeMounts:
 - mountPath: <path> # such as /opt/clouddriver/cert
 name: cert
 volumes:
 - name: cert
 secret:
 secretName: <clouddriver-secret-name> 

Configure the plugin

In the agent-plugin/config.yaml file, configure the plugin to use the mounted certs. Note that trustCertCollection, certificateChain, and privateKey values must in file:///filepath/filename format.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 profiles:
 clouddriver:
 kubesvc:
 cluster: redis
 grpc:
 server:
 security:
 enabled: true
 trustCertCollection: file:<path-to-CA-cert>
 certificateChain: file:<path-to-your-clouddriver-cert>
 privateKey: file:<path-to-your-clouddriver-key>
 clientAuth: REQUIRE

See the Armory Scale Agent Plugin Configuration Options page for additional options.

Agent service configuration

Create a secret

Create a secret in the target cluster namespace where the Armory Scale Agent resides.

kubectl create secret generic <agent-secret-name> \
--from-file=<your-agent-cert> --from-file=<your-agent-key>

Modify deployment configuration

Modify the Armory Scale Agent’s deployment configuration in deployment.yaml to mount the certs.

The paths that files are mounted to in the deployment.yaml file should always match the corresponding location in the armory-agent.yaml configuration file. For example, the mountPath of the CA cert in the deployment.yaml file must match the clouddriver.tls.clientCertFile location in armory-agent.yaml.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


spec:
 template:
 spec:
 containers:
 volumeMounts:
 - mountPath: <path> # for example, /opt/armory-agent/cert
 name: armoryagentcert
 - mountPath: <path> # for example, /opt/armory-agent/cacert
 name: clouddrivercacert
 volumes:
 - name: armoryagentcert
 secret:
 secretName: <agent-secret-name>
 - name: clouddrivercacert
 secret:
 secretName: <clouddriver-secret-name>
 - name: certpem
 secret:
 secretName: <CA-secret-name>

If you use a custom CA, you can install it on the Armory Scale Agent pod. The default location on that image, which uses the Alpine base, is /etc/ssl/cert.pem, so you can either append your CA cert to the trust store, which is /etc/ssl/cert.pem, or you can mount the file anywhere and configure the clouddriver.tls.cacertFile property in your YAML to point to that location.

See the Agent Options for configuration details.

Configure the service

Add the certificate information in armory-agent.yaml. Note that clientCertFile and clientKeyFile values must in file:///filepath/filename format.

1
2
3
4
5
6
7
8
9


clouddriver:
 grpc: <:443
 insecure: false
 tls:
 #serverName: <my-ca>
 insecureSkipVerify: false
 clientCertFile: <path-to-your-agent-cert> #client cert for mTLS.
 clientKeyFile: <path-to-your-agent-key>
 #clientKeyPassword:

See the Agent Options for configuration details.

x509 certificate subject filtering

If you have many Agents that want to talk to Clouddriver, and all of them have valid x509 certificates for mTLS, you can authorize a particular subset by configuring a subject filter in your clouddriver.yaml configuration. If a certificate subject line matches any of the filters, that certificate is authorized. All non-matching certificate subjects calls are rejected with an X509CertificateAuthorizationFilterException.

You can specify multiple filtering criteria. However, the order in which the criteria are read is not guaranteed because when Java reads the certificates, it does not maintain the order used in the certificate itself. Be careful when matching on two parts of a subject line, for example UID=.*O=Armory, because the UID and O attributes may not appear in that order. It might be safest to write a regular expression that can match in any order.

Plugin filter configuration

Add an grpc.auth.x509 section to your Clouddriver profile:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


spec:
 spinnakerConfig:
 profiles:
 clouddriver:
 kubesvc:
 cluster: redis
 grpc:
 auth:
 x509:
 enabled: true # must be true for filters to be applied
 filters:
 - UID=([a-z]){3}:[1-9]{3}:ksvc

See the Armory Scale Agent Plugin Configuration Options page for configuration options.

Plugins: Dynamic Accounts Tasks

Mon, 01 Jan 0001 00:00:00 +0000

Plugins: Integrate Prometheus

Mon, 01 Jan 0001 00:00:00 +0000

Available metrics

If prometheus.enabled is true in the Armory Scale Agent service configuration, the Armory Scale Agent exposes metrics on port 8008 (prometheus.port) on path /metrics or /prometheus_metrics like other Armory CD services. Both paths serve the same data.

Setting	Type	Description
`go_gc_duration_seconds`	summary	Amount of time spent in garbage collection.
`go_goroutines`	gauge	Number of go routines.
`go_memstats_alloc_bytes`	gauge	Amount of memory allocation used by Go.
`go_memstats_alloc_bytes`	gauge	Number of bytes allocated and still in use.
`go_memstats_alloc_bytes_total`	counter	Total number of bytes allocated, even if freed.
`go_memstats_buck_hash_sys_bytes`	gauge	Number of bytes used by the profiling bucket hash table.
`go_memstats_frees_total`	counter	Total number of frees.
`go_memstats_gc_cpu_fraction`	gauge	The fraction of this program's available CPU time used by the GC since the program started.
`go_memstats_gc_sys_bytes`	gauge	Number of bytes used for garbage collection system metadata.
`go_memstats_heap_alloc_bytes`	gauge	Number of heap bytes allocated and still in use.
`go_memstats_heap_idle_bytes`	gauge	Number of heap bytes waiting to be used.
`go_memstats_heap_inuse_bytes`	gauge	Number of heap bytes that are in use.
`go_memstats_heap_objects`	gauge	Number of allocated objects.
`go_memstats_heap_released_bytes`	gauge	Number of heap bytes released to OS.
`go_memstats_heap_sys_bytes`	gauge	Number of heap bytes obtained from system.
`go_memstats_last_gc_time_seconds`	gauge	Number of seconds since 1970 of last garbage collection.
`go_memstats_lookups_total`	counter	Total number of pointer lookups.
`go_memstats_mallocs_total`	counter	Total number of mallocs.
`go_memstats_next_gc_bytes`	gauge	Number of heap bytes when next garbage collection will take place.
`go_memstats_sys_bytes`	gauge	Number of bytes obtained from system.
`go_threads`	gauge	Number of Go threads.
`kubesvc_connection_count`	counter	Number of connections.
`kubesvc_disconnection_total`	counter	Number of disconnections.
`kubesvc_events_bytes_sent_total`	counter	Amount of data sent by caching events.
`kubesvc_events_sent_total`	counter	Number of caching events sent.
`kubesvc_resource_agent_total`	counter	Number of watched/polled resources.
`process_cpu_seconds_total`	counter	Total user and system CPU time spent in seconds.
`process_max_fds`	gauge	Maximum number of open file descriptors.
`process_open_fds`	gauge	Number of open file descriptors.
`process_resident_memory_bytes`	gauge	Resident memory size in bytes.
`process_start_time_seconds`	gauge	Start time of the process since unix epoch in seconds.
`process_virtual_memory_bytes`	gauge	Virtual memory size in bytes.
`process_virtual_memory_max_bytes`	gauge	Maximum amount of virtual memory available in bytes.

Configure Prometheus

If you are using the Prometheus operator, you can scrape metrics with:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 labels:
 app: spin
 name: armory-agent-service-monitor
spec:
 endpoints:
 - path: /prometheus_metrics
 port: metrics
# - path: /metrics
# port: metrics
 selector:
 app: spin

Import a Grafana dashboard

You can import this Grafana dashboard definition to use with Prometheus.

Plugins: Integrate Vault with the Armory Scale Agent Service

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

This guide is for experienced Kubernetes and Armory CD users.
You have read the Armory Scale Agent overview.

Authenticate Agent with Vault

The Armory Scale Agent is compatible with properties Armory CD uses for storing secrets in HashiCorp Vault. You put configuration in armory-agent.yaml in the secrets.vault.* section. You refer to Vault secrets using the same syntax you use in configuring secrets for Armory CD. See the Referencing Secrets section for details.

This is an example of what the Kubernetes service account configuration looks like in Agent, using an encryptedFile: reference for kubeconfigFile:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


# ./armory-agent.yaml
kubernetes:
 accounts:
 - name: account01
 kubeconfigFile: encryptedFile:vault!e:secret!p:spinnaker/kubernetes!k:config

secrets:
 vault:
 enabled: true
 authMethod: KUBERNETES
 url: https://your.vault.instance
 role: spinnaker
 path: kubernetes

Dynamically load accounts from Vault

This requires you to install the Vault Injector Sidecar.

The Armory Scale Agent detects changes in the configuration file and manages new accounts that it finds. This makes it possible to use a sidecar for adding and removing accounts dynamically instead of having a static ConfigMap. The Vault guide specifies the following syntax:

vault kv put secret/kubernetes account01=@kubeconfig.yaml

Keep kubeconfig files in one Vault secret (in this case secret/kubernetes).
Each field name corresponds to an account name in Armory CD.
Each field value is the contents of the kubeconfigFile used by that account.

Configuration template

Replace the configuration files and kubeconfig files with Vault injector annotations to provide a template.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


apiVersion: apps/v1
kind: Deployment
metadata:
 name: spin-armory-agent
spec:
 template:
 metadata:
 annotations:
 vault.hashicorp.com/agent-inject: "true"
 vault.hashicorp.com/agent-inject-secret-armory-agent-local.yml: ""
 vault.hashicorp.com/secret-volume-path-armory-agent-local.yml: '/opt/armory/config'
 vault.hashicorp.com/agent-inject-file-armory-agent-local.yml: 'armory-agent-local.yml'
 vault.hashicorp.com/agent-inject-template-armory-agent-local.yml: |
 kubernetes:
 accounts:
 {{- with secret "secret/kubernetes" -}}
 {{ range $k, $v := .Data.data }}
 - kubeconfigFile: 'encryptedFile:vault!e:secret!n:kubernetes!k:{{ $k }}'
 name: {{ $k -}}
 {{- else }}
 []
 {{ end -}}
 {{- end }}
 secrets:
 vault:
 enabled: true
 authMethod: KUBERNETES
 role:
 path:
 url:
 clouddriver:
 insecure: true
 spec:
 volumes:
 - $patch: delete
 name: volume-armory-agent-config
 - $patch: delete
 name: volume-armory-agent-kubeconfigs
 containers:
 - name: armory-agent
 volumeMounts:
 - $patch: delete
 name: volume-armory-agent-config
 mountPath: /opt/armory/config
 - $patch: delete
 name: volume-armory-agent-kubeconfigs
 mounthPath:
 mountPath: /kubeconfigfiles

Make sure to include the required Vault injector annotations like vault.hashicorp.com/role or vault.hashicorp.com/agent-configmap that correspond to your environment.
Be aware of the version of Vault’s KV engine currently in your environment. This guide assumes you have the secret engine KV version 2. For KV version 1, you need to modify the template to use {{ range $k, $v := .Data }} instead. See the Templating Language’s Versioned Read section for more information.
This template expects secret/kubernets to hold the kubeconfig file: Make sure to replace both line 16 and 18 in case that’s not the case in your environment.
Make sure to include all other Agent Options that you require in your environment.

After addressing the preceding points, save the template as armory-agent-vault-patch.yaml and refer to it in your kustomization.yaml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# ./kustomization.yaml
# Pre-existing SpinnakerService resource (may be different)
namespace: spinnaker
resources:
 - spinnakerservice.yaml
bases:
# Armory Scale Agent deployment
 - armory-agent

patchesStrategicMerge:
 - armory-agent-vault-patch.yaml

Troubleshooting

Agent deployment is to appearing / There are no spin-armory-agent pods

Check the following commands for any error or warning message:
- kubectl describe desploy spin-armory-agent | sed -ne '/^Events:$/,$p'
- kubectl describe rs -l cluster=spin-armory-agent | sed -ne '/^Events:$/,$p'
Error message: Error creating: admission webhook "vault.hashicorp.com" denied the request: error validating agent configuration: no Vault role found:
- Make sure that the annotations vault.hashicorp.com/role or vault.hashicorp.com/agent-configmap are set and they correspond to your environment

Agent gets stuck in status Init

Check for logs of the injector with the following command: kubectl logs deploy/spin-armory-agent -c vault-agent-init.
Error message: [WARN] (view) vault.read(secret/kubernetes): no secret exists at secret/data/kubernetes (retry attempt 1 after "250ms"):
- Make sure to update the reference in armory-agent-vault-patch.yaml to a secret that is accessible in your environment.

Agent is in Crash loop back off

Check for logs of armory-agent with the following command kubectl logs deploy/spin-armory-agent -c armory-agent.
Error message: Error registering vault config: vault configuration error:
- Make sure to update armory-agent-vault-patch.yaml to include the properties secrets.vault.* that correspond to your environment.
Error message failed to load configuration: error fetching key \"data\":
- Your vault KV engine is using version 2. Make sure the template in armory-agent-vault-patch.yaml is using {{ range $k, $v := .Data.data }}.

Agent registers with 0 servers

Check for logs of vault injector with the following command: kubectl logs -f deploy/spin-armory-agent -c vault-agent.
Error message missing dependency: vault.read(secret/kubernetes):
- Your vault KV engine is using version 1. Make sure the template in armory-agent-vault-patch.yaml is using {{ range $k, $v := .Data }} .