Armory Docs – Work with Secrets in Spinnaker

Continuous-Deployment: Configure Hashicorp's Vault for Kubernetes Auth

Mon, 01 Jan 0001 00:00:00 +0000

Configuration of Vault for the Kubernetes auth method requires configuring both Vault and Kubernetes.

Configure Kubernetes

Create a Kubernetes Service Account.

vault-auth-service-account.yml

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
 name: role-tokenreview-binding
 namespace: default
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:auth-delegator
subjects:
- kind: ServiceAccount
 name: vault-auth
 namespace: default

# Create a service account, 'vault-auth'
$ kubectl -n default create serviceaccount vault-auth

# Update the 'vault-auth' service account
$ kubectl -n default apply --filename vault-auth-service-account.yml

Configure Vault

This guide assumes that Key/Value version 1 secret engine is enabled at secret/.

Create a read-only policy spinnaker-kv-ro in Vault.

spinnaker-kv-ro.hcl

# For K/V v1 secrets engine
path "secret/spinnaker/*" {
 capabilities = ["read", "list"]
}
# For K/V v2 secrets engine
path "secret/data/spinnaker/*" {
 capabilities = ["read", "list"]
}

$ vault policy write spinnaker-kv-ro spinnaker-kv-ro.hcl

Set environment variables required for Vault configuration.

# Set VAULT_SA_NAME to the service account you created earlier
$ export VAULT_SA_NAME=$(kubectl -n default get sa vault-auth -o jsonpath="{.secrets[*]['name']}")

# Set SA_JWT_TOKEN value to the service account JWT used to access the TokenReview API
$ export SA_JWT_TOKEN=$(kubectl -n default get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode; echo)

# Set SA_CA_CRT to the PEM encoded CA cert used to talk to Kubernetes API
$ export SA_CA_CRT=$(kubectl -n default get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)

# Look in your cloud provider console for this value
$ export K8S_HOST=<https://your_API_server_endpoint>

NOTE on TTL and Token Renewal

The Kubernetes Vault Auth Secrets Engine does not currently support token renewal. As such the spinnaker role created below provides a TTL of two months.

Note By default, Vault has a max_ttl parameter set to 768h0m0s - that’s 32 days. If you want to set the TTL to a higher value, you need to modify this parameter.

Important: Spinnaker must be redeployed sometime during the defined TTL window – Armory recommends this be done by updating to a new version of Spinnaker and running kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest>.

Next, configure Vault’s Kubernetes auth method.

# Enable the Kubernetes auth method at the default path ("kubernetes")
$ vault auth enable kubernetes

# Tell Vault how to communicate with the Kubernetes cluster
$ vault write auth/kubernetes/config \
 token_reviewer_jwt="$SA_JWT_TOKEN" \
 kubernetes_host="$K8S_HOST" \
 kubernetes_ca_cert="$SA_CA_CRT"

# Create a role named, 'spinnaker' to map Kubernetes Service Account to
# Vault policies and default token TTL
$ vault write auth/kubernetes/role/spinnaker \
 bound_service_account_names=default \
 bound_service_account_namespaces='*' \
 policies=spinnaker-kv-ro \
 ttl=1440h

Verify Configuration

It is time verify that the Kubernetes auth method has been properly configured.

Deploy Armory’s debug container into your cluster – this container has the Vault cli pre-installed.

This should be deployed into the same namespace as your Spinnaker installation.

kubectl apply -f https://raw.githubusercontent.com/armory/troubleshooting-toolbox/master/docker-debugging-tools/deployment.yml

exec into the pod.

POD_NAME=$(kubectl get pod -l app=debugging-tools -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' --sort-by=".status.startTime" | tail -n 1)

kubectl exec -it $POD_NAME bash

Test the auth method.

export VAULT_ADDR='http://your.vault.address:port'
SA_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

vault write auth/kubernetes/login role=spinnaker jwt=$SA_TOKEN

This command should return output like the following

Key Value
--- -----
token s.bKSSrYOcETCADGvGxhbDaaaD
token_accessor 0ybx2CEPZqxBEwFk8jUPkBk7
token_duration 24h
token_renewable true
token_policies ["default" "spinnaker-kv-ro"]
identity_policies []
policies ["default" "spinnaker-kv-ro"]
token_meta_role spinnaker
token_meta_service_account_name default
token_meta_service_account_namespace default
token_meta_service_account_secret_name default-token-h9knn
token_meta_service_account_uid 13cee6Dbc-0bc2-11e9-9fd2-0a32f8e530cc

Using the token from the output above allows for the following:

vault login s.bKSSrYOcETCADGvGxhbDaaaD

Once logged in you should be able to read secrets:

vault kv get secret/spinnaker/test

As a reminder, the policy we created provides RO access only so you will need to have written the secret using a separate authenticated client.

Continuous-Deployment: Store Spinnaker Secrets in HashiCorp Vault

Mon, 01 Jan 0001 00:00:00 +0000

In this example, you use the default KV secret engine called secret and store GitHub credentials, a kubeconfig file, and a Java keystore for SAML SSO.

Authentication with Vault servers

We currently support two methods of authentication with Vault servers.

1. Kubernetes service account (recommended)

You’ll need to configure Vault to authenticate with Kubernetes per our Vault Configuration Guide or HashiCorp’s documentation.

Note: If multiple clusters need to access the same Vault server, you’ll need to use the -path flag and give each cluster a different path name. This becomes <cluster auth path> in the example below. If using just one cluster, you can use the default vault auth enable kubernetes command, in which case your path will be kubernetes.

After configuring authentication on the Vault side, use the following configuration to enable Vault secrets in Spinnaker:

Add the following snippet to the SpinnakerService manifest:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 armory:
 secrets:
 vault:
 enabled: true
 authMethod: KUBERNETES # Method used to authenticate with the Vault endpoint. Must be either KUBERNETES for Kubernetes service account auth or TOKEN for Vault token auth. The TOKEN method will require a VAULT_TOKEN environment variable set for Operator and the services. 
 url: <Vault server URL>:<port, if required> # URL of the Vault endpoint from Spinnaker services.
 role: <Vault role> # (Applies to KUBERNETES authentication method) Name of the role against which the login is being attempted.
 # path: <k8s cluster path> (Optional; default: kubernetes) Applies to KUBERNETES authentication method) Path of the kubernetes authentication backend mount. Default is "kubernetes"

2. Token authentication

This method is not recommended, but it is supported if you choose to use it. Armory recommends this for testing and development purposes only. For token authentication, you need to have a VAULT_TOKEN environment variable set in the Halyard container of the Operator pod as well as each of the services.

Use the following configuration to enable Vault secrets using token auth:

Add the following snippet to the SpinnakerService manifest:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 armory:
 secrets:
 vault:
 enabled: true
 authMethod: TOKEN # Method used to authenticate with the Vault endpoint. Must be either KUBERNETES for Kubernetes service account auth or TOKEN for Vault token auth. The TOKEN method will require a VAULT_TOKEN environment variable set for Operator and the services. 
 url: <Vault server URL>:<port if required> # URL of the Vault endpoint from Spinnaker services.

Configuring the Operator to use Vault secrets

If you are using the Armory Operator, set up a custom Halyard configuration with this content:

secrets:
 vault:
 enabled: true
 url: <Vault server URL>
 authMethod: KUBERNETES
 role: <Vault role>
 path: <k8s cluster path>

Once you’ve mounted your ConfigMap to the spinnaker-operator deployment, it restarts the Halyard container with your Vault config.

Storing secrets

To store a file, simply prepend the file path with @. It accepts relative paths but cannot resolve ~:

vault kv put secret/spinnaker/kubernetes config=@path/to/kube/config

The command above stores a single key-value pair at the secret/spinnaker/kubernetes path. Any updates to that path will replace the existing values even if using a different key! In order to store multiple secrets at the same path, it must be done in a single command, like so:

vault kv put secret/spinnaker/github password=<password> token=<token>

Otherwise, just store different secrets at different paths, like we’re doing in these examples.

Make sure to base64 encode any binary files:

base64 -i saml.jks -o saml.b64
vault kv put secret/spinnaker/saml base64keystore=@saml.b64

Referencing secrets

Now that secrets are safely stored in Vault, reference them in config files with the following syntax:

encrypted:vault!e:<secret engine>!p:<path to secret>!k:<key>!b:<is base64 encoded?>

Parameters

Parameters can be provided in any order.

!: required is used as a delimiter between parameters
e: required Vault’s Secret Engine.
p: required Path to your secret, ex: spinnaker/github
k: required Key of the secret.
b: optional If the value is a base64 encoded value or file, set this to true

Example of how it’s used in your YAML configs

github:
 password: encrypted:vault!e:secret!p:spinnaker/github!k:password

kubernetes:
 kubeconfigFile: encrypted:vault!e:secret!p:spinnaker/kubernetes!k:config

gate:
 javaKeyStoreBinary: encrypted:vault!e:secret!p:spinnaker/saml!k:base64keystore!b:true

Continuous-Deployment: Secrets in Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Note: Storing Spinnaker secrets in a Kubernetes secret is only supported if you’re using the Operator to deploy and manage Spinnaker. Additionally, you cannot encrypt configuration secrets for the UI (Deck) using the Kubernetes secret engine.

Creating a Kubernetes secret for Spinnaker to use

This example uses a Kubernetes secret to store GitHub credentials and a kubeconfig file.

Spinnaker^TM can read secrets only within its own namespace. It cannot access Kubernetes secrets stored in a different namespace. In this document, assume that Spinnaker lives in the namespace spinnaker.

You can store files as well as individual text values in Kubernetes secrets to be referenced by Spinnaker. To create the secret you can use this command, assuming you have a file named kubeconfig-prod where you are running the command:

kubectl -n spinnaker create secret generic spin-secrets \
 --from-file=kubeconfig-prod \
 --from-literal=github-token=aaaaaabbbbbbbbccccccccc

The command will create a secret named spin-secrets in the spinnaker namespace, having two keys: one is a kubeconfig file with key kubeconfig-prod , and the other is a text value for a GitHub token with key github-token.

Kustomize also has a secret generator, so you can automatically deploy secrets using Kustomize along with the SpinnakerService manifest. This is a kustomization.yml file that creates the same secret as above:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

secretGenerator:
 - name: spin-secrets
 files:
 - kubeconfig-prod
 literals:
 - github-token=aaaaaabbbbbbbbccccccccc

For more information on how to create secrets in Kubernetes refer to the official Kubernetes docs or the Kustomize docs.

Referencing secrets

You reference secret values in your config with the following format:

encrypted:k8s!n:<secret name>!k:<secret key>

Similarly you can reference secret files:

encryptedFile:k8s!n:<secret name>!k:<secret key>

For example, to reference the GitHub token:

encrypted:k8s!n:spin-secrets!k:github-token

And to reference the content of our kubeconfig file:

encryptedFile:k8s!n:spin-secrets!k:kubeconfig-prod

Continuous-Deployment: Secrets with Google Cloud Storage

Mon, 01 Jan 0001 00:00:00 +0000

This example uses a bucket (mybucket) to store GitHub credentials and a kubeconfig file.

Authorize Spinnaker to access the GCS bucket

Since you’re storing sensitive information, make sure to protect the bucket by restricting access and enabling encryption.

Remember to run the Operator deployment and Spinnaker services with permissions to read that content.

Storing secrets

Store your GitHub credentials in mybucket/spinnaker-secrets.yml:

github:
 password: <PASSWORD>
 token: <TOKEN>

Note: You can store the password under different keys than github.password and github.token. To do so, change how you reference the secret.

Referencing secrets

Now that secrets are securely stored in the bucket, you reference them in your config files with the following format:

encrypted:gcs!b:<bucket>!f:<path to file>!k:<optional yaml key>

For example, to reference github.password from the file above, use:

encrypted:gcs!b:mybucket!f:spinnaker-secrets.yml!k:github.password

To reference the content of our kubeconfig file:

encrypted:gcs!f:mykubeconfig!b:mybucket

Continuous-Deployment: Secrets with S3

Mon, 01 Jan 0001 00:00:00 +0000

See the S3 Getting Started Guide for more information on encryption in S3. This example uses a bucket (mybucket) in the us-west-2 region to store GitHub credentials and a kubeconfig file. You reference the bucket by its URL mybucket.us-west-2.amazonaws.com.

Authorize Spinnaker to access the S3 bucket

Since you’re storing sensitive information, make sure to protect the bucket by restricting access and enabling encryption.

Remember to run the Operator and Spinnaker^TM services with IAM roles that allow them to read the keys stored in the AWS S3 Bucket.

Storing secrets

Storing credentials

Store your GitHub credentials in mybucket/spinnaker-secrets.yml:

github:
 password: <PASSWORD>
 token: <TOKEN>

Note: You could choose to store the password under different keys than github.password and github.token. You’d just need to change how to reference the secret further down.

Storing sensitive files

Some Spinnaker configuration uses information stored as files. For example, upload the kubeconfig file of your Kubernetes account directly to mybucket/mykubeconfig:

aws s3 cp /path/to/mykubeconfig s3://mybucket/mykubeconfig

Referencing secrets

Now that secrets are safely stored in the bucket, you reference them from your config files with the following format. The S3 specific parameters (r:<region>, b:<bucket>, etc) can be in any order:

encrypted:s3!r:<region>!b:<bucket>!f:<path to file>!k:<optional yaml key>

For example, to reference github.password from the file above, we’ll use:

encrypted:s3!r:us-west-2!b:mybucket!f:spinnaker-secrets.yml!k:github.password

And to reference the content of our kubeconfig file:

encryptedFile:s3!r:us-west-2!b:mybucket!f:mykubeconfig

Continuous-Deployment: Secrets with AWS Secrets Manager

Mon, 01 Jan 0001 00:00:00 +0000

See the AWS Secrets Manager User Guide for how to set up AWS Secrets Manager,

Authorize Spinnaker to access the AWS Secrets Manager

Remember to run the Operator and Spinnaker^TM services with IAM roles that allow them to read the keys stored in the AWS Secrets Manager. The following example policy enables access to the AWS Secrets Manger and the KMS store:

 {
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "VisualEditor0",
 "Effect": "Allow",
 "Action": [
 "kms:ListKeys",
 "kms:ListAliases",
 "kms:DescribeKey"
 ],
 "Resource": "*"
 },
 {
 "Sid": "VisualEditor1",
 "Effect": "Allow",
 "Action": [
 "secretsmanager:GetSecretValue",
 "secretsmanager:ListSecretVersionIds",
 "secretsmanager:ListSecrets"
 ],
 "Resource": "*",
 "Condition": {
 "ForAnyValue:StringEquals": {
 "secretsmanager:VersionStage": "AWSCURRENT"
 }
 }
 }
 ]
}

Referencing secrets stored in AWS Secrets Manager

You can reference a KeyStore or KeyStore password stored in AWS Secrets Manager. Based on which type of secret you want to reference, use one of the following formats:

Keystore

 keyStore: encryptedFile:secrets-manager!r:<some region>!s:<secret name>

Keystore password

 keyStorePassword: encrypted:secrets-manager!r:<some region>!s:<secret name>!k:some-key

encryptedFile or encrypted - Required. Indicates that this is an encrypted file or an encrypted string, respectively.
secrets-manager - Required. Indicates that secrets are stored in AWS Secrets Manager
! - Required. Delimiter between parameters.
r:<AWS region> - Required. The AWS region your secret is stored in. For example, use r:us-west-2 for a secret stored in the us-west-2 region.
s:<Secret name> - Required. The name of the secret stored in AWS Secrets Manager
k<some-key> - Required for encrypted strings. The Secret key. Omit for KeyStores.

For example, the following example references a KeyStore stored in us-west-2:

encryptedFile:secrets-manager!r:us-west-2!s:dev--cert