Armory Docs – Armory Admin

Continuous-Deployment: Add a Kubernetes Account Deployment Target in Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

How Spinnaker operates when deploying to Kubernetes targets

At a high level, Spinnaker operates in the following way when deploying to Kubernetes:

Spinnaker is configured with one or more “Cloud Provider” Kubernetes accounts (which you can think of as deployment targets)
For each Kubernetes account, Spinnaker is provided a kubeconfig to connect to that Kubernetes cluster
The kubeconfig should have the following contents:
- A Kubernetes kubeconfig cluster, which is typically a URL and potentially certificate validation information (CA cert or a flag to skip validation)
- A Kubernetes kubeconfig user, which is basically a way to get a set of credentials to connect to the Kubernetes cluster. This can be a token, a set of commands to run to get a token, or a client certificate
- A Kubernetes kubeconfig context, which tells the kubectl tool which credential set to use with which cluster, as well as other information such as default namespace
- Metadata such as which context to use by default
Each Kubernetes account is configured in the SpinnakerService manifest under spec.spinnakerConfig.config.providers.kubernetes.accounts key if using the Operator. Each entity has these (and other) fields:
- name: A Spinnaker-internal name used to refer to the Kubernetes account. This is the item you will select in various Kubernetes stages to indicate that you would like to deploy to this particular Kubernetes account (which again, consists of a Kubernetes cluster/credential/context set)
- kubeconfigFile: A file path referencing the contents of the kubeconfig file for connecting to the target cluster. When using the Operator, this can be any name that should match an entry in spec.spinnakerConfig.files where the file contents is copied. This field supports referencing files stored in external secret engines
- onlySpinnakerManaged: When true, Spinnaker only caches and displays applications that have been created by Spinnaker. Before placing in a false state, you should review the Kubernetes cluster configuration. When false, Spinnaker analyzes the cluster and automatically attempts to configure and populate applications for resources already present in Kubernetes, unless limited with omitNamespaces. You should note the increased possibilities of misconfigured Autogenerated Application Placeholders in the deployments.
- namespaces: An array of namespaces that Spinnaker will be allowed to deploy to. If this is left blank, Spinnaker will be allowed to deploy to all namespaces
- omitNamespaces: If namespaces is left blank, you can blacklist specific namespaces to indicate to Spinnaker that it should not deploy to those namespaces
Spinnaker uses kubectl under the hood, so any semantics that kubectl specifies, Spinnaker will also use
kubectl (and the corresponding kubeconfig) will be run from the Spinnaker Clouddriver Kubernetes pod
If the kubeconfig is properly referenced and available, Operator will take care of the following for you:
- Creating a Kubernetes secret (with a dynamically-generated name) containing your kubeconfig in the namespace where Spinnaker lives
- Dynamically generating a clouddriver.yml file that properly references the kubeconfig from where it is mounted within the Clouddriver container
- Creating/Updating the Kubernetes Deployment (spin-clouddriver) which runs Clouddriver so that it is aware of the secret and properly mounts it in the Clouddriver pod
- Creating a secret containing clouddriver.yml (and optionally, clouddriver-local.yml) and including it in the spin-clouddriver deployment

So here are some takeaways and guiding principles that result from the above:

Spinnaker relies on the kubeconfig to authenticate against and access your Kubernetes cluster.
You can use a command-based or auth-provider-based kubeconfig user (such as aws-iam-authenticator), but this will rely on the command binary and all relevant files existing in the Clouddriver, and all authentication (such as IAM roles) being properly attached to the Clouddriver container.
Alternately (and perhaps, preferably), you can do the following:
- Create a Kubernetes service account in your Kubernetes cluster. This service account will exist in some specific namespace.
- Grant the service account permissions on your cluster. This can be one or more of the following:
  - cluster-admin access, to be able to access all namespaces in your cluster
  - Full wildcard (apiGroup * / resource * / verb *) access to one or more namespaces in your cluster
  - More specific permissions in your cluster (this is not covered in the scope of this document)
It is preferable to create a distinct kubeconfig file for each Spinnaker Cloud Provider Kubernetes account; each of these kubeconfigs should have one cluster, one user, and one context referencing the cluster and user. Its default context should reference the single context

Before you begin

This document assumes the following:

Your Armory CD or Spinnaker instance is running
If using Spinnaker, you installed and configured Spinnaker using Armory’s Spinnaker Operator. You have access to the SpinnakerService manifest and have the kubeconfig for the Spinnaker cluster.
You have a valid kubeconfig that currently has Cluster Admin access to your target Kubernetes cluster so you can create the relevant Kubernetes entities (service account, role, and rolebinding) in the target cluster; you have kubectl and are able to use it to interact with your target Kubernetes cluster.

The first several steps of this document take place on a system that has kubectl and the kubeconfig.

Workflow for adding a Kubernetes Service Account

Create the namespace in which the Service Account will live (if it does not yet exist)
Create the service account in the Service Account namespace
If granting cluster admin (cluster-admin), a clusterrolebinding attaching the cluster-admin ClusterRole to the service account
If granting namespace-specific access, the following:
- For each namespace, the namespace (if it does not exist)
- For each target namespace, an admin role
- For each target namespace, a rolebinding attaching the admin role to the service account
Create a minified kubeconfig containing only the service account
Add the account to Spinnaker

Setting up the Kubernetes Service Account

In this section, you create the following:

A namespace to create the service account in
A service account in that namespace
A role and rolebinding in that namespace, granting permissions to the service account
A kubeconfig containing credentials for the service account

This document uses the newly-created Armory spinnaker-tools Go CLI (available on Github) to create many of these resources. There are separate instructions to perform these steps manually.

Download the latest `spinnaker-tools` release

Go to https://github.com/armory/spinnaker-tools/releases to download the latest release for your operating system (OSX and Linux available). You can also use curl:

# If you're not already in the directory
cd ~/eks-spinnaker
# Replace this with the correct version and link for your workstation (use https://github.com/armory/spinnaker-tools/releases/download/latest/spinnaker-tools-linux if you're on Linux instead of Mac)
curl -L https://github.com/armory/spinnaker-tools/releases/download/latest/spinnaker-tools-darwin -o spinnaker-tools
chmod +x spinnaker-tools

Set up bash parameters

Set up bash environment variables that you will use later.

# If you're using a different source kubeconfig, specify it here
export SOURCE_KUBECONFIG=${HOME}/.kube/config

# Specify the name of the kubernetes context that has permissions in your target cluster.
# To get context names, you can run "kubectl config get-contexts".
export CONTEXT="aws-armory-dev"

# Enter the namespace where you want the Spinnaker service account to live
export SPINNAKER_NAMESPACE="spinnaker-system"

# Enter the name of the service account you want to create in the target namespace.
# If you are creating multiple Kubernetes Cloud Provider Accounts to point to
# different namespaces in to the same Kubernetes cluster, make sure you use a
# unique service account name.
export SPINNAKER_SERVICE_ACCOUNT_NAME="spinnaker-dev-sa"

# Specify where you want the new kubeconfig to be created
export DEST_KUBECONFIG=${PWD}/kubeconfig-dev-sa

# If you want to deploy to a specific set of namespaces, enter the namespaces
# that you want to deploy to (space-delimited).
# These namespaces can already exist, or you can create them.
export TARGET_NAMESPACES_COMMA_SEPARATED=dev-1,dev-2

Option 1: Create the service account with cluster-admin permissions

./spinnaker-tools create-service-account \
 --kubeconfig ${SOURCE_KUBECONFIG} \
 --context ${CONTEXT} \
 --output ${DEST_KUBECONFIG} \
 --namespace ${SPINNAKER_NAMESPACE} \
 --service-account-name ${SPINNAKER_SERVICE_ACCOUNT_NAME}

Important

For Kubernetes v1.24+, service account tokens are no longer generated when you execute this script. See the LegacyServiceAccountTokenNoAutoGeneration entry in Kubernetes 1.24’s Urgent Upgrade Notes for details and the new process to follow for creating service account tokens.

Option 2: Create the service account with namespace-specific permissions

./spinnaker-tools create-service-account \
 --kubeconfig ${SOURCE_KUBECONFIG} \
 --context ${CONTEXT} \
 --output ${DEST_KUBECONFIG} \
 --namespace ${SPINNAKER_NAMESPACE} \
 --service-account-name ${SPINNAKER_SERVICE_ACCOUNT_NAME} \
 --target-namespaces ${TARGET_NAMESPACES_COMMA_SEPARATED}

Add the kubeconfig and cloud provider to Spinnaker

Add the following configuration to the SpinnakerServce manifest, replacing values as needed:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 providers:
 kubernetes:
 enabled: true
 accounts:
 - name: spinnaker-dev # Account name you want Spinnaker to use to identify the deployment target
 requiredGroupMembership: []
 providerVersion: V2
 permissions: {}
 dockerRegistries: []
 configureImagePullSecrets: true
 cacheThreads: 1
 namespaces: [] # Change if you only want to deploy to specific namespaces
 omitNamespaces: []
 kinds: []
 omitKinds: []
 customResources: []
 cachingPolicies: []
 oAuthScopes: []
 onlySpinnakerManaged: true
 kubeconfigFile: kubeconfig-spinnaker-dev
 primaryAccount: spinnaker-dev # Change to a desired account from the accounts array
 features:
 artifacts: true # Not strictly necessary for Kubernetes but will be useful in general
 files:
 kubeconfig-spinnaker-dev: |
  <FILE CONTENTS HERE>

Finally, apply the changes

kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest file>

Verify the Kubernetes account appears in the Spinnaker UI

After you apply your changes, you should see the new Kubernetes account in your Spinnaker UI and be able to deploy to it. It may take a while for the Clouddriver to start up and read information from your new Kubernetes account.

Don’t forget to clear your browser cache / hard refresh your browser (cmd-shift-r or control-shift-r)

Continuous-Deployment: Administration for Cloud Foundry

Mon, 01 Jan 0001 00:00:00 +0000

Continuous-Deployment: Administration on AWS

Mon, 01 Jan 0001 00:00:00 +0000

Continuous-Deployment: Auth Propagation in Spinnaker Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Overview of Authentication and Authorization in Spinnaker

Both Armory Continuous Deployment and Spinnaker^TM provide the same functionality for authentication (“authn”) and authorization (“authz”). You can find a full reference of how to set up both in the Spinnaker documentation.

Authorization & Manual Judgments

The Spinnaker docs explain that you can limit users’ access to both “accounts” and “applications” but doesn’t talk much about the interaction of the two.

In short, if you have access to an application, you can view the pipelines, and kick off a manual execution (even if you have “read only” access). However, if those pipelines need to do something in your cloud environments, you will still need to have read/write access to those environments. Since the pipeline will run its stages “as the user” that initiated the pipeline, the stages that attempt to write changes to the environment will fail if that user doesn’t have access to those environments.

There is one exception to this rule, and that is for Manual Judgment stages. You can configure a Manual Judgment stage to “Propagate Authentication”:

Checking this box will cause the pipeline to use the identity and authorizations of the user who approved the stage for all subsequent stages. By inserting a Manual Judgment stage with this option enabled into your pipeline before the actual deploy, you can allow users with limited access to kick off pipelines safely; a user with full access to the environment can then continue the pipeline successfully after approval.

Continuous-Deployment: Bake and Share Amazon Machine Images Across Accounts

Mon, 01 Jan 0001 00:00:00 +0000

In many environments, Spinnaker^TM runs under a different AWS account than the target deployment account. This guide shows you how to configure Spinnaker to share an AMI created where Spinnaker lives with the AWS account where your applications live. This guide is assuming that AWS roles are already properly setup for talking to the target account.

You can add the following snippet to your SpinnakerService manifest and apply it after replacing the example values with ones that correspond to your environment. The example adds an AWS account and configures the baking service (Rosco) with default values:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 aws:
 enabled: true
 accounts:
 - name: my-aws-account
 requiredGroupMembership: []
 providerVersion: V1
 permissions: {}
 accountId: 'aws-account-id' # Use your AWS account id
 regions: # Specify all target regions for deploying applications
 - name: us-west-2
 assumeRole: role/SpinnakerManagedProfile # Role name that worker nodes of Spinnaker cluster caassume in the target account to make deployments and scan infrastructure
 primaryAccount: my-aws-account
 bakeryDefaults:
 baseImages: []
 defaultKeyPairTemplate: '{{"{{"}}name{{"}}"}}-keypair'
 defaultRegions:
 - name: us-west-2
 defaults:
 iamRole: BaseIAMRole
 ... # Config omitted for brevity
 service-settings:
 rosco:
 env:
 SPINNAKER_AWS_DEFAULT_REGION: "us-west-2" # Replace by default bake region
 SPINNAKER_AWS_DEFAULT_ACCOUNT: "target-aws-account-id" # Target AWS account id
 ... # Config omitted for brevity

Spinnaker pipeline Bake stage configuration

Make sure to check the Show Advanced Options checkbox. Then where it says Template File Name use aws-multi-ebs.json as the value.

Then add an Extended Attribute. Have the key be share_with_1 and the value being the target AWS account ID that was used for SPINNAKER_AWS_DEFAULT_ACCOUNT. share_with_1 is for ami_users inside Packer.

You can also copy the resulting AMI to different regions by overriding the copy_to_1 values. These match up to ami_regions inside Packer.

Continuous-Deployment: Clouddriver Caching Agents in Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

How Spinnaker discovers and caches infrastructure elements

Clouddriver is the Spinnaker™ service responsible for discovering and caching cloud provider infrastructure, such as AWS EC2 instances, Kubernetes pods, and Docker images. Clouddriver has a caching agent scheduler that runs caching agents for providers in separate threads at scheduled intervals. These agents query the infrastructure, index the data, and save the results in a cache store, which is usually a Redis or a SQL datastore.

The caching agent

Clouddriver providers and caching agents architecture

Caching agents query your cloud infrastructure for resources and store the results in a cache store. Each provider has its own set of caching agents that Spinnaker instantiates per account and, sometimes, per region. Each caching agent specializes in one type of resource, such as server groups, load balancers, security groups, or instances.

The number of caching agents varies greatly between providers and Clouddriver configurations. For example, AWS might have between 16 and 20 agents per region performing tasks such as caching the status of IAM roles, instances, and VPCs. AWS might also have some agents operating globally for tasks like cleaning up detached instances. Kubernetes, on the other hand, might have a few agents per cluster, caching things like custom resources and Kubernetes manifests.

The cache store

The cache store is where Clouddriver stores cloud resources. You can use different types:

Redis - the default and most popular implementation.
SQL - recommended store. For how to configure Clouddriver to use SQL, see the Configure Clouddriver to use a SQL Database.
an in-memory cache that is not used for actual Spinnaker deployments.

With the exception of the in-memory store, these stores work across multiple Clouddriver instances.

One or more instances of Clouddriver update this single cache store.

The caching agent scheduler

The caching agent scheduler runs caching agents at regular intervals across all Clouddriver instances. There are multiple types of schedulers:

the Redis-backed scheduler that locks agents by reading/writing a key to Redis.
the Redis-backed sort scheduler that locks agents by reading/writing a key to Redis and manages the order of agents being executed.
the SQL-backed scheduler that locks agents by inserting a row in a table with a unique constraint - not very efficient, prefer other schedulers.
the default scheduler that doesn’t lock. Don’t use this if you expect to run more than one Clouddriver instance.

The cache store does not dictate the type of agent scheduler. For instance, you could use the SQL cache store along with the Redis-backed scheduler.

If you read the Clouddriver source code, you see references to cats (Cache All The Stuff), which is the framework that manages agent scheduler, agents, and cache store.

How the cache store, scheduler, and agent work together

When Clouddriver starts, it inspects its configuration and instantiates the cache store and the agent scheduler. For each provider enabled, Clouddriver instantiates agents per account and region and then adds them to the scheduler.

When the scheduler runs, Clouddriver contacts agents not currently running on its own instance. The scheduler attempts to obtain a lock on the agent type/account/region to ensure that only a single Clouddriver instance caches a given resource at any given time. If Clouddriver obtains a lock, the agent is then run in a separate thread. When the agent finishes, Clouddriver updates the lock Time To Live (TTL) to match the next desired execution time.

On-demand caching agents

Most cloud mutating operations are not synchronous. For example, when Clouddriver sends a request to AWS to launch a new EC2 instance, the API call returns successfully but the EC2 instance takes a while to be ready. This is when Spinnaker uses on-demand caching agents.

On-demand caching agents are, as their name implies, created on demand by the client (Orca) in tasks such as Force Cache Refresh or Wait for Up Instances. They are used to ensure cache freshness and keep things up to date when a resource is created or effectively deleted.

When using a cache store like Redis that works across multiple Clouddriver instances, Clouddriver waits for the next regular caching agent of the same type to run before declaring the cache consistent. It gives the cache store one more chance to replicate its state to other replicas in the case of Redis.

Next steps

Configure Clouddriver Caching Agents in Spinnaker
If you are using Kubernetes and would like to use a different method for cataloging your Kubernetes infrastructure, see Scale Agent for Spinnaker and Kubernetes.

Continuous-Deployment: Configure Clouddriver Caching Agents in Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Caching agents in Spinnaker

See the Clouddriver Caching Agents in Spinnaker page for detailed content on caching agents.

Depending on how large your infrastructure is and how many elements it has, you may need to increase Clouddriver’s CPU and memory limits, increase the number of running Clouddriver instances, or both.

The following configuration settings affect the behavior of the caching agents and can be used to adjust them depending on the size of the infrastructure being cached. If using the Spinnaker Operator, these settings live under the key .spec.spinnakerConfig.profiles.clouddriver of SpinnakerService manifest.

SQL global caching agents configuration

Key	Description	Default Value
`sql.scheduler.enabled`	Enable or disable the sql scheduler	false
`sql.agent.max-concurrent-agents`	Indicates the maximum amount of agents to run at the same time when using the sql scheduler	100
`sql.agent.disabled-agents`	List of agent names to disable from running	(empty)
`sql.agent.enabled-pattern`	Regex of agent names enabled for running	.*
`sql.agent.release-threshold-ms`	Maximum amount of time for releasing agent locks after they finish running. If this value is higher than agent’s execution cycle, the agents keep running immediately after finishing	500
`sql.agent.agent-lock-acquisition-interval-seconds`	How often the scheduler checks for the next batch of agents to run	1
`sql.agent.poll.interval-seconds`	Default time for how often to run caching agents	30
`sql.agent.poll.error-interval-seconds`	Default time for when to run caching agents after an execution fails	30
`sql.agent.poll.timeout-seconds`	Maximum time to hold a lock of an agent execution. If an agent takes longer to finish than this, it’s possible to have concurrent executions of the same agent	300

See the Configure Clouddriver to use a SQL Database page for how to configure Clouddriver.

Redis global caching agents configuration

Key	Description	Default Value
`redis.scheduler.enabled`	Enable or disable the redis scheduler	true
`redis.agent.max-concurrent-agents`	Indicates the maximum amount of agents to run at the same time when using the redis scheduler	1000
`redis.agent.enabled-pattern`	Regex of agent names enabled for running	.*
`redis.agent.agent-lock-acquisition-interval-seconds`	How often the scheduler checks for the next batch of agents to run	1
`redis.poll.interval-seconds`	Default time for how often to run caching agents	30
`redis.poll.error-interval-seconds`	Default time for when to run caching agents after an execution fails	30
`redis.poll.timeout-seconds`	Maximum time to hold a lock of an agent execution. If an agent takes longer to finish that this, it’s possible to have concurrent executions of the same agent	300

Considerations

Using the SQL scheduler has known issues where some agents may not be running in some versions of Spinnaker.
The setting for max-concurrent-agents is directly correlated with how much CPU and memory Clouddriver needs to cache infrastructure. Higher values make each replica consume more resources. Higher values also make it possible for Clouddriver to reduce the time spent caching all infrastructure.

Continuous-Deployment: Configure Automated Canary Deployments in Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Kayenta Overview

Kayenta is the Spinnaker service that performs automated canary analysis. The goal of Kayenta is to provide you with confidence that a deployment is safe through automation and intelligence.

Enable Kayenta

Add the following to your SpinnakerService manifest:

spec:
 spinnakerConfig:
 config:
 canary:
 enabled: true # Enable/disable canary analysis

Configure Kayenta

The following example is a SpinnakerService manifest. The example config uses Datadog as the metrics provider and stores canary configs and analysis in a GCS bucket:

apiversion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 canary:
 enabled: true # Enable/disable canary analysis
 serviceIntegrations:
 - name: datadog
 enabled:
 accounts:
 - name: test-account
 endpoint:
 baseUrl: <some-URL> # The base URL to the Datadog server.
 apiKey: <encrypted-secret> # Your org’s unique Datadog API key. See https://app.datadoghq.com/account/settings#api. Supports encrypted value.
 applicationKey: <encrypted-secret> #Your Datadog application key. See https://app.datadoghq.com/account/settings#api. Supports encrypted value.
 supportedTypes:
 - METRICS_STORE
 - CONFIGURATION_STORE
 - OBJECT_STORE
 - name: google
 enabled: true # Enable/disable Google provider
 accounts:
 - name: my-google-account
 project: my-project-id # The Google Cloud Platform project the Canary service uses to consume GCS and Stackdriver.
 jsonPath: gcp-sa.json # File name of a JSON service account that Spinnaker uses for credentials. This is only needed if Spinnaker is not deployed on a Google Compute Engine VM or needs permissions not afforded to the VM it is running on. See https://cloud.google.com/compute/docs/access/service-accounts for more information. This field supports using "encryptedFile" secret references.
 bucket: my-bucket # The name of a storage bucket that your specified account has access to. If you specify a globally unique bucket name that doesn't exist, Kayenta creates that bucket.
 bucketLocation: us-central-1 # Required if the bucket you specify doesn't exist. In that case, the bucket gets created in that location. See https://cloud.google.com/storage/docs/managing-buckets#manage-class-location.
 rootFolder: kayenta # The root folder in the chosen bucket to place all of the Canary service's persistent data in (Default: kayenta).
 supportedTypes: # Array of: METRICS_STORE, CONFIGURATION_STORE, OBJECT_STORE
 - CONFIGURATION_STORE
 - OBJECT_STORE
 gcsEnabled: true # Whether or not GCS is enabled as a persistent store (Default: false).
 stackdriverEnabled: false # Whether or not Stackdriver is enabled Stackdriver as a metrics service (Default: false).
 metadataCachingIntervalMS: 60000 # Number of milliseconds to wait between caching the names of available metric types for use in building canary configs. (Default: 60000)
 reduxLoggerEnabled: true # Whether or not to enable redux logging in the canary module in deck (Default: true).
 defaultJudge: NetflixACAJudge-v1.0 # Name of canary judge to use by default (Default: NetflixACAJudge-v1.0).
 stagesEnabled: true # Whether or not to enable canary stages in deck (Default: true).
 templatesEnabled: true # Whether or not to enable custom filter templates for canary configs in deck (Default: true).
 showAllConfigsEnabled: true # Whether or not to show all canary configs in deck, or just those scoped to the current application (Default: true).
 ... # rest of config omitted for brevity
 files:
 gcp-sa.json: |
  <JSON CONTENT HERE. WATCH YOUR SPACING>

For an overview of how to configure metrics providers see Canary Config. For Dynatrace and AWS Cloudwatch providers, see Use Canary Analysis with Dynatrace or AWS CloudWatch Integration Plugin.

What’s next

For information about how to use canary deployments, see Use Automatic Canary Analysis in Spinnaker.

Continuous-Deployment: Configure Clouddriver to use a SQL Database

Mon, 01 Jan 0001 00:00:00 +0000

Advantages of using an RDMS with Clouddriver

Since Armory Continuous Deployment version 2.5.x (Spinnaker 1.14.x), Clouddriver can store its data (task, infrastructure, etc) in a MySQL-compatible database. Similar to Orca, the main advantage of doing this is to improve performance and remove Redis as a single point of failure.

Database compatibility:

MySQL	PostgreSQL
5.7; AWS Aurora	10+

Armory recommends MySQL 5.7 or AWS Aurora.

Base configuration

You can find a complete description of the options in the open source documentation.

Database setup

You can skip this step if you create the database during provisioning - for instance with Terraform.

Once you’ve provisioned your RDBMS and ensured connectivity with Spinnaker, you need to create the database:

Configure the RDBMS driver exactly as described in Set up Clouddriver to use SQL - Database Setup. The MySQL database schema must be configured to with:

Default character set utfmb4
Default collate utf8mb4_unicode_ci

CREATE SCHEMA `clouddriver` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Then we’ll grant authorization to the clouddriver_service and clouddriver_migrate users:


 GRANT
 SELECT, INSERT, UPDATE, DELETE, CREATE, EXECUTE, SHOW VIEW
 ON `clouddriver`.*
 TO 'clouddriver_service'@'%';

 GRANT
 SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, LOCK TABLES, EXECUTE, SHOW VIEW
 ON `clouddriver`.*
 TO `clouddriver_migrate`@'%';

This configuration grants authorization from any host. You can restrict it to the cluster in which Spinnaker runs by replacing the % with the IP address of Clouddriver pods from MySQL.

Depending on your TLS settings, you may need to configure TLS 1.2. For more information, see the Knowledge Base articles Disabling TLS 1.1 in Spinnaker and Specifying the Protocols to be used and How to fix TLS error “Reason: extension (5) should not be presented in certificate_request”.

Deployment

You have two options for deploying Clouddriver with MySQL: a simpler deployment, which involves downtime, or a three-step method that avoids downtime. Pick the method that best fits your requirements.

Simple deployment

If you are not worried about downtime or if Spinnaker is not currently executing any pipelines, you can run a simple deployment by adding the following snippet to SpinnakerService manifest under spec.spinnakerConfig.profiles.clouddriver:

sql:
 enabled: true
 taskRepository:
 enabled: true
 cache:
 enabled: true
 # These parameters were determined to be optimal via benchmark comparisons
 # in the Netflix production environment with Aurora. Setting these too low
 # or high may negatively impact performance. These values may be sub-optimal
 # in some environments.
 readBatchSize: 500
 writeBatchSize: 300
 scheduler:
 enabled: true
 connectionPools:
 default:
 # additional connection pool parameters are available here,
 # for more detail and to view defaults, see:
 # https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt
 default: true
 jdbcUrl: jdbc:mysql://your.database:3306/clouddriver
 user: clouddriver_service
 # password: depending on db auth and how spinnaker secrets are managed
 # The following tasks connection pool is optional. At Netflix, clouddriver
 # instances pointed to Aurora read replicas have a tasks pool pointed at the
 # master. Instances where the default pool is pointed to the master omit a
 # separate tasks pool.
 tasks:
 user: clouddriver_service
 jdbcUrl: jdbc:mysql://your.database:3306/clouddriver
 migration:
 user: clouddriver_migrate
 jdbcUrl: jdbc:mysql://your.database:3306/clouddriver

redis:
 enabled: false
 cache:
 enabled: false
 scheduler:
 enabled: false
 taskRepository:
 enabled: false

No downtime deployment

To avoid downtime for your deployment, use the following three steps:

Step 1: Warm up the cache

The first step is to start a Clouddriver that is not accessible from other services to validate the installation and warm up the cache.

You can do it manually or by using the following script. Make sure tables are properly created and being populated by these instances of Clouddriver.

Step 2: Use MySQL to back new tasks

After waiting a few minutes (from 2 to 10 minutes depending on how many accounts are connected), we’ll update Spinnaker to use MySQL but remain aware of task statuses in Redis.

We’re deploying Spinnaker with the following configuration in SpinnakerService manifest under the key spec.spinnakerConfig.profiles.clouddriver:

sql:
 enabled: true
 taskRepository:
 enabled: true
 cache:
 enabled: true
 # These parameters were determined to be optimal via benchmark comparisons
 # in the Netflix production environment with Aurora. Setting these too low
 # or high may negatively impact performance. These values may be sub-optimal
 # in some environments.
 readBatchSize: 500
 writeBatchSize: 300
 scheduler:
 enabled: true
 connectionPools:
 default:
 # additional connection pool parameters are available here,
 # for more detail and to view defaults, see:
 # https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt
 default: true
 jdbcUrl: jdbc:mysql://your.database:3306/clouddriver
 user: clouddriver_service
 # password: depending on db auth and how spinnaker secrets are managed
 # The following tasks connection pool is optional. At Netflix, clouddriver
 # instances pointed to Aurora read replicas have a tasks pool pointed at the
 # master. Instances where the default pool is pointed to the master omit a
 # separate tasks pool.
 tasks:
 user: clouddriver_service
 jdbcUrl: jdbc:mysql://your.database:3306/clouddriver
 migration:
 user: clouddriver_migrate
 jdbcUrl: jdbc:mysql://your.database:3306/clouddriver

redis:
 cache:
 enabled: false
 scheduler:
 enabled: false

executionRepository:
 dual:
 enabled: true
 primaryName: sqlExecutionRepository
 previousClass: redisExecutionRepository

At this point, you can stop the pods you created in step 1. If you used the preceding script, just delete the spin-clouddriver-sql deployment.

Step 3: Remove Redis

After waiting a few minutes so that Redis tasks are no longer relevant, finish by removing Redis entirely:

sql:
 enabled: true
 taskRepository:
 enabled: true
 cache:
 enabled: true
 # These parameters were determined to be optimal via benchmark comparisons
 # in the Netflix production environment with Aurora. Setting these too low
 # or high may negatively impact performance. These values may be sub-optimal
 # in some environments.
 readBatchSize: 500
 writeBatchSize: 300
 scheduler:
 enabled: true
 connectionPools:
 default:
 # additional connection pool parameters are available here,
 # for more detail and to view defaults, see:
 # https://github.com/spinnaker/kork/blob/master/kork-sql/src/main/kotlin/com/netflix/spinnaker/kork/sql/config/ConnectionPoolProperties.kt
 default: true
 jdbcUrl: jdbc:mysql://your.database:3306/clouddriver
 user: clouddriver_service
 # password: depending on db auth and how spinnaker secrets are managed
 # The following tasks connection pool is optional. At Netflix, clouddriver
 # instances pointed to Aurora read replicas have a tasks pool pointed at the
 # master. Instances where the default pool is pointed to the master omit a
 # separate tasks pool.
 tasks:
 user: clouddriver_service
 jdbcUrl: jdbc:mysql://your.database:3306/clouddriver
 migration:
 user: clouddriver_migrate
 jdbcUrl: jdbc:mysql://your.database:3306/clouddriver

redis:
 enabled: false
 cache:
 enabled: false
 scheduler:
 enabled: false
 taskRepository:
 enabled: false

Continuous-Deployment: Configure Dynamic Kubernetes Accounts

Mon, 01 Jan 0001 00:00:00 +0000

Overview of external account configuration

If you add, delete, or modify Kubernetes deployment targets on a regular basis, you may find that redeploying Clouddriver to pull in new account configuration impacts your teams. Spinnaker’s External Account Configuration feature allows you to manage account configuration externally from Spinnaker and then read that configuration change without requiring a redeployment of Clouddriver.

External Account Configuration is only supported for Kubernetes and Cloud Foundry provider accounts. This document describes how this works with Kubernetes accounts.

Armory does not recommend using External Account Configuration with Spring Cloud Config Server. If you have an existing environment with Spring Cloud Config Server with this setup, see the following KB article: Configure Spinnaker’s External Account Configuration with Vault.

Continuous-Deployment: Configure Gate and Deck to Run on the Same Hostname

Mon, 01 Jan 0001 00:00:00 +0000

Overview

The Spinnaker^TM microservice Gate serves as the API gateway for Armory and is leveraged by the Deck microservice to display various Armory data via a user interface. In order to successfully use the Deck UI, both Deck and Gate must be accessible via DNS. The standard approach to configuring access to both Deck and Gate is to set DNS routing to each service on separate hostnames or separate sub-domains of the same hostname. While this approach is typical, it increases the complexity of both DNS management and Ingress management into Kubernetes.

To simplify both DNS management and Ingress management, Armory can be configured to serve the Gate microservice on the same hostname as the Deck UI but located at a sub-path. We recommend configuring the Gate microservice so that it is served from the /api/v1 of the Deck UI hostname when using a single hostname for both Deck and Gate.

Prerequisites for running on the same host

Deck is accessible at https://spinnaker.example.com
Gate is accessible at https://spinnaker.example.com/api/v1
Kubernetes Ingress and Service is used to route traffic from these paths to port 8084 on Gate and 9000 on Deck.

Note: Configuring a Kubernetes Ingress and Service is outside the scope of this document.

Configure Spinnaker

Set Gate’s server servlet to be aware of its context path at /api/v1 in your SpinnakerService config.

Add the following under the spec.spinnakerConfig.profiles section:
```
gate:
 server:
 servlet:
 context-path: /api/v1
```
Set Gate’s health endpoint to /api/v1/health in your SpinnakerService config.

Add the following under the spec.spinnakerConfig.service-settings section:
```
gate:
 healthEndpoint: /api/v1/health
```
Update Deck’s URL in your SpinnakerService config.

Add the following under the spec.config.security section:
```
uiSecurity:
 overrideBaseUrl: https://spinnaker.example.com
```
Update Gate’s URL in your SpinnakerService config.

Add the following under the spec.config.security section:
```
apiSecurity:
 overrideBaseUrl: https://spinnaker.example.com/api/v1
```
Deploy your SpinnakerService config using either kubectl or kustomize command syntax

Continuous-Deployment: Configure GitHub OAuth for Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

You have the ability to modify developer settings for your GitHub organization.
You have a Spinnaker^TM instance with DNS and SSL configured.

Configure GitHub OAuth in GitHub

Follow the instructions in GitHub’s Creating an OAuth App guide.

Homepage URL: This is the URL of your the Gate service; for example, https://gate.spinnaker.acme.com.
Authorization callback URL: The URL needs login appended to your Gate endpoint; for example, https://gate.spinnaker.acme.com/login or https://spinnaker.acme.com/gate/login.

Configure GitHub OAuth in Spinnaker

Add the following snippet to your SpinnakerService manifest under the spec.spinnakerConfig.config.security.authn level:

oauth2:
 enabled: true
 client:
 clientId: a08xxxxxxxxxxxxx93
 clientSecret: 6xxxaxxxxxxxxxxxxxxxxxxx59 # Secret Enabled Field
 scope: read:org,user:email
 provider: GITHUB

Review the Armory Continuous Deployment Manifest Configuration Reference for additional configuration options.

What’s next

Continuous-Deployment: Configure mTLS for Spinnaker Services

Mon, 01 Jan 0001 00:00:00 +0000

Overview of mTLS

mTLS is a transport level security measure. When a client connects to a server, as in a TLS connection:

The server responds with its certificate. Additionally, the server sends a certificate request and a list of Distinguished Names the server recognizes.
The client verifies the certificate of the server and responds with its own certificate and the Distinguished Name its certificate was (in)directly signed with.
The server verifies the certificate of the client.

To set up TLS, provide the following:

For a server:
- Certificate and private key
- Chain of certificates to validate clients
For a client:
- Certificate and private key to present to the server
- Chain of certificates to validate the server (if self signed)

For information about TLS, see Configure TLS for Spinnaker Services.

What you need

The following table lists the Armory and Spinnaker services, their type (Java or Golang), and which certificates they need:

Service	Type	Server	Client
Clouddriver	Java	Yes	Yes
Deck	N/A	-	-
Dinghy*	Golang	Yes	Yes
Echo	Java	Yes	Yes
Fiat	Java	Yes	Yes
Front50	Java	Yes	Yes
Gate	Java	Maybe	Yes
Kayenta	Java	Yes	Yes
Igor	Java	Yes	Yes
Orca	Java	Yes	Yes
Rosco	Java	Yes	Yes
Terraformer*	Golang	Yes	Yes

Dinghy is the service for Pipelines-as-Code.
Terraformer is the service for the Armory Terraform Integration.

Note: Gate may be handled differently if you already terminating SSL at Gate. If not, make sure the load balancer and ingress you are using supports self-signed certificates.

In the following sections, you need to have the following information available:

ca.pem (all Golang servers): the CA certificate in PEM format
[service].crt (each Golang server): the certificate and (optionally) the private key of the Golang server in PEM format
[service].key (each Golang server): the private key of the Golang server if not bundled with the certificate you’re using
[GOSERVICE]_KEY_PASS (each Golang server): the password to the private key of the server
truststore.p12 (all Java clients): a PKCS12 truststore with CA certificate imported
TRUSTSTORE_PASS (all Java clients): the password to the truststore you’re using
[service].p12 (each Java server): a PKCS12 keystore containing the certificate and private key of the server
[SERVICE]_KEY_PASS (each Java server): the password to the keystore you’re using

The server certificate will serve as its client certificate to other services. You can generate different certificates and use them in ok-http-client.key-store* (Java) and http.key* (Golang).

To learn how to generate these files, see the Generate Certificates for Spinnaker guide.

Configuring Java services

Add the following to each Java service under profiles in the SpinnakerService’s profiles:

# Only needed for "server" role
server:
 ssl:
 enabled: true
 key-store: <reference to [service].p12>
 key-store-type: PKCS12
 key-store-password: <[SERVICE]_KEY_PASS>
 trust-store: <reference to ca.p12>
 trust-store-type: PKCS12
 trust-store-password: <TRUSTSTORE_PASS>
 # Roll out with "want" initially
 client-auth: need

# Needed for all Java services
ok-http-client:
 key-store: <reference to [service].p12>
 key-store-type: PKCS12
 key-store-password: [SERVICE]_KEY_PASS
 trust-store: <reference to truststore.p12>
 trust-store-type: PKCS12
 trust-store-password: <TRUSTSTORE_PASS>

Configuring Golang services

server:
 ssl:
 enabled: true
 certFile: <reference to [service].crt>
 keyFile: <reference to [service].key if not included in the certFile's PEM>
 keyPassword: <[GOSERVICE]_KEY_PASS if required>
 cacertFile: <reference to ca.pem>
 # Roll out with "want" initially
 clientAuth: need

http:
 cacertFile: <reference to ca.pem>
 clientCertFile: <reference to [service].crt>
 clientKeyFile: <reference to [service.key]>
 clientKeyPassword: <[GOSERVICE]_KEY_PASS if required>

Changing service endpoints

Use the Operator to change Spinnaker’s service endpoints.

Change the SpinnakerService custom resource:

kind: SpinnakerService
...
spec:
 spinnakerConfig:
 service-settings:
 clouddriver:
 baseUrl: https://spin-clouddriver.<NAMESPACE>:7002
 dinghy:
 baseUrl: https://spin-dinghy.<NAMESPACE>:8081
 echo:
 baseUrl: https://spin-echo.<NAMESPACE>:8089
 fiat:
 baseUrl: https://spin-fiat.<NAMESPACE>:7003
 front50:
 baseUrl: https://spin-front50.<NAMESPACE>:8080
 gate:
 baseUrl: https://spin-gate.<NAMESPACE>:8084
 kayenta:
 baseUrl: https://spin-kayenta.<NAMESPACE>:8090
 orca:
 baseUrl: https://spin-orca.<NAMESPACE>:8083
 igor:
 baseUrl: https://spin-igor.<NAMESPACE>:8088
 rosco:
 baseUrl: https://spin-rosco.<NAMESPACE>:8087
 terraformer:
 baseUrl: https://spin-terraformer.<NAMESPACE>:7088

Changing readiness probe

Change the readiness probe used by Kubernetes from an HTTP request to a TCP probe.

Add the following snippet to each service in SpinnakerService manifest:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 service-settings:
 <service>:
 kubernetes:
 useTcpProbe: true

Deployment

Apply your changes to your Spinnaker deployment:

kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest>

If Spinnaker services are already using HTTPS, you can roll out mTLS without interruption by making the client certificate optional (want) in server.ssl.client-auth (Java) and server.ssl.clientAuth. Then once all the services are stable, rolling out a new configuration with that value set to need.

Continuous-Deployment: Configure Auth for Spinnaker Using Okta SAML

Mon, 01 Jan 0001 00:00:00 +0000

Configure a Spinnaker application in Okta

Select Applications > Applications from the top menu.

Click the Add Application button.

Click the Create New App button.

In the dialog Create a New Application Integration, select the following values:

Platform > Web
Sign on method > SAML 2.0

Then click the Create button.

On the Create SAML Integration screen, enter an app name and click the Next button.

On the Configure SAML screen, configure the following settings:

Single sign on URL: Enter the URL for your Gate service, with the path /saml/SSO. For example, https://oktaspinnaker.spinnaker.armory.io:8084/saml/SSO
Audience URI (SP Entity ID): Enter a unique entity id. For example, io.armory.spinnaker.oktatest
Name ID format: For example, “EmailAddress”
Application username: For example, “Email”

In the GROUP ATTRIBUTE STATEMENTS section:

Name = memberOf, Name format = Unspecified, Filter = Regex: .*

Then, click the Next button.

On the Create SAML Integration screen, select I’m an Okta customer adding an internal app and then click the Finish button.

This takes you to the Sign On screen of the application you just created.

Click the View Setup Instructions button. This displays the page with information necessary to configure Spinnaker.

In the Optional section, copy the contents of IDP metadata and save to file. For example, under /Users/armory/.hal/saml/metadata.xml.

Configure Spinnaker to use Okta

1: Create a SAML keystore file

Make sure the Java version used to generate the keystore is the same version used to run Armory Continuous Deployment or Spinnaker.

Generate a keystore and key:

KEYSTORE_PATH=/Users/armory/.hal/saml/saml.jks
keytool -genkey -v -keystore KEYSTORE_PATH -alias saml -keyalg RSA -keysize 2048 -validity 10000 -storetype JKS

2: Configure Spinnaker to use SAML

The value you enter for issuerId must match the value entered in Audience URI (SP Entity ID) when configuring the app in Okta

Add the following snippet to SpinnakerService manifest. This references secrets stored in a Kubernetes secrets in the same namespace as Spinnaker, but secrets can be stored in any of the supported secret engines:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 security:
 authn:
 saml:
 enabled: true
 keyStore: encryptedFile:k8s!n:spin-secrets!k:saml.jks
 keyStoreAliasName: saml
 keyStorePassword: encrypted:k8s!n:spin-secrets!k:keystorePassword
 metadataLocal: encryptedFile:k8s!n:spin-secrets!k:metadata.xml
 issuerId: io.armory.spinnaker.oktatest # The identity of the Spinnaker application registered with the SAML provider.
 serviceAddress: https://<gate-URL> # The address of the Gate server that will be accesible by the SAML identity provider. This should be the full URL, including port, e.g. https://gate.org.com:8084/. If deployed behind a load balancer, this would be the laod balancer's address.

Create the Kubernetes secret holding the Spinnaker secrets:

kubectl -n <spinnaker namespace> create secret generic spin-secrets \
 --from-file=saml.jks \
 --from-file=metadata.xml \
 --from-literal=keystorePassword=<password-entered-in-step-1>

Apply your changes to the SpinnakerService manifest:

kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest>

Troubleshooting

Make sure the DNS is correctly pointing to the load balancers of gate-URL and deck-URL.

Verify that the gate-URL is the one entered in Okta with :8084/saml/SSO appended to it.

Validate that the service-address-url in your configuration is the gate-URL.

Continuous-Deployment: Configure Spinnaker's Orca Service to Use SQL RDBMS

Mon, 01 Jan 0001 00:00:00 +0000

Advantages to using an RDMS with Orca

By default, Spinnaker’s task orchestration service, Orca, uses Redis as its backing store. You can configure Orca to use a relational database instead of Redis to store its pipeline execution. The main advantage of doing so is a gain in performance and the removal of Redis as a single point of failure.

Armory recommends MySQL 5.7. For AWS, you can use Aurora.

Base configuration

You can find a complete description of configuration options in the Open Source Spinnaker documentation.

You can configure your SQL database by adding the following snippet to SpinnakerService manifest under spec.spinnakerConfig.profiles.orca:

sql:
 enabled: true
 connectionPools:
 default:
 default: true
 jdbcUrl: jdbc:mysql://<DB CONNECTION HOSTNAME>:<DB CONNECTION PORT>/<DATABASE NAME>
 user: orca_service
 password: <orca_service password>
 connectionTimeoutMs: 5000
 maxLifetimeMs: 30000
 maxPoolSize: 50
 migration:
 jdbcUrl: jdbc:mysql://<DB CONNECTION HOSTNAME>:<DB CONNECTION PORT>/<DATABASE NAME>
 user: orca_migrate
 password: <orca_migrate password>

# Ensure we're only using SQL for accessing execution state
executionRepository:
 sql:
 enabled: true
 redis:
 enabled: false

monitor:
 activeExecutions:
 redis: false

Depending on your TLS settings, you may need to configure TLS 1.2. For more information, see the Knowledge Base articles Disabling TLS 1.1 in Spinnaker and Specifying the Protocols to be used and How to fix TLS error “Reason: extension (5) should not be presented in certificate_request”.

Create the `orca` database and configure authorization

Once you’ve provisioned your RDBMS and ensured connectivity from Spinnaker, you need to create the database. You can skip this step if you created the database during provisioning.

CREATE SCHEMA `orca` DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Grant authorization to the orca_service and orca_migrate users:


 GRANT
 SELECT, INSERT, UPDATE, DELETE, EXECUTE, SHOW VIEW
 ON `orca`.*
 TO 'orca_service'@'%';

 GRANT
 SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, REFERENCES, INDEX, ALTER, LOCK TABLES, EXECUTE, SHOW VIEW
 ON `orca`.*
 TO 'orca_migrate'@'%';

The above configuration grants authorization from any host. You can restrict it to the cluster in which Spinnaker runs by replacing the % with the IP address of Orca pods from MySQL.

Migrate from Redis to SQL to keep existing execution history

The above configuration will point Orca to your database. You have the option to run a dual repository by adding dual in profiles/orca-local.yml.

Armory v2.18+:

executionRepository:
 dual:
 enabled: true
 primaryName: sqlExecutionRepository
 previousName: redisExecutionRepository
 sql:
 enabled: true
 redis:
 enabled: true

Armory versions prior to v2.18:

executionRepository:
 dual:
 enabled: true
 primaryClass: com.netflix.spinnaker.orca.sql.pipeline.persistence.SqlExecutionRepository
 previousClass: com.netflix.spinnaker.orca.pipeline.persistence.jedis.RedisExecutionRepository

 sql:
 enabled: true
 redis:
 enabled: true

However, this configuration won’t migrate your existing execution history to your new database. This will make your Spinnaker instance run on both the SQL and Redis backend. Spinnaker will only write the new execution on SQL but will continue to read the data on Redis. To migrate the data from Redis to SQL, you need to add the following

pollers:
 orchestrationMigrator:
 enabled: true
 intervalMs: 1800000
 pipelineMigrator:
 enabled: true
 intervalMs: 1800000 # After how much time the migration process is going to start

Once everything has been migrated (you will see logs in the orca pod about the migration process) you can remove these settings.

Database maintenance

Each new version of Orca may potentially migrate the database schema. This is done with the orca_migrate user defined above.

Pipeline executions are saved to the database. Each execution can add between a few KBs to hundreds of KBs of data depending on the size of your pipeline. It means that after a while, data will grow large and you’ll likely want to purge older executions.

Note: We recommend saving past executions to a different data store for auditing purposes. You can do it in a variety of ways:

During the purge, by marking, exporting, then deleting older records.
By saving execution history from Echo’s events and just delete older records from your database.

Continuous-Deployment: Configure Slack Notifications in Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Create a Slack application

Go to the Apps Management URL and click on the “Create New App” button. Once done, you get access to the basic configuration pane. You might want to customize some settings, like the color or the logo of your application at the bottom of it.

Create a bot

Create a bot after your Slack application has been created. Next, select the “Add features and functionality” menu and then select “Bots”.

Enter the following fields:

The Bot Display Name
The Bot Username
The “Always Show My Bot as Online” to On

Deploy the bot

Select the “OAuth & Permissions” menu, copy the Bot User OAuth Access Token this is the token needed in Spinnaker configuration, also on the Bot Token Scopes section add a scope of type chat:write like in the screenshot below:

Select the “Install your app to your workspace” from the Bot “Basic Information” page and deploy it.

Invite the bot to a channel

Spinnaker only requires publishing on a channel to interact with Slack. All you have to do is connect to a channel or create a new channel and name the bot you’ve just created. Slack proposes to invite the bot. Accept the invitation.

Register the Slack token with Spinnaker

You are now ready to configure Spinnaker with the bot you’ve just registered.

Add the following snippet to the SpinnakerService manifest:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 notifications:
 slack:
 enabled: true
 botName: spinnaker # The name of your slack bot.
 token: xoxb-xxxxxxxx-xxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxx # Your slack bot token. This field supports "encrypted" secret references

Apply the changes:

kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest>

Test Spinnaker

You should then make sure Spinnaker can send the notifications as expected. You can configure a notification within a channel you have invited your bot in and test by running a test pipeline. See example below:

Continuous-Deployment: Configure TLS for Spinnaker Services

Mon, 01 Jan 0001 00:00:00 +0000

Switching from plain HTTP to HTTPS will cause some short disruption to the services as they become healthy at different times.

Overview of TLS

When a client attempts to communicate with a server over SSL:

the server must present a certificate to the user.
the client must validate that certificate by checking it against its known valid certificate authorities (CA).

To properly set up TLS between services, we need to provide each service with:

a certificate signed by a CA
the CA certificate to verify these certificates

Note that distributing a CA public key is only needed if you sign certificates with a CA that is not bundled in most systems. In this document, we assume that you are using a self-signed CA.

Java

Java services can present #1 as a keystore and #2 as a trust store in PKCS12 (preferred) or JKS format.

Golang

Golang services need a X509 certificate (PEM format) and a private key for #1 as well as the X509 certificate of the CA for #2.

What you need

The following table lists the Armory and Spinnaker services, their type (Java or Golang), and which certificates they need:

Service	Type	Server	Client
Clouddriver	Java	Yes	Yes
Deck	N/A	-	-
Dinghy*	Golang	Yes	Yes
Echo	Java	Yes	Yes
Fiat	Java	Yes	Yes
Front50	Java	Yes	Yes
Gate	Java	Maybe	Yes
Kayenta	Java	Yes	Yes
Igor	Java	Yes	Yes
Orca	Java	Yes	Yes
Rosco	Java	Yes	Yes
Terraformer*	Golang	Yes	Yes

Dinghy is the service for Pipelines-as-Code.
Terraformer is the service for the Armory Terraform Integration.

Note: Gate may be handled differently if you already terminating SSL at Gate. If not, make sure the load balancer and ingress you are using supports self-signed certificates.

In the following sections, you need to have the following information available:

ca.pem (all Golang servers): the CA certificate in PEM format
[service].crt (each Golang server): the certificate and (optionally) the private key of the Golang server in PEM format
[service].key (each Golang server): the private key of the Golang server if not bundled with the certificate you’re using
[GOSERVICE]_KEY_PASS (each Golang server): the password to the private key of the server
truststore.p12 (all Java clients): a PKCS12 truststore with CA certificate imported
TRUSTSTORE_PASS (all Java clients): the password to the truststore you’re using
[service].p12 (each Java server): a PKCS12 keystore containing the certificate and private key of the server
[SERVICE]_KEY_PASS (each Java server): the password to the keystore you’re using

The server certificate will serve as its client certificate to other services. You can generate different certificates and use them in ok-http-client.key-store* (Java) and http.key* (Golang).

To learn how to generate these files, see the Generate Certificates for Spinnaker guide.

Configuring Java services

Add the following to each Java service profile under profiles in the SpinnakerService’s profiles:

# Only needed for "server" role
server:
 ssl:
 enabled: true
 key-store: <reference to [service].p12>
 key-store-type: PKCS12
 key-store-password: <[SERVICE]_KEY_PASS>

# Needed for all Java services
ok-http-client:
 key-store: <reference to truststore.p12>
 key-store-type: PKCS12
 key-store-password: <TRUSTSTORE_PASS>
 trust-store: <reference to truststore.p12>
 trust-store-type: PKCS12
 trust-store-password: <TRUSTSTORE_PASS>

Currently, ok-http-client.key-store is required even though it is not used in a simple TLS setup.

Configuring Golang services

server:
 ssl:
 enabled: true
 certFile: <reference to [service].crt>
 keyFile: <reference to [service].key if not included in the certFile's PEM>
 keyPassword: <[GOSERVICE]_KEY_PASS if required>

http:
 cacertFile: <reference to ca.pem>

Changing service endpoints

Use the Operator to change Spinnaker’s service endpoints.

Change the SpinnakerService custom resource:

kind: SpinnakerService
...
spec:
 spinnakerConfig:
 service-settings:
 clouddriver:
 baseUrl: https://spin-clouddriver.<NAMESPACE>:7002
 dinghy:
 baseUrl: https://spin-dinghy.<NAMESPACE>:8081
 echo:
 baseUrl: https://spin-echo.<NAMESPACE>:8089
 fiat:
 baseUrl: https://spin-fiat.<NAMESPACE>:7003
 front50:
 baseUrl: https://spin-front50.<NAMESPACE>:8080
 gate:
 baseUrl: https://spin-gate.<NAMESPACE>:8084
 kayenta:
 baseUrl: https://spin-kayenta.<NAMESPACE>:8090
 orca:
 baseUrl: https://spin-orca.<NAMESPACE>:8083
 igor:
 baseUrl: https://spin-igor.<NAMESPACE>:8088
 rosco:
 baseUrl: https://spin-rosco.<NAMESPACE>:8087
 terraformer:
 baseUrl: https://spin-terraformer.<NAMESPACE>:7088

Deploying Spinnaker

After you finish modifying the service YAML files, run kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest> to apply your changes to your Spinnaker deployment.

Switching from plain HTTP to HTTPS will cause some short disruption to the services as they become healthy at different times.

Providing certificates and passwords to services

Secret engines

You can store secrets (and non secrets) in supported secret stores as well as in Kubernetes secrets if using the Armory Operator. This is the simplest route.

For instance, assuming all the information is stored in a bucket named mybucket on s3 that all services have access to, SpinnakerService manifest might look like:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 profiles:
 echo:
 server:
 ssl:
 enabled: true
 key-store: encryptedFile:s3!b:mybucket!r:us-west-2!f:echo.jks
 key-store-type: JKS
 key-alias: echo
 key-store-password: encrypted:s3!b:mybucket!r:us-west-2!f:passwords.yml!k:echo.keyPassword

 ok-http-client:
 key-store: encryptedFile:s3!b:mybucket!r:us-west-2!f:truststore.jks
 key-store-type: JKS
 key-store-password: encrypted:s3!b:mybucket!r:us-west-2!f:passwords.yml!k:truststorePassword
 trust-store: encryptedFile:s3!b:mybucket!r:us-west-2!f:truststore.jks
 trust-store-type: JKS
 trust-store-password: encrypted:s3!b:mybucket!r:us-west-2!f:passwords.yml!k:truststorePassword

Run kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest> after you make your changes.

Manually providing information

An alternative if you cannot use one of the supported secret engine is to store the information in Kubernetes secrets and manually provide the information. Files are added through a volumeMount and passwords through an environment variable.

For instance, assuming mysecrets Kubernetes Secret is available in the same namespace as Spinnaker with the following keys:

data:
 echo.jks: <base64 encoded keystore>

We’ll now provide the following in service-settings/echo.yml:

kubernetes:
 volume:
 - name: secretvol
 mountPath: /var/mysecrets

And a reference would then be:

server:
 ssl:
 key-store: /var/mysecrets/echo.jks

Apply your changes.

Continuous-Deployment: Connect Docker Registries

Mon, 01 Jan 0001 00:00:00 +0000

Overview of connecting Spinnaker to Docker registries

Many of the commands below have additional options that may be useful, or possibly required.

Enable Docker registries in Spinnaker

If you’ve just installed Armory or Open Source Spinnaker^TM, you need to enable Docker registry providers.

Add the following snippet to SpinnakerService manifest:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 providers:
 dockerRegistry:
 enabled: true

Add a Docker registry and repositories to Spinnaker

To add a new registry, you use some variation of the following configuration. This example uses a public Docker Hub registry (armory/demoapp) and actually would not use the username or password options, since the registry is public. In most cases, you’ll be configuring a private registry and the authentication credentials will be required, so the options are shown here as an example.

Add the following snippet to SpinnakerService manifest:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 providers:
 dockerRegistry:
 enabled: true
 primaryAccount: my-docker-registry # Account with search priority. (Required when using a locally deployed registry.)
 accounts:
 - name: my-docker-registry
 requiredGroupMembership: [] # A user must be a member of at least one specified group in order to make changes to this account's cloud resources.
 providerVersion: V1
 permissions: {}
 address: https://index.docker.io # The registry address you want to pull and deploy images from. For example: index.docker.io - DockerHub quay.io - Quay gcr.io - Google Container Registry (GCR) [us|eu|asia].gcr.io - Regional GCR localhost - Locally deployed registry
 username: yourusername # Your docker registry username
 password: abc # Your docker registry password. This field support "encrypted" secret references.
 email: fake.email@spinnaker.io # Your docker registry email (often this only needs to be well-formed, rather than be a real address)
 cacheIntervalSeconds: 30 # How many seconds elapse between polling your docker registry. Certain registries are sensitive to over-polling, and larger intervals (e.g. 10 minutes = 600 seconds) are desirable if you're seeing rate limiting.
 clientTimeoutMillis: 60000 # Timeout time in milliseconds for this repository.
 cacheThreads: 1 # How many threads to cache all provided repos on. Really only useful if you have a ton of repos.
 paginateSize: 100 # Paginate size for the docker repository _catalog endpoint.
 sortTagsByDate: false # Sort tags by creation date.
 trackDigests: false # Track digest changes. This is not recommended as it consumes a high QPM, and most registries are flaky.
 insecureRegistry: false # Treat the docker registry as insecure (don't validate the ssl cert).
 # repositories: # An optional list of repositories to cache images from. If not provided, Spinnaker will attempt to read accessible repositories from the registries _catalog endpoint
 # repositoriesRegex: <regexForYourRepos> # Optional regular expression that specifies what repositories Clouddriver caches images from. This is useful if you add repos frequently. Any new repo that matches the regex gets cached automatically.
 - library/nginx
 # passwordFile: docker-pass # The path to a file containing your docker password in plaintext (not docker/config.json file). This field support "encryptedFile" secret references.
 # passwordCommand: abc # Command to retrieve docker token/password, commands must be available in environment
 # environment: dev # The environment name for the account. Many accounts can share the same environmen(e.g. dev, test, prod)

Some registries, like Docker Hub, require you to identify the repositories explicitly, like above. Some do not (such as the Google Container Registry). You can read more in the Spinnaker docs.

Amazon’s ECR requires additional configuration to work properly with Spinnaker. See the Connect Spinnaker to Amazon Elastic Container Registry guide for details.

Continuous-Deployment: Connect Spinnaker to Jenkins

Mon, 01 Jan 0001 00:00:00 +0000

Consult the Spinnaker documentation’s Jenkins page for an in-depth guide.

Enable Jenkins in Spinnaker

Add the following snippet to your SpinnakerService manifest:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 ci:
 jenkins:
 enabled: true

Create a User API Token in Jenkins

Spinnaker uses your Jenkins username and API token for authentication.

Log into Jenkins
Click on your username (in the top right)
Click on “Configure” (on the left)
Under the “API Token” section, click on “Add new Token”, and “Generate” and record the generated token
Record your username; this is the value in the current page URL between ‘user’ and ‘configure’ (http:///user//configure)

Add a Jenkins Master

You can add as many Jenkins masters as needed. Once the master is configured properly, Spinnaker will use the credentials provided to query for all available jobs and display them in the UI for triggers and stages.

Add the Jenkins master to Spinnaker:

Add the following snippet to your SpinnakerService manifest:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 ci:
 jenkins:
 enabled: true
 masters:
 - name: <jenkins-master-name>
 address: https://<jenkins-url>/ # The address your jenkins master is reachable at.
 username: <jenkins-username> # The username of the jenkins user to authenticate as.
 password: abc # The password of the jenkins user to authenticate as. This field support "encrypted" secret references.

Don’t forget to apply your changes:

kubectl -n >spinnaker namespace> apply -f <SpinnakerService manifest>

Troubleshooting Authentication / Connectivity

Igor is the service that interacts with Jenkins. You can test Spinnaker-Jenkins connectivity using curl from another pod. The Deck or Clouddriver pod is a good option since curl is already installed there.

curl https://<jenkins-url>/api/json --user <jenkins-username>:<jenkins-api-token>

This returns a JSON list of jobs.

For example:

# Exec into the Clouddriver container:
kubectl exec -it spin-clouddriver-6cf45f4db-lkg7t bash

bash-4.4$ curl https://jenkins.domain.com/api/json --user justin:1234567890abcdefghijklmnopqrstuvwx
{"_class":"hudson.model.Hudson","assignedLabels":[{"name":"master"}], [...] }

Continuous-Deployment: Connect GitHub as an Artifact Source in Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Configure a GitHub trigger in Spinnaker

Spinnaker^TM pipelines can be configured to trigger when a change is committed to a GitHub repository. This doesn’t require any configuration of Spinnaker other than adding a GitHub trigger but does require administration of the GitHub repositories to configure the webhook.

The open source Spinnaker documentation has concise instructions for configuring GitHub webhooks.

Configure GitHub as an artifact source

If you actually want to use a file from the GitHub commit in your pipeline, you’ll need to configure GitHub as an artifact source in Spinnaker.

Many of the commands below have additional options that may be useful (or possibly required).

Enable GitHub artifacts

If you haven’t done this yet (for example, if you’ve just installed Armory Spinnaker fresh), you need to enable GitHub as an artifact source:

Add the following snippet to SpinnakerService manifest:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 features:
 artifacts: true
 artifacts:
 github:
 enabled: true

Add a GitHub credential

To access private GitHub repositories, you need a GitHu bPersonal Access Token. See the GitHub docs for instructions. The token needs the repo scope.

Once you have a token, you should provide that token for Spinnaker’s Igor service as a credential to use to access GitHub.

Replace the account name github_user with the string you want to use to identify this GitHub credential.

Add the following snippet to SpinnakerService manifest:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 features:
 artifacts: true
 artifacts:
 github:
 enabled: true
 accounts:
 - name: github_user
 token: abc # GitHub's personal access token. This fields supports `encrypted` references to secrets.
 # username: abc # GitHub username
 # password: abc # GitHub password. This fields supports `encryptedreferences` to secrets.
 # usernamePasswordFile: creds.txt # File containing "username:password" to use for GitHub authentication. This fields supports `encryptedFilereferences` to secrets.
 # tokenFile: token.txt # File containing a GitHub authentication token. This fields supports `encryptedFile` references to secrets.

If you have a GitHub Personal Access Token, you only need that to authenticate against GitHub, but there are other authentication options like username/password, or specifying credentials in a file entry.

Apply your changes:

kubectl -n >spinnaker namespace> apply -f <SpinnakerService manifest>

Using the GitHub credential

You may note that the above GitHub “account” doesn’t actually have an endpoint for your GitHub. This account is basically just the credential used by Spinnaker artifacts to access GitHub. The actual GitHub API endpoint is specified in the artifact reference. The following is an example of how to use this credential.

Pulling a Kubernetes Manifest from Github

Under Expected Artifacts in your pipeline, create an artifact of type GitHub.
Specify the file path as the path within the repository to your file. For example, if your manifest is at demo/manifests/deployment.yml in the Github repository orgname/reponame , specify demo/manifests/deployment.yml.
Check the Use Default Artifact checkbox.
In the Content URL, provide the full path to the API URI for your manifest. Here are some examples of this:
- If you’re using SaaS GitHub, the URI is generally formatted like this: https://api.github.com/repos/<ORG>/<REPO>/contents/<PATH-TO-FILE>.
  - For example: https://api.github.com/repos/armory/demo/contents/manifests/deployment.yml
- If you have on-prem Github Enterprise, then the URI may be formatted like this: https://<GITHUB_URL>/api/v3/repos/<ORG>/<REPO>/contents/<PATH-TO-FILE>.
  - For example: http://github.customername.com/api/v3/repos/armory/spinnaker-pipelines/contents/manifests/deployment.yml
Create a Deploy (Manifest) stage. Rather than specifying the manifest directly in the UI, under the Manifest Source specify Artifact, and in the Expected Artifact field, select the artifact you created above.
If you have multiple Github Accounts (credentials) added to your Spinnaker cluster, there should be a dropdown to select which one to use.

Troubleshooting credentials and URIs

To verify that your token and URI are correct, you can run a curl command to test authentication (the user field doesn’t matter):

curl https://api.github.com/repos/armory/demo/contents/manifests/deployment.yml \
 -u nobody:abcdef0123456789abcdef0123456789abcdef01

If you receive metadata about your file, the credential and URI are correct.

Continuous-Deployment: Create Kubernetes Service Accounts and Kubeconfigs

Mon, 01 Jan 0001 00:00:00 +0000

What Spinnaker needs to connect to Kubernetes

When connecting Spinnaker to Kubernetes, Spinnaker needs the following:

A service account in the relevant Kubernetes cluster (or namespace in a cluster). In Kubernetes, a service account exists in a given namespace but may have access to other namespaces or to the whole cluster
Permissions for the service account to create/read/update/delete objects in the relevant Kubernetes cluster (or namespace)
A kubeconfig that has access to the service account through a token or some other authentication method.

The spinnaker-tools binary creates all of the above objects. See Add a Kubernetes Account Deployment Target in Spinnaker if you want to use the spinnaker-tools binary. Otherwise, if you want to create these objects manually or need to know what is going on, use this document.

Attention

This document primarily uses kubectl and assumes you have access to permissions that can create and/or update these resources in your Kubernetes cluster:

Kubernetes Service Account(s)
Kubernetes Roles and Rolebindings
(Optionally) Kubernetes ClusterRoles and Rolebindings

Create the Kubernetes Service Account

You can use the following manifest to create a service account. Replace NAMESPACE with the namespace you want to use and, optionally, rename the service account.

# spinnaker-service-account.yml
apiVersion: v1
kind: ServiceAccount
metadata:
 name: spinnaker-service-account
 namespace: NAMESPACE

Then create the object:

kubectl apply -f spinnaker-service-account.yml

Grant `cluster-admin` permissions

Do this only if you want to grant the service account access to all namespaces in your cluster. A Kubernetes ClusterRoleBinding exists at the cluster level, but the subject of the ClusterRoleBinding exists in a single namespace. Again, you must specify the namespace where your service account lives.

# spinnaker-clusterrolebinding.yml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
 name: spinnaker-admin
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: cluster-admin
subjects:
- kind: ServiceAccount
 name: spinnaker-service-account
 namespace: NAMESPACE

Then, create the binding:

kubectl apply -f spinnaker-clusterrolebinding.yml

Grant namespace-specific permissions

If you only want the service account to access specific namespaces, you can create a role with a set of permissions and rolebinding to attach the role to the service account. You can do this multiple times. Additionally, you will have to explicitly do this for the namespace where the service account is, as it is not implicit.

Important points:

A Kubernetes Role exists in a given namespace and grants access to items in that namespace
A Kubernetes RoleBinding exists in a given namespace and attaches a role in that namespace to some principal (in this case, a service account). The principal (service account) may be in another namespace.
If you have a service account in namespace source and want to grant access to namespace target, then do the following:
- Create the service account in namespace source
- Create a Role in namespace target
- Create a RoleBinding in namespace target, with the following properties:
  - RoleRef pointing to the Role (that is in the same namespace target)
  - Subject pointing to the service account and namespace where the service account lives (in namespace source)

Change the names of resources to match your environment, as long as the namespaces are correct, and the subject name and namespace match the name and namespace of your service account.

# spinnaker-role-and-rolebinding-target.yml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: spinnaker-role
 namespace: target # Should be namespace you are granting access to
rules:
- apiGroups: ["*"]
 resources: ["*"]
 verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 name: spinnaker-rolebinding
 namespace: target # Should be namespace you are granting access to
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: spinnaker-role # Should match name of Role
subjects:
- namespace: source # Should match namespace where SA lives
 kind: ServiceAccount
 name: spinnaker-service-account # Should match service account name, above

Then, create the object:

kubectl apply -f spinnaker-role-and-rolebinding-target.yml

Get the service account and token

Run these commands (or commands like these) to get the token for your service account and create a kubeconfig with access to the service account.

This file will contain credentials for your Kubernetes cluster and should be stored securely.

# Update these to match your environment
SERVICE_ACCOUNT_NAME=spinnaker-service-account
CONTEXT=$(kubectl config current-context)
NAMESPACE=spinnaker

NEW_CONTEXT=spinnaker
KUBECONFIG_FILE="kubeconfig-sa"


SECRET_NAME=$(kubectl get serviceaccount ${SERVICE_ACCOUNT_NAME} \
 --context ${CONTEXT} \
 --namespace ${NAMESPACE} \
 -o jsonpath='{.secrets[0].name}')
TOKEN_DATA=$(kubectl get secret ${SECRET_NAME} \
 --context ${CONTEXT} \
 --namespace ${NAMESPACE} \
 -o jsonpath='{.data.token}')

TOKEN=$(echo ${TOKEN_DATA} | base64 -d)

# Create dedicated kubeconfig
# Create a full copy
kubectl config view --raw > ${KUBECONFIG_FILE}.full.tmp
# Switch working context to correct context
kubectl --kubeconfig ${KUBECONFIG_FILE}.full.tmp config use-context ${CONTEXT}
# Minify
kubectl --kubeconfig ${KUBECONFIG_FILE}.full.tmp \
 config view --flatten --minify > ${KUBECONFIG_FILE}.tmp
# Rename context
kubectl config --kubeconfig ${KUBECONFIG_FILE}.tmp \
 rename-context ${CONTEXT} ${NEW_CONTEXT}
# Create token user
kubectl config --kubeconfig ${KUBECONFIG_FILE}.tmp \
 set-credentials ${CONTEXT}-${NAMESPACE}-token-user \
 --token ${TOKEN}
# Set context to use token user
kubectl config --kubeconfig ${KUBECONFIG_FILE}.tmp \
 set-context ${NEW_CONTEXT} --user ${CONTEXT}-${NAMESPACE}-token-user
# Set context to correct namespace
kubectl config --kubeconfig ${KUBECONFIG_FILE}.tmp \
 set-context ${NEW_CONTEXT} --namespace ${NAMESPACE}
# Flatten/minify kubeconfig
kubectl config --kubeconfig ${KUBECONFIG_FILE}.tmp \
 view --flatten --minify > ${KUBECONFIG_FILE}
# Remove tmp
rm ${KUBECONFIG_FILE}.full.tmp
rm ${KUBECONFIG_FILE}.tmp

You should end up with a kubeconfig that can access your Kubernetes cluster with the desired target namespaces.

Continuous-Deployment: DNS and SSL

Mon, 01 Jan 0001 00:00:00 +0000

Overview

In order to use Spinnaker in your organization, you’re going to want to configure your infrastructure so that users can access Spinnaker. This has several steps:

Expose the Spinnaker endpoints (Deck and Gate)
Configure TLS encryption for the exposed endpoints
Create DNS entries for your endpoints
Update Spinnaker so that it’s aware of its new endpoints.

Expose the Spinnaker endpoints

Spinnaker users need access to two endpoints within Spinnaker

Deck (the Spinnaker UI microservice), which listens on port 9000
Gate (the spinnaker API microservice), which listens on port 8084

There are a number of ways to expose these endpoints, and your configuration of these will be heavily dependent on the Kubernetes environment where Spinnaker is installed. Several common options are as follows:

Set up an ALB ingress controller within your Kubernetes environment, and add an ingress for the spin-deck and spin-gate services.
Set up an nginx ingress controller within your Kubernetes environment, and add an ingress for the spin-deck and spin-gate services.
Create Kubernetes loadbalancer services for both the spin-deck and spin-gate Kubernetes deployments

Configure TLS encryption for the exposed endpoints

You should encrypt the exposed Spinnaker endpoints. There are three high-level ways of achieving this:

Most common: Terminate TLS on the load balancer(s) in front of the endpoints, and allow HTTP traffic between the load balancer and the endpoint backends.
Terminate TLS on the load balancer(s) in front of the endpoints, and configure the load balancer and endpoint backends with TLS between them, as well.
Least common: Configure your load balancer(s) to support the SNI so that the load balancer passes the initial TLS connection to the backends.

There are a number of ways to achieve all of these - you can work with your Kubernetes, security, and networking teams to determine which methods best meet your organization’s needs.

If you need to terminate TLS on the backend containers (the second or third options), review the Open Source Spinnaker documentation regarding configuring TLS certificates on the backend microservices: (Setup/Security/SSL)[https://spinnaker.io/setup/security/ssl/].

Create a DNS Entry for your load balancer

Add a DNS Entry to your DNS management system. You should only need to add a DNS entry for the user-facing ALB, ELB, or other load balancer which is what you use to currently access Spinnaker. It typically has a name such as the one below

armoryspinnaker-prod-external-123456789.us-west-1.elb.amazonaws.com

Add a CNAME entry for the given ELB to create a simple name you will use to access your instance of Spinnaker, e.g. spinnaker.armory.io.

Update Spinnaker configuration

Update the endpoints for Spinnaker Deck (the Spinnaker UI microservice) and Spinnaker Gate (the Spinnaker API microservice)

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 security:
 apiSecurity:
 overrideBaseUrl: https://spinnaker-gate.mydomain.com
 uiSecurity:
 overrideBaseUrl: https://spinnaker.mydomain.com

Don’t forget to apply your changes:

kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest>

Continuous-Deployment: Enable Pipelines-as-Code in Armory Continuous Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Continuous-Deployment: Enable the Evaluate Artifacts Stage

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Some artifact types and their associated stages, such as Kubernetes, support using SpEL to read parameters from the Spinnaker context and inject them into an artifacts manifest at deploy time. Other stages, such as the Terraform Integration stage, do not support this out of the box. The Evaluate Artifacts Stage plugin adds supports for evaluating SpEL against any artifact within your Spinnaker pipeline.

For information about how to use the stage, see Use the Evaluate Artifacts Stage.

Before you begin

If you are new to enabling plugins, read Spinnaker’s Plugin Users Guide to familiarize yourself with how plugins to Armory Continuous Deployment work.
The Evaluate Artifacts Stage requires Armory Continuous Deployment 2.24.x or later (Spinnaker 1.24.x or later)

Setup

Add the following snippet to your Spinnaker manifest, such as spinnakerservice.yml:

 spec:
 spinnakerConfig:
 profiles:
 spinnaker:
 spinnaker: # This second `spinnaker` is required
 extensibility:
 plugins:
 Armory.EvaluateArtifactsPlugin:
 enabled: true
 version: <PLUGIN_VERSION> # Replace with the version you want to use. For example, use 0.1.0.
 repositories:
 evaluateArtifacts:
 url: https://raw.githubusercontent.com/armory-plugins/evaluate-artifacts-releases/master/repositories.json
 gate:
 spinnaker:
 extensibility:
 deck-proxy:
 enabled: true
 plugins:
 Armory.EvaluateArtifactsPlugin:
 enabled: true
 version: <PLUGIN_VERSION> # Replace with the version you want to use. For example, use 0.1.0.

Keep the following in mind when using this configuration snippet:

Make sure to include the nested spinnaker parameter. Both are required because of how Armory Continuous Deployment consumes plugin configurations.
Replace <PLUGIN_VERSION> on lines 10 and 22 with the version of the plugin that you want to use. Plugin versions can be found here.

Then, deploy your updated Armory Continuous Deployment configuration using one of the following methods:

If you are using a single manifest file: kubectl -n <namespace> apply -f <path-to-manifest-file>
If you are using Kustomize patches like the ones in the Armory kustomize repo, you need to apply the kustomization. Depending on how you have Kustomize installed (either directly or as part of kubectl), use one of the following commands:

Kustomize is installed separately from kubectl:
```
# The namespace is declared in kustomization.yml
# Run this from the same directory as your kustomization.yml file
kustomize build | kubectl apply -f -
```
Kustomize is bundled with kubectl:
```
kubctl -n <namespace> apply -k <path-to-kustomize-directory>
```

Versions

0.1.1 - Update plugin to be compatible with Armory Continuous Deployment 2.27.0 and later
0.1.0 - Improved the user experience. Execution errors for the stage now display in the UI.
0.0.10 - Initial Release

What’s next

Use the Evaluate Artifacts Stage

Continuous-Deployment: Enable the Terraform Integration Stage in Armory Continuous Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Continuous-Deployment: Expose Spinnaker API Endpoints

Mon, 01 Jan 0001 00:00:00 +0000

Overview of setting up an X509 client certificate for Spinnaker

When you have third-party authentication set up for your Spinnaker^TM cluster, automating against the Spinnaker API can be slightly more difficult. One way to achieve this is to set up X509 client certificate authentication, which can optionally be enabled on a second port on Gate (which then must be exposed to clients). This document details one way to do this.

This document details the following:

Obtaining / Creating a CA certificate. If your organization has an internal (or other) CA, you can use the organization CA certificate (you only need the certificate in PEM format). All API clients must present a certificate signed by this CA; if you are creating a self-signed certificate, you can create client certs on your own. If you are using an organization CA, you must be able to request client certs from your organization CA.
Obtaining / Creating a certificate and private key for use by Deck (Spinnaker’s UI microservice). You can either request the certificate from your organization’s CA, or use the self-signed CA created above.
Obtaining / Creating a certificate and private key for use by Gate (Spinnaker’s UI microservice). You can either request the certificate from your organization’s CA, or use the self-signed CA created above.
Creating a JKS (Java Keystore) for Gate containing these two objects:
- The CA certificate imported as a trust store. Gate will use this to validate all inbound certificate-based clients. Specifically, this operates in this fashion:
  - Spinnaker only needs the certificate, not the private key. Generally speaking, the certificate is not a sensitive piece of information (it is what is publicly presented)
  - This CA certificate can either be an organization-wide CA certificate, or a CA certificate built specifically for Spinnaker
  - Gate (Spinnaker’s API layer) will trust clients that present a certificate signed by this certificate
- Gate’s server certificate and private key, which does not necessarily need to be a valid certificate if Gate is fronted by a load balancer
Enabling Deck to use the certificate generated for Deck
Enabling Gate to use the JKS
Configuring Gate to support X509 client certificate-based authentication, on a second port.

Exposed Spinnaker endpoints

At the end of this process, you will have three exposed endpoints for Spinnaker:

A UI endpoint on the Deck microservice. You can terminate TLS on the load balancer (Ingress or other load balancer), and end up with a flow that looks like this:
```
[Browser] ---HTTPS--> [Load Balancer with TLS Termination] ---HTTP--> [Deck:9000]
```
An API endpoint on the Gate microservice for browser clients. You can terminate TLS on the load balancer (Ingress or other load balancer). Authentication will be handled by your primary authentication provider (LDAP, SAML, OAuth2.0, etc.). You may end up with a flow that looks like this:
```
[Browser] ---HTTPS--> [Load Balancer with TLS Termination] ---HTTP--> [Gate:8084]
```
An API endpoint on the Gate microservice for automation clients. Clients must present an x509 client certificate in order to use this endpoint, so you cannot terminate TLS on a load balancer in front of Gate. The data flow may look something like this:
```
[API Client] ---HTTPS--> [TCP Load Balancer] ---HTTPS--> [Gate:8085]
```
or
```
[API Client] ---HTTPS--> [TLS Pass Through Load Balancer] ---HTTPS--> [Gate:8085]
```

Decision points

Before moving on, you should decide the following:

For the CA used to validate client certificates, are you using an organization CA or creating a self-signed CA?
For the Gate and Deck certificate, are you using certificates signed by an organization CA, certificates signed by a self-signed CA, or self-signed certificates?
In order to access the API endpoint, the API endpoint (which is an extra port on the existing Gate service) must be directly exposed. Clients must be able to provide their certificates to Gate; there should be no TLS termination in front of it. This means you must do one of the following:
- Set up a Layer 4 (TCP) load balancer in front of this endpoint
- Set up a Layer 7 (HTTPS) load balancer that supports TLS passthrough.

This document borrows heavily from the Open Source Spinnaker document on SSL, found here: Setup / Security / SSL.

Getting a self-signed CA certificate

If your organization has a CA that you can use to sign client certificates, then you must obtain a copy of the CA certificate (note: this is generally the public-facing certificate, and therefore not sensitive information). You must get this in a PEM-formatted file, or convert it to a PEM-formatted file.

Alternatively, you can create a self-signed CA certificate to use to sign client certificates, and configure Gate to trust certificates signed by this CA certificate.

Obtaining the organization certificate

If your organization has a CA certificate, you should obtain the CA certificate in PEM format. You do not need the private key, as long as your organization has a way to request client certificates signed by the CA.

You should have the following items:

ca.crt: a pem-formatted certificate. Spinnaker will trust client certificates that were signed by this CA.

Creating a self-signed CA certificate

If your organization has a CA certificate or you do not have a way to request client certificates signed by the CA, you can generate a self-signed CA certificate and private key. We will use this

We will use openssl to generate a Certificate Authority (CA) key and a server certificate. These instructions create a self-signed CA and key. You might want to use an external CA, to minimize browser configuration, but it’s not necessary (and can be expensive).

Use the steps below to create a Certificate Authority. (If you’re using an external CA, skip to the next section.)

It will produce the following items:

ca.key: a pem-formatted private key, which will have a pass phrase.
ca.crt: a pem-formatted certificate, which (with the private key) acts as a self-signed Certificate Authority.

Create the CA key, which will be encrypted with a passphrase. You can remove the -passout flag to have the command prompt for the passphrase.

CA_KEY_PASSWORD=SOME_PASSWORD_FOR_CA_KEY

openssl genrsa \
 -des3 \
 -out ca.key \
 -passout pass:${CA_KEY_PASSWORD} \
 4096

Self-sign the CA certificate. You can remove the -passin flag to have the command prompt for the passphrase. This should be the pass phrase used to encrypt ca.key.

CA_KEY_PASSWORD=SOME_PASSWORD_FOR_CA_KEY

openssl req \
 -new \
 -x509 \
 -days 365 \
 -key ca.key \
 -out ca.crt \
 -passin pass:${CA_KEY_PASSWORD}

Get a server certificate for the Deck (UI) service

This step is technically optional, but it requires setting up separate exposure mechanisms for Deck and Gate if this step is skipped. For simplicity’ sake, we recommend completing this step.

You need a server certificate and private key. If you have a load balancer in front of Deck that is terminating TLS, the certificate on Deck generally won’t matter aside from the fact that you must have one. There are two main options here:

Obtaining a server certificate and private key from your organization CA
Generating a server certificate and private key from the self-signed CA

Obtaining a Deck server certificate and private key from your organization CA

If your organization has a CA certificate, you should request a server certificate for use with Deck. This should result in the following files:

deck.key: a pem-formatted private key, which will have a pass phrase.
deck.crt: a pem-formatted x509 certificate, which matches the private key and was signed by your CA.

Generating a Deck server certificate and private key from the self-signed CA

If you want to generate your own certificates (for example, from the self-signed CA certificate created above), you can follow these steps:

Create a server key for Deck. Keep this file safe!

# This will be the passphrase used to encrypt the Deck private key
DECK_KEY_PASSWORD=SOME_PASSWORD_FOR_DECK_KEY

openssl genrsa \
 -des3 \
 -out deck.key \
 -passout pass:${DECK_KEY_PASSWORD} \
 4096

Generate a certificate signing request (CSR) for Deck. Specify localhost or Deck’s eventual fully-qualified domain name (FQDN) as the Common Name (CN).

# This should be the passphrase used to encrypt the Deck private key
DECK_KEY_PASSWORD=SOME_PASSWORD_FOR_DECK_KEY

openssl req \
 -new \
 -key deck.key \
 -out deck.csr \
 -passin pass:${DECK_KEY_PASSWORD}

Use the CA to sign the server’s request and create the Deck server certificate (in pem format). If using an external CA, they will do this for you.

# This should be the passphrase used to encrypt the self-signed CA private key
CA_KEY_PASSWORD=SOME_PASSWORD_FOR_CA_KEY

openssl x509 \
 -req \
 -days 365 \
 -in deck.csr \
 -CA ca.crt \
 -CAkey ca.key \
 -CAcreateserial \
 -out deck.crt \
 -passin pass:${CA_KEY_PASSWORD}

You should end up with these two files:
- deck.key: a pem-formatted private key, which will have a pass phrase
- deck.crt: a pem-formatted x509 certificate, which matches the private key and was signed by your CA

Get a server certificate for the Gate (API) Service

You will need a server certificate and private key. This server certificate will be used for both Gate endpoints - one for the authenticated API endpoint used by browser clients, one for the certificate-authenticated (X509) API endpoint used by automated clients.

There are two main options here:

Obtaining a server certificate and private key from your organization CA
Generating a server certificate and private key from the self-signed CA

Obtaining a server certificate and private key from your organization CA

If your organization has a CA certificate, you should request a server certificate for use with Gate.

Because the X509 API is exposed directly to your automation endpoints, it may be helpful to request a certificate with a CN that matches a DNS name that resolves to your load balancer. For example, if you can point api.spinnaker.domain.com at your load balancer, you may want to request a certificate that has api.spinnaker.domain.com as its CN.

Alternatively you can configure your clients to skip server certificate validation (this is usually the -k flag for curl).

This should result in the following files:

gate.key: a pem-formatted private key, which will have a pass phrase.
gate.crt: a pem-formatted x509 certificate, which matches the private key and was signed by your CA.

Generating a server certificate and private key from the self-signed CA

If you want to generate your own certificates (for example, from the self-signed CA certificate created above), you can follow these steps:

Create a server key for Gate. Keep this file safe!

This will prompt for a pass phrase to encrypt the key.

# This will be the passphrase used to encrypt the Gate private key
GATE_KEY_PASSWORD=SOME_PASSWORD_FOR_GATE_KEY

openssl genrsa \
 -des3 \
 -out gate.key \
 -passout pass:${GATE_KEY_PASSWORD} \
 4096

Generate a certificate signing request (CSR) for Gate. Ideally, specify Gate’s eventual fully-qualified domain name (FQDN) as the Common Name (CN).

This will prompt for the pass phrase for gate.key.

# This should be the passphrase used to encrypt the Gate private key
GATE_KEY_PASSWORD=SOME_PASSWORD_FOR_GATE_KEY

openssl req \
 -new \
 -key gate.key \
 -out gate.csr \
 -passin pass:${GATE_KEY_PASSWORD}

Use the CA to sign the server’s request and create the Gate server certificate in pem format. If using an external CA, they do this for you.

This prompts for the passphrase used to encrypt ca.key.
```
# This should be the passphrase used to encrypt the self-signed CA private key
CA_KEY_PASSWORD=SOME_PASSWORD_FOR_CA_KEY

openssl x509 \
 -sha256 \
 -req \
 -days 365 \
 -in gate.csr \
 -CA ca.crt \
 -CAkey ca.key \
 -CAcreateserial \
 -out gate.crt \
 -passin pass:${CA_KEY_PASSWORD}
```
Note: if you omit the -sha256 argument, openssl x509 creates a certificate that’s not valid to use with Gate. When you import the invalid certificate into a Java KeyStore (JKS), you see Warning: <gate> uses the SHA1withRSA signature algorithm which is considered a security risk. This algorithm will be disabled in a future update. Then Gate throws an Invalid keystore format when importing the JKS.
You should end up with these two files:
- gate.key: a pem-formatted private key, which will have a pass phrase
- gate.crt: a pem-formatted x509 certificate, which matches the private key and was signed by your CA

Checkpoint

At this point, you should have these files: * deck.key (with a distinct passphrase) * deck.crt * gate.key (with a distinct passphrase) * gate.crt * ca.crt

Also, if you are using a self-signed CA, you will also have these files: * ca.key (with a distinct passphrase) * ca.srl (automatically generated, used to keep track of certificates generated by a given CA)

We will use all of these below.

Create a JKS

Gate expects all certificates in JKS (Java KeyStore) format, so we have to do some conversion and combination. Specifically, we will generate a JKS that has these two items in it:

An entry containing the Gate private key and public certificate
An entry containing the CA public certificate

Convert the pem format Gate server certificate into a PKCS12 (p12) file, which is importable into a Java Keystore (JKS).

# GATE_KEY_PASSWORD should be the passphrase you used to encrypt `gate.key`
GATE_KEY_PASSWORD=SOME_PASSWORD_FOR_GATE_KEY

# GATE_EXPORT_PASSWORD can be a new passphrase that will be used to encrypted `gate.p12`
GATE_EXPORT_PASSWORD=SOME_PASSWORD_FOR_GATE_P12

openssl pkcs12 \
 -export \
 -clcerts \
 -in gate.crt \
 -inkey gate.key \
 -out gate.p12 \
 -name gate \
 -passin pass:${GATE_KEY_PASSWORD} \
 -password pass:${GATE_EXPORT_PASSWORD}

This creates a p12 keystore file with your certificate imported under the alias “gate”.

Create a new Java Keystore (JKS) containing your p12-formatted Gate server certificate.

Because Gate assumes that the keystore password and the password for the key in the keystore are the same, we must provide both via the command line.

# GATE_EXPORT_PASSWORD should be the passphrase that was used to encrypted `gate.p12`
GATE_EXPORT_PASSWORD=SOME_PASSWORD_FOR_GATE_P12

# JKS_PASSWORD can be a new password that will be used to encrypt `gate.jks`
JKS_PASSWORD=SOME_JKS_PASSWORD

keytool -importkeystore \
 -srckeystore gate.p12 \
 -srcstoretype pkcs12 \
 -srcalias gate \
 -destkeystore gate.jks \
 -destalias gate \
 -deststoretype pkcs12 \
 -deststorepass ${JKS_PASSWORD} \
 -destkeypass ${JKS_PASSWORD} \
 -srcstorepass ${GATE_EXPORT_PASSWORD}

Import the CA certificate into the Java Keystore.

# JKS_PASSWORD should be the password that was used to encrypt `gate.jks`
JKS_PASSWORD=SOME_JKS_PASSWORD

keytool -importcert \
 -keystore gate.jks \
 -alias ca \
 -file ca.crt \
 -storepass ${JKS_PASSWORD} \
 -noprompt

Verify the Java Keystore contains the correct contents.

# JKS_PASSWORD should be the password that was used to encrypt `gate.jks`
JKS_PASSWORD=SOME_JKS_PASSWORD

keytool \
 -list \
 -keystore gate.jks \
 -storepass ${JKS_PASSWORD}

It should contain two entries:

gate as a PrivateKeyEntry
ca as a trustedCertEntry

Back Up Your Spinnaker Configuration

Backup all SpinnakerService related files.

Enable SSL for the UI

Next, configure Spinnaker’s Deck service to use the Deck certificate and private key that you generated.

The Kubernetes secret engine does not work for encrypting secrets for the UI.

Add the following snippet to the SpinnakerService manifest:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 security:
 uiSecurity:
 ssl:
 enabled: true
 sslCertificateFile: encryptedFile:k8s!n:spin-deck-secrets!k:deck.crt
 sslCertificateKeyFile: encryptedFile:k8s!n:spin-deck-secrets!k:deck.key
 sslCertificatePassphrase: abc # Your passphrase

Create a new Kubernetes secret containing the files. The following example assumes that Spinnaker is installed in the spinnaker namespace, and you are in the folder where deck.crt and deck.key are located:

kubectl -n spinnaker create secret generic spin-deck-secrets --from-file=deck.crt --from-file=deck.key

Depending on how your load balancer is configured (if you’re using an Ingress vs. a Service), you may have to change your service and/or ingress configuration. This is discussed below, in Update Load Balancers and URLs

Enable SSL for the API

Next, we will configure Spinnaker’s Gate service to use the JKS that we’ve generated.

Add the following snippet to the SpinnakerService manifest:

apiVersion: spinnaker.armory.io/{{ site.data.versions.perator-extended-crd-version }}
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 security:
 apiSecurity:
 ssl:
 enabled: true
 keyAlias: gate
 keyStore: encryptedFile:k8s!n:spin-gate-secrets!k:gate.jks
 keyStoreType: jks
 keyStorePassword: abc # The password to unlock your keystore. Due to a limitation in Tomcat, this must match your key's password in the keystore.
 trustStore: encryptedFile:k8s!n:spin-gate-secrets!k:gate.jks
 trustStoreType: jks
 trustStorePassword: abc # The password to unlock your truststore.
 clientAuth: WANT # Declare 'WANT' when client auth is wanted but not mandatory, or 'NEED', when client auth is mandatory.

Create a new Kubernetes secret having the above files. Here we assume that Spinnaker is installed in the spinnaker namespace, and you are in the folder where gate.jks is located:

kubectl -n spinnaker create secret generic spin-gate-secrets --from-file=gate.jks

Update load balancers and URLs

When you apply the above changes, Gate and Deck will change in the following ways:

Deck will continue to use port 9000, but it will be an HTTPS port instead of an HTTP port
Gate will continue to use port 8084, but it will be an HTTPS port instead of an HTTP port

Depending on your load balancer and infrastructure configuration, this will likely necessitate one of the following changes:

If you have a load balancer (such as an Ingress) in front of your Gate/Deck, you’ll likely need to change the way that Ingress communicates with Deck/Gate.
If you have no load balancer in front of your Gate/Deck and are instead using a basic Kubernetes Service, you’ll likely need to change the base URL overrides for Deck and Gate

Changing ingress

If you are using an Ingress in front of Deck and Gate, the Ingress is likely configured to communicate via HTTP to your backend services. You’ll need to change this to communicate via HTTPS.

The mechanism to achieve this will depend on what type of Ingress you are using. Here are two of the most common:

If you are using an NGINX Ingress Controller, you will need to add these annotations to the Ingress resource that is set up for Deck and gate:
```
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
```
If you are using the Amazon AWS ALB Ingress Controller, you will need to add these annotations to the Ingress resource that is set up for Deck and gate:
```
alb.ingress.kubernetes.io/backend-protocol: "HTTPS"
alb.ingress.kubernetes.io/healthcheck-protocol: "HTTPS"
```

Changing URL overrides

If you are instead using a Layer 4 (TCP) load balancer (such as a Service configured as a LoadBalancer in EKS), or directly exposing your Kubernetes Service objects using a NodePort configuration, then you’ll need to change the base URL overrides.

In the SpinnakerService manifest you need to change the overrideBaseUrl settings of Deck and Gate in the following way:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 security:
 apiSecurity:
 overrideBaseUrl: https://spinnaker.domain.com/api/v1 # URL to access Gate, using HTTPS scheme
 uiSecurity:
 overrideBaseUrl: https://spinnaker.domain.com # URL to access Deck, using HTTPS scheme

Although Spinnaker also supports using a different DNS name, Armory recommends that you use the same DNS but different paths for Deck and Gate. Tasks such as Cross-Origin Resource Sharing (CORS) between your Gate and Deck endpoints and securing Gate and Deck are much easier when both services use the same DNS name.

Apply SSL Changes

Before enabling the API endpoint, we should apply the changes that we’ve made so far and make sure everything continues to work.

kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest>

Apply your Ingress / Service changes, as indicated abvoe in the Changing Ingress section.
Verify that you’re still able to access Spinnaker. You may have to switch existing URLs from http to https.
Verify that you can still see Spinnaker applications and pipelines as before.
If you have any issues, perform various troubleshooting steps (such as those related to HTTPS in our KB), or restore your prior backup:

Apply the backed up manifests files with kubectl -n <spinnaker namespace> apply -f ...

Gate-local

Once you’ve verified that your existing Gate and Deck endpoints continue to work as expected, we can enable the API endpoint. This requires the following steps:

Enable x509 Authentication

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 security:
 authn:
 x509:
 enabled: true

Configure Gate to use a second port for the x509 API port.

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 profiles:
 gate:
 default:
 apiPort: 8085

Apply your changes

kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest>

Gate will now have a second API port set up listening on port 8085, which will expect an x509 client certificate from all clients trying to communicate with it. We have to expose this port externally.

Update load balancers and URLs

You must expose port 8085 on your Gate containers externally, and you should not terminate TLS in front of them. Depending on how your Kubernetes cluster lives, you may be able to use a LoadBalancer or NodePort Service. Alternatively, if your Ingress Controller is configured to support TLS pass-through, you can use that.

Operator automatically exposes Gate’s API port.

For example, with the LoadBalancer configuration in EKS, you will get an ELB endpoint (get this with kubectl -n NAMESPACE get svc -owide). With the NodePort configuration, you can use any instance in your cluster with the generated NodePort port.

You can verify that the client certificate is set up properly by curling the endpoint with the -v and -k (verbose, no validation) flags. You should get an alert for bad certificate, since the endpoint is expecting a client certificate and you are not providing one:

curl https://<endpoint>:8085 -v -k

For example:

curl https://abcd-123456789.us-west-2.elb.amazonaws.com:8085 -v -k
* Rebuilt URL to: https://abcd-123456789.us-west-2.elb.amazonaws.com:8085/
* Trying 35.0.0.1...
* TCP_NODELAY set
* Connected to abcd-123456789.us-west-2.elb.amazonaws.com (35.0.0.1) port 8085 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
 CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS alert, Server hello (2):
* error:1401E412:SSL routines:CONNECT_CR_FINISHED:sslv3 alert bad certificate
* stopped the pause stream!
* Closing connection 0
curl: (35) error:1401E412:SSL routines:CONNECT_CR_FINISHED:sslv3 alert bad certificate

Signing client certificates using the self-signed CA

If you created a self-signed CA, you can use that CA to sign certificates for your API clients to use.

Create the client key. Keep this file safe!

# This will be the passphrase used to encrypt the client private key
CLIENT_PASSWORD=SOME_CLIENT_PASSPHRASE

openssl genrsa \
 -des3 \
 -out client.key \
 -passout pass:${CLIENT_PASSWORD} \
 4096

Generate a certificate signing request for the client. Ensure the Common Name is set to a non-empty value.

# This should be the same passphrase used to encrypt the client private key
CLIENT_PASSWORD=SOME_CLIENT_PASSPHRASE

openssl req \
 -new \
 -key client.key \
 -out client.csr \
 -passin pass:${CLIENT_PASSWORD}

Use the CA to sign the client’s request.

# This should be the passphrase used to encrypt the self-signed CA private key
CA_KEY_PASSWORD=SOME_PASSWORD_FOR_CA_KEY

openssl x509 \
 -req \
 -days 365 \
 -in client.csr \
 -CA ca.crt \
 -CAkey ca.key \
 -CAcreateserial \
 -out client.crt \
 -passin pass:${CA_KEY_PASSWORD}

You should end up with these two files:
- client.key: a pem-formatted private key, which will have a pass phrase
- client.crt: a pem-formatted x509 certificate, which matches the private key and was signed by your CA

Connecting to the API with the client certificate

Now that you have a client certificate (with key), you can use it to connect to the Spinnaker API. For example, to get a list of Spinnaker applications from Gate:

curl https://abcd-123456789.us-west-2.elb.amazonaws.com:8085//applications \
 -k \
 --cert client.crt:SOME_CLIENT_PASSPHRASE\
 --key client.key
[{"accounts":"spinnaker","name":"ingress"},{"accounts":"spinnaker","name":"ldap"},{"accounts":"spinnaker","name":"spin"}]

Using Authorization with Spinnaker, and other Configurations

If your Spinnaker is configured with Authorization (based on groups), it can be helpful to encode groups into the client Certificate that will allow the API client access to things that are restricted to specific groups. This is achieved by doing these two things:

Configuring Gate to look at a specific OID in the client certificate for groups
Adding the groups at that specific OID in the client certificate

Configuring Gate to use a specific OID for groups

The OID 1.2.840.10070.8.1 is used to identify roles for a given user, so it is perfect for identifying the groups that a given client certificate will have access to.

You can configure Gate to look for a list of endline-delimited groups in the 1.2.840.10070.8.1 OID:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 security:
 authn:
 x509:
 roleOid: 1.2.840.10070.8.1

kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest>

Adding groups and signing certificates with the OID extensions

In order to allow a client certificate access to specific groups, when you’re generating the CSR and the certificate, you can specify custom fields in OpenSSL by providing them via an OpenSSL configuration file. For example, to add these two groups:

spinnaker-example0
spinnaker-example1

To a client certificate, construct a string with the groups delimited by \n, like this: spinnaker-example0\nspinnaker-example1.

Then, add that to a configuration file that looks like this (replacing GROUP_STRING with your group string)

# group.conf
[ req ]
distinguished_name = req_distinguished_name
attributes = req_attributes
req_extensions = v3_req

[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_min = 2
countryName_max = 2
stateOrProvinceName = State or Province Name (full name)
localityName = Locality Name (eg, city)
0.organizationName = Organization Name (eg, company)
organizationalUnitName = Organizational Unit Name (eg, section)
commonName = Common Name (eg, fully qualified host name)
commonName_max = 64
emailAddress = Email Address
emailAddress_max = 64

[ req_attributes ]
challengePassword = A challenge password
challengePassword_min = 4
challengePassword_max = 20

[ v3_req ]
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
1.2.840.10070.8.1 = ASN1:UTF8String:GROUP_STRING

Here’s an example of how to use this:

Create the configuration file

tee group.conf <<-'EOF'
[ req ]
distinguished_name = req_distinguished_name
attributes = req_attributes
req_extensions = v3_req

[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_min = 2
countryName_max = 2
stateOrProvinceName = State or Province Name (full name)
localityName = Locality Name (eg, city)
0.organizationName = Organization Name (eg, company)
organizationalUnitName = Organizational Unit Name (eg, section)
commonName = Common Name (eg, fully qualified host name)
commonName_max = 64
emailAddress = Email Address
emailAddress_max = 64

[ req_attributes ]
challengePassword = A challenge password
challengePassword_min = 4
challengePassword_max = 20

[ v3_req ]
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
1.2.840.10070.8.1 = ASN1:UTF8String:spinnaker-example0\nspinnaker-example1
EOF

Create the client key. Keep this file safe!

# This will be the passphrase used to encrypt the client private key
CLIENT_PASSWORD=SOME_CLIENT_PASSPHRASE

openssl genrsa \
 -des3 \
 -out client.key \
 -passout pass:${CLIENT_PASSWORD} \
 4096

Specify the configuration file when you generate the CSR with the -config group.conf

# This should be the same passphrase used to encrypt the client private key
CLIENT_PASSWORD=SOME_CLIENT_PASSPHRASE

openssl req \
 -new \
 -key client.key \
 -out client.csr \
 -passin pass:${CLIENT_PASSWORD} \
 -config group.conf

Specify the configuration file when you use the CA to sign the certificate with the -extensions v3_req and -extfile group.conf flags:

# This should be the passphrase used to encrypt the self-signed CA private key
CA_KEY_PASSWORD=SOME_PASSWORD_FOR_CA_KEY

openssl x509 \
 -req \
 -days 365 \
 -in client.csr \
 -CA ca.crt \
 -CAkey ca.key \
 -CAcreateserial \
 -out client.crt \
 -passin pass:${CA_KEY_PASSWORD} \
 -extensions v3_req \
 -extfile group.conf

Now, when you use the generated certificate and key, your API client will be able to access Spinnaker items that are restricted to those groups.

Configuring Spinnaker to parse out usernames from client certificate(s)

When looking at audit logs, it can be helpful to differentiate different API clients. One way to achieve this is to parse out a “username” from the client for each API client certificate. This is achieved by configuring Spinnaker to use regex to pull out a subject from the certificate. This is set up with the subject principal regex field, which is configured like this:

Operator

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 security:
 authn:
 x509:
 subjectPrincipalRegex: DESIRED_REGEX # For example, if you want to use the "Email Address" field from the certificate, the regex would be: EMAILADDRESS=(.*?)(?:,|$)

kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest>

Verify the x509 certificate

Make a call to https://${GATE_FQDN}/auth/user to verify the setup is correct. The call should return the certificate’s Spinnaker login and authorization information. For example, here’s a call using curl:

curl --cacert ca.crt --key client.key --cert client.crt https://${GATE_FQDN}/auth/user

Continuous-Deployment: Generate Certificates for Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

Make sure you are using the same version of Java that Armory Continuous Deployment is using.
You need a recent version of OpenSSL.

Generating self-signed certificate authority

Generate a key for our certificate authority:

openssl genrsa -aes256 -passout pass:TRUSTSTORE_PASS -out ca.key 2048

Replace TRUSTSTORE_PASS with your own CA password.

Important: Keep ca.key secure and do not distribute it.

Next, generate the certificate of the CA:

openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -out ca.pem -passin pass:TRUSTSTORE_PASS -subj /C=US/ST=California/O=Acme Corp/OU=Devops/CN=mydomain.com

Replace the values of the subj parameter with your own. Only the CN part is required and doesn’t need to match a real domain.

Generating a PKCS12 truststore (Java)

Import the ca.pem file you generated into a Java truststore:

keytool -importcert -storetype PKCS12 -keystore services/ca.p12 -storepass TRUSTSTORE_PASS -alias ca -file ca.pem -noprompt

Generating a keystore for a Java server

The CN attribute must match the hostname of the service. It will generally be spin-[service].[namespace] or spin-[service], for example spin-clouddriver.spinnaker.

Generate the keystore with the following commands:

openssl genrsa -aes256 -passout pass:KEY_PASSWORD -out svc.key 2048
openssl req -new -key svc.key -out svc.csr -subj /C=US/CN=spin-svc.spinnaker -passin pass:KEY_PASSWORD
openssl x509 -req -in svc.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out svc.crt -days 3649 -sha256 -passin pass:TRUSTSTORE_PASS
openssl pkcs12 -export -out svc.p12 -inkey svc.key -in svc.crt -passout pass:KEY_PASSWORD -passin pass:KEY_PASSWORD

Alternate names

If you want the Spinnaker service to support alternate names, replace the openssl pkcs12 command in the previous section with:

openssl x509 -req -sha256 -in svc.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out svc.crt -days 3650 -extfile svc-cert.ext -passin pass:TRUSTSTORE_PASS

The svc-cert.ext referenced in the command should contain the following (example for Clouddriver):

authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = spin-clouddriver.spinnaker
DNS.2 = spin-clouddriver

Note: If you are setting up mTLS, you can use the same command (or the same file) for client-side certificates.

Generating keys and certificates (Golang)

openssl genrsa -aes256 -passout pass:KEY_PASSWORD -out svc.key 2048
openssl req -new -key svc.key -out svc.csr -subj /C=US/CN=spin-svc.spinnaker -passin pass:KEY_PASSWORD
openssl x509 -req -in svc.csr -CA ca.pem -CAkey ca.key -CAcreateserial -out svc.crt -days 3650 -sha256 -passin pass:TRUSTSTORE_PASS

Putting it together (TLS)

The following script generates these files in the services directory:

Self-signed CA (ca.pem) and its PKCS12 truststore (ca.p12)
Keystore files for each Java service (clouddriver.p12, …)
Certificate and key files for each Golang services (terraformer.crt and terraformer.key, …)
a tls-passwords file containing all the passwords. You can store as-is in a bucket.

#!/bin/bash -e

# You can change it to a different method
newPassword() {
 echo $(openssl rand -base64 32)
}

# Namespace where you're installing Spinnaker. Used for the hostname.
NAMESPACE="spinnaker"

CA_PASSWORD="password"
TRUSTSTORE_PASSWORD=$(newPassword)
JAVA_SVCS=("clouddriver" "orca" "echo" "fiat" "igor" "rosco" "front50" "kayenta" "gate")
# JAVA_SVCS=("clouddriver-rw" "clouddriver-caching" "clouddriver-ro" "clouddriver-ro-deck" "orca" "echo-scheduler" "echo-worker" "fiat" "igor" "rosco" "front50" "kayenta" "gate")
GOLANG_SVCS=("dinghy" "terraformer")

echo "Cleaning up..."
rm -rf services/*
mkdir -p services/

echo "Generating CA key..."
openssl genrsa -aes256 -passout pass:${CA_PASSWORD} -out services/ca.key 4096

echo "Generate self signed root certificate"
openssl req -x509 -new -nodes -key services/ca.key -sha256 -days 3650 -out services/ca.pem -passin pass:${CA_PASSWORD} -subj /C=US/CN=Test

echo "Generate trust store as PKCS12"
keytool -importcert -storetype PKCS12 -keystore services/ca.p12 -storepass $TRUSTSTORE_PASSWORD -alias ca -file services/ca.pem -noprompt

echo "truststore: ${TRUSTSTORE_PASSWORD}" > services/tls-passwords

for i in ${!JAVA_SVCS[@]};
do
 svc=${JAVA_SVCS[$i]}
 password=$(newPassword)
 echo "Generating $svc key and certificate..."
 openssl genrsa -aes256 -passout pass:${password} -out services/${svc}.key 4096
 openssl req -new -key services/${svc}.key -out services/${svc}.csr -subj /C=US/CN=spin-${svc}.${NAMESPACE} -passin pass:${password}
 cat <<EOF>services/${svc}.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = spin-${svc}.${NAMESPACE}
DNS.2 = spin-${svc}.${NAMESPACE}.svc.cluster.local
EOF
 openssl x509 -req -in services/${svc}.csr -CA services/ca.pem -CAkey services/ca.key -CAcreateserial -out services/${svc}.crt -days 3649 -sha256 -passin pass:${CA_PASSWORD} -extfile services/${svc}.ext
 openssl pkcs12 -export -out services/${svc}.p12 -inkey services/${svc}.key -in services/${svc}.crt -passout pass:${password} -passin pass:${password}
 echo "${svc}: ${password}" >> services/tls-passwords
done


for i in ${!GOLANG_SVCS[@]};
do
 svc=${GOLANG_SVCS[$i]}
 password=$(newPassword)
 echo "Generating $svc key and certificate..."
 openssl genrsa -aes256 -passout pass:${password} -out services/${svc}.key 4096
 cat <<EOF>services/${svc}.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = spin-${svc}.${NAMESPACE}
DNS.2 = spin-${svc}.${NAMESPACE}.svc.cluster.local
EOF
 openssl req -new -key services/${svc}.key -out services/${svc}.csr -subj /C=US/CN=spin-${svc}.${NAMESPACE} -passin pass:${password}
 openssl x509 -req -in services/${svc}.csr -CA services/ca.pem -CAkey services/ca.key -CAcreateserial \
 -out services/${svc}.crt -days 3650 -sha256 -passin pass:${CA_PASSWORD} -extfile services/${svc}.ext

 echo "${svc}: ${password}" >> services/tls-passwords
done

Troubleshooting

You may encounter the following error if the version of Java you are using to generate the certificates is not the same version that Armory Continuous Deployment is using:

java.io.IOException: Integrity check failed: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available

Continuous-Deployment: Integrate ServiceNow with Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Overview of ServiceNow

ServiceNow provides several workflow solutions (ITSM, PPM, Security Response, ITOM, etc). Instead of defining a single use case for an integration that satisfies some use cases, this guide reviews the various building blocks available to create integrations through a generic webhook stage or a custom webhook stage.

Consuming Webhooks in ServiceNow requires some configuration. You can read more about the process in the ServiceNow docs.

Using a custom webhook stage to create a Change Request in ServiceNow

Potential uses for creating a change request in ServiceNow include:

A deployment happens and a ticket needs to be filed in ServiceNow for record keeping
A canary deployment is successful, and you need a change ticket to be created and approved before full deployment into production.

In ServiceNow, you need to create a Scripted Web Service. At a high level, this service performs the following actions:

Receives the webhook from Spinnaker
Processes the contents of the payload
Creates a Change Request.

A very similar approach could be taken to create a different type of ticket.

To create the Scripted Web Service, perform the following task:

Search for Scripted REST APIs in the ServiceNow Navigator.
Create a new Scripted Web Service and give it a descriptive name. For example, you can name it “SpinnakerWebhookListener”.
Submit the information to create the service.
Create a Resource that contains the endpoint that Spinnaker and the script to create the ticket.

You can create multiple Resources, each with its own specific URL and script to do something. In this example, the “SpinnakerWebhookListener” is the Scripted REST API that handles all the various calls that Spinnaker makes to this specific ServiceNow Instance. You define Resources to handle each specific call. For example, you may want to create separate calls to create different type of tickets or perform different actions within ServiceNow.
Create a new resource and give it a meaningful name. The example below uses create-change-ticket.
Set the HTTP method to POST.

If you are planning on having only one resource in the REST API, you can leave the Relative Path as is.

If you plan to create multiple Resources, set the relative path to be specific to this resource. This will be appended to the API Path. The resulting path is appended to your ServiceNow instance address, and that is the URL Spinnaker uses.

Note: For testing the integration, you can opt to not require authentication.
Optionally, provide the script. You can also provide one at a later time.

Example script

The following is an example script you can use when configuring ServiceNow:

(function process(/RESTAPIRequest/ request, /RESTAPIResponse/ response) {
// implement resource here
//gs.info(request.body.dataString);
response = request.body.dataString;
var gr = new GlideRecord('change_request');
gr.initialize();
var parser = new JSONParser();
var parsedData = parser.parse(response);
//gs.info(parsedData);
gs.info(parsedData.application);

gr.short_description = parsedData.application;
gr.description = parsedData.description;
//gr.change_request = current.sys_id;
gr.insert();
})(request, response);

Keep in mind that this only works if the webhook sends the proper data.

In the example script, the response object is the JSON payload from the webhook. Whenever you see ‘gs.info()’, it is how you can write to the ServiceNow System Log. Use this as a debug tool as it allows you to write to the log the values found in the payload. The example script expects at least two key-pair values in the JSON payload, one key being application and the other description. In this example, the value for the application key is being set to the Short Description of the Change. The value associated with the description key goes in the Description field.

ServiceNow REST API

ServiceNow also has a REST API. You can use ServiceNow’s REST API Explorer to view sample code for any REST API call, including creating a change request.

ServiceNow Workflows

While most people are familiar with ServiceNow as a ‘ticketing system’, it also has an automation engine. RunBooks for this engine are created as workflows. These workflows can automate internal ServiceNow operations (like handling approval routing) as well as calling other systems APIs (called Orchestrations), like provisioning VMs on-premise or in the cloud.

These workflows can be called from a ServiceNow script in a Scripted REST API.

Setting Up the Webhook Stage in Spinnaker

The next step is to either create a generic webhook stage or a custom stage.

To create a custom stage, perform the following steps:

In /home/spinnaker/.hal/default/profiles on your Halyad pod, edit orca-local. If the file does not exist, create it. Below are the contents of my orca-local.yaml that generated my custom ServiceNow stage:

webhook:
 preconfigured:
 - # stage name in stage selector drop down
 label: "Create Change Request in ServiceNow"
 enabled: true
 # UI description for stage
 description: Creates Change Request in ServiceNow
 # stage 'type' in stage definition
 type: createChangeSNOW
 # Webhook type
 method: POST
 # URL for webhook call
 url: https://dev102110.service-now.com/api/77303/spinnakerwebhooklistener
 customHeaders:
 # Any necessary headers (supports S3 / Vault)
 # Authorization: Basic c2VydmljZWFjY291bnQ6cGFzc3dvcmQK
 Content-Type: application/json
 # Payload for the request
 payload: |-
 {
 "application": "${trigger\['artifacts'\][0]['reference']}"
 "description": "${parameterValues['description']}"
 }
 # Zero or more parameters
 parameters:
 - label: Description (optional)
 name: description
 description: Enter a description for the ServiceNow Change Request
 type: string

Note that the example payload contains the two key/pair values that ServiceNow is expecting.

The application is going to be the artifact that is triggering the pipeline. In the example, the pipeline is triggered when a new version of a container for a given organization or application is pushed to the configured Docker Registry. Artifactory is configured as the Docker Registry, so the application is set to something that follows a similar format: <jfrog server-reponame-jfrog.io/<orgname>/ <application>:<tag>.

The above is an example of using Pipeline Expressions to pass dynamic values. The description is an example of passing values that the user can enter in the stage when it is configured.

Apply your changes to Spinnaker: hal deploy apply.
Once you apply your changes, open Deck, Spinnaker’s UI.
When you create or edit a pipeline, a new stage available is available:

If the new stage does not appear, perform a hard refresh or clear your browser cache and reopen Deck.

Other Spinnaker users can use the ServiceNow stage you created.

The following screenshot shows a Change Request in ServiceNow created by the custom webhook stage:

Triggering a pipeline based on webhooks

Another use case is to trigger a pipeline when a ticket gets approved. For many users, a deployment into production cannot happen unless there is an approval in ServiceNow.

At it’s simplest implementation, an approval can trigger a pipeline execution. A more complex implementation could take fields from the ticket being approved and passed them to the pipeline as parameters to guide the deployment.

The following screenshot shows a pipeline with that gets triggered based on a webhook:

You can add payload constraints to avoid webhooks without the right payload from being processed. Armory recommends testing the webhook without contraints first.

On the ServiceNow side, you need something to call this URL. In the example environment, there is a button on the Change Request form in ServiceNow that calls the URL.

The first step is to define the REST call you need to make. To do this, type ‘REST’ on the navigation filter box and, from the results, select System Web Services > Outbound > REST Message:

Note: On the menu shown above, select REST API Explorer to get code samples on how to use the ServiceNow REST API to create, search, or edit objects in ServiceNow.

In the REST Messages window, create a new REST Message. The following image shows an example to trigger the pipeline.

Once you have the REST Message set up, configure something to call it. Examples include when a ticket transitions as a step in a workflow or orchestration or a button or link on a form. This example uses a button in the form. Type “change” in the navigation filter as shown in the following image:

Click on Change > Open to get a list of all open Change Requests. Open any of Change Request from the list and right click on the top of the form as shown below to select Configure > UI Actions:

The following screenshot shows how a UI action can be configured:

Note that you can use the Condition field to determine under which conditions the button should be active, such as when the status is “Ready for Approval”.

Continuous-Deployment: Observability

Mon, 01 Jan 0001 00:00:00 +0000

Continuous-Deployment: Rate Limiting the Spinnaker API

Mon, 01 Jan 0001 00:00:00 +0000

How Spinnaker monitors a deployment

Spinnaker polls the entire state of managed cloud resources every 30 seconds through the Clouddriver service. The polling can cause cloud providers, such as AWS, to throttle the requests on your account. If you have a large number of Auto-Scaling Groups and Elastic Load Balancers in your account or other services commonly querying the same APIs then you can expect to see throttling exceptions in your Spinnaker logs.

How to alleviate AWS throttling exceptions

There are several things you can do to help reduce the effects of throttling:

Set fine tune rate limits within Spinnaker.
Adjust Spinnaker’s retry limit per request.
Decrease the polling interval.

Fine-grained rate limits

Spinnaker queries your Cloud Provider (AWS, GCP, Azure, Kubernetes, etc) frequently to understand the state of your existing infrastructure and current deployments. However, this might cause you to run into rate limits imposed by the Cloud Provider. To help avoid this Spinnaker provides controls to limit the number of requests it generates. The unit used for these controls is “requests per second” (a double float value). Global defaults are 10.0 max requests per second.

Below is an example configuration for global rate limits for all services that you would place in your SpinnakerService manifest under section spec.spinnakerConfig.profiles.clouddriver:

serviceLimits:
 defaults:
 rateLimit: 10.0 # default max req/second

If you have multiple Cloud Providers, you can limit each one differently:

serviceLimits:
 cloudProviderOverrides:
 aws:
 rateLimit: 10.0 # default max req/second

You can provide account specific overrides as well in case you have significantly more resources in one account while others have less:

serviceLimits:
 accountOverrides:
 my-test:
 rateLimit: 10.0 # default max req/second
 my-prod:
 rateLimit: 10.0 # default max req/second

And finally, you can have more fine-grained control for particular AWS endpoints that might have a different rate limits.

We’ve found that this formula works pretty well:

max_req_second = num_of_x_resources / clouddriver_30s_poll_interval

For example, if we have 90 load balancers and clouddriver polls every 30 seconds, then we’ll end up with a rate limit of 3 reqs/second for AmazonElasticLoadBalancing.

Here’s the list of rate limits you can adjust created from AmazonClientProvider.java@v5.36.0 on 05/14/2019 by using the regex \w+\.class:

serviceLimits:
 implementationLimits:
 AWSApplicationAutoScaling:
 defaults:
 rateLimit: 10.0 # default max req/second
 AWSApplicationAutoScalingClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AWSLambda:
 defaults:
 rateLimit: 10.0 # default max req/second
 AWSLambdaAsync:
 defaults:
 rateLimit: 10.0 # default max req/second
 AWSLambdaAsyncClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AWSLambdaClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AWSSecretsManager:
 defaults:
 rateLimit: 10.0 # default max req/second
 AWSSecretsManagerClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AWSServiceDiscovery:
 defaults:
 rateLimit: 10.0 # default max req/second
 AWSServiceDiscoveryClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AWSShield:
 defaults:
 rateLimit: 10.0 # default max req/second
 AWSShieldClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonAutoScaling:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonAutoScalingClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonCloudFormation:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonCloudFormationClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonCloudWatch:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonCloudWatchClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonEC2:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonEC2ClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonECR:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonECRClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonECS:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonECSClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonElasticLoadBalancing:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonElasticLoadBalancingClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonIdentityManagement:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonIdentityManagementClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonRoute53:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonRoute53ClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonS3:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonS3ClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonSNS:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonSNSClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonSQS:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonSQSClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonSimpleWorkflow:
 defaults:
 rateLimit: 10.0 # default max req/second
 AmazonSimpleWorkflowClientBuilder:
 defaults:
 rateLimit: 10.0 # default max req/second

Using these settings will help you avoid hitting the AWS rate limits. They can also help Spinnaker be more responsive since the cloud provider clients will not implement their back-off strategy to continue to query the infrastructure.

Request retry

You can set the number of retries per request with the following setting:

aws:
 client:
 maxErrorRetry: 3 # default

This is the number of retries before the request fails. It’s used with an exponential backoff, maxing out at 20 seconds.

Fiat hitting rate limits

If Fiat is configured to poll Github or Google, you may end up seeing rate limits when Fiat does its polling for user groups. Some symptoms that you might see are:

You can’t log into Spinnaker anymore

Your Fiat logs contain lines similar to:

GithubTeamsUserRolesProvider : [] HTTP 403 Forbidden. Rate limit info: X-RateLimit-Limit
GoogleDirectoryUserRolesProvider : [] Failed to fetch groups for user x: Rate Limit Exceeded

To address this issue, adjust poll cycle time and/or timeouts in your SpinnakerService manifest under section spec.spinnakerConfig.profiles.fiat (beware: you will end up with two nested fiat sections). See (FiatServerConfigurationProperties.java for more details).:

fiat:
 writeMode:
 # Poll cycle interval, "check if a user belongs to a group every X ms"
 syncDelayMs: 600000 # the default 600000 (10 mins) is usually fine

 # How much time to between retries of dependent resource providers if they are down.
 retryIntervalMs: 10000

FAQ

Q: Why doesn’t Spinnaker use AWS Config to update its state?

A: AWS Config does not support all resource types. The two most rate limited APIs are Auto Scaling and Classic Elastic Load Balancing, neither are supported by AWS Config. Additionally, there is a non-trivial delay from the time a resource is created and the time a notification is created by AWS Config.

Continuous-Deployment: Restrict Application Creation

Mon, 01 Jan 0001 00:00:00 +0000

Requirements and guidelines for using Fiat

Armory 2.17 (Open Source Spinnaker^TM 1.17) or later
Fiat must be enabled and configured to work with an identity provider. For more information, see Authorization (RBAC).

When managing roles for Spinnaker, keep the following in mind:

Roles are case insensitive. All roles are changed to lowercase in Fiat’s internal model.
You must explicitly configure permissions for each user role. The default for a user role is no permissions, which means it cannot perform any actions.

Restrict application creation

To restrict which users can create applications in Spinnaker, perform the following steps:

Add the line auth.permissions.provider.application: aggregate to SpinnakerService manifest under key spec.spinnakerConfig.profiles.fiat.

Add prefixes as a source:

auth.permissions.source.application.prefix:
 enabled: true

Define the permissions for a prefix:

- prefix: <some_prefix>
 permissions:
 READ:
 - "<user role 1>"
 - "<user role 2>"
 - "<user role n>"
 WRITE:
 - "<user role n>"
 EXECUTE:
 - "user role n>"

Here is an example configuration with in-line comments:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 profiles:
 fiat:
 # Enables Fiat to read from new sources.
 auth.permissions.provider.application: aggregate
 # Sets `prefix` as one of these new sources
 auth.permissions.source.application.prefix:
 enabled: true
 prefixes:
 # Defines the prefix `apptest-x`.
 - prefix: "apptest-*"
 permissions:
 # Defines permission requirements for all applications that match the prefix `apptest-*` based on roles.
 # role-one and role-two have READ permission
 READ:
 - "role-one"
 - "role-two"
 # role-one has write permission
 WRITE:
 - "role-one"
 # role-one has execute permission
 EXECUTE:
 - "role-one"

As a result, any application that matches the prefix apptest-* has restrictions on who can perform actions. For example, a user with the user role role-two only has READ permission.

To restrict application creation specifically, add fiat.restrictApplicationCreation at the top of fiat config and set it to true.

Note: Currently, the prefix source is the only source that support the CREATE permission.

The following example builds upon the example from the previous steps. In-line comments describe additions:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 profiles:
 fiat:
 # Add CREATE as a permission
 fiat.restrictApplicationCreation: true
 auth.permissions.provider.application: aggregate
 auth.permissions.source.application.prefix:
 enabled: true
 prefixes:
 - prefix: "*"
 permissions:
 # Assign CREATE permission to role-one
 CREATE:
 - "role-one"
 READ:
 - "role-one"
 - "role-two"
 WRITE:
 - "role-one"
 EXECUTE:
 - "role-one"

The above example assigns CREATE permission to users with the role-one role. Users without the role-one role cannot create any applications in Spinnaker.

Apply your configuration changes to Spinnaker by running the following command: kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest>.

The following screenshot shows what happens when a user without sufficient permissions attempts to create an application in Deck, Spinnaker’s UI:

Continuous-Deployment: Spinnaker Production Environments

Mon, 01 Jan 0001 00:00:00 +0000

Architecture

Use MySQL compatible database engine
- Managed services like Aurora provide cross-region replication
Kubernetes
- See System Requirements for more information
Each service has at least 2 replicas to provide basic availability at the Kubernetes level
Monitoring is optional but strongly recommended
Armory provides optional log Spinnaker aggregation for troubleshooting but we also recommend customers to have a log management solution in place

Kubernetes Considerations

The Kubernetes cluster sizing recommendations assume that only Spinnaker, its monitoring, and the Kubernetes operator run in the cluster. It also provides extra room for rolling deployments of Spinnaker itself.
If available, use a cluster with nodes in different availability zones.
It’s generally better to go with more smaller nodes than fewer larger nodes for the cluster to be more resilient to the loss of nodes. Make sure that nodes are still able to handle the largest pods in terms of CPU and memory.

Database Considerations

If available, use cross-region replication to ensure durability of the data stored.
- Front50’s database contains pipeline definitions and needs to be properly backed up.
- Orca’s database contains pipeline execution history that is displayed in Spinnaker’s UI.
- Clouddriver’s database contain your infrastructure cache. If lost, it will need to be re-cached which depending on the size of your infrastructure may take a while. It doesn’t have long term value.
Make sure the network latency between Spinnaker and the database cluster is reasonable. It often just means located in the same datacenter.
Clouddriver, Orca, and Front50 services must each use a different database. They can be in different database clusters or in the same. A single cluster is easier to manage and more cost effective but the number of connections used by Spinnaker will be added across all services.
- Your database cluster must support the number of open connections from Spinnaker and any other tool you need. For numbers refer to the database connections chart in the profiles below.
- Clouddriver connection pools can be tuned via sql.connectionPools.cacheWriter.maxPoolSize and sql.connectionPools.default.maxPoolSize. Both values default to 20 and need to be increased to handle more tasks per Clouddriver.

Redis Considerations

Most services rely on Redis for lightweight storage and/or task coordination. Spinnaker does not store many items in Redis as is reflected in the following recommendations. Redis being single threaded doesn’t need more than one CPU.

When available, a managed Redis (like ElastiCache) can be used. A shared Redis can be used for ease of management.

Spinnaker Settings

To support a high number of API requests, we advise to set the following settings in gate and front50 profiles:

hystrix.threadpool.default.coreSize: x

Where x is given by: maximum API request per seconds * mean response time

We estimated the maximum API request per seconds in the tests below, refer to the Average API Response Time graph below. For instance for 30 request/sec, the coreSize value could be set to: 300 * 0.25 = 75.

Recommendations

Installation Types

Many factors come to sizing Spinnaker, for instance:

Number of active users will impact how to size Gate service.
Complex pipelines will impact the amount of work the Orca service has to do.
Different providers (Kubernetes, GCP, AWS,…) come with very different execution profiles for the Clouddriver service.

This document makes the following assumptions:

Pipelines used to evaluate Spinnaker are simple and made of a Deploy and 2 Wait stages for stage scheduling. If you expect your pipelines to be complex, divide the supported executions by the number of non-trivial expected stages (baking, deploying) in your pipelines.
API requests simulate potential tool requests as well as user activity. We give number of concurrent users.
All services run with at least 2 replicas for basic availability. It is possible to run with fewer replicas at the cost of potential outages.

Base Profile - Kubernetes Deployments

Base profile recommendations

A base deployment of Spinnaker targets organizations with:

50 applications
250 deployments per day over 5 hour window.
30 req/s coming from browser sessions or tools
10x burst for both pipelines and API calls.

Service	Replicas	CPU request	CPU limit	Memory request	Memory limits
Clouddriver	2	2000m	3000m	2.0Gi	2.5Gi
Deck	2	150m	300m	32Mi	64Mi
Dinghy	2	500m	1000m	0.5Gi	1.0Gi
Echo	2	500m	1000m	1.0Gi	1.5Gi
Fiat	2	500m	1000m	0.5Gi	1.0Gi
Front50	2	500m	1000m	1.0Gi	1.5Gi
Gate	2	750m	1000m	1.0Gi	1.5Gi
Kayenta	2	500m	1000m	0.5Gi	1.0Gi
Igor	2	500m	1000m	0.5Gi	1.0Gi
Orca	2	1000m	1500m	1.0Gi	1.5Gi
Rosco	2	500m	1000m	0.5Gi	1.0Gi
Terraformer	2	500m	1000m	0.5Gi	1.0Gi
Redis	1	500m	1000m	0.5Gi	1.0Gi
Total		16300m	28600m	18.56Gi	31.125Gi

Load Test: Base Profile (Kubernetes)

Overview

Armory Spinnaker	API request/sec - baseline (burst)	Pipeline trigger/minute - baseline (burst)
2.17.1	30 (300)	0.834 (8.34)

General Service Health

Database Connections

Per Service CPU and Memory

Clouddriver Health

Orca Health

Gate Health

Continuous-Deployment: Work with Secrets in Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Overview of storing secrets

Storing Spinnaker^TM configs in a git repository is a great solution for maintaining versions of your configurations, but storing secrets in plain text is a bad security practice. If you’re using the Operator to deploy Spinnaker, separating your secrets from your configs through end-to-end secrets management is already supported. All you need to do is replace secrets in the configuration files with the syntax described here, and Spinnaker will decrypt them as needed.

We can now store secrets (tokens, passwords, sensitive files) separately from the Spinnaker configurations. We’ll provide references to these secrets to services that need them.

Spinnaker services that support decryption will decrypt these secrets upon startup.
Operator can decrypt these secrets when it needs to use them (e.g. when validating resources).
Operator can send secret references to the services that support decryption or send decrypted secrets if the service does not support it.

Using secrets

Secret format

When referencing string secrets (passwords, tokens) in configs, use the following general format:

encrypted:<secret engine>!<key1>:<value1>!<key2>:<value2>!...

When referencing files, the same parameters are used but with the encryptedFile prefix:

encryptedFile:<secret engine>!<key1>:<value1>!<key2>:<value2>!...

The keys and values making up the string vary with each secret engine. Refer to the specific documentation for each engine for more information.

In main configuration

This applies to section spec.spinnakerConfig.config of the SpinnakerService manifest when using the Operator.

Operator can understand the secrets you provide. If the service you are deploying is able to decrypt secrets, Operator will pass the reference directly. Otherwise it will decrypt the configuration before sending it.

For instance, after replacing the GitHub token in our main config with the encrypted syntax:

...
 github:
 enabled: true
 accounts:
 - name: github
 token: encrypted:s3!r:us-west-2!b:mybucket!f:spinnaker-secrets.yml!k:github.token
...

You find the following in /opt/spinnaker/config/clouddriver.yml inside clouddriver pod:

...
 github:
 enabled: true
 accounts:
 - name: github
 token: encrypted:s3!r:us-west-2!b:mybucket!f:spinnaker-secrets.yml!k:github.token
...

And for an older release of Clouddriver that does not support decryption:

...
 github:
 enabled: true
 accounts:
 - name: github
 token: <TOKEN>
...

In other configuration

You can also provide secret references directly in SpinnakerService manifest under section spec.spinnakerConfig.profiles when using the Operator as well as directly in Spinnaker services.

Supported secret engines

Is there a secret engine you’d like us to support? Submit a feature request here!

Armory Docs – Armory Admin

Continuous-Deployment: Add a Kubernetes Account Deployment Target in Spinnaker

How Spinnaker operates when deploying to Kubernetes targets

Before you begin

Workflow for adding a Kubernetes Service Account

Setting up the Kubernetes Service Account

Download the latest spinnaker-tools release

Set up bash parameters

Option 1: Create the service account with cluster-admin permissions

Important

Option 2: Create the service account with namespace-specific permissions

Add the kubeconfig and cloud provider to Spinnaker

Verify the Kubernetes account appears in the Spinnaker UI

Continuous-Deployment: Administration for Cloud Foundry

Continuous-Deployment: Administration on AWS

Continuous-Deployment: Auth Propagation in Spinnaker Pipelines

Overview of Authentication and Authorization in Spinnaker

Authorization & Manual Judgments

Continuous-Deployment: Bake and Share Amazon Machine Images Across Accounts

Overview of sharing AMIs across accounts

Spinnaker configuration for sharing baked AMIs

Spinnaker pipeline Bake stage configuration

Continuous-Deployment: Clouddriver Caching Agents in Spinnaker

How Spinnaker discovers and caches infrastructure elements

The caching agent

The cache store

The caching agent scheduler

How the cache store, scheduler, and agent work together

On-demand caching agents

Next steps

Continuous-Deployment: Configure Clouddriver Caching Agents in Spinnaker

Caching agents in Spinnaker

SQL global caching agents configuration

Redis global caching agents configuration

Considerations

Continuous-Deployment: Configure Automated Canary Deployments in Spinnaker

Kayenta Overview

Enable Kayenta

Configure Kayenta

What’s next

Continuous-Deployment: Configure Clouddriver to use a SQL Database

Advantages of using an RDMS with Clouddriver

Base configuration

Database setup

Deployment

Simple deployment

No downtime deployment

Step 1: Warm up the cache

Step 2: Use MySQL to back new tasks

Step 3: Remove Redis

Continuous-Deployment: Configure Dynamic Kubernetes Accounts

Overview of external account configuration

Continuous-Deployment: Configure Gate and Deck to Run on the Same Hostname

Overview

Prerequisites for running on the same host

Configure Spinnaker

Continuous-Deployment: Configure GitHub OAuth for Spinnaker

Before you begin

Configure GitHub OAuth in GitHub

Configure GitHub OAuth in Spinnaker

What’s next

Continuous-Deployment: Configure mTLS for Spinnaker Services

Overview of mTLS

What you need

Configuring Java services

Configuring Golang services

Changing service endpoints

Changing readiness probe

Deployment

Continuous-Deployment: Configure Auth for Spinnaker Using Okta SAML

Configure a Spinnaker application in Okta

Configure Spinnaker to use Okta

1: Create a SAML keystore file

2: Configure Spinnaker to use SAML

Troubleshooting

Continuous-Deployment: Configure Spinnaker's Orca Service to Use SQL RDBMS

Advantages to using an RDMS with Orca

Base configuration

Create the orca database and configure authorization

Migrate from Redis to SQL to keep existing execution history

Download the latest `spinnaker-tools` release

Create the `orca` database and configure authorization

Grant `cluster-admin` permissions