Armory Docs – User Guides

Continuous-Deployment: Spinnaker's Application Screen

Mon, 01 Jan 0001 00:00:00 +0000

An application within Spinnaker^TM is a combination of clusters and load balancers.

To get to the application details screen, select the ‘Applications’ tab on the top navigation bar in Spinnaker. Then pick an application from the list. It should load to a screen that looks like:

This is a view of all of the server groups (which make up your cluster), grouped by stack. In the image above, I have three server groups. Two server groups are in the ‘hellodeploy-cron’ stack, namely V948 and V947. Notice that V947 is greyed out and doesn’t contain any instances. In other words, it is disabled and scaled down to zero. The last server group, V013, is in the ‘hellodeploy-preprod’ stack. It has one instance.

From here, we can click on different things:

Cluster tab

Active Server group selected:

Disabled server group selected:

Instance selected:

Load balancer selected:

Load balancer tab

Load balancer selected:

Pipelines

If you select the ‘Pipelines’ tab, you will see something like:

This page shows two pipelines, ‘Deploy’ and ‘Cron Deploy’. The ‘Deploy’ pipeline shows two executions, both labeled ‘BUILD #5’, this indicates that they were triggered by Jenkins’ build #5. Clicking on the build number will take you to the Jenkins job. The red block on the second execution indicates that it has failed. If I click on the red part of the execution, it will expand into:

Now I can see that the execution failed because of a subnet issue.

For more information about pipelines, check out the pipeline guide.

Deleting an application

To delete an application, first click on the ‘Applications’ tab on Spinnaker’s top navigation bar. Select the application you would like to delete and enter its application detail page. Now select the ‘Config’ tab. Scroll all the way down to the ‘Delete Application’ section.

Example:

You can see that in order to delete the application, you first need to delete its server groups. To delete its server groups, select the ‘Clusters’ tab. Now select a server group to delete by clicking on it.

Example of V952 selected:

Now press the ‘Server Group Actions’ from the bar on the right hand side and press ‘Destroy’. Spinnaker asks you to confirm that you are going to delete this server group.

Once all of the server groups have been deleted, navigate back to the ‘Config’->‘Delete Application’ section and press “Delete Application’.

Continuous-Deployment: Artifact Progression Through Environments in Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

Configure Github to post push events

Add two Kubernetes accounts (staging and prod)

Overview of promoting artifacts between environments

One of the more powerful and common uses of Spinnaker is the promotion of an artifact through a series of environments, i.e. from staging to prod.

In such a workflow, a pipeline might initially be triggered by the creation or modification of an artifact. While the artifact itself may be a manifest, a Docker image, or a number of other possibilities, what is important is that once it is initially deployed, the artifact’s subsequent promotion is dependent on the result of the artifact’s deployment to the previous environment.

Determining promotion success can be accomplished through the completion of integration tests or another automated method but it can also be determined through manual judgement. We make use of the latter case in our example below.

Below is an example where we deploy an artifact to our staging environment. The successful completion of this pipeline then triggers another pipeline configured with a Manual Judgement stage. If manual judgement succeeds, a third stage is triggered which takes the same artifact we initially deployed into our staging environment and deploys it to our production environment.

Deploy to Staging

We begin by creating a new application named artifact-promotion-demo. Once you have created your application, click on Pipelines and then Configure a new pipeline.

We want this initial pipeline to be responsible for our artifact’s deployment to staging. With this in mind, we have named it Deploy to Staging

You should now configure the expected artifacts for your pipeline and specify how your pipeline should be triggered. In our example we have specified a manifest (stored in our Github repo) as our artifact and set our pipeline to trigger automatically when changes to our manifest are detected.

Next, click Add stage. Provide your new stage a descriptive name – we have chosen Deploy to Staging as ours. In order to deply our artifact, we choose Deploy (Manifest) as our Stage Type and specify Artifact as our Manifest Source. Finally, we select our arfifact from the Expected Artifact dropdown.

Save your changes.

Validate your Deployment

Now that we have a pipeline deploying to staging, we want validate that the deployment was successful before promoting our artifact to production.

Click Pipelines and then click + Create to create a new pipeline. We have named ours Staging Judgement.

Add an automated trigger to the pipeline where the type is Pipeline. Select the Deploy to Staging pipeline from the Pipeline dropdown. Check both successful and Trigger Enabled from the remaining options.

Now click, Add stage and select Manual Judgement as the Stage Type. In Execution Options select halt the entire pipeline for If stage fails. Additionally, you can provide additional instructions in the Instructions text-area.

Save your changes.

Deploy to Production

We are now ready to create the pipeline responsible for deploying our artifact to production.

Click Pipelines and then click + Create to create a new pipeline. We have named this one Deploy to Production and have made use of the Copy From dropdown in order to copy our Deploy to Staging pipeline.

You will want to delete the Automated Trigger copied from our Deploy to Staging pipeline. In its place, create a new one of Type Pipeline. In the Pipeline dropdown select Staging Judgement. Lastly, select successful for Pipeline Status and select Trigger Enabled

Finally, add a new stage of type Deploy (Manifest). The configuration for this stage should mirror the configuration of the Deploy (Manifest) stage of the Deploy to Staging pipeline with the exception that specified account should correspond to your production environment.

Summary

We have just shown an example of promoting a single artifact through a series of environments. While we relied on a Manual Judgement stage in order to validate the deployment of our artifact to staging before deploying it to production, we could also have used automated methods such as the successful completion of integration tests or through the use of canary analysis. Finally, although the example relied on only two distinct environments, adding additional environments is trivial.

Continuous-Deployment: Automated Kubernetes Rollbacks in Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Creating a main deployment pipeline

Let’s start by creating a new pipeline that will deploy a simple nginx container:

We’ll be supplying the manifest directly in the Deploy (Manifest) stage as text for simplicity, but it can also be taken from a version control system like Github:

This is the sample deployment manifest, which will create two nginx pods in Kubernetes under the “german-env1” namespace (the namespace must exist already):

apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx-deployment
 namespace: german-env1
spec:
 selector:
 matchLabels:
 app: nginx
 replicas: 2
 template:
 metadata:
 labels:
 app: nginx
 spec:
 containers:
 - name: nginx
 image: nginx:1.14.0
 ports:
 - containerPort: 80

This is all that is required in the deploy pipeline. We can now save and execute manually the pipeline and it should be successful.

Generating historic deployment versions for testing rollbacks

We’ll test rollbacks by making a change to the deployment manifest and running the pipeline again, so that we’ll end up with two historic versions of the deployment. First we’ll edit the manifest to change nginx image version from nginx:1.14.0 to nginx:1.15.05-alpine and then run the pipeline. Now under infrastructure view we’ll see that we have two historic versions of the deployment:

The first version deployed (V001) corresponds to nginx 1.14.0, while the second version (V002) corresponds to nginx 1.15.5-alpine.

Creating a rollback pipeline

Now we’ll create a rollback pipeline that triggers automatically when deploy-nginx pipeline fails:

This pipeline just contains a single stage of type Undo Rollout (Manifest) with the following configuration:

This configuration is important, the fields Namespace, Kind and Name are the coordinates of the Kubernetes deployment, and should match what is defined in the deployment manifest on the deploy pipeline. Revisions Back indicate how many deployment versions to rollback. In our historic running we currently have V001 and V002. Each time a deployment occurs, this version is automatically increased. Click here for detailed information on how Undo Rollout (Manifest) stage works.

Testing rollback logic

Now we’ll do a rollback test by changing the manifest, providing an invalid value for the namespace to force a failure. In deploy-nginx we’ll change the namespace to an invalid value:

We’ll run the pipeline again to see the rollback pipeline kick in automatically to rollback one version of the deployment:

If we go to the infrastructure view we can see that the active deployment was rolled back to the previous version:

In this case, V003 was generated by the rollback operation with the contents of a previous deployment. This is the summary of events:

Versions before failed deployment	Versions after failed deployment
V001 (nginx:1.14.0)	V002 (nginx:1.15.5-alpine)
V002 (nginx:1.15.5-alpine) → ACTIVE	V003 (nginx:1.14.0) → ACTIVE

Here we can see one gotcha of the Undo Rollout (Manifest) stage: it will rollback a version no matter if the failed pipeline deployed the application or not. So make sure that this stage only executes after a successful deployment of the artifact to rollback, and then some other logic marks the pipeline as failed, otherwise this may have the unintended consequence of rolling back a healthy version. One use case of this may be a pipeline that first deploys some configuration changes to Kubernetes, then deploys an updated version of a service. We can have a rollback pipeline execute to rollback the config changes if the main service deployment fails.

What happens if we don’t have a rollback pipeline? In the previous case where the namespace is invalid nothing is deployed and we can end up with a clean state, but if something goes wrong after a deployment we’ll end up with a failed server group. To see this in action let’s change nginx deployment to a non-existent version (1000), disable the rollback pipeline and see how ends up the infrastructure view after a failed deployment:

This is clearly not what we want, although the previous version (V003 - nginx:1.14.0) is still active and running, we have a failed server group (replica set) that cannot be started, so the rollback pipeline is still necessary to clean this up. In the next section we’ll look at a more advanced way for detecting when to apply a rollback.

Advanced configuration: Protecting against unwanted rollbacks

A deployment pipeline can fail because of many different reasons, but a rollback needs to be executed only in case something broken was deployed. One way to detect this is by using conditional checks with Spring Expressions to detect the case where something broken was deployed. In the rollback pipeline we’ll add two more stages of type “Check Preconditions” before the Undo Rollout stage:

This allow us to write expressions that continue the pipeline only if a broken deployment occurred. Every pipeline execution has a “context” object holding information about all aspects of execution. We can see this information clicking the “Source” link inside the details of a previous execution:

This opens a new tab showing a json similar to this:

We’re looking for two pieces on information inside this json:

A stage of type deployManifest in the trigger pipeline execution that failed.
This stage should have created an artifact.

The corresponding Spring Expression for the “Deploy stage failed?” stage is the following:

${ trigger['parentExecution']['stages'].?[type == 'deployManifest'][0]['status'].toString() == 'TERMINAL' }

And for the “Artifact created?” stage:

${ trigger['parentExecution']['stages'].?[type == 'deployManifest'][0]['outputs']['outputs.createdArtifacts'] != null }

In these expressions we are looking into the trigger pipeline execution for the first stage of type deployManifest whose status is TERMINAL (failed stage) AND that created some artifacts. Here’s the full reference documentation for pipeline expressions.

Also we make sure to uncheck “Fail Pipeline”, so that the pipeline doesn’t fail when there’s nothing to rollback, just stops execution. Using Spring Expressions we can also specify the coordinates of the artifact to rollback in Undo Rollout stage, taking them from the above outputs.createdArtifacts object:

Namespace: ${ trigger['parentExecution']['stages'].?[type == 'deployManifest'][0]['outputs']['outputs.createdArtifacts'][0]['location'] }

Name: ${ trigger['parentExecution']['stages'].?[type == 'deployManifest'][0]['outputs']['outputs.createdArtifacts'][0]['name'] }

Kind: ${ trigger['parentExecution']['stages'].?[type == 'deployManifest'][0]['outputs']['outputs.createdArtifacts'][0]['type'].substring('kubernetes/'.length()) }

This allow us to have an Undo Rollout stage a bit more resilient in case the artifact coordinates change:

Now we can repeat the tests and see that the rollback only executes when changing nginx version to a non existent version, but it doesn’t execute if the namespace is invalid, even though both scenarios fail the deployment pipeline.

Continuous-Deployment: Automating Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

API docs

People often ask how they can write scripts and use Spinnaker™ programmatically. Spinnaker is a collection of subservices that all expose a RESTful API. You can see a list (with descriptions) of all of the endpoints by navigating to:

http(s)://<your-gate-url>/swagger-ui.html

Note: you may need to append your url with the gate port, :8084.

You should see a screen that looks like:

You can click on the controller you are interested in to see endpoints related to it. You can even test out hitting these different endpoints right in the UI.

Auth

Being able to access the API when auth is enabled requires a certain configuration by your Spinnaker Administrator. They will need to enable programmatic access via mutual tls (x509 certs). Then you will need to use a cert when communicating with the API.

Continuous-Deployment: AWS Guides for Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Continuous-Deployment: Azure Guides for Armory CD and Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Continuous-Deployment: Canary Analysis in Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Continuous-Deployment: Cloud Foundry Best Practices

Mon, 01 Jan 0001 00:00:00 +0000

Buildpacks and Procfiles

Buildpacks typically define processes that run with some default values. These default values include things like starting process and commands. Some buildpacks, like Python, do not specify any default processes. During staging, the following error occurs if there are no processes defined: StagingError - Staging error: No process types returned from stager. Any buildpacks that do not specify a default process run into the same issue.

When developers use the COMMAND option in a manifest with the Cloud Foundry (CF) CLI, this command gets propagated to the process, and it gets used as expected. This is, in part, due to the fact that the CF CLI uses the v2 endpoints for most of the operations. Spinnaker, however, is configured to use various v3 endpoints. These allow more granular control over the deployment.

The solution is to use Procfiles if the buildpack does not include any default processes, like the Python buildpack. These Procfiles live at the root of an application and are evaluated during staging. This ensures that the staging does not fail because no processes are present.

Once you have a Procfile, additional customization can be added by using the COMMAND option in manifests. Any options supplied this way override the Procfile. But, start with a Procfile to ensure that stages do not fail because of missing commands.

Continuous-Deployment: Debian Packages

Mon, 01 Jan 0001 00:00:00 +0000

Why use Debian packages

While Spinnaker is flexible enough to use any dependency management system, it is predisposed to manage Debian packages due to its default settings with Rosco, Orca, and Jenkins.

Out-of-the-box settings for Spinnaker look for an archived package with a .deb suffix within Jenkins. Spinnaker also gets the version from the package and automatically appends it to the package name within Rosco. This makes it easy to specify your package in Rosco without the version number, mycompany-app. However, during the bake provisioning process Spinnaker installs the version that was specified by the Jenkins build: mycompany-app.3.24.9-3.
Debian packaging allows service teams to easily add their app specific configuration to common Packer templates. If you’re using any Debian-based system (Ubuntu, DSL, Astra, etc), you’ll likely be using Debian packages for your system configuration and dependency management. So it’s a natural extension to use a Debian package for your own applications. Using Debian packages helps reduce the variations in Packer templates, or variables passed to Packer templates, during the bake process.

Creating Debian packages

You can create a Debian package by using various open source packaging tools. If you’re using Java, use the OS Package library. You can also use the packaging tools provided by Debian.

Language	Tool	Package Types
Java	OS Package	deb/rpm
Python	stdeb	deb
Node	node-deb	deb
PHP	php-deb-packager	deb
Any	pkgr	deb/rpm
Any	fpm	deb/rpm/others

Example: Debian package with OSPackage Gradle plugin

Begin by creating a build.gradle. You also need to create a config/scripts/post-install.sh file in your project directory.

Below is an example of what a Gradle file might look like for an app that builds a war. This uses the gradle-ospackage-plugin package. Basic usage of the Deb Plugin in the Deb Plugin docs.

buildscript {
 repositories {
 jcenter()
 maven { url "https://plugins.gradle.org/m2/" }
 }
 dependencies {
 classpath 'com.netflix.nebula:gradle-ospackage-plugin:8.5.6'
 }
}

apply plugin: 'nebula.ospackage'

ospackage {
 packageName = "mycompanyname-service"
 version = "1.10.3"

 requires('nginx')

 postInstall file('config/scripts/post-install.sh')

 from('build/application.war') {
 into '/opt/application/'
 }
}

Then build your Debian package based on your Gradle build file:

gradle buildDeb

If the build succeeds then you should find a Debian package in the following path:

./build/distributions/mycompanyname-service.1.10.3_all.deb

Continuous-Deployment: Deploy a Docker Image to Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Kubernetes Deployments

Spinnaker delegates the deployment of containers to Kubernetes. Kubernetes then proceeds with a rolling update deployment which is effectively a rolling blue/green deployment. Pods are created in batches and when a pod is deemed healthy it starts receiving traffic.

Example

This example focuses on the manifest-based Kubernetes V2 provider.

We’ve defined a simple pipeline where we first define an artifact (a Docker image) with a version coming from a pipeline parameter. You’d usually get the image from a trigger via container registry or a Jenkins job with the same result.

The second step defines the deployment of that image. It’s a simple “Deploy (Manifest)” stage. Here we’re adding the static deployment manifest via a text field but you’d usually retrieve it as an artifact directly or via a Helm bake stage.

Note

For more information, watch this video about Deploying Helm Charts with Armory – Example of artifact promotion through environments managed by a single pipeline.

As a matter of fact, the deployment manifest is not entirely static: Spinnaker will replace the image name with the actual tagged name from the bound artifact.

Let’s see how a new version of the container gets deployed with this pipeline:

Can we change how containers are deployed?

In our example so far, we observed that we go down to 3 running pods and up to 5 non Terminating pods existing simultaneously. You actually have some control over how Kubernetes handles the pod creation. Two parameters will help you:

maxSurge sets a limit on how many pods can be created over the desired number of pods. Default is 25%.
maxUnavailable sets a limit on how many pods can be non-running. Default is also 25%.

Let’s modify our deployment manifest and ensure that we always have 4 pods running during deployment:

Let’s see how we deploy with our new configuration:

We keep 4 pods running throughout the deployment and never more than 5 non Terminating pods (4 pods + 25%) existing at any given time.

Continuous-Deployment: Deploy Applications to Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Overview of Spinnaker’s Kubernetes V2 Provider

The Kubernetes V2 provider is centered on delivering and promoting Kubernetes manifests to different Kubernetes environments. These manifests are delivered to Spinnaker using artifacts and applied to the target cluster using kubectl in the background. Currently, there is support to supply artifacts through Git, GCS and Google Pub/Sub. The V2 provider manages the version of the artifacts deployed to Kubernetes by appending a string such as v421 to the resource that was deployed to Kubernetes. This allows for easy rollbacks and management of resources.

Current limitations

The only supported services for artifact delivery are Github, GCS, or Google Pub/Sub. S3, SQS, and SNS are currently not supported.
Native Spinnaker deployment strategies such as blue/green, highlander, rolling blue/green deployment are not supported. If these strategies are desired consider using the deployment object in your manifests.
While you’re able to deploy all Kubernetes resource types, the V2 provider only considers containers and configMaps for binding to the deployed manifest. secrets and other resource types are coming soon.
You cannot manually trigger the pipeline, you have to use Github triggers.

Available manifest-based stages

There are 4 stages that are available:

Deploy Manifest - Uses kubectl apply to deploy a manifest. Spinnaker will wait until the manifest stabilizes in the Kubernetes cluster or reach the expected capacity on healthy pods.
Delete Manifests - Removes the manifests and their corresponding artifacts in kubernetes based on different types and labels.
Scale Manifests - Scales replica sets to a given static size.
Rollback Manifest - Allows rollback of a given Kubernetes artifact by a given number of versions. Helpful in cases of a failed deployment or failed manual QA.

Creating a Kubernetes V2 pipeline

Configuring the pipeline trigger

We’ll begin by creating a pipeline that is triggered from Kubernetes artifacts and delivered through Github. Below we’ll define two artifacts that will be deployed as Kubernetes manifests: deployment.yml and config-map.yml which are valid Kubernetes manifests. Make sure to select the source as Github.

After configuring the artifacts we’ll need to associate them with a Github trigger so the pipeline is triggered whenever there are modifications pushed to Github. For example, the pipeline below will only trigger when either deployment.yml or config-map.yml are modified and pushed to the repository. If the manifest isn’t modified it’ll use the latest version that was deployed.

Note: If you haven’t already, you’ll need to configure Github webhooks for your Spinnaker instance.

Configuring the config map manifest delivery

We’ll configure the configMap to be deployed first. Add a Deploy (Manifest) stage to your pipelines.

Once you’ve added the stage, select Artifact from the Manifest Source below and it will allow you to choose one of the expected artifacts that we configured in the previous section. Choose config-map.yml and hit save. Spinnaker will deploy the chosen artifact but append a version to the name of the artifact. For our example config map. So for the name k8-v2-config-map it will appear in the Kubernetes cluster with k8-v2-config-map-v001.

Configuring deployment manifest delivery

Next we’ll configure a new Deploy (Manifest) stage to deploy the deployment.yml manifest. This manifest references our config-map as a volume and it’s source will be replaced by the versioned artifact deployed in the previous step: k8-v2-config-map-v001. So if our deployment.yml contains the following:

volumes:
 - name: k8-config
 configMap:
 name: k8-v2-config-map

the name will be replaced with the properly versioned artifact:

volumes:
 - name: k8-config
 configMap:
 name: k8-v2-config-map-v000

Executing the pipeline

Your final pipeline should look similar to the one below.

In order to execute your pipeline the first time you’ll need to edit both the config-map.yml and deployment.yml, commit the changes to git and push the changes to Github. The pipeline should trigger and execute.

Continuous-Deployment: Manage Spinnaker Application Secrets Using HashiCorp Vault

Mon, 01 Jan 0001 00:00:00 +0000

Overview of storing secrets

Managing application secrets, such as database passwords and API keys, can be tricky. These secrets should not be saved into the application’s source code, but they do need to be made available to the application when it runs.

Several tools have been built to address this issue. In this document, we’re going to focus on how one might use HashiCorp Vault to act as our secrets management utility, and then set up our application and clusters to pull secrets from there.

In this guide, we’ll be using Vault as our example secrets manager, but a similar pattern should be applicable to similar tools, such as Amazon Secrets Manager.

This guide presumes you’ve already installed and configured Vault; if you are using Vault and have not done so, check Vault’s Getting Started guide.

Protecting application secrets

Armory recommends that you do not pass secrets through Spinnaker in plain text as this is not safe from a security standpoint. If your Spinnaker deployment gets breached and secrets were passed through it, intruders now have all applications secrets that were passed.

Instead, use a secret store and only pass the location of or references to the secret. The best practice for using application secrets is for the application to fetch the secret during application startup. For VMs, this is during the VM bootstrap or application startup process. For Kubernetes, you usually do this using an init-container, sidecar, or both.

For Vault, refer to the following resources about injecting application secrets securely into Kubernetes pods:

HashiCorp - Injecting Vault Secrets Into Kubernetes Pods via a Sidecar
Banzai Cloud - Inject secrets directly into Pods from Vault revisited
IT Next - Dynamic Vault Secrets — Agent Sidecar on Kubernetes

For more general information, see the following pages about AWS Secrets Manager and Vault:

GoDaddy Engineering blog about their use of Kubernetes External Secrets
GoDaddy GitHub page about their open sourced implementation of Kubernetes External Secrets

Basic outline

The basic plan here is to allow developers to write code that references “secrets” but which aren’t checked into the codebase. Often this will be via a configuration file, or environment variables. For testing purposes, this data may be changed out; developers may have their own personal set of “secrets” for their development environment. The code just relies on the file or env vars being set properly.

An operations person (or team) is then responsible for the actual production systems, including the passwords used. It’s their job to maintain those secrets and make sure they are set correctly in the production environment.

The plan, then, is to use a Kubernetes secret (which the developers should not have access to) to store the Vault auth token (managed solely by an Operations person). The app can use the auth token to retrieve the application secrets from Vault at runtime. Spinnaker acts as a limited-use interface such that the developer can kick off deployments to production, but may not have actual direct access to the credentials for production.

The following diagram attempts to outline this design, showing the barrier between the developer and the actual secrets their code uses:

NOTE: Vault (and other tools) have taken this security a step further by connecting the secrets service with the backend systems such that the actual password being used is cycled automatically and is never actually known to any specific person. If your environment permits this setup, it’s definitely a more secure mechanism of password management, but it’s also a much more complicated topic, and may not be possible depending on what services you are attempting to secure.

Operations-side configuration

The first step to securing the secrets is to create a policy in Vault, apply the policy to the Vault path(s) that contain the secrets, and create a service account that is associated with that policy to access it. The authorization token for this account is what we’ll push into the deployed pods via a Kubernetes Secret, and will allow that pod to request the current secrets from Vault.

Set up a secret path

Within Vault, set up your secrets path (such as /secrets or /prod/secrets or whatever works for you). Don’t put any secrets in yet; you’ll want to secure the path first (next step)

Secure path with a policy

Create a policy by name that has access to the path you created. By default, paths are open to anyone authenticated, but as soon as you add a policy to the path, it’s locked down to only those accounts with that policy.

You can read more about Vault policies here.

Create service account token

Use vault token create to create a token attached to the policy you just created (reference). Your app, using this token, should then have access to the actual secrets.

You can now put in your production secrets into that “bucket” safely.

Create a Kubernetes secret

Using kubectl you’ll want to create a Secret within your namespace that contains this token. The Kubernetes documentation discusses in detail the different ways you may create a secret. Also note this documentation discusses how to then mount/use the secret within the pod.

Bringing it all together

Now that your ops team has secured your secrets in Vault, and set up the Kubernetes secret, the next step is to have your Kubernetes deployment mount the auth token secret, and then finally, arrange for your app to use Vault to retrieve the actual production secrets.

Configure your manifest

To make the Vault token available to your application, you’ll need to mount the secret within your manifest. The Kubernetes documentation shows a number of ways in which you can configure your pods to get access to the Kubernetes secrets either as a mounted volume or as an environment variable.

This manifest reference may be visible to developers, but provided developers do not have direct access to the Kubernetes cluster, they won’t be able to retrieve the actual token from the secret; Spinnaker allows the developers to still manage the deployments to production, but not to retrieve the secret itself.

Configure your application

Now your code (or perhaps just a bootup script) can be written/configured to grab the Vault token that was mounted in your Manifest, and, combining that with the Vault paths, can retrieve your secrets safely.

One way to do this is with a Kubernetes init container intended to run the Vault image (which provides the vault command line tool) to log in, retrieve the secrets and write them to disk, where your application can pick them up.

Alternatively, your code may be able to access Vault directly with the access token, storing the secrets retrieved in application memory, where it’s not readily accessible by someone who manages to gain access to the container’s shell. Hashicorp provides documentation on the Vault API

Alternatives

We’ve only scratched the surface of ways in which app secrets can be managed and then merged at deploy time by Spinnaker. We’ve used Vault as an example because it’s being used more and more by our customers, but there are other products (like Amazon Secrets Manager) and, of course, homebrewed solutions.

In all cases, Spinnaker provides a terrific way to give developers the ability to manage their own deployments while still securing production passwords separately.

Continuous-Deployment: Migrating Armory CD from Operator to Kustomize Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Migrating Armory CD from Operator to Kustomize Deployment

Introduction

This document provides step-by-step instructions for migrating your Spinnaker installation from using the Operator deployment method to a native Kubernetes deployment using Kustomize. This approach gives you more direct control over your Spinnaker resources and removes the dependency on the Operator.

Important

Please thoroughly test this migration in a non-production environment before deploying to production.

Prerequisites

Kubectl command-line tool installed and configured to access your cluster
Basic understanding of Kubernetes resources (deployments, services, configmaps)
Access to the current Spinnaker namespace

Migration Process Overview

Download current configuration files and Kubernetes resources
Set up Kustomize structure for native deployment
Remove Operator ownership from services
Scale down the Operator
Deploy using Kustomize
Validate the deployment
Remove Operator and CRDs (after confirming stability)

Step 1: Download Current Configuration

The script provided below will download:

All configuration files located in /opt/spinnaker/config from each service
All deployment, service, and statefulset YAML files for each service

How to Use the Download Script

Save the script at the bottom of this document to a file named download_spinnaker_configs.sh
Make the script executable: chmod +x download_spinnaker_configs.sh
Run the script with your Spinnaker namespace: ./download_spinnaker_configs.sh your-spinnaker-namespace
The script will create an operator-migration directory containing all needed files

What Gets Downloaded

Deployments: YAML files for each service deployment
Services: YAML files for all Spinnaker services
StatefulSets: YAML files for any statefulsets (like front50)
Configuration Files: All files from /opt/spinnaker/config in each pod

Step 2: Set Up Kustomize Structure

Create a Kustomize directory structure for your Spinnaker deployment:

Move the downloaded deployments and services to their respective directories
Create configmaps from the downloaded configuration files
Set up the kustomization.yaml files

Tip

You can use the GitHub - spinnaker/spinnaker-kustomize: Spinnaker installation via kustomize as a reference for Kustomize structure

Step 3: Remove Operator Ownership from Services

This step detaches the Operator’s control while keeping services running.

Identify resources owned by the Operator: kubectl get all -n your-spinnaker-namespace -o json | jq '.items[] | select(.metadata.ownerReferences[]? | .apiVersion=="spinnaker.armory.io/v1alpha2" and .kind=="SpinnakerService") | {name: .metadata.name, kind: .kind}'
Remove ownership references using patch commands: kubectl patch deployment spin-deck -n your-spinnaker-namespace --type json -p='[{"op": "remove", "path": "/metadata/ownerReferences"}]'
Repeat for all resources with Operator ownership

Note
This breaks the connection between the Operator and services but keeps everything running

Step 4: Verify Ownership Removal

Confirm that no resources are still owned by the Operator: kubectl get all -n your-spinnaker-namespace -o json | jq '.items[] | select(.metadata.ownerReferences[]? | .apiVersion=="spinnaker.armory.io/v1alpha2" and .kind=="SpinnakerService") | {name: .metadata.name, kind: .kind}' The command should return empty if all ownership references have been removed.

Step 5: Extra Precautions Before Deployment

Compare current resources with your Kustomize configurations: kubectl diff -f <(kustomize build ./overlays/prod)

Review the differences carefully. Look for:

Immutable field changes (might require special handling)
Configuration changes that could affect service behavior
Missing resources that should be included

Perform a Dry Run

Test your deployment without actually applying changes: kubectl apply --dry-run=client -f <(kustomize build ./overlays/prod)

Step 6: Scale Down the Operator

Prevent the Operator from interfering with your deployment: kubectl scale deployment spinnaker-operator -n your-spinnaker-namespace --replicas=0

Step 7: Deploy Using Kustomize

Apply your Kustomize configurations:

kubectl apply -f <(kustomize build ./overlays/prod)

Step 8: Validate and Monitor

Check that all pods are running: kubectl get pods -n your-spinnaker-namespace
Verify Spinnaker services are accessible:
- Access the Spinnaker UI
- Test a simple pipeline
- Check integrations are working
Monitor the environment for stability over the next few days

Step 9: Remove Operator and CRDs

Once stability is confirmed (at least 24 hours later):

Remove the Operator CRDs: kubectl delete crd spinnakerservices.spinnaker.armory.io
Remove the Operator deployment if still present: kubectl delete deployment spinnaker-operator -n your-spinnaker-namespace

Rollback Plan

If issues arise during migration:

Scale up the Operator: kubectl scale deployment spinnaker-operator -n your-spinnaker-namespace --replicas=1
Reapply the previous SpinnakerService resource: kubectl apply -f original-spinnakerservice.yaml
Allow the Operator to reconcile and restore the previous state

Download Script

#!/bin/bash

# Script to download ONLY files from /opt/spinnaker/config in Spinnaker pods

set -e

if [ -z "$1" ]; then
 echo "Please provide a namespace"
 echo "Usage: $0 <namespace>"
 exit 1
fi

NAMESPACE=$1
OUTPUT_DIR="operator-migration"

echo "Creating output directory: $OUTPUT_DIR"
rm -rf "$OUTPUT_DIR"
mkdir -p "$OUTPUT_DIR"

# 1. First download deployments and services
echo "Downloading Kubernetes deployments and services..."

# Download Deployments
echo "Downloading deployments..."
mkdir -p "$OUTPUT_DIR/deployments"
kubectl get deployments -n "$NAMESPACE" -o name | while read -r deployment; do
 deployment_name=$(echo "$deployment" | cut -d/ -f2)
 echo "Downloading deployment: $deployment_name"
 kubectl get deployment "$deployment_name" -n "$NAMESPACE" -o yaml > "$OUTPUT_DIR/deployments/$deployment_name.yaml"
done

# Download Services
echo "Downloading services..."
mkdir -p "$OUTPUT_DIR/services"
kubectl get services -n "$NAMESPACE" -o name | while read -r service; do
 service_name=$(echo "$service" | cut -d/ -f2)
 echo "Downloading service: $service_name"
 kubectl get service "$service_name" -n "$NAMESPACE" -o yaml > "$OUTPUT_DIR/services/$service_name.yaml"
done

# Download StatefulSets if any
echo "Checking for statefulsets..."
if kubectl get statefulsets -n "$NAMESPACE" 2>/dev/null | grep -q .; then
 mkdir -p "$OUTPUT_DIR/statefulsets"
 kubectl get statefulsets -n "$NAMESPACE" -o name | while read -r statefulset; do
 statefulset_name=$(echo "$statefulset" | cut -d/ -f2)
 echo "Downloading statefulset: $statefulset_name"
 kubectl get statefulset "$statefulset_name" -n "$NAMESPACE" -o yaml > "$OUTPUT_DIR/statefulsets/$statefulset_name.yaml"
 done
fi

# 2. Now download files from /opt/spinnaker/config
echo "Downloading files ONLY from /opt/spinnaker/config..."

# Get all pods
PODS=$(kubectl get pods -n "$NAMESPACE" -o name | cut -d/ -f2)
for POD in $PODS; do
 SERVICE=$(echo "$POD" | sed -E 's/([a-z-]+)-[0-9a-z-]+.*/\1/')

 echo "Processing pod: $POD (service: $SERVICE)"
 mkdir -p "$OUTPUT_DIR/$SERVICE"

 # Check ONLY for /opt/spinnaker/config
 if kubectl exec -n "$NAMESPACE" "$POD" -- ls -la /opt/spinnaker/config &>/dev/null; then
 echo "Found /opt/spinnaker/config directory in $POD"

 # List all files first
 CONFIG_FILES=$(kubectl exec -n "$NAMESPACE" "$POD" -- find /opt/spinnaker/config -type f 2>/dev/null)
 if [ -z "$CONFIG_FILES" ]; then
 echo "No files found in /opt/spinnaker/config for $POD"
 continue
 fi

 echo "Found $(echo "$CONFIG_FILES" | wc -l | tr -d ' ') files in /opt/spinnaker/config for $POD"

 # Download each file
 for FILE in $CONFIG_FILES; do
 FILENAME=$(basename "$FILE")
 echo "Downloading $FILENAME from $POD"

 FILE_CONTENT=$(kubectl exec -n "$NAMESPACE" "$POD" -- cat "$FILE" 2>/dev/null)
 if [ $? -eq 0 ] && [ -n "$FILE_CONTENT" ]; then
 echo "$FILE_CONTENT" > "$OUTPUT_DIR/$SERVICE/$FILENAME"
 echo "Saved $FILENAME to $OUTPUT_DIR/$SERVICE/$FILENAME"
 else
 echo "Failed to download $FILENAME or file is empty"
 fi
 done

 # Check if we downloaded any files
 if [ -z "$(ls -A "$OUTPUT_DIR/$SERVICE" 2>/dev/null)" ]; then
 echo "No files were successfully downloaded from $POD"
 else
 echo "Successfully downloaded $(ls -1 "$OUTPUT_DIR/$SERVICE" | wc -l | tr -d ' ') files from $POD"
 fi
 else
 echo "No /opt/spinnaker/config directory found in $POD"
 fi
done

echo "======================================"
echo "Download completed. Files saved to: $OUTPUT_DIR"
echo "Summary of downloaded files by service:"

# Generate summary
for DIR in $(find "$OUTPUT_DIR" -mindepth 1 -maxdepth 1 -type d | sort); do
 SERVICE=$(basename "$DIR")
 FILE_COUNT=$(find "$DIR" -type f | wc -l)

 echo "- $SERVICE: $FILE_COUNT files"
 if [ "$FILE_COUNT" -gt 0 ]; then
 ls -1 "$DIR" | sort | while read -r file; do
 echo " - $file"
 done
 fi
done

echo "Script execution complete!"

Conclusion

By following these steps, you’ll successfully migrate from the Spinnaker Operator to a native Kubernetes deployment using Kustomize. This approach gives you more direct control over your Spinnaker resources and eliminates dependency on the Operator. If you encounter any issues during the migration process, please submit a support ticket for assistance.

Continuous-Deployment: Spinnaker Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Overview of Spinnaker pipelines

Pipelines are a combination of stages that enable very sophisticated coordination and branching. They are the key to orchestrating deploys in Spinnaker and each one is specific to an application. To see an application’s pipelines, select ‘Applications’ from Spinnaker’s top navigation bar, click on an application’s name, and then press the ‘Pipelines’ tab. The result from a pipeline running is called an execution.

Take this screenshot for example:

There is a pipeline called ‘Deploy’ with two executions, both labeled ‘Manual Start’. The top execution is marked as ‘Succeeded’ while the bottom is marked as ‘Cancelled’.

For more information on creating bake and deploy pipelines on AWS, checkout the baking and deploying guides.

Manual execution

You can re-run an execution by pressing the ‘Start Manual Execution’.

If your pipeline has a Jenkins’ trigger, you can select which Jenkins’ build number to use for running the pipeline.

The artifacts produced by the build you select will be used in the pipeline. If your pipeline bakes an image, a cached image will be used if available. To force a rebuild, make sure you specify such before pressing the ‘Run’ button.

Enabling notifications

Spinnaker supports several methods of notification. Notifications can be made when a pipeline runs, succeeds, or fails. You can be contacted via SMS, email, slack, hipchat, and/or pagerduty. Each of these outlets need to be configured within Spinnaker by your Spinnaker Administrator. Once it is configured, you can enable it in your pipeline.

To enable it, navigate to the configuration screen for your pipeline. Make sure you have the ‘Configuration’ stage selected. Scroll down to the ‘Notifications’ section.

Press ‘Add Notifications Preference’. For example’s sake, I have selected to receive a notification via Slack in the #engineering channel whenever my pipeline fails.

Finally press ‘Update’ to finish. Don’t forget to press ‘Save Changes’ on your pipeline configuration.

Pipeline JSON

Pipelines are represented as JSON behind the scenes. The JSON is interpreted and displayed to you in the UI. However, sometimes it is helpful to view or edit the JSON directly. To access the JSON:

Click ‘Configure’ on your pipeline:

Press the ‘Pipeline Actions’ button in the upper right to display a dropdown menu.

There are two JSON related options on this dropdown menu:

a. If you select ‘Edit as JSON’ then you should see something like:

From this screen you can edit the JSON directly. Remember to always save your changes, so they will be used in the next execution of your pipeline.

b. If you select ‘Show Revision History’ then you should see something like:

You can select different revisions using the dropdown menu labeled ‘Revision’ in the top left. You can compare it to different versions using the ‘compare to’ dropdown menu in the upper right.

Troubleshooting

Hanging or timed out pipelines

A lot of the time pipelines hang because of a misconfigured stage. This is a common occurrence when a server group does not complete its deploy because the deployed instances never pass the health check. This happens both when the health check is misconfigured and/or when the image doesn’t bake as expected. These two areas should be investigated first. For more information you can see the troubleshooting topic in the deployment guide.

Continuous-Deployment: Spinnaker Best Practices

Mon, 01 Jan 0001 00:00:00 +0000

Configuration management

Baking configuration

It is common practice to bake the bulk of your application’s configuration into your AMI. Secrets should not be included. Configurations that frequently change and service discovery is often managed by another system. Some common system choices include but are not limited to: Consul, Archaius, Zookeeper, and etcd.

If you are running the default installation of Spinnaker, then every AWS instance will have details about its environment/stack stored in /etc/default/server-env. This can be used to load the correct configuration.

Service discovery

While it is not feasible for every type of application, using a load balancer can greatly simplify things. Spinnaker gives load balancers first-class citizenship and will not need additional integrations to work right out of the box.

For applications and workloads that cannot utilize a load balancer, the tools listed above may help.

Secret management

Managing Secrets is a major part of deploying software. Spinnaker does not directly do this for you. However, there are still some guiding principles to help you along the way.

Do not to bake Secrets into your images. If you have the default installation of Armory, then you can find /etc/default/server-env on your instance. This file will have information (in the format of key-value pairs) about the environment and stack you are running on. You can use this to load the correct Secrets.

Loading Secrets will have to be done at runtime since Secrets are not baked into images.

Specifically for Secure Sockets Layer (SSL), it can be beneficial to terminate SSL at the Elastic Load Balancer (ELB) whenever feasible. Amazon has the Key Management Service (KMS) for this purpose. If you need to handle certificate management at the application level, you might want to check out Netflix’s Lemur project.

Some other general purpose Secret Management tools include:

Using Vault With Spinnaker on AWS

Issuing a Vault token with AWS is an automated process that uses AWS as a trusted third party to initiate authorization.

One piece of “dynamic metadata” available to the EC2 instance, is the instance identity document, a JSON representation of a collection of instance metadata. AWS also provides PKCS#7 signature of the instance metadata document, and publishes the public keys (grouped by region) which can be used to verify the signature.

You can issue an authentication by issuing the following:

vault auth-enable aws-ec2

And then proceeding with any additional vault commands and execution that needs to happen. These steps need to happen at startup you’ll want add this script as a base 64 encoded users-data in your deployment steps.

Isolate delivery pipelines from integration

There are several ways to trigger a deployment pipeline. However, depending on the asset you are delivering, some methods are easier to work with than others.

Docker Images

It is best to have Spinnaker trigger off of a push to a Docker registry instead of triggering off of a GitHub push or Jenkins job. Doing so frees up the development teams to restructure their build systems and validation as they desire because everything in the delivery process can remain the same as long as the Docker image makes it up to the repo.

Sometimes you may need extra information if you’re triggering off of Docker images. For instance you might want to release anything on the master branch to production, but release any other branch to the staging area. In order to do so, put the extra information into the tag, and the pipeline triggers in Spinnaker can use regular expression matches on the tag name in Docker to determine which pipeline to execute.

Continuous-Deployment: Use the Spring Expression Language (SpEL) in Spinnaker Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Overview of SpEL in Spinnaker

The Spring Expression Language is a powerful tool that you can use to add logic and decision-making to your pipelines. While a lot of the time you will probably use it to evaluate variables, it can do a lot more. You can write straight Java/Groovy into it. This means you can do transformations, filters, maps, etc. You can use it to branch your pipeline into different directions.

Some of the most common uses include:

Getting build information from Jenkins
Passing image names from one stage to another
Retrieving a user’s manual judgement response

Take a look at the open source Spinnaker Pipeline Expressions Guide for a detailed overview of dynamically setting and accessing variables during pipeline execution.

Common techniques

Dynamically defining User-Data

If you are creating a deployment configuration for AWS, Spinnaker gives you the option to provide user-data. As you can see here:

The user-data field needs to be base64 encoded. It is possible to create this dynamically with the built in expression language. To do this you can use the ${ #toBase64() } command. For example, You can pass the build number to the user-data via:

Examples

You can find the expression language used in the examples within the baking images, deploying, working with Jenkins and finding images guides.

Troubleshooting

Sometimes using the expression language doesn’t go as anticipated. Here are some of the common issues:

Sometimes the UI doesn’t display the autocomplete popup menu. This is because it doesn’t have the context to do so. Try filling in as much of the details as you can for the stages in your pipeline, then run the pipeline. After it has ran (it’s okay if it failed), go back to where you were trying to use the expression language and see if the autocomplete popup menu is displayed.

Expression doesn’t get evaluated

Sometimes your expression is just printed out plainly and not evaluated. Usually this happens when the expression is invalid and/or can not be resolved and does not mean that Spinnaker isn’t trying to evaluate it. Try double checking that what you are referencing does exist. To check that it does exist, go to your pipeline’s execution details and click the ‘Source’ link in the very bottom right hand corner.

For example:

You should see a page full of JSON. It doesn’t print in a very readable format, so you may want to copy and paste it into a text editor or another tool that will help you read it (I usually just curl this URL and pipe it to jq). You can navigate to the JSON field ‘stages’ for a list of stages in your pipeline. These stages are not necessarily in order. In the stage you’ll see another field called ‘context’. This is the information avaliable to the expression language. Make sure what you are referencing is in the context of the appropriate stage.

Testing your pipeline expressions

The best way to test a pipeline expression is to create a sample pipeline and see if your SPEL expression does what you expect it to do before you add it to your production pipeline. Another feature that is helpful for writing SPEL is if the pipeline has ran in the past, the UI will show you autocomplete options based on the previous execution.

Continuous-Deployment: Use Docker Images in Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Triggering a pipeline with a Docker Registry update

Before you start, you need to configure a Docker registry. If you don’t see your Docker registry as an option, or you’re missing your organization or image in the UI, you’ll need to double-check your Spinnaker is configured to use that registry (and/or repository).

To add a Docker trigger to your pipeline, go to your configurations stage and select “Add Trigger”, then select “Docker Registry” from the Type dropdown menu. You should then be able to select the Registry to use, and find your Organization(s) and Image(s).

The Tag field is optional; if left empty, any new Docker image posted to the registry (for that image) will trigger the pipeline. You can enter a regular expression here to limit triggering to tag pattern names (for example, master-.* will only trigger when a new image with a tag starting with “master-” is uploaded).

A Regular Expression (“regex”) is similar to, but different than, a wildcard. You may be familiar with wildcards on your command line, where good* would list all files that start with “good”. In regexes, a * character simply “matches 0 or more of the preceding character” – good* would then match “goo”, “good”, and “goodddddd”, but wouldn’t match “goodbar”. If you need help with the regex syntax, this is a good introduction.

Referencing the new image

When a new Docker image (that matches the Tag pattern, if set) is detected, the Docker Registry trigger will fill in some context, which can be used in SpEL Expressions elsewhere in the pipeline (for example, in a Kubernetes manifest).

If you click on the Source link for the pipeline that was triggered, you can find the trigger section of the JSON. Below is a partial block of JSON as an example of the fields that will be filled in.

 "trigger": {
 "account": "my-docker-registry",
 "artifacts": [
 {
 "name": "index.docker.io/armory/demoapp",
 "reference": "index.docker.io/armory/demoapp:master-29",
 "type": "docker/image",
 "version": "master-29"
 }
 ],
 "repository": "armory/demoapp",
 "tag": "master-29",
 "type": "docker",
 },

You can reference the Docker tag that triggered the pipeline with the expression ${trigger['tag']}, which may be all you need, as in this image spec line from a Kubernetes manifest:

- image: "docker.io/armory/demoapp:${trigger['tag']}"

Continuous-Deployment: Use GitHub in Spinnaker Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Trigger a Pipeline with a GitHub commit

Before you start, you’ll need to configure your GitHub repositories. You’ll be able to configure a pipeline trigger without having configured your GitHub webhook, but the trigger won’t fire until Spinnaker can receive those calls from GitHub.

To add a GitHub trigger to your pipeline, go to your configurations stage and select “Add Trigger”, then select “Git” from the Type dropdown menu. Then select “github”. You can then enter your organization (ex. “armory”) and the repository name to monitor (ex. “demoapp”). Branch and Secret are optional, although it’s recommended you set Branch to whatever the name of your production branch is (usually master) so you only trigger pipelines when code is committed to the production branch. The Branch field also supports regular expressions, so you can limit the trigger to several branches with common patterns or partial matches.

A Regular Expression (“regex”) is similar to, but different than, a wildcard. You may be familiar with wildcards on your command line, where good* would list all files that start with “good”. In regexes, a * character simply “matches 0 or more of the preceding character” – good* would then match “goo”, “good”, and “goodddddd”, but wouldn’t match “goodbar”. If you need help with the regex syntax, this is a good introduction.

Continuous-Deployment: Use GitHub Artifacts in Spinnaker Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Pulling a Kubernetes Manifest from Github

Under “Expected Artifacts” in your pipeline, create an artifact of type “Github”.
Specify the “file path” as the path within the repository to your file. For example, if your manifest is at demo/manifests/deployment.yml in the Github repository orgname/reponame , specify demo/manifests/deployment.yml.
Check the “Use Default Artifact” checkbox.
In the “Content URL”, provide the full path to the API URI for your manifest. Here are some examples of this:
- If you’re using SaaS Github (www.github.com), the URI is generally formatted like this: https://api.github.com/repos/<ORG>/<REPO>/contents/<PATH-TO-FILE>.
  - For example: https://api.github.com/repos/armory/demo/contents/manifests/deployment.yml
- If you have an on-prem Github Enterprise, then the URI may be formatted like this: https://<GITHUB_URL>/api/v3/repos/<ORG>/<REPO>/contents/<PATH-TO-FILE>.
  - For example: http://github.customername.com/api/v3/repos/armory/spinnaker-pipelines/contents/manifests/deployment.yml
Create a “Deploy (Manifest)” stage. Rather than specifying the manifest directly in the UI, under the “Manifest Source” specify “Artifact”, and in the “Expected Artifact” field, select the artifact you created above.
If you have multiple Github Accounts (credentials) added to your Spinnaker cluster, there should be a dropdown to select which one to use.

Continuous-Deployment: Use Jenkins in Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Triggering a pipeline with Jenkins

Before you start, you need to configure Jenkins. If you don’t see Jenkins as an option, or you’re not seeing the correct master/job combination in the UI, you’ll need to double-check your Spinnaker is configured to use that resource.

To add a Jenkins trigger to your pipeline, go to your configurations stage and select “add trigger”, then select “Jenkins” from the Type dropdown menu. Select a Master from the Master category list and then select a Job to trigger from the pipeline.

Note: Make sure you archive your package files and your properties file in Jenkins.

Property File

The property file is a way to transfer information about your build from Jenkins to Spinnaker. The file needs to be archived by the Jenkins job and should contain key value pairs.

As an example, if you had your Jenkins job create and archive a file named build.properties which looks like:

COMMITER_NAME=andrew
BRANCH_NAME=mybranch
CONFIG=config-3059cad.tar.gz

Then in the property files field in the Spinnaker Jenkins trigger, fill it in with build.properties.

Now that those variables are in Spinnaker, we can access them elsewhere in our pipeline by using the built-in Spinnaker expression language.

In any given stage we can use the expression ${ trigger.properties['BRANCH_NAME']} to access the property value of the variable named BRANCH_NAME.

Note: For more elaborate instructions on expression language, please refer to the Spinnaker Expression Language Guide.

How to trigger a Jenkins job through Spinnaker

Step 1: In pipelines, click on ‘Add stage’.

Step 2: Select ‘Jenkins’ from Type. Name the Stage Name and Depends On if you need it.

Step 3: Configure your Master and Job. If your Job is parameterized then Spinnaker will display a Parameters form for your input.

Step 4: The Property File input works the same way as above - however you cannot use the same expression above to access it. Instead, you would access it with ${ #stage('Jenkins')['context']['BRANCH_NAME'] }, supposing you wanted to access the BRANCH_NAME variable.

Example

In this example, we will create a pipeline with a Jenkin’s job stage and a manual judgement that repeats back the value of a property in a properties file from the Jenkins’ job.

First we navigate to my Jenkins Master and create a new Jenkins 2.0 Pipeline Job. As you can see in the image below, this job creates and archives a file named build.properties which contains the key value pair KEY=VAL.

Then we go back to Spinnaker and create a new pipeline. We will skip over the Jenkins trigger for this example and create a Jenkins stage. Make sure to name the stage Jenkins. Make sure to type build.properties into the Properties field so that Spinnaker know where to source the property values.

To demonstrate accessing our properties file, let’s create a new Manual Judgement stage. In the Instructions text box input:

{ #stage('Jenkins')['context']['KEY'] }

Spinnaker has added the key value pairs from build.properties in the Jenkins’ job to the context of the Jenkins stage. The above expression allows us to access that information.

Don’t forget to press the save button in the lower right corner. Your pipeline configuration should look like:

Finally, navigate back to the pipeline execution screen an start a manual execution of the pipeline you just configured.

When the execution gets to the manual judgement stage, you should see the word VAL in the intructions text area.

Running scripts on Spinnaker

You can also run arbitrary scripts on Spinnaker (behind the scenes it is pushing this work off to a Jenkins master). Simply select the Script type stage while configuring a pipeline. You should see a screen like:

Continuous-Deployment: Use Kustomize for Manifest-Based Kubernetes Deployments in Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

Overview of Kustomize

Kustomize is a tool that lets you create customized Kubernetes deployments without modifying underlying YAML configuration files. Since the files remain unchanged, others are able to reuse the same files to build their own customizations. Your customizations are stored in a file called kustomization.yaml. If you need to make configuration changes, the underlying YAML files and kustomization.yaml can be updated independently of each other. To learn more about Kustomize and how to define a kustomization.yaml file, see the following links:

In the context of Spinnaker, Kustomize lets you generate a custom manifest, which can be deployed in a downstream Deploy (Manifest) stage. This manifest is tailored to your requirements and built on existing configurations. Spinnaker uses the latest non-kubectl version of Kustomize.

Using Kustomize

Kustomize works by running kustomize build against a kustomization.yaml file located in a Git repository. This file defines all of the other files needed by Kustomize to render a fully hydrated manifest.

Kustomize

Requirements

Kustomize requires the git/repo artifact type.

Configure git/repo artifact with Operator

Add the following settings to the SpinnakerService manifest:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 artifacts:
 gitrepo:
 enabled: true
 accounts:
 - name: gitrepo
 username: # Git username.
 token: encrypted:k8s!n:spin-secrets!k:github-token # Your github access token from a K8s secret (here secret='spin-secrets', key='github-token')

Build the Pipeline

For this example, you can use the helloWorld example from the Kustomize public repository.

Step 1 - Add an Expected Artifact

Add a git/repo Expected Artifact in the Configuration section:

Account (Required): The git/repo account to use.
URL (Required): The location of the Git repository.
Branch (Optional): The branch of the repository you want to use. Defaults to master.
Subpath (Optional): By clicking Checkout subpath, you can optionally pass in a relative subpath within the repository. This provides the option to checkout only a portion of the repository, thereby reducing the size of the generated artifact.

In order to execute the pipeline manually, it is necessary to select Use Default Artifact and also fill the fields (same information above).

Step 2 - Add a Bake (Manifest) Stage

Add a Bake (Manifest) stage and choose the Render Engine KUSTOMIZE. Then, select the Expected Artifact you created in step 1 and specify the path for the kustomization.yaml file.

Step 3 - Produce the Artifact

Spinnaker returns the manifest in a Base64 encoded file, so it is necessary to Produce a single Base64 Artifact in this Bake (Manifest) stage:

Step 4 - Deploy

Add a Deploy (Manifest) stage. Make sure to select the Manifest Source: Artifact and select the Base64 Artifact produced by the Bake (Manifest) stage.

Note: As we are deploying a manifest without a specified namespace, we need to override the namespace by checking the “Override Namespace” option in the deployment stage.

Run the Pipeline

After you execute the pipeline, you can see the manifest generated in YAML format by clicking on the Baked Manifest YAML link:

Continuous-Deployment: Use Max Concurrent Pipeline Executions

Mon, 01 Jan 0001 00:00:00 +0000

What max concurrent pipeline executions does

Max concurrent pipeline executions allows you to throttle the number of maximum parallel pipeline executions, so you can configure your pipeline to maximize deployment frequency without encountering any performance degradations or timeouts. This effectively allows you to continue accelerating your deployment frequency and velocity.

If max concurrent pipeline executions is enabled, pipelines queue when the max concurrent pipeline executions is reached. Any queued pipelines are allowed to run once the number of running pipeline executions drops below the max. If the max is set to 0, then pipelines do not queue.

How to enable and use max concurrent pipeline executions

You can find the max concurrent pipeline executions feature in the your pipeline’s Configuration section. It is disabled by default. Uncheck the Disable concurrent pipeline executions (only run one at a time). option. Then enter an integer in the Maximum concurrent pipeline executions field and save your changes.

Continuous-Deployment: Use the Evaluate Artifacts Stage

Mon, 01 Jan 0001 00:00:00 +0000

Before you begin

The Evaluate Artifacts Stage is a plugin for Armory Continuous Deployment. You must have the plugin enabled. For more information, see Enable the Evaluate Artifacts Stage Plugin.

Using the stage

At a high level, using this stage involves the following steps:

In the Armory Continuous Deployment UI, navigate to the pipeline you want to modify.
Add any parameters you might use within the pipeline.
Add the stage called Evaluate Artifacts stage to your pipeline.
Under the Evaluate Artifacts Configuration, select Add Artifact to the stage. A window appears where you can enter your artifact definition.
Enter your artifact definition:
- Provide a descriptive name for the artifact.
- Enter the artifact definition in the Contents section. When entering the definition, you can use a SpEL expression to parameterize it to use the input from step 2.
Save your changes.

When the pipeline runs, the UI prompts the user to enter a value for the parameter, and you can now reference this artifact in subsequent stages or other pipelines.

Example

This example combines Armory’s Terraform Integration and the Evaluate Artifact stage to insert values a user inputs into artifacts when the pipeline runs. The example uses the Evaluate Artifact stage to insert the app name, namespace and number of replicas into the sample Terraform script. Then, the Terraform Integration deploys the infrastructure (NGINX in this example).

To follow along with this example, you need to have the Terraform Integration stage enabled and have permission to provision a Kubernetes cluster and any associated resources.

In the Armory UI, create a new pipeline.
For the Configuration phase of the pipeline, add the following parameter:
- nameAndSpace
- replicas and set the Default Value to 2.
For both parameters, set them to required. Optionally, pin the parameters.
Save your changes.
Add a stage to the pipeline and set the type to Evaluate Artifacts.
Provide a name for the stage.
Under Evaluate Artifacts Configuration, add an artifact:

This example creates a tfvar file, so name it testvariables.tfvar.
For the Contents section, insert the following snippet:
```
namespace="${#readJson(parameters['nameAndSpace'])['space']}" # SpEL expression
deployName="${#readJson(parameters['nameAndSpace'])['name']}" # SpEL expression
replicas=${parameters.replicas}
```
All three fields were added previously as parameters for the pipeline. The namespace and deployName fields are SpEL expressions that evaluate JSON that you provide when the pipeline runs. replicas accepts an integer value for the number of replicas you want. You can provide a value when the pipeline runs or use the default value of 2.

Create the artifact.
Wait until the Save Changes button becomes available. Then, save your artifact.
Add a second artifact for Evaluate Artifacts Configuration. Use the following example:

Name the artifact main.tf and use the following snippet for the Contents:

Show the Terraform script that deploys an NGINX container in a Kubernetes cluster

variable "namespace" {
 type = string
}
variable "deployName" {
 type = string
}
variable "replicas" {
 type = number
}
resource "kubernetes_namespace" "test" {
 metadata {
 name = var.namespace
 }
}
resource "kubernetes_deployment" "test" {
 metadata {
 name = var.deployName
 namespace = kubernetes_namespace.test.metadata.0.name
 }
 spec {
 replicas = var.replicas
 selector {
 match_labels = {
 app = "MyTestApp"
 }
 }
 template {
 metadata {
 labels = {
 app = "MyTestApp"
 }
 }
 spec {
 container {
 image = "nginx"
 name = "nginx-container"
 port {
 container_port = 80
 }
 }
 }
 }
 }
}

Note that the variable block at the beginning of the script declares the variables namespace, deployName, and replicas for the script, which are configured as parameters for the pipeline.

This example adds the Terraform script directly to the pipeline. You can also store it in source control, such as GitHub, and reference it that way.

Add a Terraform stage to the pipeline with the following characteristics:

Set the Action to Plan.
Select Save Plan Output
For Main Terraform Artifact > Expected Artifact, select main.tf. This is the Terraform script.
For Terraform Artifacts > Expected Artifact, select testvariables.tfVar. This is the variables file for Terraform.
For Produces Artifacts, add an artifact named planfile of type embedded artifact.

The plan file is the result of the stage and can be consumed by Terraform stages that perform the apply action.

Optionally, add a Manual Judgment stage. Although not required, it is good practice to add a Manual Judgment after a plan stage so that you can confirm the stage does what you expect it to do.
Add a second Terraform stage with the following characteristics:

Set the Action to Apply.
For Main Terraform Artifact > Expected Artifact, select main.tf. This is the Terraform script.
For Terraform Artifacts > Expected Artifact, select planfile. This is the artifact that the Terraform Plan stage produced.

Save your changes. This is what the complete pipeline looks like:
Start a manual execution of the pipeline.
The UI prompts you for values that get substitued into the pipeline:

nameAndSpace: {"name":"test-deployment","space":"test-space-param"} is a SpEL expression.
replicas is an integer value.

The pipeline creates values for variables in the Terraform script and the number of replicas. These are stored in a tfVar file.

Approve the Manual Judgment stage, which then starts the Terraform apply stage.

When the pipeline completes, you can query your Kubernetes cluster: kubectl get all -namespace test-space-param. The command returns all the resources the Terraform stage created.

Known issues

Problem saving artifacts

You may run into an issue where it seems like artifacts (or changes to them) are not being saved even though you click Save Changes. This issue occurs because of how the UI handles updates to artifacts in relation to changes to other configurations.

To avoid this issue, use the following workflow when you want to modify artifacts in a stage:

Save any changes you have made to the pipeline before you modify artifacts.
Make changes to the artifacts for the stage.
Wait for the status in the bottom right of the UI to change from In sync to the server to the action buttons. This wait period is important. If you make other changes before the artifact is ready, the artifact will not be saved.
Save your changes.
Continue making other changes.

Continuous-Deployment: Use Webhooks in Spinnaker Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

How does Spinnaker use webhooks?

Spinnaker uses “webhooks” in two ways – as a trigger for pipeline execution and as a stage that can make arbitrary calls to another service. If you’re looking for information on configuring a webhook trigger that you can use to run a pipeline, the open source community has a very good guide for that.

Spinnaker has a stage type called “Webhook” which allows the stage to call out to APIs as part of running a pipeline:

Setting up the a webhook stage

The basic configuration is what you might expect. Fill in the URL to make the request against, the HTTP method to use, and depending on the request type the payload and/or additional headers:

Of particular note is that you can use the Spinnaker pipeline expression language both as part of the URL field and within the payload, making it easy to pass anything that’s available as part of the pipeline context.

In this simple configuration the stage will be marked as successful if it gets a 2XX status code back, and will fail on anything else. If the return value from the request itself isn’t enough to determine the overall success you can check the “Wait for completion” checkbox and get a set of additional configuration:

Wait for completion using status field

There are three different techniques Spinnaker can use to lookup the overall status:

“GET method against webhook URL” means that once a request is sent using the method specified in the stage, Spinnaker will swap to polling the same webhook URL but using the GET method instead
“From the Location header” means that Spinnaker will look for the URL to poll in the headers of the original request, and use that URL to poll for the final status
“From webhook’s response” means Spinnaker will issue the original request, and will parse the response from that call to find a new URL it’ll use to poll for the final status

Spinnaker will use one of those mechanisms to find the status URL, and then repeatedly issue requests against that URL until it finds values that tell it what the final stage status should be. The “Status JsonPath” and mapping fields tell Spinnaker how to interpret the payloads that come back from the status URL. The “Status JsonPath” field is a JsonPath expression that Spinnaker uses to pull a single value from the payload for the status response. It compares the value from the payload to the values given in the SUCCESS, CANCELLED, and TERMINAL status mapping fields. Once there’s a match the overall state of the stage is set.

Webhook execution

Spinnaker records the URL used as part of the webhook, the payload, and the status URL as part of the stage details. If the webhook transaction can run for a long time and there’s information available from the API, you can set the “Progress location” expression to also extract info to give some feedback about status in the Spinnaker UI. The “Progress location” value shows up in the Info field of the stage details:

Once the webhook stage is complete the payload is attached to the stage context as “buildInfo”. So if you need you can pull info out of the webhook response to pass into future stages using a pipeline expression. For instance, if the response from our stage above contains the value “threshold” that we want to use in another stage we can reference it like this:

{ #stage('Webhook')['context']['buildInfo']['threshold'] }

Continuous-Deployment: Spinnaker Video Tutorials

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker is the continuous delivery platform that codifies the software delivery best practices that put Netflix and Google a decade ahead of most other companies.

Overview tab

What is Spinnaker

Concepts / Naming Conventions

How To Create a Pipeline

How to Build a Pipeline

Various Pipeline Stages

Bake Stage
Deploy Stage
Manual Judgement Stage
Check Preconditions
Adding a Parallel Stage

Armory Docs – User Guides

Continuous-Deployment: Spinnaker's Application Screen

Cluster tab

Load balancer tab

Pipelines

Deleting an application

Continuous-Deployment: Artifact Progression Through Environments in Spinnaker

Prerequisites

Overview of promoting artifacts between environments

Deploy to Staging

Validate your Deployment

Deploy to Production

Summary

See Also

Continuous-Deployment: Automated Kubernetes Rollbacks in Spinnaker

Creating a main deployment pipeline

Generating historic deployment versions for testing rollbacks

Creating a rollback pipeline

Testing rollback logic

Advanced configuration: Protecting against unwanted rollbacks

Continuous-Deployment: Automating Spinnaker

API docs

Auth

Continuous-Deployment: AWS Guides for Spinnaker

Continuous-Deployment: Azure Guides for Armory CD and Spinnaker

Continuous-Deployment: Canary Analysis in Spinnaker

Continuous-Deployment: Cloud Foundry Best Practices

Buildpacks and Procfiles

Continuous-Deployment: Debian Packages

Why use Debian packages

Creating Debian packages

Example: Debian package with OSPackage Gradle plugin

Continuous-Deployment: Deploy a Docker Image to Kubernetes

Kubernetes Deployments

Example

Note

Can we change how containers are deployed?

Continuous-Deployment: Deploy Applications to Kubernetes

Overview of Spinnaker’s Kubernetes V2 Provider

Current limitations

Available manifest-based stages

Creating a Kubernetes V2 pipeline

Configuring the pipeline trigger

Configuring the config map manifest delivery

Configuring deployment manifest delivery

Executing the pipeline

Continuous-Deployment: Manage Spinnaker Application Secrets Using HashiCorp Vault

Overview of storing secrets

Protecting application secrets

Basic outline

Operations-side configuration

Set up a secret path

Secure path with a policy

Create service account token

Create a Kubernetes secret

Bringing it all together

Configure your manifest

Configure your application

Alternatives

See also

Continuous-Deployment: Migrating Armory CD from Operator to Kustomize Deployment

Migrating Armory CD from Operator to Kustomize Deployment

Introduction

Important

Prerequisites

Migration Process Overview

Step 1: Download Current Configuration

How to Use the Download Script

What Gets Downloaded

Step 2: Set Up Kustomize Structure

Tip

Step 3: Remove Operator Ownership from Services

Note

Step 4: Verify Ownership Removal

Step 5: Extra Precautions Before Deployment

Perform a Dry Run

Step 6: Scale Down the Operator

Step 7: Deploy Using Kustomize

Step 8: Validate and Monitor

Step 9: Remove Operator and CRDs