Armory Docs – Administration on AWS

Continuous-Deployment: Bake Machine Images on AWS Using Packer

Mon, 01 Jan 0001 00:00:00 +0000

Overview of baking images in Spinnaker

Spinnaker^TM uses the open source Packer tool to bake images, which is included in the Spinnaker Rosco microservice. For example, if you are deploying to AWS, you can use Spinnaker to bake Amazon Machine Images (AMIs) from the artifacts that were produced by your CI tool.

This section focuses on configuring Packer scripts to build machine images (such as AMIs). If you’re only deploying to Kubernetes, you can skip this section.

Configuring AWS bake credentials

If you’ve configured Spinnaker to deploy to AWS, then you’ve likely set up a set of IAM credentials and permissions for Clouddriver to be able to deploy to AWS (using the AWS API). Since Packer is run from a different microservice in Spinnaker (Rosco), configuration of Bake credentials is separate from the configuration of Deploy credentials.

AMI baking primarily occurs in a single AWS account; you can configure baked AMIs to be shared to additional desired AWS accounts

AWS bake credentials using IAM credentials

If you’ve configured Spinnaker to interact with AWS using explicit credentials (an AWS Access Key and Secret Access Key), you can likewise configure Rosco to use a set of AWS credentials. What you essentially need are an IAM user with permissions to do the things that Packer needs to do, and then you can pass those credentials to Rosco.

The AWS account that you’re baking in must also match an account configured as a Managed Account, and that Managed Account must be configured as the primary AWS account within Spinnaker.

This User must have all permissions necessary to bake (for example, PowerUserAccess and associated PassRoles)

This User may be, but does not have to be, the same as the Managing Account User.

Spinnaker will always bake with this user. If you need to deploy to other accounts, update your Packer template to support sharing the baked image with other accounts. For example, add this to your builder configuration in your packer template (and add the custom packer template following the instructions in the Spinnaker Packer documentation:

 "ami_users": ["222222222222","333333333333"]

Create an IAM user

If you don’t have an IAM user with the desired permissions, you can create one by doing the following:

Log into the AWS account where Spinnaker lives, into the browser-based AWS Console
Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”)
Click on “Users” on the left side
Click on “Add user”
Specify a logical user name, such as “SpinnakerBake”
Check the “Programmatic access” checkbox
Select “Attach existing policies directly”
Find and attach the AWS “PowerUserAccess” Policy
Click “Next: Tags”
Optionally, add tags that will identify this user
Click “Next: Review”
Click “Create user”
Copy the “Access key ID” and “Secret access key” (you’ll have to click “Show”).
Click “Close”
Click on the “User name” for the user that you just created.
Click on “Add inline policy” (on the right).

Click on the “JSON” tab, and paste in this:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": [
 "iam:PassRole"
 ],
 "Resource": [
 "*"
 ],
 "Effect": "Allow"
 }
 ]
}

Click on “Review Policy”
Call it “PassRole” and then click “Create Policy”

Add IAM user credentials to Rosco

Using the “Access key ID” and “Secret access key” for your generated user, run this command in Halyard:

# You will be prompted for the secret access key
hal config provider aws bakery edit --aws-access-key "YOUR_ACCESS_KEY" --aws-secret-key

Then deploy your changes with this:

hal deploy apply

AWS bake credentials using IAM profiles

If you’re using IAM Instance Roles, you need to provide credentials to Spinnaker to use to Bake by adding additional policies to the EC2 instances where Spinnaker is running. The AWS account that you’re baking in must also be configured as a Managed Account, and that Managed Account must be configured as the primary AWS account within Spinnaker.

These policies must have all permissions necessary to bake (for example, PowerUserAccess and associated PassRoles)

Spinnaker will always bake with the EC2 instance role (unless you specify explicit baking creds). If you need to deploy to other accounts, update your Packer template to support sharing the baked image with other accounts. For example, add this to your builder configuration in your packer template (and add the custom packer template following the instructions in the Spinnaker Packer documentation:

 "ami_users": ["222222222222","333333333333"]

If you don’t configure Rosco with explicit AWS credentials to use, Packer will default to the AWS permissions available to the Rosco container. In general, this means that Packer will use the IAM Role/Profile attached to the Kubernetes nodes where Spinnaker is running. In order for this to work, the IAM Role/Profile attached to your Kubernetes cluster will need a set of permissions to be able to create and interact with EC2 instances (and assign roles to those EC2 instances).

First, identify the IAM Profile attached to the Kubernetes cluster where Spinnaker is running. If you’re running EKS, this will be the role attached to the EKS EC2 worker nodes.
Go to the Role Summary for the IAM Profile, and click “Attach policies”
Find and select the AWS “PowerUserAccess” Policy
Click “Attach policy”
Click on “Add inline policy” (on the right).

Click on the “JSON” tab, and paste in this:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": [
 "iam:PassRole"
 ],
 "Resource": [
 "*"
 ],
 "Effect": "Allow"
 }
 ]
}

Click on “Review Policy”
Call it “PassRole” and then click “Create Policy”

This role should be immediately available to your Rosco instance.

Configuring AWS networks

In addition to providing permissions for Rosco’s packer to do Bake stages, if your AWS account doesn’t have a default VPC / subnet, you can specify a default subnet to bake in with this:

hal config provider aws bakery edit --aws-subnet subnet-0123456789
hal deploy apply

Or, on a per-bake basis, you can specify what VPC and subnet to bake by adding these two extended attributes to the Bake stage:

aws_vpc_id: vpc-123456
aws_subnet_id: subnet-0123456789

What exactly are Packer scripts?

Spinnaker works best when deploying immutable artifacts to immutable machine images. When working with machine images, packer scripts are used during the Bake Stage to create an immutable machine image.

Spinnaker has a microservice called Rosco, which uses Packer to bake machine images (such as Amazon Machine Images or AMIs). Out of the box, it comes with the packer templates and scripts listed here.

By default, Rosco performs the following actions in a “Bake” stage:

Takes a list of desired packages specified in the pipeline definition
Identifies the deb files produced by your CI pipeline and matches those to the desired package
Creates a set of Packer variables using the name/repository of the matched deb file(s)
Bakes an AMI using the packer template (visible here
Runs the install_packages.sh script (visible here) to install the identified deb packages into the AMI
Make the AMI ID available to later stages in the Spinnaker pipeline (such as deployments)

In a bake stage configuration, you can specify other packer templates to use.

If your app is using zip, tarballs or you need some customization, you need to create a new Packer script (see below).

Adding custom Packer scripts to Armory

Out of the box, Armory comes with these built-in Packer templates and scripts: https://github.com/spinnaker/rosco/tree/master/rosco-web/config/packer

If you’d like to add additional Packer template or script files, you can add them via the Armory Operator .

Add any Packer template and supporting scripts as string-formatted entries under the spec.spinnakerConfig.files section of the SpinnakerService config.

If you have a template named example-packer-config.json containing the following:

{
 "packerSetting" : "someValue"
}

and a script file named my-custom-script.sh containing the following:

#!/bash/bash -e
echo "Hello world!"

make the following entries into the SpinnakerService config:

files:
 profiles__rosco__packer__example-packer-config.json: |
 {
 "packerSetting" : "someValue"
 }
 profiles__rosco__packer__my-custom-script.sh: |
 #!/bash/bash -e
 echo "Hello world!"

NOTE: The Armory Operator interprets the double underscores in the file names as slashes indicating the directory path where the Operator saves the file.

See the “Export Packer template files” step in the Migrating from Halyard to Operator section of the Armory Operator document for more examples.

Continuous-Deployment: Configure AWS for Spinnaker Using IAM Instance Roles

Mon, 01 Jan 0001 00:00:00 +0000

Overview of deploying applications to AWS

This document will guide you through the following:

Understanding AWS deployment from Spinnaker^TM
Configuring Spinnaker to use AWS IAM Instance Roles (if Spinnaker is running on AWS, either via AWS EKS or installed directly on EC2 instances)
- Creating a Managed Account IAM Role in each of your target AWS Accounts
- Creating the default BaseIamRole for use when deploying EC2 instances
- Creating a Managing Account IAM Policy in your primary AWS Account
- Adding the Managing Account IAM Policy to the existing IAM Instance Role on the AWS nodes
- Configuring the Managed Accounts IAM Roles to trust the IAM Instance Role from the AWS nodes
- Adding the Managed Accounts to Spinnaker
- Adding/Enabling the AWS Cloud Provider to Spinnaker

Prerequisites for deploying to AWS

You installed Spinnaker with Operator.
You have access to the Spinnaker config files, and a way to apply them (kubectl for Operator).
If you’re using Operator, you have a access to run kubectl commands against the cluster where Spinnaker is installed.
You have permissions to create IAM roles using IAM policies and permissions, in all relevant AWS accounts.
- You should also be able to set up cross-account trust relationships between IAM roles.
If you want to add the IAM Role to Spinnaker via an Access Key/Secret Access Key, you have permissions to create an IAM User.
If you want to add the IAM Role to Spinnaker via IAM instance profiles/policies, you have permissions to modify the IAM instance.

All configuration with AWS in this document will be handled via the browser-based AWS Console. All configurations could alternatively be configured via the aws CLI, but this is not currently covered in this document.

Also - you will be granting AWS Power User Access to each of the Managed Account Roles. You could optionally grant fewer permissions, but those more limited permissions are not covered in this document.

Background: Understanding AWS Deployment from Spinnaker

Even if Spinnaker is installed in Kubernetes, it can be used to deploy to other cloud environments, such as AWS. Rather than granting Spinnaker direct access to each of the target AWS accounts, Spinnaker will assume a role in each of the target accounts.

Deploying to AWS EC2

Spinnaker is able to deploy EC2 instances via Auto Scaling Groups.

Spinnaker’s Clouddriver Pod should be able to assume a Managed Account Role in each deployment target AWS account, and use that role to perform any AWS actions. This may include one or more of the following:
- Create AWS Launch Configurations and Auto Scaling Groups to deploy AWS EC2 instances
- Run ECS Containers
- Run AWS Lambda Actions (alpha/beta as of the time of this document)
- Create AWS CloudFormation Stacks (alpha/beta as of the time of this document)
Clouddriver is configured with direct access to a “Managing Account” Policy (it may be helpful to think of this as the Master or Source Policy), which is accomplished in one of two ways:
- If Spinnaker is running in AWS (either in AWS EKS, or with Kubernetes nodes running in AWS EC2), the Managing Account Policy can be made available to Spinnaker by adding it to the AWS nodes (EC2 instances) where the Spinnaker Clouddriver pod(s) are running.
  - (You can also use Kube2IAM or similar capabilities, but this is not covered in this document)
- An IAM User with access to the Managing Account Policy can be passed directly to Spinnaker via an Access Key and Secret Access Key
For each AWS account that you want Spinnaker to be able to deploy to, Spinnaker needs a “Managed Account” Role in that AWS account, with permissions to do the things you want Spinnaker to be able to do (it may be helpful to think of this as a Target Role)
The Managing Account Role (Source/Master Role) should be able to assume each of the Managed Account Roles (Target Roles). This requires two things:
- The Managing Account Role needs a permission string for each Managed Account it needs to be able to assume. It may be helpful to think of this as an outbound permission.
- Each Managed Account needs to have a trust relationship with the Managing Account User or Role to allow the Managing Account User or Role to assume it. It may be helpful to think of this as an inbound permission.

In addition, if you are deploying EC2 instances with AWS, you will need to provide an IAM role for each instance. If you do not specify a role, Spinnaker will attempt to use a role called BaseIAMRole. So you should create a BaseIAMRole (potentially with no permissions).

Deployment scenario

Here’s an example situation:

We would like Armory to deploy to three AWS accounts, with account IDs 111111111111, 222222222222, and 333333333333. Each of these is a Managed Account
Choose one account (111111111111), that Armory will log into directly. This is the Managing Account
We will end up with four IAM entities:
- A Managing Account Policy in account 111111111111 (arn:aws:iam::111111111111:user/managingAccount)
- A Managed Account Role in account 111111111111 (arn:aws:iam::111111111111:role/spinnakerManaged)
- A Managed Account Role in account 222222222222 (arn:aws:iam::222222222222:role/spinnakerManaged)
- A Managed Account Role in account 333333333333 (arn:aws:iam::333333333333:role/spinnakerManaged)
The Managing Account Policy needs these:
- The sts:AssumeRole permission for each of the Managed Account Roles
- The ec2:DescribeAvailabilityZones permission
- The ec2:DescribeRegions permission
- It should be attached to the IAM Instance Role where Armory is running
Each Managed Account Role needs these:
- PowerUserAccess
- The iam:PassRole permission for roles that will be assigned to EC2 instances that are being deployed
- A trust relationship with the IAM Instance Role attached to the EC2 instances where Armory is running (to allow Armory to assume the Managed Account Role)

Spinnaker configuration examples

Here’s a sample SpinnakerService manifest block that supports the above:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 providers:
 aws:
 enabled: true
 accounts:
 - name: aws-1
 requiredGroupMembership: []
 providerVersion: V1
 permissions: {}
 accountId: '111111111111'
 regions:
 - name: us-east-1
 - name: us-west-2
 assumeRole: role/spinnakerManaged
 - name: aws-2
 requiredGroupMembership: []
 providerVersion: V1
 permissions: {}
 accountId: '222222222222'
 regions:
 - name: us-east-1
 - name: us-west-2
 assumeRole: role/spinnakerManaged
 - name: aws-3
 requiredGroupMembership: []
 providerVersion: V1
 permissions: {}
 accountId: '333333333333'
 regions:
 - name: us-east-1
 - name: us-west-2
 assumeRole: role/spinnakerManaged
 # Because we're baking in 111111111111, this must match the accountName that is associated with 111111111111
 primaryAccount: aws-1
 bakeryDefaults:
 templateFile: aws-ebs-shared.json
 baseImages: []
 awsAssociatePublicIpAddress: true
 defaultVirtualizationType: hvm
 defaultKeyPairTemplate: '{{name}}-keypair'
 defaultRegions:
 - name: us-west-2
 defaults:
 iamRole: BaseIAMRole

Configuring Armory to use AWS IAM Instance Roles

If you are running Armory on AWS (either via AWS EKS or installed directly on EC2 instances), you can use AWS IAM roles to allow Clouddriver to interact with the various AWS APIs across multiple AWS Accounts.

Instance Role Part 1: Creating a Managed Account IAM Role in each your target AWS Accounts

In each account that you want Armory to deploy to, you should create an IAM role for Armory to assume.

For each account you want to deploy to, perform the following:

Log into the browser-based AWS Console
Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”)
Click on “Roles” on the left hand side
Click on “Create role”
For now, for the “Choose the service that will use this role”, select “EC2”. We will change this later, because we want to specify an explicit consumer of this role later on.
Click on “Next: Permissions”
Search for “PowerUserAccess” in the search filter, and select the Policy called “PowerUserAcces”
Click “Next: Tags”
Optionally, add tags that will identify this role.
Click “Next: Review”
Enter a Role Name. For example, “DevSpinnakerManagedRole”. Optionally, add a description, such as “Allows Armory Dev Cluster to perform actions in this account.”
Click “Create Role”
In the list of Roles, click on your new Role (you may have to scroll down or filter for it).
Click on “Add inline policy” (on the right).

Click on the “JSON” tab, and paste in this:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": [
 "iam:ListServerCertificates",
 "iam:PassRole"
 ],
 "Resource": [
 "*"
 ],
 "Effect": "Allow"
 }
 ]
}

Click “Review Policy”
Call it “PassRole-and-Certificates”, and click “Create Policy”
Copy the Role ARN and save it. It should look something like this: arn:aws:iam::123456789012:role/DevSpinnakerManagedRole. This will be used in the section “Instance Role Part 3”, and in section “Instance Role Part 6”

You will end up with a Role ARN for each Managed / Target account. The Role names do not have to be the same (although it is a bit cleaner if they are). For example, you may end up with roles that look like this:

arn:aws:iam::123456789012:role/DevSpinnakerManagedRole
arn:aws:iam::123456789013:role/DevSpinnakerManagedRole
arn:aws:iam::123456789014:role/DevSpinnakerManaged

Instance Role Part 2: Creating the BaseIAMRole for EC2 instances

When deploying EC2 instances, Armory currently requires that you attach a role for each instance (even if you don’t want to grant the instance any special permissions. If you do not specify an instance role, Armory will default to a role called BaseIAMRole, and it will throw an error if this does not exist. Therefore, you should at a minimum create an empty role called BaseIAMRole.

Log into the browser-based AWS Console
Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”)
Click on “Roles” on the left side
Click “Create role”
Select “EC2”, and click “Next: Permissions”
Click “Next: Tags”
Optionally, add tags if required by your organization. Then, click “Next: Review”.
Specify the Role Name as “BaseIAMRole”

Instance Role Part 3: Creating a Managing Account IAM Policy in your primary AWS Account

In the account that Armory lives in (i.e., the AWS account that owns the EKS cluster where Armory is installed), create an IAM Policy with permissions to assume all of your Managed Roles.

Log into the AWS account where Armory lives, into the browser-based AWS Console
Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”)
Click on “Policies” on the left hand side
Click on “Create Policy”

Click on the “JSON” tab, and paste in this:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "ec2:DescribeAvailabilityZones",
 "ec2:DescribeRegions"
 ],
 "Resource": [
 "*"
 ]
 },
 {
 "Action": "sts:AssumeRole",
 "Resource": [
 "arn:aws:iam::123456789012:role/DevSpinnakerManagedRole",
 "arn:aws:iam::123456789013:role/spinnakerManaged",
 "arn:aws:iam::123456789014:role/DevSpinnakerManaged"
 ],
 "Effect": "Allow"
 }
 ]
}

Update the sts:AssumeRole block with the list of Managed Roles you created in Instance Role Part 1.
Click on “Review Policy”
Create a name for your policy, such as “SpinnakerManagingPolicy”. This policy will be attached to your Armory instance, so give it a name describing your Armory instance. Optionally, add a descriptive description. Copy the name of the policy. This will be used in the next section, “Instance Role Part 4”
On the list policies, click your newly-created Policy.

(This policy could also be attached inline directly to the IAM Instance Role, rather than creating a standalone policy)

Instance Role Part 4: Adding the Managing Account IAM Policy to the existing IAM Instance Role on the AWS nodes

Log into the AWS account where Armory lives, into the browser-based AWS Console
Navigate to the EC2 page (click on “Services” at the top, then on “EC2” under “Compute”)
Click on “Running Instances”
Find one of the nodes which is part of your EKS or other Kubernetes cluster, and select it.
In the Instance details section of the screen (in the lower half), find the “IAM Role” and click on it to go to the Role page.
Click on “Attach Policies”
Search for the Policy that you created, and select it.
Click “Attach Policy”
Back on the screen for the Role, copy the node role ARN. It should look something like this: arn:aws:iam::123456789010:role/node-role. This will be used in the next section, “Instance Role Part 5”

Note: If your instances do not have an IAM instance profile and role attached to them, you can follow these steps to create and attach one:

Log into the AWS account where Armory lives, into the browser-based AWS Console
Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”
Click on “Roles”
Click on “Create role”
Select “EC2” for the service that will use the role, and click “Next: Permissions”
In the policy filter, enter the name of the managing policy you created in step 3. Click “Next: Tags”
Add any relevant tags. Click “Next: Review”
Give the role a name, such as “Armory”. This role will be attached to your Armory instance, so give it a name describing your Armory instance. Optionally, add a descriptive description. Copy the name of the role.
Navigate to the EC2 page (click on “Services” at the top, then on “EC2” under “Compute”)
Click on “Running Instances”
Find one of the nodes which is part of your EKS or other Kubernetes cluster, and select it.
Click “Actions” at the top, then select “Instance Settings” and then “Attach/Replace IAM Role”
In the “IAM role” dropdown, select the IAM role that you just created
Click “Apply”
Repeaat the last four steps for each of the other instances in your EKS or Kubernetes cluster.

Instance Role Part 5: Configuring the Managed Accounts IAM Roles to trust the IAM Instance Role from the AWS nodes

Now that we know what role will be assuming each of the Managed Roles, we must configure the Managed Roles (Target Roles) to trust and allow the Managing (Assuming) Role to assume them. This is called a “Trust Relationship” and is configured each of the Managed Roles (Target Roles).

For each account you want to deploy to, perform the following:

Log into the browser-based AWS Console
Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”)
Click on “Roles” on the left hand side
Find the Managed Role that you created earlier in this account, and click on the Role Name to edit the role.
Click on the “Trust relationships” tab.
Click on “Edit trust relationship”

Replace the Policy Document with this (Update the ARN with the node role ARN from “Instance Role Part 4”)

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": {
 "AWS": [
 "arn:aws:iam::123456789010:role/node-role"
 ]
 },
 "Action": "sts:AssumeRole"
 }
 ]
}

Click “Update Trust Policy”, in the bottom right.

Instance Role Part 6: Adding the Managed Accounts to Armory

The Clouddriver pod(s) should be now able to assume each of the Managed Roles (Target Roles) in each of your Deployment Target accounts. We need to configure Armory to be aware of the accounts and roles its allowed to consume.

For each of the Managed (Target) accounts you want to deploy to, add a new entry to the accounts array in SpinnakerService manifest as follows:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
name: spinnaker
spec:
spinnakerConfig:
 config:
 providers:
 aws:
 enabled: true
 accounts:
 - name: aws-dev-1 # Should be a unique name which is used in the Armory UI and API to identify the deployment target. For example, aws-dev-1 or aws-dev-2
 requiredGroupMembership: []
 providerVersion: V1
 permissions: {}
 accountId: '111111111111' # Should be the account ID for the Managed Role (Target Role) you are assuming. For example, if the role ARN is arn:aws:iam::123456789012:role/ DevSpinnakerManagedRole, then ACCOUNT_ID would be 123456789012
 regions: # Configure the regions you want to deploy to
 - name: us-east-1
 - name: us-west-2
 assumeRole: role/spinnakerManaged # Should be the full role name within the account, including the type of object (role). For example, if the role ARN is arn:aws:iam::123456789012:role/DevSpinnakerManagedRole, then ROLE_NAME would be role/DevSpinnakerManagedRole
 primaryAccount: aws-dev-1
 bakeryDefaults:
 templateFile: aws-ebs-shared.json
 baseImages: []
 awsAssociatePublicIpAddress: true
 defaultVirtualizationType: hvm
 defaultKeyPairTemplate: '{{name}}-keypair'
 defaultRegions:
 - name: us-west-2
 defaults:
 iamRole: BaseIAMRole

Instance Role Part 7: Adding/Enabling the AWS CloudProvider configuration to Armory

Apply the changes done in Spinnakerservice manifest:

kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest file>

Continuous-Deployment: Configure Spinnaker to Access AWS Using IAM User Roles

Mon, 01 Jan 0001 00:00:00 +0000

Overview of deploying applications to AWS

This document will guide you through the following:

Understanding AWS deployment from Spinnaker^TM
Configuring Spinnaker to use AWS IAM Instance Roles (if Spinnaker is running on AWS, either via AWS EKS or installed directly on EC2 instances)
- Creating a Managed Account IAM Role in each of your target AWS Accounts
- Creating the default BaseIamRole for use when deploying EC2 instances
- Creating a Managing Account IAM Policy in your primary AWS Account
- Adding the Managing Account IAM Policy to the existing IAM Instance Role on the AWS nodes
- Configuring the Managed Accounts IAM Roles to trust the IAM Instance Role from the AWS nodes
- Adding the Managed Accounts to Spinnaker
- Adding/Enabling the AWS Cloud Provider to Spinnaker

Prerequisites for deploying to AWS

You installed Spinnaker with Operator.
You have access to the Spinnaker config files, and a way to apply them (kubectl for Operator).
If you’re using Operator, you have a access to run kubectl commands against the cluster where Spinnaker is installed.
You have permissions to create IAM roles using IAM policies and permissions, in all relevant AWS accounts.
- You should also be able to set up cross-account trust relationships between IAM roles.
If you want to add the IAM Role to Spinnaker via an Access Key/Secret Access Key, you have permissions to create an IAM User.
If you want to add the IAM Role to Spinnaker via IAM instance profiles/policies, you have permissions to modify the IAM instance.

All configuration with AWS in this document will be handled via the browser-based AWS Console. All configurations could alternatively be configured via the aws CLI, but this is not currently covered in this document.

Background: Understanding AWS Deployment from Spinnaker

Deploying to AWS EC2

Spinnaker is able to deploy EC2 instances via Auto Scaling Groups.

Spinnaker’s Clouddriver Pod should be able to assume a Managed Account Role in each deployment target AWS account, and use that role to perform any AWS actions. This may include one or more of the following:
- Create AWS Launch Configurations and Auto Scaling Groups to deploy AWS EC2 instances
- Run ECS Containers
- Run AWS Lambda Actions (alpha/beta as of the time of this document)
- Create AWS CloudFormation Stacks (alpha/beta as of the time of this document)
Clouddriver is configured with direct access to a “Managing Account” Policy (it may be helpful to think of this as the Master or Source Policy), which is accomplished in one of two ways:
- If Spinnaker is running in AWS (either in AWS EKS, or with Kubernetes nodes running in AWS EC2), the Managing Account Policy can be made available to Spinnaker by adding it to the AWS nodes (EC2 instances) where the Spinnaker Clouddriver pod(s) are running.
  - (You can also use Kube2IAM or similar capabilities, but this is not covered in this document)
- An IAM User with access to the Managing Account Policy can be passed directly to Spinnaker via an Access Key and Secret Access Key
For each AWS account that you want Spinnaker to be able to deploy to, Spinnaker needs a “Managed Account” Role in that AWS account, with permissions to do the things you want Spinnaker to be able to do (it may be helpful to think of this as a Target Role)
The Managing Account Role (Source/Master Role) should be able to assume each of the Managed Account Roles (Target Roles). This requires two things:
- The Managing Account Role needs a permission string for each Managed Account it needs to be able to assume. It may be helpful to think of this as an outbound permission.
- Each Managed Account needs to have a trust relationship with the Managing Account User or Role to allow the Managing Account User or Role to assume it. It may be helpful to think of this as an inbound permission.

Deployment scenario

Here’s an example situation:

We would like Spinnaker to deploy to three AWS accounts, with account IDs 111111111111, 222222222222, and 333333333333. Each of these is a Managed Account
Choose one account (111111111111), that Spinnaker will log into directly. This is the Managing Account
We will end up with four IAM entities:
- A Managing Account User in account 111111111111 (arn:aws:iam::111111111111:user/managingAccount)
- A Managed Account Role in account 111111111111 (arn:aws:iam::111111111111:role/spinnakerManaged)
- A Managed Account Role in account 222222222222 (arn:aws:iam::222222222222:role/spinnakerManaged)
- A Managed Account Role in account 333333333333 (arn:aws:iam::333333333333:role/spinnakerManaged)
The Managing Account User needs these permissions:
- The sts:AssumeRole permission for each of the Managed Account Roles
- The ec2:DescribeAvailabilityZones permission
- The ec2:DescribeRegions permission
Each Managed Account Role needs these:
- PowerUserAccess
- The iam:PassRole permission for roles that will be assigned to EC2 instances that are being deployed
- A trust relationship with the Managing Account User (to allow the Managing Account User to assume the Managed Account Role)

Spinnaker configuration examples

Here’s a sample SpinnakerService manifest block that supports the above:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 providers:
 aws:
 enabled: true
 accounts:
 - name: aws-1
 requiredGroupMembership: []
 providerVersion: V1
 permissions: {}
 accountId: '111111111111'
 regions:
 - name: us-east-1
 - name: us-west-2
 assumeRole: role/spinnakerManaged
 - name: aws-2
 requiredGroupMembership: []
 providerVersion: V1
 permissions: {}
 accountId: '222222222222'
 regions:
 - name: us-east-1
 - name: us-west-2
 assumeRole: role/spinnakerManaged
 - name: aws-3
 requiredGroupMembership: []
 providerVersion: V1
 permissions: {}
 accountId: '333333333333'
 regions:
 - name: us-east-1
 - name: us-west-2
 assumeRole: role/spinnakerManaged
 # Because we're baking in 111111111111, this must match the accountName that is associated with 111111111111
 primaryAccount: aws-1
 bakeryDefaults:
 templateFile: aws-ebs-shared.json
 # These creds are for our Baking IAM user in account 111111111111
 awsAccessKey: ABC123
 awsSecretKey: abc123
 baseImages: []
 awsAssociatePublicIpAddress: true
 defaultVirtualizationType: hvm
 accessKeyId: DEF456
 secretAccessKey: def456
 defaultKeyPairTemplate: '{{name}}-keypair'
 defaultRegions:
 - name: us-west-2
 defaults:
 iamRole: BaseIAMRole

Configuring Spinnaker to use AWS IAM Roles

If you are not running Spinnaker on AWS, or if you do not want to use AWS IAM roles (or don’t have the ability to modify the roles attached to your Kubernetes instances), you can create an AWS IAM user and provide its credentials to Clouddriver to allow Clouddriver to interact with the various AWS APIs across multiple AWS Accounts.

IAM User Part 1: Creating a Managed Account IAM Role in each your target AWS Accounts

In each account that you want Spinnaker to deploy to, you should create an IAM role for Spinnaker to assume.

For each account you want to deploy to, perform the following:

Log in to the browser-based AWS Console
Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”)
Click on “Roles” on the left hand side
Click on “Create role”
For now, for the “Choose the service that will use this role”, select “EC2”. We will change this later, because we want to specify an explicit consumer of this role later on.
Click on “Next: Permissions”
Search for “PowerUserAccess” in the search filter, and select the Policy called “PowerUserAcces”
Click “Next: Tags”
Optionally, add tags that will identify this role.
Click “Next: Review”
Enter a Role Name. For example, “DevSpinnakerManagedRole”. Optionally, add a description, such as “Allows Spinnaker Dev Cluster to perform actions in this account.”
Click “Create Role”
In the list of Roles, click on your new Role (you may have to scroll down or filter for it).
Click on “Add inline policy” (on the right).

Click on the “JSON” tab, and paste in this:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": [
 "iam:ListServerCertificates",
 "iam:PassRole"
 ],
 "Resource": [
 "*"
 ],
 "Effect": "Allow"
 }
 ]
}

Click “Review Policy”
Call it “PassRole-and-Certificates”, and click “Create Policy”
Copy the Role ARN and save it. It should look something like this: arn:aws:iam::123456789012:role/DevSpinnakerManagedRole. This will be used in the section “IAM User Part 3” and in section “IAM User Part 6”

arn:aws:iam::123456789012:role/DevSpinnakerManagedRole
arn:aws:iam::123456789013:role/DevSpinnakerManagedRole
arn:aws:iam::123456789014:role/DevSpinnakerManaged

IAM User Part 2: Creating the BaseIAMRole for EC2 instances

When deploying EC2 instances, Spinnaker currently requires that you attach a role for each instance (even if you don’t want to grant the instance any special permissions. If you do not specify an instance role, Spinnaker will default to a role called BaseIAMRole, and it will throw an error if this does not exist. Therefore, you should at a minimum create an empty role called BaseIAMRole.

Log into the browser-based AWS Console
Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”)
Click on “Roles” on the left side
Click “Create role”
Select “EC2”, and click “Next: Permissions”
Click “Next: Tags”
Optionally, add tags if required by your organization. Then, click “Next: Review”.
Specify the Role Name as “BaseIAMRole”

IAM User Part 3: Creating a Managing Account IAM Policy in your primary AWS account

In the account that Spinnaker lives in (i.e., the AWS account that owns the EKS cluster where Spinnaker is installed), create an IAM Policy with permissions to assume all of your Managed Roles.

Log into the AWS account where Spinnaker lives, into the browser-based AWS Console
Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”)
Click on “Policies” on the left hand side
Click on “Create Policy”

Click on the “JSON” tab, and paste in the following:

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "ec2:DescribeAvailabilityZones",
 "ec2:DescribeRegions"
 ],
 "Resource": [
 "*"
 ]
 },
 {
 "Action": "sts:AssumeRole",
 "Resource": [
 "arn:aws:iam::123456789012:role/DevSpinnakerManagedRole",
 "arn:aws:iam::123456789013:role/spinnakerManaged",
 "arn:aws:iam::123456789014:role/DevSpinnakerManaged"
 ],
 "Effect": "Allow"
 }
 ]
}

Update the sts:AssumeRole block with the list of Managed Roles you created in IAM User Part 1.
Click on “Review Policy”
Create a name for your policy, such as “DevSpinnakerManagingPolicy”. Optionally, add a descriptive description. Copy the name of the policy. This will be used in the next section, “IAM User Part 4”
On the list policies, click your newly-created Policy.

(This policy could also be attached inline directly to the IAM User, rather than creating a standalone policy)

IAM User Part 4: Creating a Managing Account IAM User with access to the Managing Account Policy

The IAM user we’re creating can be in any AWS account, although it may make sense to place it in the same account where Spinnaker lives if Spinnaker is installed in AWS.

Log into the AWS account where Spinnaker lives, into the browser-based AWS Console
Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”)
Click on “Users” on the left side
Click on “Add user”
Specify a logical user name, such as “DevSpinnakerManagingAccount”
Check the “Programmatic access” checkbox
Select “Attach existing policies directly”
Find the policy you created in “IAM User Part 2”. and select it with the checkbox.
Click “Next: Tags”
Optionally, add tags that will identify this user
Click “Next: Review”
Click “Create user”
Copy the “Access key ID” and “Secret access key” (you’ll have to click “Show”). This will be used later, in “IAM User Part 6”
Click “Close”
Click on the “User name” for the user that you just created.
Copy the “User ARN”. This will look something like this: arn:aws:iam::123456789010:user/DevSpinnakerManagingAccount. This will be used in the next section, “IAM User Part 5”

IAM User Part 5: Configuring the Managed Accounts to trust the Managing Account IAM User

Now that we know what user will be assuming each of the Managed Roles, we must configure the Managed Roles (Target Roles) to trust and allow the Managing (Assuming) User to assume them. This is called a “Trust Relationship” and is configured each of the Managed Roles (Target Roles).

For each account you want to deploy to, perform the following:

Log into the browser-based AWS Console
Navigate to the IAM page (click on “Services” at the top, then on “IAM” under “Security, Identity, & Compliance”)
Click on “Roles” on the left hand side
Find the Managed Role that you created earlier in this account, and click on the Role Name to edit the role.
Click on the “Trust relationships” tab.
Click on “Edit trust relationship”

Replace the Policy Document with this (Update the ARN with the User ARN from “IAM User Part 4”)

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": {
 "AWS": [
 "arn:aws:iam::123456789010:user/DevSpinnakerManagingAccount"
 ]
 },
 "Action": "sts:AssumeRole"
 }
 ]
}

Click “Update Trust Policy”, in the bottom right.

IAM User Part 6: Adding the Managing Account User and managed accounts to Spinnaker

The Clouddriver pod(s) should be now able to assume each of the Managed Roles (Target Roles) in each of your Deployment Target accounts. We need to configure Spinnaker to be aware of the accounts and roles it is allowed to consume.

For each of the Managed (Target) accounts you want to deploy to, add a new entry to the accounts array in SpinnakerService manifest as follows:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 providers:
 aws:
 enabled: true
 accounts:
 - name: aws-dev-1 # Should be a unique name which is used in the Spinnaker UI and API to identify the deployment target. For example, aws-dev-1 or aws-dev-2
 requiredGroupMembership: []
 providerVersion: V1
 permissions: {}
 accountId: '111111111111' # Should be the account ID for the Managed Role (Target Role) you are assuming. For example, if the role ARN is arn:aws:iam::123456789012:role/ DevSpinnakerManagedRole, then ACCOUNT_ID would be 123456789012
 regions: # Configure the regions you want to deploy to
 - name: us-east-1
 - name: us-west-2
 assumeRole: role/spinnakerManaged # Should be the full role name within the account, including the type of object (role). For example, if the role ARN is arn:aws:iam::123456789012:role/DevSpinnakerManagedRole, then ROLE_NAME would be role/DevSpinnakerManagedRole
 primaryAccount: aws-dev-1
 bakeryDefaults:
 templateFile: aws-ebs-shared.json
 baseImages: []
 awsAssociatePublicIpAddress: true
 defaultVirtualizationType: hvm
 # These creds are for our Baking IAM user in account 111111111111
 awsAccessKey: ABC123
 awsSecretKey: abc123
 accessKeyId: DEF456 # AWS access key and secret access key from "IAM User Part 4"
 secretAccessKey: def456 # AWS access key and secret access key from "IAM User Part 4"
 defaultKeyPairTemplate: '{{name}}-keypair'
 defaultRegions:
 - name: us-west-2
 defaults:
 iamRole: BaseIAMRole

IAM User Part 7: Adding/Enabling the AWS CloudProvider configuration to Spinnaker

Apply the changes done in Spinnakerservice manifest:

kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest file>

Continuous-Deployment: Configure Spinnaker on AWS for Disaster Recovery

Mon, 01 Jan 0001 00:00:00 +0000

Spinnaker disaster recovery

The following guide describes how to configure Spinnaker^TM deployment on AWS to be more resilient and perform disaster recovery (DR). Spinnaker does not function in multi-master mode, which means that active-active is not supported at this time. Instead, this guide describes how to achieve an active-passive setup. This results in two instances of Spinnaker deployed into two regions that can fail independently.

Requirements

The passive instance will have the same permissions as the active instance
The active instance is configured to use AWS Aurora and S3 for persistent storage
Your Secret engine/store has been configured for disaster recovery
All other services integrated with Spinnaker, such as your Continuous Integration (CI) system, are configured for disaster recovery

What a passive instance is

A passive instance means that the deployment:

Is not reachable by its known endpoints while passive (external and internal)
Does not schedule pipelines
Cannot have pipelines triggered by CI jobs

Storage considerations

Note

The storage you use should be replicated across regions since these contain all the application and pipeline definitions.

Armory recommends using a relational database for Orca and Clouddriver. For Orca, a relational database helps maintain integrity. For Clouddriver, it reduces the time to recovery. Even though any MySQL version 5.7+ database can be used, Armory recommends using AWS Aurora MySQL for the following reasons:

More performant than RDS MySQL
Better high availability than RDS MySQL
Less downtime for patching and maintenance
Support for cross-region replication

Note the following guidelines about storage and caching:

S3 buckets should be set up with cross-region replication turned on. See Replication in the AWS documentation.
Consider the following if you plan to use Aurora MySQL:
Redis - Each service should be configured to use its own Redis. With Armory services configured to use a relational database or S3 as a permanent backing store Redis is now used for caching. For disaster recovery purposes it is no longer required that Redis is recoverable. A couple things to note are:
- Gate - Users will need to login again
- Fiat - Will need to sync user permissions and warmup
- Orca - Will lose pending executions
- Rosco - Will lose bake logs
- Igor - Will lose last executed Jenkins job cursor

Kubernetes guidelines

Keep the following guidelines in mind when configuring Kubernetes.

Control plane

The Kubernetes control plane should be configured to use multiple availability zones in order to handle availability zone failure. For EKS clusters they are available across availability zones by default.

Workers

The following guidelines are meant for EKS workers:

The Kubernetes cluster should be able to support the Armory load. Use the same instance type and configure the same number of worker nodes as the primary.
There needs to be at least 1 node in each availability zone the cluster is using.
The autoscaling group has to have a proper termination policy. Use one or all of the following policies: OldestLaunchConfiguration, OldestLaunchTemplate, OldestInstance. This allows the underlying worker AMIs to be rotated more easily.
Ideally, Armory pods for each service that do not have a replica of 1 should be spread out among the various workers. This means that pod affinity/anti-affinity should be configured. With this configuration Armory will be able to handle availability zone failures better.

DNS considerations

A good way to handle failover is to set up DNS entries as a CNAME for each Armory installation.

For example:

Active Spinnaker accessible through us-west.spinnaker.acme.com and api.us-west-spinnaker.acme.com load balancers.
Passive Spinnaker accessible through us-east.spinnaker.acme.com and api.us-east-spinnaker.acme.com load balancers.
Add DNS entries spinnaker.acme.com with a CNAME pointing to us-west-spinnaker.acme.com (same for api subdomain) and a small TTL (1 minute to 5 minute).

In this setup, point your CNAME to us-east when a disaster event happens.

Note

Armory does not recommend setting up DNS with a backup IP address when manual steps are required for failover.

Setting up a passive Spinnaker

To make a passive version of Spinnaker, use the same configuration files as the current active installation for your starting point. Then, modify it to deactivate certain services before deployment.

To keep the configurations in sync, set up automation to create a passive Spinnaker configuration every time a configuration is changed for the active Spinnaker. An easy way to do this is to use Kustomize Overlays.

Configuration modifications

Make sure you set replicas for all Spinnaker services to 0. Example in SpinnakerService manifest for service gate:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 deploymentEnvironment:
 customSizing: # Configure, validate, and view the component sizings for the Armory services.
 gate:
 replicas: 0

Once you’re done configuring for the passive Spinnaker, run kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest> to deploy.

Note

Armory recommends performing a DR exercise run to make sure the passive Armory is set up correctly. Ideally, the DR exercise should include both failing over to the DR region and failing back to the primary region.

Performing disaster recovery

If the active Spinnaker is failing, the following actions need to be taken:

Activating the passive Spinnaker

Perform the following tasks when you make the passive Spinnaker into the active Spinnaker:

Use the same version of Operator to deploy the passive Spinnaker installation that was used to deploy the active Spinnaker.
AWS Aurora
- Promote another cluster in the global database to have read/write capability.
- Update SpinnakerService manifest to point to the promoted database if the database endpoint and/or the database credentials have changed.
Create the Redis clusters.
Activate the passive instance.
- Set the replicas to more than 0. Ideally, this should be set to the same number of replicas that the active Armory used.
Change the DNS CNAME if it is not already pointing to the passive Armory installation.
If the Armory that is not working is accessible, it should be deactivated

Recovery Time Objective (RTO)

Restoration time is dependent on the time it takes to restore the database, the Spinnaker services, and the time it takes to update DNS. Most Spinnaker services that fail should recover within a 10-minute timeframe. Clouddriver may take longer especially when at scale because it needs to reconnect to all configured cloud accounts. Note that services are limited to local resources, which are configured to be redundant (databases, nodes, etc.) or highly available. In addition to Clouddriver, the following services may also take additional time to restore since Redis needs time to warm up the cache:

Orca
Igor
Echo
Fiat

Recovery Point Objective (RPO)

This is the state to which Spinnaker will recover the affected systems in case of a failure, such as database corruption. The current Spinnaker RPO target is 24 hours maximum, tied to the last snapshot of the database.

Other resources

Continuous-Deployment: Configure AWS Lambda for Spinnaker™

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Armory supports using AWS Lambda as a deployment target for your apps and includes a variety of Lambda specific stages. Enabling the full suite of features for Lambda support requires updating the configurations for core Spinnaker services and adding the Lambda Plugin. Depending on how you manage Spinnaker, this requires Operator config updates.

Requirements

AWS Lambda support requires either Spinnaker 1.23+ or Armory 2.23+.

Configuration

If you are using the Armory Operator, check out the Spinnaker Kustomize Patches repo for an example on how to easily add the configurations required to enable AWS Lambda.

Enabling AWS Lambda

First, enable Lambda as a deployment target for your apps by updating the settings for Clouddriver and the UI (Deck).

In the spinnakerservice manifest, update the spinnakerConfig section to include the properties for Lambda:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 profiles:
 deck: # Enables Lambda Functions UI
 settings-local.js: |
  window.spinnakerSettings.feature.functions = true
 clouddriver: # Enables Lambda Functions in "Infrastructure"
 aws:
 features:
 lambda:
 enabled: true
 accounts:
 - name: aws-dev # NOTE: This merge is Index based - so if you do not want to overwrite spinnakerConfig.config.providers.aws.accounts you must create another account in the list
 lambdaEnabled: true
 accountId: "xxxxxxxx" # (Required)
 assumeRole: role/spinnaker # (Required)

Adding AWS Lambda Plugin

Next, add the Lambda Plugin to include the Lambda stages (Delete, Deploy, Invoke, and Route) in the UI.

#-----------------------------------------------------------------------------------------------------------------
# Example configuration for adding AWS Lambda plugin to spinnaker.
#
# Documentation: https://github.com/spinnaker-plugins/aws-lambda-deployment-plugin-spinnaker
#-----------------------------------------------------------------------------------------------------------------
apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 profiles:
 gate:
 spinnaker:
 extensibility:
 deck-proxy:
 enabled: true
 plugins:
 Aws.LambdaDeploymentPlugin:
 enabled: true
 version: 1.0.1
 repositories:
 awsLambdaDeploymentPluginRepo:
 url: https://raw.githubusercontent.com/spinnaker-plugins/aws-lambda-deployment-plugin-spinnaker/master/plugins.json
 orca:
 spinnaker:
 extensibility:
 plugins:
 Aws.LambdaDeploymentPlugin:
 enabled: true
 version: 1.0.1
 # extensions:
 # Aws.LambdaDeploymentStage:
 # enabled: true
 repositories:
 awsLambdaDeploymentPluginRepo:
 id: awsLambdaDeploymentPluginRepo
 url: https://raw.githubusercontent.com/spinnaker-plugins/aws-lambda-deployment-plugin-spinnaker/master/plugins.json

Applying config updates

Once you make the required config changes, apply them by running the command for Operator:

Assuming the Armory instance lives in the spinnaker namespace, run the following command to apply the changes:

kubectl -n spinnaker apply -f spinnakerservice.yml

Known issues

Lambda UI issue

There is a UI bug related to the caching agent that prevents Lambda functions from being displayed in the UI when there are no other clusters associated with the Application. In other words, in order for the function to show up in “Functions” tab, there needs to be a cluster (such as an AWS ASG/EC2 instance) deployed for that application.

Affected versions: 2.23.0 (1.23.0) - 2.26.2 Fixed version: 2.26.3

References

See the following links for more information:

Continuous-Deployment: Configure AWS Networking for Spinnaker

Mon, 01 Jan 0001 00:00:00 +0000

AWS resources

AWS VPC guides
AWS VPCs and subnets

Configuring subnets

Spinnaker groups subnets into a single subnet name across multiple availability zones. This makes it simpler for end-users of Spinnaker to choose a group of subnets within a VPC that have a given purpose such as ec2-subnets, elb-subnets or public-subnets. This allows Spinnaker to place the machines within that group and ensure equal redundancy across zones. Below is a logical representation of how Spinnaker groups multiple subnets together. If you want to make a subnet accessible to Spinnaker you’ll have to add a tag and value to the subnet with the following: immutable_metadata={"purpose":"example-purpose"}

Conceptually, this is how Spinnaker groups subnets logically.

Verifying subnet configuration

Once you configured the purpose of your subnets you can use the Spinnaker API to double check that settings have been noticed. It takes between 30 seconds and 2 minutes for the changes to be picked up. After that time period you can run:

curl http://<YOUR_GATE_ENDPOINT>/subnets/aws

You can expect to receive a response similar to:

[
 {
 "account": "default-aws-account",
 "availabilityZone": "us-west-1b",
 "availableIpAddressCount": 4088,
 "cidrBlock": "172.31.0.0/20",
 "deprecated": false,
 "id": "subnet-7bd69322",
 "purpose": "external",
 "region": "us-west-1",
 "state": "available",
 "target": null,
 "type": "aws",
 "vpcId": "vpc-63327b06"
 }
]

If the purpose field is non-null then things are configured correctly.

I don’t see my subnets or VPCs

Spinnaker caches as much as possible to keep performance through the UI responsive. If you don’t see the subnets and you believe you configured them correctly, then make sure to refresh the cache. You can find the cache going to the config section of your application and clicking refresh all caches. You should also make sure to refresh your browswer cache by using your browser’s development tools and deleting any browser databases.

Continuous-Deployment: Configure Amazon Simple Storage Service (S3) Artifacts

Mon, 01 Jan 0001 00:00:00 +0000

S3 artifact configuration

The example on this page describes how to reference a Helm chart tarball for later use during deployment.

This is a quick walkthrough of how to configure Spinnaker^TM and Armory to access an S3 bucket as a source of artifacts. Many of the configurations below have additional options that may be useful (or possibly required).

Enable S3 artifacts

If you’ve just installed Spinnaker or Armory, you need to enable S3 as an artifact source.

Add the following snippet to SpinnakerService manifest:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 features:
 artifacts: true
 artifacts:
 s3:
 enabled: true

Add S3 account

You only need to configure the S3 credentials as an account – all buckets that account has access to can be referenced after that.

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 features:
 artifacts: true
 artifacts:
 s3:
 enabled: true
 accounts:
 - name: my-s3-account
 region: us-west-2 # S3 region
 awsAccessKeyId: ABCDEF01234... # Your AWS Access Key ID. If not provided, Spinnaker will try to find AWS credentials as described at http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default
 awsSecretAccessKey: abc # Your AWS Secret Key. This field supports "encrypted" secret references

Apply your changes with kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest>.

Continuous-Deployment: Connect Spinnaker to Amazon Elastic Container Registry

Mon, 01 Jan 0001 00:00:00 +0000

Adding ECR as a Docker registry

When configuring a registry, you normally use standard SpinnakerService configuration when using the Operator.

Update your Spinnaker installation

The configuration below must go under spinnakerConfig.config.providers, as explained in Connect Docker Registries.

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 providers:
 dockerRegistry:
 enabled: true
 primaryAccount: dockerhub
 accounts:
 - name: dockerhub
 requiredGroupMembership:
 providerVersion: V1
 address: 012345678910.dkr.ecr.us-east-1.amazonaws.com
 username: AWS
 passwordCommand: "aws --region us-east-2 ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken' | base64 -d | sed 's/^AWS://'"

Success! Now you can use ECR as a Docker registry in the configuration stage.

Continuous-Deployment: Expose Spinnaker on AWS EKS

Mon, 01 Jan 0001 00:00:00 +0000

DNS Preparation

In this tutorial, you set up two CNAME entries in your DNS. You won’t be able to actually configure the DNS until you get an A record from AWS after creating the LoadBalancer, but you need to select the names in order to configure the LoadBalancer. This example uses demo.armory.io to be the Deck service (the UI), and gate.demo.armory.io to be the Gate service (the API).

Exposing Armory on EKS with a public Load Balancer

Create a LoadBalancer service

While there are many ways to expose Armory, we find the method described in this post to be the easiest way to get started. If your organization has other requirements, this post may be helpful as you start working through the process.

Update your SpinnakerService manifest with the following example expose configuration, which will automatically create one Kubernetes service LoadBalancer for the API (Gate) and one for the UI (Deck):

apiversion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 ... # rest of config omitted for brevity
 expose:
 type: service
 service:
 type: LoadBalancer

Save and apply the configuration. After some time, you can see the LoadBalancer CNAMEs that were created:

NAMESPACE={spinnaker namespace}
API_URL=$(kubectl -n $NAMESPACE get spinsvc spinnaker -o jsonpath='{.status.apiUrl}')
UI_URL=$(kubectl -n $NAMESPACE get spinsvc spinnaker -o jsonpath='{.status.uiUrl}')

Secure with SSL on EKS

This tutorial presumes you’ve already created a certificate in the AWS Certificate Manager.

Update and apply the SpinnakerService manifest to specify the DNS names for Gate and Deck, and to provide annotations specific for EKS LoadBalancers:

apiversion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 security:
 apiSecurity:
 overrideBaseUrl: https://spinnaker-gate.armory.io # Specify your DNS name for Gate with https scheme
 uiSecurity:
 overrideBaseUrl: https://spinnaker.armory.io # Specify your DNS name for Deck with https scheme
 ... # rest of config omitted for brevity
 expose:
 type: service
 service:
 type: LoadBalancer
 annotations:
 service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <ACM CERT ARN> # Replace with your cert ARN
 service.beta.kubernetes.io/aws-load-balancer-ssl-ports: 80,443

Assuming that Armory is installed in spinnaker namespace:

kubectl -n spinnaker apply -f spinnakerservice.yml

Enabling sticky sessions

If your Armory installation will be using authentication and you expect to scale the API server (Gate) beyond more than one instance you’ll want to enable sticky sessions. This will ensure that clients will connect and authenticate with the same server each time. Otherwise, you may be forced to reauthenticate if you get directed to a new server. To enable sticky sessions, you’ll want to enable session affinity on the Gate service created above.

GATE_SVC=<spin-gate/spin-gate-public> # spin-gate
kubectl -n ${NAMESPACE} patch service/$GATE_SVC --patch '{"spec": {"sessionAffinity": "ClientIP"}}'

For more details about session affinity, see the Kubernetes documentation on Services.

Exposing Armory on EKS with an internal Load balancer

In this option the goal is to use AWS ALB’s of type internal for exposing Armory only within an organization’s private VPC. This consists of 3 steps: configuring Kubernetes services of type NodePort, creating AWS internal ALB’s and updating Armory with final DNS names.

Step 1: Create Kubernetes NodePort services

A NodePort Kubernetes service opens the same port (automatically chosen) on all EKS worker nodes, and forwards requests to internal pods. In this case we’ll be creating two services: one for Deck (Armory’s UI) and one for Gate (Armory’s API).

apiversion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 ... # rest of config omitted for brevity
 expose:
 type: service
 service:
 type: NodePort
 ... # rest of config omitted for brevity

Assuming that Armory is installed in spinnaker namespace:

kubectl -n spinnaker apply -f spinnakerservice.yml

After a few seconds you can view which ports were opened in EKS worker nodes, you’ll need them in the next step:

DECK_PORT=$(kubectl get service spin-deck -o jsonpath='{.spec.ports[0].nodePort}')
GATE_PORT=$(kubectl get service spin-gate -o jsonpath='{.spec.ports[0].nodePort}')

Step 2: Create AWS internal load balancers

We’ll describe how to create these load balancers from AWS console, but you can use any preferred method for provisioning infrastructure. We’ll create a Load Balancer for Deck and other for Gate.

Navigate to AWS EC2 management console, in Load Balancers section, and click on Create New Load Balancer

We’ll be creating a new Application Load Balancer:

Make sure to select internal scheme, and if you have a SSL certificate available, use HTTPS protocol:

Select the VPC and subnets where EKS worker nodes live:

If you selected HTTPS for the protocol, you can configure here the ACM certificate:

In the next screen you can either select an existing security group or create a new one for your load balancer:

Now you want to create a new target group that points to DECK_PORT or GATE_PORT, taken from the NodePort created in the previous step:

Finally, you need to select all EKS worker nodes to be registered with the load balancer target, review, and save the changes:

If for some reason you get Unhealthy status in the target group you created, make sure that EKS worker nodes security groups allow traffic to the target ports, at least from Load Balancer’s security groups.

Finally repeat the same steps for creating Gate Load balancer.

Step 3: Update Armory configuration

Armory needs to know which url’s are used to access it. After you have updated your DNS with the Load Balancers CNAME’s created in the previous step, the next step is to update Armory configuration:

Update and apply the SpinnakerService manifest:

apiversion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 security:
 apiSecurity:
 overrideBaseUrl: https://spinnaker-gate.armory.io # Specify your DNS name for Gate with https scheme
 uiSecurity:
 overrideBaseUrl: https://spinnaker.armory.io # Specify your DNS name for Deck with https scheme
 ... # rest of config omitted for brevity

Assuming that Armory is installed in spinnaker namespace:

kubectl -n spinnaker apply -f spinnakerservice.yml

Exposing Armory on GKE with Ingress

Setting up HTTP Load Balancing with Ingress

GKE has a “built-in” ingress controller and that’s what we will use.

First create a file called basic-ingress.yaml and paste it the following

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: basic-ingress
spec:
 rules:
 - host: demo.armory.io
 http:
 paths:
 - backend:
 serviceName: spin-deck
 servicePort: 80
 path: /
 - host: gate.demo.armory.io
 http:
 paths:
 - backend:
 serviceName: spin-gate
 servicePort: 80
 path: /

Then apply this kubectl apply -f basic-ingress.yaml

Find out the external IP address of the load balancer serving your application by running: kubectl get ingress basic-ingress

Output:

NAME HOSTS ADDRESS PORTS AGE
basic-ingress demo.armory.io, gate.demo.armory.io 203.0.113.12 80 2m

Note: It may take a few minutes for GKE to allocate an external IP address and set up forwarding rules until the load balancer is ready to serve your application. In the meanwhile, you may get errors such as HTTP 404 or HTTP 500 until the load balancer configuration is propagated across the globe.

You need to update your DNS records to have the demo.armory.io host point to the IP address generated.

Now tell Armory about its external endpoints:

Update and apply the SpinnakerService manifest:

apiversion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 config:
 security:
 apiSecurity:
 overrideBaseUrl: http://gate.demo.armory.io # Specify your DNS name for Gate
 uiSecurity:
 overrideBaseUrl: http://demo.armory.io # Specify your DNS name for Deck
 ... # rest of config omitted for brevity

Assuming that Armory is installed in spinnaker namespace:

kubectl -n spinnaker apply -f spinnakerservice.yml

After doing that you can visit http://demo.armory.io/ to view Armory.

Secure with SSL on GKE

To enable SSL and configure your certificates you can follow this guide: https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-multi-ssl

HTTP/HTTPS Redirects

You must enable HTTP/HTTPS redirects when your Armory deployment fits the following description:

TLS encryption for Deck (UI) and Gate (API) for Armory
A load balancer (service, ingress, etc.) in front of your Deck/Gate that terminates TLS and forwards communications to the Armory microservices.

To enable redirects, complete the following steps:

Update the SpinnakerService manifest with the following:

apiversion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 profiles:
 gate:
 server:
 tomcat:
 protocolHeader: X-Forwarded-Proto
 remoteIpHeader: X-Forwarded-For
 internalProxies: .*
 httpsServerPort: X-Forwarded-Port
 ... # rest of config omitted for brevity

Assuming that Armory is installed in spinnaker namespace:

kubectl -n spinnaker apply -f spinnakerservice.yml

Finally, clear your cache.

For an alternative solution, see the following Knowledge Base article: Troubleshooting http/https redirects with authentication.