Armory Docs – Advanced Installation and Configuration

Plugins: Armory Scale Agent Service Deployment Modes

Mon, 01 Jan 0001 00:00:00 +0000

Deployment modes

Spinnaker Service mode

In this mode, you install the Armory Scale Agent service as a new Spinnaker service (spin-armory-agent), so you can configure it like other services.

If you provision clusters automatically, the Armory Scale Agent service can dynamically reload accounts when armory-agent.yaml changes. You could, for example, configure accounts in a ConfigMap mounting to /opt/armory/config/armory-agent-local.yaml. The Agent service reflects ConfigMap changes within seconds after etcd sync.

Pros

You do not have to configure external Network/Application Load Balancers to expose the Clouddriver gRPC port unless you want to use the Dynamic Accounts REST API.
The Agent service can be managed centrally, so it is easy to get logs or configs and upgrade.

Cons

You need to create kubeconfig file for each Kubernetes account.
The API servers of the target clusters need to be accessible from the Armory CD (Spinnaker) cluster.

Infrastructure mode

In infrastructure mode, multiple Agent service deployments handle different groups of Kubernetes clusters. Each service deployment is configured separately.

Keep the following pros and cons in mind when deciding if Infrastructure mode fits your use case:

Pro

The Agent service can be managed centrally, so it is easy to get logs or configs and upgrade.

Cons

You need to create a kubeconfig file for each Kubernetes account.
The API servers of the target clusters need to be accessible from the Armory Scale Agent cluster.
You need to expose gRPC port for Clouddriver through an external load balancer capable of handling HTTP/2 for gRPC communication.

Kubernetes account names must be unique across all your infrastructure. Clouddriver rejects new accounts with a name that matches a different cluster.

Agent mode

In this mode, the Armory Scale Agent service acts as a piece of infrastructure. It authenticates using a service account token. You use RBAC service account permissions to configure what the Armory Scale Agent service is authorized to do.

If the Clouddriver plugin is unable to communicate with the Scale Agent service, the plugin attempts to reconnect during a defined grace period. If the plugin still can’t communicate with the Scale Agent service after the grace period has expired, the plugin removes the cluster associated with that Scale Agent service from Armory CD.

In this mode, Armory CD never gets credentials, and Kubernetes account registration is dynamic.

Keep the following pros and cons in mind when deciding if Agent mode fits your use case:

Pros

Agent mode is scalable because each agent only manages one Kubernetes account.
Initial setup is easier because there’s no need to create kubeconfig files.
Target clusters can remain private or in separate VPCs from the Armory CD (Spinnaker) cluster because Scale Agents initiate the connection to Armory CD (Spinnaker).

Cons

It is difficult to get agent logs, upgrade the Agent service, or check configurations because agents (often) run in third-party clusters that the DevOps team operating Armory CD (Spinnaker) doesn’t have access.
There is no authentication/authorization, so any team can start an Agent and register itself with Armory CD (Spinnaker). mTLS encryption can be used so that only agents with the right certificate can register. For information about how to configure mTLS, see Configure Mutual TLS Authentication.
You need to expose gRPC port for Clouddriver through an external load balancer capable of handling HTTP/2 for gRPC communication.

Scalability

Each Agent can scale to hundreds of Kubernetes clusters. The more types of Kubernetes objects the Armory Scale Agent has to watch, the more memory it uses. Memory usage is bursty. You can control burst with budget. See Agent options for configuration information.

Scaling the Armory Scale Agent can mean:

Scaling the Kubernetes Deployment the Armory Scale Agent service is part of.
Sharding Kubernetes clusters into groups that can in turn be scaled as described in Infrastructure mode above.

You can also mix deployment strategies if you have complex Kubernetes infrastructure and permissions:

Run as a Spinnaker service
Run in target clusters
Run next to traditional Spinnaker Kubernetes accounts

Plugins: Armory Scale Agent Service Deployment Methods

Mon, 01 Jan 0001 00:00:00 +0000

Plugins: Armory Scale Agent Service Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Where to configure the Armory Scale Agent

If you install the Armory Scale Agent service using kubectl, configure the Armory Scale Agent using a ConfigMap. See the kubectl installation instruction’s Configure the service section for an example.
See Deploy the Armory Scale Agent Service Using a Helm Chart for both CLI and values.yml configuration options.

Configure Kubernetes accounts

You can configure Kubernetes accounts in Spinnaker in multiple places:

Clouddriver configuration files: clouddriver.yml, clouddriver-local.yml, spinnaker.yml, spinnaker-local.yml
Spring Cloud Config Server reading accounts from Git, Vault, or another supported backend
Plugins

Behavior when reading Kubernetes account configuration from multiple sources:

When you configure different accounts in Agent and Armory CD, Agent merges both sources.
If you configure an account with the same name in Agent as well as Armory CD (Clouddriver or Spring Cloud Config Server backends), Agent account configuration always overrides the Armory CD configuration.
If you configure an account with the same name in Agent as well as in the Clouddriver plugin, the account that is used depends on the order of precedence defined in the plugin’s CredentialsDefinitionSource interface. The Agent has an order precedence of 1000. Check with your plugin provider for the plugin’s order of precedence.
- If the plugin has a higher precedence than the Armory Scale Agent, the plugin’s account is used. For example, if the plugin’s precedence is 500, the plugin’s account is used.
- If the plugin has no precedence defined or a lower precedence than the Armory Scale Agent, the account defined in the Armory Scale Agent is used.

Migrate accounts from Clouddriver to Agent

Copy the account definition from its original source, such as Clouddriver, to Agent’s configuration. Depending on how you installed Agent, this configuration could be in the armory-agent.yml data section of a ConfigMap or in the armory-agent.yml file in the Armory Scale Agent pod.

Agent may not use all the properties you copy from the original source definition. Unused properties are ignored.

Agent mode

In Agent mode, your configuration should look similar to this:

kubernetes:
 accounts:
 - name: account-01
 serviceAccount: true
 ...

Spinnaker Service and Infrastructure modes

You set up multiple accounts per Agent in these modes. Your configuration should look similar to this:

kubernetes:
 accounts:
 - name: account-name-01
 kubeconfigFile: /kubeconfigfiles/kubecfg-account01.yaml
 - name: account-name-02
 kubeconfigFile: /kubeconfigfiles/kubecfg-account02.yaml
 - ...

Automatically migrate accounts from Clouddriver to Agent

The Dynamic Accounts feature enables manual account migration using a REST API or automatic account migration via request interception or a scanning mechanism. See Dynamic Accounts Architecture and Features.

Migrate accounts from Agent to Clouddriver

Follow these steps to migrate accounts from Agent to Clouddriver:

Delete the account definition from your Agent configuration. Depending on how you installed Agent, this configuration could be in the armory-agent.yml data section of a ConfigMap or in the armory-agent.yml file in the Armory Scale Agent pod.
Add the account definition to the source that Clouddriver uses.

Permissions format

Permissions for the Armory Scale Agent use a format that is slightly different than the format that Clouddriver uses for permissions. Define your permissions like this:

kubernetes:
 accounts:
 - name: my-k8s-account
 permissions:
 - READ: ['role1', 'role2']
 - WRITE: ['role3', 'role4']

Make sure that there are no whitespaces in the role configurations under the READ or WRITE tags. The permissions are not applied if there are whitespaces in the configuration. This means that all users will have access to the account.

Restricted environments

Network access

The Agent needs access to its control plane (Spinnaker) as well to the various Kubernetes clusters it is configured to monitor. You can control which traffic should go through an HTTP proxy via the usual HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables.

A common case is to force the connection back to the control plane via a proxy but bypass it for Kubernetes clusters. In that case, define the environment variable HTTPS_PROXY=https://my.corporate.proxy and use the kubernetes.noProxy: true setting to not have to maintain the list of Kubernetes hosts in NO_PROXY.

Kubernetes authorization

The Agent should be configured to access each Kubernetes cluster it monitors with a service account. You can limit what Spinnaker can do via the role you assign to that service account. For example, if you’d like Spinnaker to see NetworkPolicies but not deploy them:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: agent-role
rules:
- apiGroups: ["networking.k8s.io"]
 resources: ["networkpolicies"]
 verbs: [ "get", "list", "watch"]
...

Namespace restrictions

You can limit the Armory Scale Agent to monitoring specific namespaces by listing them under namespaces. If you need to prevent the Armory Scale Agent from accessing cluster-wide (non-namespaced) resources, use the onlyNamespacedResources setting.

A side effect of disabling cluster-wide resources is that CustomResourceDefinitions won’t be known (and therefore deployable by Spinnaker). CustomResourceDefinitions are cluster-wide resources, but the custom resources themselves may be namespaced. To workaround the limitation, you can define customResourceDefinitions. Both namespaces and CRDs are sent to Spinnaker as “synthetic” resources. They won’t be queried or watched, but Spinnaker knows about their existence.

kubernetes:
 accounts:
 - name: production
 ...
 # Restricts the agent to namespaces `ns1` and `ns2`
 namespaces:
 - ns1
 - ns2
 # Prevents the Armory Scale Agent from querying non-namespaced resources
 onlyNamespacedResources: true
 # Whitelist CRDs so Spinnaker
 customResourceDefinitions:
 - kind: ServiceMonitor.monitoring.coreos.com
 - kind: SpinnakerService.spinnaker.armory.io

What’s next

Armory Scale Agent Service Configuration Options

Plugins: Install the Scale Agent Plugin Using a Docker Image

Mon, 01 Jan 0001 00:00:00 +0000

Halyard local config

Warning

The Scale Agent plugin extends Clouddriver. When Halyard adds a plugin to a Spinnaker installation, it adds the plugin repository information to each service. This means that when you restart Spinnaker, each service restarts, downloads the plugin, and checks if an extension exists for that service. Each service restarting is not ideal for large Spinnaker installations due to service restart times. To avoid each service restarting and downloading the plugin, configure the plugin in Clouddriver’s local profile.

Create a custom service setting that mounts an empty volume inside Clouddriver for the plugin image. Add the following to your .hal/default/service-settings/clouddriver.yml file:
```
kubernetes:
 volumes:
 - id: kubesvc-plugin-vol
 type: emptyDir
 mountPath: /opt/clouddriver/lib/plugins
```
Define an init container for the plugin. In the deploymentConfigurations.name.deploymentEnvironment.initContainers section of your .hal/config, add the following:
```
spin-cloudriver:
- name: kubesvc-plugin
 image: docker.io/armory/kubesvc-plugin:<version>
 volumeMounts:
 - mountPath: /opt/plugin/target
 name: kubesvc-plugin-vol
```
Be sure to use the plugin version compatible with your Spinnaker version.

Add the plugin to your clouddriver-local.yml.

spinnaker:
 extensibility:
 plugins:
 Armory.Kubesvc:
 enabled: true
 extensions:
 armory.kubesvc:
 enabled: true
 pluginsRootPath: /opt/clouddriver/lib/plugins
kubesvc:
 cluster: kubernetes
kubernetes:
 enabled: true
 # enable Clouddriver Account Management if you are using Spinnaker 1.28+
 account:
 storage:
 enabled: true

Apply your changes by running hal deploy apply.

Validate plugin installation

Confirm the plugin Docker image exists locally.
```
docker images | grep kubesvc
```
Output is similar to:
```
armory/kubesvc-plugin 0.11.32
```
Find the name of the new Clouddriver pod.
```
kubectl -n spinnaker get pods
```

Confirm the Clouddriver service is using the Docker image.

kubectl -n spinnaker describe pod <clouddriver-pod-name> | grep Image:

Output is similar to:

Image: docker.io/armory/kubesvc-plugin:0.11.32

View the Clouddriver log to verify that the plugin has started.

kubectl -n spinnaker logs deployments/spin-clouddriver | grep "Plugin"

Output is similar to:

org.pf4j.AbstractPluginManager : Plugin 'Armory.Kubesvc@0.11.32' resolved
org.pf4j.AbstractPluginManager : Start plugin 'Armory.Kubesvc@0.11.32'
io.armory.kubesvc.KubesvcPlugin : Starting Kubesvc plugin...

Armory Operator or Spinnaker Operator

You can find the Kustomize file plugin-container-patch.yml in the spinnaker-patches-kustomize repo.

Change the value for metadata.name if your Armory CD service is called something other than “spinnaker”.

Then include the file under the patchesStrategicMerge section of your kustomization file.

patchesStrategicMerge:
 - <path>/plugin-container-patch.yml

Apply your changes.