Armory Docs – Armory Scale Agent for Spinnaker and Kubernetes Concepts

Plugins: Armory Scale Agent Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Key Components

Scale Agent plugin for Clouddriver

The Scale Agent plugin runs inside Clouddriver and manages migrated accounts and accounts created specifically for Scale Agent management. The plugin does not engage in outbound communication. Through Clouddriver, the plugin exposes an internally-accessible REST API that you can use to dynamically migrate and manage Kubernetes accounts.

Scale Agent service for Kubernetes

You Scale Agent service monitors your Kubernetes clusters and sends information to the Scale Agent plugin running inside Clouddriver.

The Agent service sends the following information about the cluster it is watching to the Clouddriver plugin:

Account properties as configured in kubernetes.accounts[].
Kubernetes API server host, certificate fingerprint, and version.
All the Kubernetes objects it is configured to watch and has permissions to access. You can ignore certain Kubernetes kinds (kubernetes.accounts[].omitKinds) or configure specific kinds to watch (kubernetes.accounts[].kinds).

The Agent service always scrubs data from Secret in memory before it is sent and even before that data makes it onto the Armory Scale Agent’s memory heap.

Kubernetes permissions for the service

The Agent service should have ClusterRole authorization if you need to deploy pods across your cluster. If you only deploy pods only to a single namespace, the service needs Role authorization.

See Kubernetes Permissions for the Armory Scale Agent for detailed information.

Communication and networking

Communication from the Scale Agent service to the Clouddriver plugin occurs over gRPC port 9091. Communication between the service and the plugin must be http/2. http/1.1 is not compatible and causes communication issues between the Armory Agent service and Clouddriver plugin.

Except for a local health check, the Agent service makes outbound calls only to the Clouddriver plugin over a single gPRC connection. The connection can be over TLS or mTLS. You can terminate TLS:

On Clouddriver: in the case of running the Armory Scale Agent in Spinnaker Service mode or if declaring spin-clouddriver-grpc as a network load balancer.
On a gRPC proxy that directs request to the spin-clouddriver-grpc service.

The Clouddriver plugin uses the bidirectional communication channel to receive changes from Kubernetes accounts as well as send operations to the Armory Scale Agent service.

See the Armory Scale Agent Communication With Clouddriver Instances in Kubernetes page for detailed information.

Security

Since the Armory Scale Agent service does outbound calls only, you can have Agent services running on-premises or in public clouds such as AWS, GCP, Azure, Oracle, or Alibaba.

What the Scale Agent service can do in the target cluster is limited by what it is running as:

a serviceAccount in Agent mode
a kubeconfig setup for infrastructure or Spinnaker service mode

Communications are secured with TLS and optionally mTLS.

What’s next

Plugins: Armory Scale Agent Communication With Clouddriver Instances in Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

How the Armory Scale Agent plugin communicates with Clouddriver instances

At startup, Clouddriver registers a watch for the kind Endpoints in the Kubernetes cluster where it is running for the namespace where Clouddriver is running. Objects of kind Endpoints are automatically generated based on a Kubernetes Service. The plugin knows that there’s a Clouddriver Service for routing HTTP traffic to Clouddriver pods (usually named spin-clouddriver). This means that the Clouddriver pod needs to mount a Kubernetes Service Account that has permissions to list and watch the kind Endpoints in the current namespace. The Agent plugin does the equivalent of these calls at startup:

kubectl auth can-i list endpoints
kubectl auth can-i watch endpoints

If permissions are in place, the plugin then does the equivalent of this call:

kubectl get endpoints

From the returned collection, the plugin filters the entries based on these conditions:

The endpoint Name should have the prefix spin-clouddriver. This is configurable and is a prefix to be able to match several entries like the ones used in HA.
Each entry should have a port named http, which is configurable. This is to be able to differentiate between the port that Clouddriver uses to receive REST requests and the port it uses to listen for gRPC connections from Agent.

Information about discovered Clouddriver instances is kept in memory on each Clouddriver pod and is automatically updated by the watch mechanism. No polling is needed. The Kubernetes API server notifies the plugin when there are any changes.

Operation request and response forwarding

When a Clouddriver instance receives an operation for an account or Agent that is registered to a different Clouddriver instance, the plugin uses a combination of a database table and an in-memory map to determine where to forward the operation. Forwarding is done via REST requests.

Exposed endpoints

GET <clouddriver host:port>/armory/clouddrivers

Sample response:

[
 {
 baseUrl: "http://10.0.15.246:7002",
 id: "spin-clouddriver-766c678c6c-scg68",
 lastUpdated: "2022-02-23T20:41:39.333Z",
 ready: true
 }
]

The field ready matches the column Ready of kubectl -n <spinnaker ns> get pods for Clouddriver pods.

POST <clouddriver host:port>/armory/agent/operations

Sample request body:
```
{
 "operation": "<toString() of protobuf Operation object>"
}
```
POST <clouddriver host:port>/armory/agent/operationresults

Sample request body:
```
{
 "result": "<toString() of protobuf OperationResult object>"
}
```

Plugins: Dynamic Accounts Architecture and Features

Mon, 01 Jan 0001 00:00:00 +0000

Overview of Dynamic Accounts

Rather than manually configure each Scale Agent service with the Kubernetes accounts it should manage, you can use the Dynamic Accounts feature to migrate and manage your accounts. Dynamic Accounts provides:

Manual migration of Clouddriver Kubernetes accounts to the Scale Agent using a REST API
Automatic migration of Clouddriver Kubernetes accounts using Clouddriver Account Management
- Automatic migration requires Armory Continuous Deployment 2.28+ or Spinnaker 1.28+.
- Clouddriver Account Management is not enabled by default in Spinnaker or Armory Continuous Deployment. See Spinnaker’s Clouddriver Account Management for how to enable this feature in your Spinnaker instance.

REST API

The Dynamic Accounts REST API provides endpoints to create, delete, get, migrate, and update Kubernetes accounts. You can’t access these endpoints through Gate. You should have kubectl access to your Spinnaker cluster and port-forward to be able to call the API.

kubectl port-forward deployment/spin-clouddriver 7002:7002 -n spinnaker

You can then access endpoints via http://localhost:7002.

How to enable and use Dynamic Accounts

First, familiarize yourself with the architecture and features in this guide. Then you can:

Dynamic Accounts glossary

Account: an abstraction of a target cluster or target set of namespaces within a cluster
Credentials source: any source from which credentials/accounts are read
Endpoint: the URL segment after the Clouddriver root
Migrate an account: move a Clouddriver-managed account to the Scale Agent for management
Request: an instruction that isn’t fulfilled immediately and can have different outcomes; a request can be done through HTTP by the admin or internally by one of the services.

Architecture

The Scale Agent stores account data in a dedicated table called clouddriver.kubesvc_accounts. It does not modify or delete the account data in the original credential source.

An account has the following lifecycle states:

Non-transient:
- INACTIVE: This is the initial state when a user adds an account. The account is waiting for a migration operation.
- ACTIVE: Scale Agent watches and manages the account.
- FAILED: Scale Agent failed to add or delete an account.
- ORPHANED: Neither Scale Agent nor Clouddriver is managing the account. It is inactive. You usually see this state when restarting or bringing down all of the replicas managing that account.
Transient
- TO_MIGRATE: The account is waiting for migration.
- ACTIVATING: The account transferred to Scale Agent for activation.
- TO_DEACTIVATE: Indicates there is an instruction to deactivate the account.
- DEACTIVATING: A request to stop watching the account has been sent to the Scale Agent.

Manual account migration

Migration of an account is the combination of taking the snapshot from a credential source and then activating the accounts.

sequenceDiagram
actor User
participant Plugin
participant Service
User->>Plugin: POST /armory/accounts
Plugin->>Plugin: Store in clouddriver.kubesvc_accounts
User->>Plugin: PATCH /armory/accounts
Plugin->>Service: gRPC AddAccounts
Service-->>Plugin: return
Plugin->>Plugin: Update status in clouddriver.kubesvc_accounts

POST PATCH

POST	PATCH
`curl --request POST \ --url http://clouddriver:7002/armory/accounts \ --header 'Content-Type: application/json' \ --data '[ { "name": "account-01" }, { "name": "account-02", "zoneId": "agent-1_namespace1", "kubeconfigFile": "encryptedFile:k8s!n:kubeconfig!k:config!ns:spinnaker" } ]'`	`curl --request PATCH \ --url http://localhost:7002/armory/accounts \ --header 'Content-Type: application/json' \ --data '[ { "name": "account-01", "state": "ACTIVE" }, { "name": "account-02", "state": "ACTIVE", "zoneId": "agent-1_namespace1", "kubeconfigFile": "encryptedFile:k8s!n:kubeconfig!k:config!ns:spinnaker" } ]'`

 curl --request POST \
 --url http://clouddriver:7002/armory/accounts \
 --header 'Content-Type: application/json' \
 --data '[
 {
 "name": "account-01"
 },
 {
 "name": "account-02",
 "zoneId": "agent-1_namespace1",
 "kubeconfigFile": "encryptedFile:k8s!n:kubeconfig!k:config!ns:spinnaker"
 }
 ]'

curl --request PATCH \
 --url http://localhost:7002/armory/accounts \
 --header 'Content-Type: application/json' \
 --data '[
 {
 "name": "account-01",
 "state": "ACTIVE"
 },
 {
 "name": "account-02",
 "state": "ACTIVE",
 "zoneId": "agent-1_namespace1",
 "kubeconfigFile": "encryptedFile:k8s!n:kubeconfig!k:config!ns:spinnaker"
 }
]'

See Migrate Clouddriver Kubernetes Accounts to the Armory Scale Agent for detailed instructions and examples.

What happens when you initiate a migration request

If you do not include a zoneId, the plugin sends the request to every connected Scale Agent service in an attempt to find one that can process the request. The account’s zoneId is updated to that of the Agent service that is able to process the add request.
If you do provide a zoneId, the plugin forwards the request only to matching Agent services.
Before sending account data to the Agent service, the plugin decrypts the kubeconfig secret and encodes it in Base64. Then the plugin adds the encoded content as an attribute of the account. The secret data remains encoded in the database; it is not stored in plain text.
The plugin stores the account data in the clouddriver.kubesvc_accounts before sending the accounts to the relevant Scale Agent services for processing.

To reduce overhead, create the kubeconfig with only the minimum requirements.

What happens when the service receives a request to add accounts

After receiving the set of accounts, the Scale Agent service parses each kubeconfig and fetches the certificate information from the specified cluster. After the fetch succeeds, the Scale Agent service initiates a process that discovers every Kubernetes kind in the target cluster for initiation of a Kubernetes watch.
Next, the Scale Agent service creates a new gRPC connection as a response to tell the Scale Agent plugin which accounts are now active. The plugin then updates account data in the clouddriver.kubesvc_accounts table.

The plugin does not inform you of the operation results due to potentially long processing time. The more accounts you send, the longer the operation takes to complete. You can check for an ACTIVE account state in the clouddriver.kubesvc_accounts table by querying the database directly or by calling /agents/kubernetes/accounts/{accountName}.

Automatic account migration

This feature requires you to enable Clouddriver Account Management in Spinnaker.

You can configure the Scale Agent to scan for newly created Clouddriver accounts that match a specified pattern and then migrate those accounts to Scale Agent management. This replaces manual requests to migrate and activate accounts.

Clouddriver Account Management API request interception

This feature requires you to enable Clouddriver Account Management in Spinnaker.

The Scale Agent can intercept an account creation requests sent to the Cloudddriver Account Management API POST <GATE_URL>/credentials endpoint and add those accounts to Scale Agent management.

The intercept feature provides the following additional functionality:

Account modification: When you send an update request to PUT <GATE_URL>/credentials, the Scale Agent checks to see if the account exists in the clouddriver.kubesvc_accounts table and if found, updates the account and send the changes to the associated Scale Agent service.
Account deletion: When you send a request to DELETE <GATE_URL>/credentials, the Scale Agent checks to see if the account exists in the clouddriver.kubesvc_accounts table and if found, deletes the account.

Account management

Dynamic Accounts provides create, read, update, and delete endpoints for managing accounts stored in clouddriver.kubesvc_accounts. The following illustrates what happens when you initiate a deletion request.

sequenceDiagram
actor User
participant Plugin
participant Service
User->>Plugin: DELETE /armory/accounts
Plugin->>Plugin: checks for the account in kubesvc_accounts
Plugin->>Service: gRPC DeleteAccounts
Service-->>Plugin: return
Plugin->>Plugin: delete in clouddriver.kubesvc_accounts

See Manage Kubernetes Accounts in the Armory Scale Agent for detailed instructions and examples.

Failures and retry mechanism

If the Scale Agent cannot add an account, it updates the account state to FAILED in the clouddriver.kubesvc_accounts table. You can find the reason in the error_message column and the number of attempts in the failed_count column.

The Scale Agent plugin has an automatic retry mechanism for FAILED accounts. You can configure the max retries and the frequency of retries by setting kubesvc.dynamicAccounts.retryFrequencySeconds (default: 5) and kubesvc.dynamicAccounts.maxRetries (default: 3) in the plugin configuration.

If an account is manually patched to ACTIVE using the API, the failed_count resets and the retries can start over.

Automatic recovery of orphaned accounts

When all of the Scale Agent services managing a specific account die, the account goes into an ORPHANED state. The account state changes back to ACTIVE the next time the managing Scale Agent service becomes active. The same happens when a new replica is scaled up for an ACTIVE account. That way consistency is maintained across replicas.

What’s next

Plugins: Kubernetes Permissions for the Armory Scale Agent

Mon, 01 Jan 0001 00:00:00 +0000

Permissions

The Scale Agent service can use a kubeconfig file loaded as a Kubernetes secret when deploying to a remote cluster. Also, you can configure Agent permissions using a Kubernetes Service Account when deploying to the cluster the Armory Scale Agent resides in.

The Scale Agent service should have ClusterRole authorization if you need to deploy pods across your cluster or Role authorization if you deploy pods only to a single namespace.

If Scale Agent service is running in Agent Mode, then the ClusterRole or Role is the one attached to the Kubernetes Service Account mounted by the Armory Scale Agent pod.
If Scale Agent service is running in any of the other modes, then the ClusterRole or Role is the one the kubeconfigFile uses to interact with the target cluster. kubeconfigFile is configured in armory-agent.yml of the Armory Scale Agent pod.

Example configuration for deploying Pod manifests:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: agent-role
rules:
- apiGroups: ""
 resources:
 - pods
 - pods/log
 - pods/finalizers
 verbs:
 - get
 - list
 - watch
 - create
 - update
 - patch
 - delete

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: agent-role
rules:
- apiGroups: ""
 resources:
 - pods
 - pods/log
 - pods/finalizers
 verbs:
 - get
 - list
 - watch
 - create
 - update
 - patch
 - delete

See the Kubernetes Using RBAC Authorization guide for details on configuring ClusterRole and Role authorization.