Armory Docs – Armory Scale Agent Service Release Notes

Plugins: v1.0.84 Armory Agent Service (2025-03-07)

Fri, 07 Mar 2025 00:00:00 +0000

Fixes:

fixes an out of memory error when having an unstable connection with too many k8s events.

Plugins: v1.0.81 Armory Agent Service (2024-02-29)

Thu, 29 Feb 2024 00:00:00 +0000

Helm Chart Fixes:

Fixes in the namespace section.

Plugins: v1.0.82 Armory Agent Service (2024-02-29)

Thu, 29 Feb 2024 00:00:00 +0000

Helm Chart Update:

Updating Helm chart to support SSA(serverSideApply) config.

Plugins: v1.0.83 Armory Agent Service (2024-02-29)

Thu, 29 Feb 2024 00:00:00 +0000

Helm Chart Update:

Adding cacheGroup options in the Helm chart.

Plugins: v1.0.80 Armory Agent Service (2024-02-27)

Tue, 27 Feb 2024 00:00:00 +0000

Fixes

Better support for correct resolution of kinds with the same name and different API group.

This fix is important in k8s engines like OpenShift where different kinds have the same name such as Ingress.

Plugins: v1.0.79 Armory Agent Service (2024-02-09)

Fri, 09 Feb 2024 00:00:00 +0000

Fixes:

Additional coverage for resolution of the k8s REST api for different kubernetes kinds.

Plugins: v1.0.77 Armory Agent Service (2024-01-29)

Mon, 29 Jan 2024 00:00:00 +0000

Fixes:

Better REST mapping search when discovery had misses.

This fix prevents an agent restart in the case of discovery (during start) missing one k8s kind.

Plugins: v1.0.78 Armory Agent Service (2024-01-29)

Mon, 29 Jan 2024 00:00:00 +0000

Fixes:

Additional improvements to live k8s REST mapping fallback methods.

Plugins: v1.0.76 Armory Agent Service (2024-01-24)

Wed, 24 Jan 2024 00:00:00 +0000

Fixes

Improves detection on config file changes, prevents false positives such as permissions being modified.

Plugins: v1.0.75 Armory Agent Service (2024-01-22)

Mon, 22 Jan 2024 00:00:00 +0000

Fixes:

Prevents rest-mapper overwrite and deletion when config changes happen too soon.

Plugins: v1.0.74 Armory Agent Service (2024-01-11)

Thu, 11 Jan 2024 00:00:00 +0000

Fixes

Prevents duplicate discovery requests when config file is modified before current process completes.

Plugins: v1.0.73 Armory Agent Service (2023-12-01)

Fri, 01 Dec 2023 00:00:00 +0000

Fixes nil pointer error that happened when parsing certain CRD events sent from k8s.

Plugins: v1.0.72 Armory Agent Service (2023-11-20)

Mon, 20 Nov 2023 00:00:00 +0000

Fixes:

prevent race condition panic error while starting and cleaning watchers

Plugins: v1.0.71 Armory Agent Service (2023-11-10)

Fri, 10 Nov 2023 00:00:00 +0000

Fixes

Fixes a rare issue that could lead to a crash when a CRD event is not able to be parsed by the k8s watch mechanism.

Plugins: v1.0.70 Armory Agent Service (2023-11-08)

Wed, 08 Nov 2023 00:00:00 +0000

Fixes

Fixes an issue with operations when onlyNamespacedResources = true and two kinds with the same name exist. For example having batch/v1 Job and armory.runnable Job.

Plugins: v1.0.69 Armory Agent Service (2023-11-07)

Tue, 07 Nov 2023 00:00:00 +0000

Changes:

use delete operation wait ms when gracefulPeriod is not specified, as the maximum time to assert delete operations

Plugins: v1.0.68 Armory Agent Service (2023-10-02)

Mon, 02 Oct 2023 00:00:00 +0000

Changes:

agent is able to receive multiple registration response, ensuring the correct watchers are created without duplicates

Plugins: v1.0.67 Armory Agent Service (2023-09-28)

Thu, 28 Sep 2023 00:00:00 +0000

Changes:

Improve validation for operations on Kubernetes Custom Resources.

Plugins: v1.0.66 Armory Agent Service (2023-09-18)

Mon, 18 Sep 2023 00:00:00 +0000

Fixes

fixes issue that caused the service to crash when using kubernetes.cacheGroupSize and kubernetes.cacheGroupDelayMs.

Plugins: v1.0.64 Armory Agent Service (2023-09-11)

Mon, 11 Sep 2023 00:00:00 +0000

Fixes

Fixed issue in which Agent might crash OOM when reconnecting multiple times to Clouddriver

Plugins: v1.0.65 Armory Agent Service (2023-09-07)

Thu, 07 Sep 2023 00:00:00 +0000

Changes

Improvement in logic to prevent issue in which apply operations are unable to delete container sidecars, volumes, volumeMounts, or container envs.

Plugins: v1.0.63 Armory Agent Service (2023-08-31)

Thu, 31 Aug 2023 00:00:00 +0000

Changes

Agent is now available under the container names ‘armory/agent-k8s’ and ‘armory/scale-agent’.

Plugins: v1.0.62 Armory Agent Service (2023-08-29)

Tue, 29 Aug 2023 00:00:00 +0000

Updates:

Introduces an optional rate-limiting feature to minimize CPU strain during the initialization or expiration of watchers. This feature is off by default and can be activated by configuring both kubernetes.cacheGroupSize and kubernetes.cacheGroupDelayMs.

Note: This feature is particularly useful for setups with a large number of accounts, where CPU resources may become a bottleneck in the DB and the service itself.

Adjust settings based on the number of accounts, for example kubernetes.cacheGroupSize: 5 and kubernetes.cacheGroupDelayMs: 5000 waits 5 seconds between initializing the kubernetes watchers every 5 accounts.

Plugins: v1.0.61 Armory Agent Service (2023-08-15)

Tue, 15 Aug 2023 00:00:00 +0000

Feature

Updated admin endpoint :8082/accounts with filters and pagination
Current filters uses query params:
- Config.Name
- Config.Dynamic = true | `false
- State.Name = DiscoveryError | Healthy | ConfigLoaded
Pagination uses query string params limit offtset

Plugins: v1.0.60 Armory Agent Service (2023-08-03)

Thu, 03 Aug 2023 00:00:00 +0000

Fixes

Compatibility with Kubernetes v1.27.x (resulted in SIGSEGV)

Plugins: v1.0.59 Armory Agent Service (2023-08-02)

Wed, 02 Aug 2023 00:00:00 +0000

Features:

Restart the connection to clouddriver agent plugin, if agent has not been receiving any operation in the last clouddriver.keepAliveOperationSeconds property specified seconds. To enable this feature, need to: update agent plugin version >= (0.13.9|0.12.10|0.11.47), and put a value different than zero on clouddriver.keepAliveOperationSeconds agent property

Plugins: v1.0.58 Armory Agent Service (2023-06-09)

Fri, 09 Jun 2023 00:00:00 +0000

Changes:

rename the operation logging namespace contextual key to operationNamespace
expose logging contextual key-value pairs in config file
```
logging:
 fields:
 keyOne: valueOne
 keyTwo: valueTwo
 keyN: valueN
```
The following two keys could be detected automatically when a value is not provided:
```
logging:
 fields:
 agentCluster:
 agentNamespace:
```
The computed value when it is not specified for the following keys will be:
1. agentCluster: Cluster name where agent is running (if KUBERNETES_SERVICE_NAME environment variable is set), Cluster Host where agent is running (if KUBERNETES_SERVICE_HOST environment variable is set), empty if any environment variable is specified
2. agentNamespace: Namespace where agent is running

Example: Having the following configuration:

logging:
 fields:
 agentCluster: test
 agentNamespace:
 keyOne: valueOne
 keyTwo: valueTwo
 keyN: valueN

The log will look like this:

INFO[0099] Operation Delete processed (status 204 No Content) in 2.03850709s account=account1 agentCluster=test** agentId=7affa394-7ee1-456d-9528-a4339fb56051 **agentNamespace=spinnaker keyN=valueN keyOne=valueOne keyTwo=valueTwo operationId=01H2RBKP1SHP7PDXA1JEMMA6DZ operationNamespace=default operationType=Delete

Plugins: v1.0.57 Armory Agent Service (2023-06-02)

Fri, 02 Jun 2023 00:00:00 +0000

Changes:

Handle delete operation lowercase propagation policy
Stop truncating operation errors to 255 chars; in order to see the whole error on logs and UI

Plugins: v1.0.56 Armory Agent Service (2023-05-18)

Thu, 18 May 2023 00:00:00 +0000

Fixes

Prevent application hangup when defining a big number of accounts.

Plugins: v1.0.55 Armory Agent Service (2023-05-08)

Mon, 08 May 2023 00:00:00 +0000

Changes:

add the following properties to configure grpc connection backoff configuration:
- clouddriver.backoff.baseDelay the amount of time to backoff after the first failure
- clouddriver.backoff.multiplier the factor with which to multiply backoffs after a failed retry. Should ideally be greater than 1.
- clouddriver.backoff.jitter the factor with which backoffs are randomized.
- clouddriver.backoff.maxDelay is the upper bound of backoff delay.

Could find more details on https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md, also check default values on https://pkg.go.dev/google.golang.org/grpc/backoff

Plugins: v1.0.54 Armory Agent Service (2023-04-25)

Tue, 25 Apr 2023 00:00:00 +0000

Fixes gRPC connection management to prevent trying connections during an incorrect state.

Plugins: v1.0.47 Armory Agent Service (2023-02-27)

Wed, 08 Mar 2023 00:00:00 +0000

Summary:

Manifests deployed and updated with kubectl<1.26 and agent versions v1.0.32 to v1.0.46 in tandem, may have an inconsistent state which prevents apply operations from deleting container sidecars, volumes, volumeMounts, container envs, and other lists representing maps. Running either kubectl apply --server-side --force-conflicts with kubectl>=1.26 or agent v1.0.47 with kubernetes.serverSideApply.enabled: always option in the config file will migrate the manifest to consistent state again.

Context:

Server Side Apply

Original customer blocker: Default Client-Side apply is unable to handle some of the configMaps through spinnaker because the manifest starting hitting annotation length hard limits and CSA requires a copy of the previous manifest in the annnoation named last-applied-configuration
Serverside Apply includes a new structure called field managers under .metadata.managedFields which allow to store the same information as last-applied-configuration without relying on annotations. Each field manager tell who set what fields and who can modify them (i.e. end-users, k8s controllers, kubeapi)
Serverside apply uses a similar syntax to StrategicMerge (i.e. CSA payloads) but it removes the $patch directive. The only option to delete a side-car is to remove it from the payload
Serverside apply removes field managers that have no corresponding field included in the payload
The last two points combine to prevent SSA from deleting a side-car and own the side in order to be allowed to deleted it
~The only workaround it to clear or the fieldManagers so that the next apply becomes the sole owner of the complete manifest. Whoever that approach might lead to issues that the CSA’s last-applied-configuration annotation and the SSA’s field manager are trying to prevent down the road.~
There’s a new bookkeeping logic in kubectl 1.26.0 that handles both csa annotation and field managers to keep both consistent and allow CSA and SSA to coexist. In case a third fieldManager (i.e. a k8s controller) insists on retaining ownership, you can clear the field managers in order to become the sole owner. But it is unlikely to be necessary. Using kubectl apply --server-side --force-conflicts to deploy the lastest baked manifest is the manual fix to keep CSA and SSA consistent
Earlier versions of k8s (e.g. 1.18) some optional fields with defaults are expected to be assigned by kubectl, and agent now is able to send those defaults as well. However, it is not recommended to do so, since in SSA not including fields means yielding ownership.

TLS Insecure Changes:

Uses the correct value when the clouddriver.insecure flag is not included based on the mTLS settings.

Server Side Apply Changes:

ConfigMap and Secrets default to SSA to allow handle large payloads for them; in addition this manifests are immutable so that prevents multiple ownership.
All other manifests are ClientSide-Apply by default unless an annotation is set:
- agent-k8s.armory.io/serverside-apply: true will send the payload through SSA. otherwise will use CSA
- agent-k8s.armory.io/ssa-manifest-defaults: true will include missing fields with known defaults (e.g. containerPort = TCP) Only necesary for some older k8s cluster versions e.g. 1.18
- agent-k8s.armory.io/ssa-clear-managed-fields: true will clear the field managers

Add:

Per Manifest-Kind configuration:
- kubernetes.serverSideApply.kinds[].kind Kind name to override.
- kubernetes.serverSideApply.kinds[].enabled: always|never|allowed always|never: Override annotations to Always/Never use SSA. allowed: Use CSA unless annotation agent-k8s.armory.io/serverside-apply: true is present
- kubernetes.serverSideApply.kinds[].manifestDefaults: always|never|allowed always|never: Override annotations to Always/Never modify applied manifests to include field defaults. allowed: Send manifest as-is unless annotation agent-k8s.armory.io/ssa-manifest-defaults: true is present
- kubernetes.serverSideApply.kinds[].clearManagedFields: always|never|allowed always|never: Override annotations to Always/Never modify applied manifests to clear managed fields and become sole owner, Partial manifess will fail, and information from controller might be lost. allowed: Respect managed fields unless annotation agent-k8s.armory.io/ssa-manifest-defaults: true is present
Agent global configuration
- kubernetes.serverSideApply.enabled: always|never|allowed
  always|never: Override annotations and kind list to Always/Never use SSA. allowed: Use CSA unless annotation agent-k8s.armory.io/serverside-apply: enabled is present or config kubernetes.serverSideApply.kinds[].enabled is present in configuration file.
- kubernetes.serverSideApply.manifestDefaults: always|never|allowed always|never: Override annotations and kind list to Always/Never modify applied manifests to include field defaults. allowed: Send manifest as-is unless annotation agent-k8s.armory.io/ssa-manifest-defaults: enabled is present or kubernetes.serverSideApply.kinds[].manifestDefaults is present in configuration file
- kubernetes.serverSideApply.clearManagedFields: always|never|allowed always|never: Override annotations and kind list to Always/Never modify applied manifests to include field defaults. allowed: Send manifest as-is unless annotation agent-k8s.armory.io/ssa-manifest-defaults: enabled is present or kubernetes.serverSideApply.kinds[].manifestDefaults is present in configuration file

Defaults:

kubernetes.serverSideApply.enabled=allowed
kubernetes.serverSideApply.manifestDefaults=allowed
kubernetes.serverSideApply.clearManagedFields=allowed

CSA migration logic: https://github.com/kubernetes/kubernetes/issues/107980 https://github.com/kubernetes/kubernetes/issues/108081 https://github.com/kubernetes/kubernetes/issues/107417 https://github.com/kubernetes/kubernetes/issues/112826 https://github.com/kubernetes/kubernetes/pull/112905 https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.26.md#bug-or-regression-5

For kubectl, –server-side now migrates ownership of all fields used by client-side-apply to the specified –fieldmanager. This prevents fields previously specified using kubectl from being able to live outside of server-side-apply’s management and become undeleteable. (#112905, @alexzielenski) [SIG API Machinery, CLI and Testing]

Summary:

Using server side apply now ensures that all fields are managed by the server, making it easier to manage Kubernetes objects and avoid data inconsistencies.

Plugins: v1.0.45 Armory Agent Service (2023-02-27)

Mon, 27 Feb 2023 00:00:00 +0000

Changes:

Introduces a mechanism to make sure old caching events are not sent to clouddriver.

The refactor was applied after noticing that the agent version 1.0.41 could introduce a delay on agent registration if there are too many outdated caching events queued up.

Plugins: v1.0.46 Armory Agent Service (2023-02-27)

Mon, 27 Feb 2023 00:00:00 +0000

Fixes:

Truncates error messages from all kubernetes operations to a max of 255 characters.

Plugins: v1.0.44 Armory Agent Service (2023-02-14)

Tue, 14 Feb 2023 00:00:00 +0000

Fixes:

Memory leak when CR watches are enabled.

Plugins: v1.0.43 Armory Agent Service (2023-02-09)

Thu, 09 Feb 2023 00:00:00 +0000

Fixes:

fixes a problem with the ability to override default boolean properties with a false value.

Plugins: v1.0.41 Armory Agent Service (2023-01-25)

Wed, 25 Jan 2023 00:00:00 +0000

Changes:

Ensures there are not pending caching tasks or events before connecting or reconnecting to clouddriver, thereby optimizing performance and minimizing resource usage.

Plugins: v1.0.42 Armory Agent Service (2023-01-25)

Wed, 25 Jan 2023 00:00:00 +0000

Changes:

Enables Agent to start without any provided config file.

Plugins: v1.0.40 Armory Agent Service (2023-01-23)

Mon, 23 Jan 2023 00:00:00 +0000

Changed:

Now serverside apply sets default values from kubernetes 1.26.1 library (for example Deploy’s spec.containers[].containerPorts.protocol=TCP) Added:
Config flags: kuberentes.accounts[].skipManifestDefaults, kubernetes.accounts[].skipServerSideApply to opt to use client side apply without default values (Might be required for kubernetes clusters in version =<1.21)

Plugins: v1.0.39 Armory Agent Service (2023-01-19)

Thu, 19 Jan 2023 00:00:00 +0000

Added:

Helm settings for podLabels, podAnnotations, and podResources

Plugins: v1.0.38 Armory Agent Service (2023-01-18)

Wed, 18 Jan 2023 00:00:00 +0000

Changes:

Improve zoneID calculation by taking the deployment information from the HOSTNAME environment variable when the provided service account has no access to list deployments.

Plugins: v1.0.37 Armory Agent Service (2023-01-06)

Fri, 06 Jan 2023 00:00:00 +0000

Improvements:

Prevent unnecessary disconnects when Agent starts with 0 accounts

Plugins: v1.0.35 Armory Agent Service (2022-12-30)

Fri, 30 Dec 2022 00:00:00 +0000

Changes:

Bump gRPC and proto library versions
Optimize gRPC connection states

Plugins: v1.0.34 Armory Agent Service (2022-12-19)

Mon, 19 Dec 2022 00:00:00 +0000

Changes

Added /version and /accounts/{name}/version endpoints for troubleshooting.

Plugins: v1.0.33 Armory Agent Service (2022-12-12)

Mon, 12 Dec 2022 00:00:00 +0000

Changes:

Use protobuf encoding for: Logs, Rollbacks, RevisionHistory & Discovery

Plugins: v1.0.32 Armory Agent Service (2022-12-07)

Wed, 07 Dec 2022 00:00:00 +0000

Changes:

DeployManifest task now uses server-side apply to prevent error “last-applied metadata.annotations: Too long: must have at most 262144 bytes”

Plugins: v1.0.31 Armory Agent Service (2022-12-02)

Fri, 02 Dec 2022 00:00:00 +0000

Changes:

Fit patch operation response to 255 characters.

Plugins: v1.0.30 Armory Agent Service (2022-12-01)

Thu, 01 Dec 2022 00:00:00 +0000

Fixes:

Fixes a bug that caused Custom Resource watchers to not get created if CRD was created before agent started running.

Plugins: v1.0.29 Armory Agent Service (2022-11-30)

Wed, 30 Nov 2022 00:00:00 +0000

Feat:

Restart operations grpc stream connection when clouddriver stops responding a ping event.

Plugins: v1.0.28 Armory Agent Service (2022-11-28)

Mon, 28 Nov 2022 00:00:00 +0000

Fixes:

Prevents an unnecessary registration call to clouddriver when no accounts have been discovered.