Armory Docs – Observability

Continuous-Deployment: Armory Continuous Deployment Monitoring Indicators

Mon, 01 Jan 0001 00:00:00 +0000

Common Metrics - All Services

Service Level Indicators (SLI)

For all java based services, Spinnaker exposes common metrics like HTTP and JVM ones.

Controller latency

sum(rate($service:controller:invocations__totalTime_total[1m])) by
(controller, method) / sum(rate($service:controller:invocations__count_total[1m])) by
(controller, method)

→ Y axis is in nanoseconds.

To get milliseconds:

totalTime query / 1000000 / count query

Indicates how long it takes for each HTTP controller to respond.

Service Level Objective (SLO): Average rate below 10 seconds.

Troubleshoot: Determine if slowness is caused by downstream, external services or by the affected service itself, by looking at other metrics.

Outgoing requests latency

sum(rate($service:okhttp:requests__totalTime_total[1m])) by (requestHost) / sum(rate($service:okhttp:requests__count_total[1m])) by (requestHost)

→ Y axis is in nanoseconds.

To get milliseconds:

totalTime query / 1000000 / count query

Shows how long it takes for downstream Spinnaker services to respond.

SLO: Average rate below 10 seconds.

Troubleshoot: Look at specific metrics of slow service.

5xx errors sent

sum(rate($service:controller:invocations__count_total{status="5xx"}[1m])) by
(controller, method, statusCode)

Indicates the number of 5xx errors produced by the service.

SLO: Average rate below 0.25 errors per second.

Troubleshoot: Investigate in logs and metrics for other services, if the errors are caused in the affected service or by downstream, external services.

5xx errors received

sum(rate($service:okhttp:requests__count_total{status="5xx"}[1m])) by
(requestHost)

Shows the number of errors received from other Spinnaker services.

SLO: Average rate below 0.25 errors per second.

Troubleshoot: Look at logs and metrics of the service throwing the errors.

Hystrix fallbacks

sum(rate($service:hystrix:countFallbackSuccess[1m])) by (metricGroup, metricType)

sum(rate($service:hystrix:countFallbackFailure[1m])) by (metricGroup, metricType)

sum(rate($service:hystrix:isCircuitBreakerOpen[1m])) by (metricGroup, metricType)

Not all services have Hystrix. It is a disaster recovery library, meaning that if the original request to an external service fails, Hystrix will execute a “Plan B” to produce an acceptable response. If errors accumulate and the downstream service is still not healthy, Hystrix circuit breaker opens and errors start to flow upstream.

SLO: Average rate below 0.10 Hystrix events per second.

Troubleshoot: Investigate log files of affected service, it’s possible that downstream external services are either failing or are too slow to respond. Look at “Outgoing requests latency” metric.

JVM memory usage

sum($service:jvm:memory:used{memtype="HEAP"}) by (instance)

Shows memory consumption of a java based service.

SLO: No pods restarting due to OutOfMemory errors.

Troubleshoot: There’s no ideal value identified, but pod memory requests/limits MUST be 2x max JVM heap value. This is because services use -XX:MaxRAMFraction=2 flag, which means that the JVM memory will be limited to take at most half of the available RAM to the pod. For example if the pod memory request/limit is 2GB, the java service will take up to 1GB before OutOfMemory errors appear and the container gets killed. Increase pod memory requests/limits.

Configurations affecting performance

OkHttp client timeout

Add the following config to profiles/<service>-local.yml (such as clouddriver-local.yml) to change the default timeout of the OkHttp client. This can be a temporary, quick workaround to fix increased 5xx errors and Hystrix fallbacks.

ok-http-client:
 connectTimeoutMs: <TIMEOUT MS>
 readTimeoutMs: <TIMEOUT MS>

JVM memory requests/limits

Example in main hal config:

deploymentEnvironment:
  customSizing:
   spin-clouddriver:
    requests:
     memory: 8Gi
    limits:
     memory: 8Gi

Clouddriver Metrics

Average caching agent time

avg(label_replace(
 delta(clouddriver:executionTime__totalTime_total{}[1m])
, "agentType", "$1", "agent", "^.*/(\\w+)\\[.*$")) by (agent)

Indicates how long it takes for clouddriver caching agents to complete a caching cycle.

SLO: Max time below 1 minute.

Troubleshoot: Usually caching cycles are done every 30 seconds. Caching time can be improved by increasing clouddriver pods and memory/cpu requests and limits. Number of pods are related with number of caching threads. More pods, more caching threads can be used to balance workload among pods. Example:

providers:
  kubernetes:
   accounts:
   - kubernetes
    cacheThreads: 3
----------------
 deploymentEnvironment:
  customSizing:
   spin-clouddriver:
    replicas: 5

Also using SQL instead of Redis may improve caching agent times. When this metric is too high, it can cause “Deploy (Manifest)” stage to last 12 “Force Cache Refresh task during deployments taking up to 12 minutes to complete. Long Cloudddriver startup times if it checks for permissions on startup (more than 30 minutes). 500 timeouts on UI during deployments.

Number of caching agents

count(label_replace(
 delta(clouddriver:executionTime__totalTime_total{}[1m])
, "agentType", "$1", "agent", "^.*/(\\w+)\\[.*$")
) by (agent)

Shows how many caching agents are running in clouddriver.

SLO: No deviation in time, the number of agents should stay constant.

Troubleshoot: When this metric varies it typically means a pod was killed.

Kubernetes, kubectl latency

sum(rate(clouddriver:kubernetes:api__totalTime_total{success="true"}[1m])) by
(account, action) / sum(rate($service:kubernetes:api__count_total{success="true"}[1m])) by
(account, action)

→ Y axis is in nanoseconds.

To get seconds: totalTime query / 1000000000 / count query

Shows how long kubectl takes for completing requests like kubectl get.

SLO: This metric is impacted by the target Kubernetes cluster and is not directly responsibility of Spinnaker, but a latency below 5 seconds average is good.

Troubleshoot: You can log in to clouddriver pod and list processes, there are likely kubetcl running processes from which you can grab the full command used by clouddriver:

bash-4.4$ ps -f | cat
PID USER  TIME COMMAND
17103 spinnake 0:00 kubectl --kubeconfig=/Users/german/.hal/default/staging/dependencies/1062698630-kubecfg-german --namespace=istio-system -o json get configMap,controllerRevision,cronJob,daemonSet,deployment,event,horizontalpodautoscaler,ingress,job,limitRange,networkPolicy,persistentVolumeClaim,pod,podDisruptionBudget,replicaSet,role,roleBinding,secret,service,serviceAccount,statefulSet

If you repeat this command with --v=10 flag, you’ll get all requests, responses and internal cache hits done by the command. From there, you can know where’s spending most of the time. kubectl has an internal cache expiring every 10 minutes or so in ~/.kube.

Output of the kubectl --v=10 get xxx command can be analyzed with this script.

AWS failed requests

sum(rate(clouddriver:aws:request:requestCount_total{error="true"}[1m])) by (AWSErrorCode, requestType)

Indicates the amount of failed requests from Spinnaker to AWS infrastructure.

SLO: Average rate below 0.10 errors per second.

Troubleshoot: Adjust Cloudddriver rate limits.

SQL blocked connections

clouddriver:sql:pool:default:blocked{}

Shows how many SQL connections are hanging waiting for processes to complete.

SLO: Zero blocked connections.

Troubleshoot: Take a look at database provider specific metrics to see if database is reaching maximum resource capacity, or maximum number of allowed concurrent active connections.

Redis latency

sum(rate(clouddriver:redis:command:latency:sscan__count_total[1m])) /
sum(rate(clouddriver:redis:command:latency:sscan__totalTime_total[1m]))

There are around 10 different redis actions and each one has its own metric. This is the time it takes for a redis read or write operation to complete.

SLO: Average rate below 100 milliseconds.

Troubleshoot: Investigate overall network latency from clouddriver to redis, and if redis is reaching its capacity limit.

Configurations affecting performance

checkPermissionsOnStartup

Kubernetes account level. When set to false, clouddriver startup time is much faster because permissions are checked on demand. Bad side effects: unknown.

liveManifestCalls

Kubernetes account level. When set to true, “Force Cache Refresh” task during Deploy stage execution is reduced to zero, because Spinnaker will not try to refresh the cache to check if the manifest was deployed. Bad side effects: many more kubectl calls than needed during deployments. Ideally this flag should always be false (default).

cacheThreads

Kubernetes account level. This should only be increased when the number of pods increases, it allows to spread the load of caching across different pods and threads.

metrics

Kubernetes account level. When set to false, Spinnaker will not try to constantly retrieve memory and cpu usage of each and every pod, reducing the caching cycle time of the metrics caching agent to almost zero. Side effects: on Infrastructure screen, when you click a pod, on the right side panel there’s information about its cpu/memory usage. When this flag is set to false, those values show as unknown.

replicas

deploymentEnvironment.customSizing.<service>.replicas. Indicates the number of pods for each service.

cpu/memory requests/limits

deploymentEnvironment.customSizing.<service>.requests.cpu, deploymentEnvironment.customSizing.<service>.limits.cpu. Indicates the max value for cpu and memory usage for pods.

onDemandClusterThreshold

File: default/profiles/settings-local.js setting: window.spinnakerSettings.onDemandClusterThreshold. This is a Deck setting that controls the threshold at which the Infrastructure screen will change to not show all server groups at the same time, and instead show a dropdown to select the server group to show. This setting has a huge impact on improving performance because the get server groups call is usually one of the most expensive ones, and is done periodically as long as the UI is open.

max-concurrent-agents

File: clouddriver-local.yml

redis.agent.max-concurrent-agents, default: 1000.

sql.agent.max-concurrent-agents, default: 100.

This is the maximum amount of caching agents running simultaneously. Limiting this value may benefit to avoid having spikes of high resource usage, specially when clouddriver handles a lot of accounts. Example for kubernetes provider: having 10 accounts and 5 max agents, will spawn a maximsuum of 5 kubectl simultaneous calls, whereas using the default value may spawn 10 * 2 = 20 simultaneous kubectl calls, because each account has at least a CoreCachingAgent and a CustomResourceDefinition caching agent.

jobs.local.timeout-minutes

File: clouddriver-local.yml

jobs.local.timeout-minutes . Default: 10. The maximum amount of time for spawned kubectl processes to finish before being killed. A change is proposed to Spinnaker 1.24 to send a SIGKILL signal to those processes (kill -9) instead of just the usual SIGTERM, which may still leave misbehaving kubectl processes running.

Continuous-Deployment: Configure Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Configure monitoring using the Observability Plugin

Caution

Before configuring monitoring, read and understand the following information about the security implications. If any of your services, typically Gate (API), are exposed to the open internet, there is a risk that you can publicly expose information. Armory recommends that you filter these paths at your edge layer in some manner. Be aware of any endpoints you expose. Spring boot exposes the health endpoint by default though with some restrictions on what information is exposed. When auth is enabled, Gate restricts access to the endpoints other than /health, preventing access to metric data.

For more information on Spring actuators, see the Monitoring and Management.

Armory recommends that you monitor your systems by using the Armory Observability Plugin. This is an open source solution for monitoring Armory Continuous Deployment. The plugin supports the following:

Adding Prometheus (OpenMetrics) endpoints to Armory Continuous Deployment pods.
Sending data to NewRelic (see the plugin’s README).

The Observability Plugin removes the service name from the metric. This is incompatible with the behavior of the open source Spinnaker monitoring daemon system, which was the default monitoring solution in versions earlier than 2.20 and is now deprecated.

Install the plugin

You have two options for installing the Observability plugin:

Add the plugin configuration for all services the spinnaker profile section.
Add the plugin configuration to only the services you want to monitor.

The following example adds the Observability Plugin configuration for all services to the spinnaker-local.yml file and enables Prometheus:

# These lines are spring-boot configuration to allow access to the metrics
# endpoints. This plugin adds the "aop-prometheus" endpoint on the
# "<service>:<port>/aop-prometheus" path.

management:
 endpoints:
 web:
 # Read the security warning at the start of this section about what gets exposed!!
 exposure.include: health,info,aop-prometheus
spinnaker:
 extensibility:
 plugins:
 Armory.ObservabilityPlugin:
 enabled: true
 version: <VERSION>
 # This is the basic configuration for prometheus to be enabled
 config.metrics:
 prometheus:
 enabled: true
 repositories:
 armory-observability-plugin-releases:
 url: https://raw.githubusercontent.com/armory-plugins/armory-observability-plugin-releases/master/repositories.json

You can find more options for management endpoints and the plugin in the plugin’s README.

Continuous-Deployment: Configure Support Diagnostics for Armory Continuous Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Advantages to enabling Armory Diagnostics

When you engage Armory Support, the support team might ask you about enabling Armory Diagnostics. This sends the log and event data from the your system to Armory so that the support team can remotely investigate what might be going on with your system, resulting in a faster turnaround on solutions.

UUIDs are not checked for uniqueness. As a result, if you have two environments and you provide both with the same UUID, the logs sent to Armory will conflict with each other. Please ensure unique UUIDs for each environment.

Turning on Armory Diagnostics

Generate a unique UUID before adding the information. You can do this at a number of 3rd party websites that you can find via a Google search.

Add to your SpinnakerService.yml file:

spec:
 spinnakerConfig:
 config:
 armory:
 diagnostics:
 enabled: true
 uuid: # Enter uniquely Generated UUID
 logging:
 enabled: # true/false to enable
 endpoint: # Enter Endpoint, e.g. https://debug.armory.io/v1/logs

kubectl -n spinnaker apply -f spinnakerservice.yml

You can find a description of diagnostic parameters in the “Diagnostics Parameters” section of the Operator armory reference.

Migrating Diagnostics to Operator from Halyard

Follow the same steps as above, but you do not have to generate a UUID, as you already have one in your Halyard config. Please use the same UUID to ensure diagnostics continue.

What data gets sent to Armory

You’re probably wondering exactly what gets sent to Armory when you’re sharing your log output with us.

There are actually two streams of output that we will receive from an enabled system. The first is the easiest, it’s a copy of what gets logged in the pod (the same output you get from running kubectl logs -f <pod_id>). This tends to be status messages, but we will also capture any software failures that might have gotten caught. This often gives us the best insight into why something is going wrong.

The second stream is actually the set of events being passed internally from Spinnaker microservice to microservice. These are JSON payloads that are then tagged on our side with your unique customer ID (so we can isolate your instances), a timestamp, and then two primary keys, “details” and “content”.

The Details key maps some very common elements for the event:

application – The name of the Spinnaker application
created – Timestamp of when the event was created
requestHeaders – HTTP request headers for the internal request
source – The microservice that generated the event (ex. ‘clouddriver’)
type – An identifier as to what kind of event this was (ex. orca:pipeline:starting)

The Content section is variable, and depends on what kind of event is being tracked. Here are some of the more common events:

Web access events contain your basic web event data, the request headers, URL, User Agent, encoding directives, etc.
Build events pass along the name of the build job (from Jenkins, Travis, etc), the success/failure, duration of the job.
Webhook events (like commits from Github) will contain whatever the webhook has sent along in its payload. For most SCMs, this is just the name of the organization, the repo name, a hash representing the commit, and a timestamp. If you’ve set up any other webhook events that you using to trigger Spinnaker, take care with what data is being sent.
Pipeline events will contain all the same data you would see in Spinnaker if you click on “Source” under a pipeline’s stage. This includes the name of the application, pipeline, the build information, the trigger information, and the status and results of all the stages included in that pipeline (including stages that have not been executed yet).

Other events may be the result of performing operations within Spinnaker, such as creating a new load balancer, or resizing a server group.

How Armory uses the data

Armory’s Technical Account Manager and Support teams may use this data to generate reports for customers and provide summary information to customers. Additionally, Armory may look at environment configuration data and view error logs in order to reduce resolution time on open support cases.

Technical Account Manager Report Example

Customer Configuration Example

A Note on Private Data Exposure Risks

The data passed out of Spinnaker is based on payloads controlled by the open source Spinnaker community (except for those proprietary microservices provided by Armory specifically). Armory is not aware of any cases where private data (passwords, source code, etc.) would be passed out of Spinnaker as part of an event payload during normal operation. Armory is not responsible for secrets a customer has entered into an otherwise unprotected field being transmitted (for example, entering a password into a “Notes” field of a pipeline stage).

Continuous-Deployment: Enable Logging in Armory Continuous Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Metrics and logging overview

Metrics allow you to investigate the health of your Armory Continuous Deployment system. Armory’s Observability Plugin provides a good overview of data over time about what is happening in particular services. You can monitor data points such as overall load, number of errors, and connection speeds.

Logging, on the other hand, is more useful for tracing data about what individual accounts and functions are doing within Armory Continuous Deployment. This data provides an overview on applications and errors and is primarily for auditing purposes. Armory Continuous Deployment captures logging data in the Echo service, so you can configure Echo to send logging data to your data aggregator.

Before you begin

You have set up a data aggregator with a unique endpoint. You need the endpoint and an authorization token when you enable logging in Armory Continuous Deployment. Documentation for some popular data aggregators:

Enable logging data

The following configuration example shows how to configure Echo to send data to Splunk. If you are using another data aggregator, replace the url value with the endpoint for your aggregator.

Add the rest block to the spec.spinnakerConfig.profiles.echo section of your Operator manifest.

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 profiles:
 echo:
 rest:
 enabled: true
 endpoints:
 - wrap: true
 url: "http://<SERVICE_URL>:<PORT>/services/collector/event?"
 headers:
 Authorization: "<TOKEN>"
 template: '{"event":{{event}} }'
 insecure: true

SERVICE_URL: the FQDN of the service; for example https://splunk.armory.io.
PORT: the TCP listener port available in your aggregator.
TOKEN: the authorization token for your aggregator.

The default port for Splunk’s HTTP Event Collector (HEC) is either 8088 or 8089 depending on your Splunk version. This port is used if you do not configure a TCP listener. Any TCP listener config is a custom port assignment. See the Metric data does not flow into Splunk despite setting up HEC Connection KB article for more on Splunk HEC and ports.

The SERVICE_URL:PORT can be HTTPS, which would remove the need for the insecure TOKEN. Consult your specific data aggregator provider’s documentation for how to configure an HTTPS endpoint.

Verify data flow to your aggregator

You can either wait for events to flow or run a new pipeline to test that Echo is sending logging details to your aggregator. You should now be able to search the Armory Continuous Deployment index and see events coming in. If you don’t see any data, Echo’s logs may point to the source of the problem.

Creating dashboards

After data is sent to your aggregator, you can set up dashboards to display useful data. Consult your data aggregator’s documentation for details on creating dashboards.

Splunk dashboard examples

If you are using Splunk, you can create a dashboard and then define the dashboard characteristics using a JSON file. The following examples show different dashboards created from logging data. Each example includes the JSON file that defines the dashboard characteristics.

Developer insights

Show me the JSON

{
 "visualizations": {
 "viz_chart_1": {
 "type": "viz.pie",
 "options": {
 "drilldown": "all",
 "refresh.display": "progressbar"
 },
 "dataSources": {
 "primary": "ds_search_2"
 },
 "title": "Top Artifacts fetched by Spinnaker"
 },
 "viz_table_1": {
 "type": "splunk.table",
 "options": {
 "drilldown": "none",
 "refresh.display": "progressbar"
 },
 "dataSources": {
 "primary": "ds_search_3"
 },
 "title": "Top Pipeline Executions by User"
 },
 "viz_table_2": {
 "type": "splunk.table",
 "options": {
 "drilldown": "none",
 "refresh.display": "progressbar"
 },
 "dataSources": {
 "primary": "ds_search_4"
 },
 "title": "Top Spinnaker Kubernetes Deployments"
 },
 "viz_chart_2": {
 "type": "viz.column",
 "options": {
 "axisLabelsX.majorLabelStyle.rotation": 0,
 "axisTitleX.visibility": "visible",
 "axisTitleY.visibility": "visible",
 "axisTitleY2.visibility": "visible",
 "axisY.abbreviation": "none",
 "axisY.scale": "linear",
 "chart.showDataLabels": "none",
 "chart.stackMode": "stacked",
 "drilldown": "all",
 "legend.placement": "bottom",
 "refresh.display": "progressbar",
 "trellis.enabled": 0
 },
 "dataSources": {
 "primary": "ds_search_5"
 },
 "title": "Spinnaker Deployments by Application"
 },
 "viz_chart_3": {
 "type": "viz.column",
 "options": {
 "chart.stackMode": "default",
 "drilldown": "none",
 "refresh.display": "progressbar"
 },
 "dataSources": {
 "primary": "ds_search_6"
 },
 "title": "Top Spinnaker Executions"
 }
 },
 "dataSources": {
 "ds_search_1": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker | fields content.execution.application | dedup content.execution.application | table content.execution.application | rename content.execution.application as \"AppName\"",
 "queryParameters": {
 "earliest": "0"
 }
 }
 },
 "ds_search_2": {
 "type": "ds.search",
 "options": {
 "query": "index = Spinnaker \"event.details.type\"=\"orca:pipeline:starting\" event.content.execution.application=$field2$ | top limit=20 \"event.content.execution.stages{}.context.inputArtifacts{}.artifact.name\" | rename event.content.execution.stages{}.context.inputArtifacts{}.artifact.name as \"Artifact Name\"",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_3": {
 "type": "ds.search",
 "options": {
 "query": "index=\"Spinnaker\" event.content.execution.application=$field2$ | top limit=20 \"event.content.execution.authentication.user\" | rename event.content.execution.authentication.user as \"Spinnaker Users\"",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_4": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker event.content.execution.application=$field2$ | top limit=20 \"event.content.execution.stages{}.context.deployedManifests{}.name\" | rename event.content.execution.stages{}.context.deployedManifests{}.name as \"Top Deployed Manifests\"",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_5": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker event.content.execution.application=$field2$ | fields event.content.execution.application | timechart count by event.content.execution.application limit=10 usenull=f",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_6": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker \"event.details.type\"=\"orca:pipeline:starting\" event.content.execution.application=$field2$ | top limit=20 \"event.content.execution.name\" | rename event.content.execution.name as \"Top Spinnaker Executions\"",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 }
 },
 "defaults": {
 "dataSources": {
 "ds.search": {
 "options": {
 "queryParameters": {}
 }
 }
 }
 },
 "inputs": {
 "input_1": {
 "type": "input.timerange",
 "title": "Select Time",
 "options": {
 "token": "field1",
 "defaultValue": "-7d@h,now"
 }
 },
 "input_2": {
 "type": "input.dropdown",
 "title": "Spinnaker Application Name",
 "options": {
 "token": "field2",
 "defaultValue": "*",
 "items": [
 {
 "value": "*",
 "label": "All"
 }
 ]
 },
 "encoding": {
 "value": "primary.AppName",
 "label": "primary.AppName"
 },
 "dataSources": {
 "primary": "ds_search_1"
 }
 }
 },
 "layout": {
 "type": "absolute",
 "options": {
 "submitButton": false,
 "display": "auto-scale",
 "height": 500
 },
 "structure": [
 {
 "item": "viz_chart_1",
 "type": "block",
 "position": {
 "x": 0,
 "y": 0,
 "w": 400,
 "h": 250
 }
 },
 {
 "item": "viz_table_1",
 "type": "block",
 "position": {
 "x": 400,
 "y": 0,
 "w": 400,
 "h": 250
 }
 },
 {
 "item": "viz_table_2",
 "type": "block",
 "position": {
 "x": 800,
 "y": 0,
 "w": 400,
 "h": 250
 }
 },
 {
 "item": "viz_chart_2",
 "type": "block",
 "position": {
 "x": 0,
 "y": 250,
 "w": 600,
 "h": 250
 }
 },
 {
 "item": "viz_chart_3",
 "type": "block",
 "position": {
 "x": 600,
 "y": 250,
 "w": 600,
 "h": 250
 }
 }
 ],
 "globalInputs": [
 "input_1",
 "input_2"
 ]
 },
 "description": "",
 "title": "Developer Insights"
}

Software deployments

Show me the JSON

{
 "visualizations": {
 "viz_single_1": {
 "type": "splunk.singlevalue",
 "options": {
 "drilldown": "all",
 "refresh.display": "progressbar",
 "trendInterval": "-24h",
 "underLabel": "Compared to a day before",
 "shouldUseThousandSeparators": true
 },
 "dataSources": {
 "primary": "ds_search_1"
 },
 "title": "Deployment Frequency (DF)"
 },
 "viz_chart_1": {
 "type": "viz.pie",
 "options": {
 "drilldown": "none"
 },
 "dataSources": {
 "primary": "ds_search_2"
 },
 "title": "Top Deployments by Application"
 },
 "viz_table_1": {
 "type": "splunk.table",
 "options": {
 "drilldown": "none",
 "refresh.display": "progressbar"
 },
 "dataSources": {
 "primary": "ds_search_3"
 },
 "title": "Production Approvals - Manual Judgements"
 },
 "viz_chart_2": {
 "type": "viz.column",
 "options": {
 "axisTitleX.text": "Mouse over chart (click to drilldown)",
 "drilldown": "all"
 },
 "dataSources": {
 "primary": "ds_search_4"
 },
 "title": "Software Deployments by Application"
 },
 "viz_Cgz4gtYq": {
 "type": "viz.pie",
 "dataSources": {
 "primary": "ds_1xOoneRR"
 },
 "title": "Failed Deployments"
 }
 },
 "dataSources": {
 "ds_search_1": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker \"event.details.type\"=\"orca:task:complete\" | fields event.details.type | timechart count",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_2": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker \"event.details.type\"=\"orca:pipeline:complete\"| top limit=20 \"event.details.application\"",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_3": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker \"event.content.execution.stages{}.name\"=\"Manual Judgment\" | stats count by event.content.execution.trigger.user | rename event.content.execution.trigger.user as \"Approval Users\"",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_4": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker \"event.details.type\"=\"orca:pipeline:complete\"| timechart count by event.details.application limit=10",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_1xOoneRR": {
 "type": "ds.search",
 "options": {
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 },
 "query": "index=Spinnaker \"event.details.type\"=\"orca:pipeline:failed\"| top limit=20 \"event.details.application\""
 },
 "name": "Failed Deployments"
 }
 },
 "defaults": {
 "dataSources": {
 "ds.search": {
 "options": {
 "queryParameters": {}
 }
 }
 }
 },
 "inputs": {
 "input_1": {
 "type": "input.timerange",
 "title": "Select Time",
 "options": {
 "token": "field1",
 "defaultValue": "-30d@d,now"
 }
 }
 },
 "layout": {
 "type": "absolute",
 "options": {
 "submitButton": false,
 "display": "auto-scale",
 "height": 500
 },
 "structure": [
 {
 "item": "viz_single_1",
 "type": "block",
 "position": {
 "x": 0,
 "y": 0,
 "w": 400,
 "h": 250
 }
 },
 {
 "item": "viz_chart_1",
 "type": "block",
 "position": {
 "x": 400,
 "y": 0,
 "w": 400,
 "h": 250
 }
 },
 {
 "item": "viz_table_1",
 "type": "block",
 "position": {
 "x": 800,
 "y": 0,
 "w": 400,
 "h": 250
 }
 },
 {
 "item": "viz_chart_2",
 "type": "block",
 "position": {
 "x": 0,
 "y": 250,
 "w": 610,
 "h": 250
 }
 },
 {
 "item": "viz_Cgz4gtYq",
 "type": "block",
 "position": {
 "x": 610,
 "y": 250,
 "w": 590,
 "h": 250
 }
 }
 ],
 "globalInputs": [
 "input_1"
 ]
 },
 "description": "",
 "title": "Software Deployments"
}

Armory Continuous Deployment platform

Show me the JSON

{
 "visualizations": {
 "viz_chart_1": {
 "type": "viz.area",
 "options": {
 "chart.stackMode": "stacked",
 "drilldown": "all"
 },
 "dataSources": {
 "primary": "ds_search_1"
 },
 "title": "Gate Sessions with Errors - Spinnaker API"
 },
 "viz_chart_2": {
 "type": "viz.column",
 "options": {
 "chart.stackMode": "stacked",
 "refresh.display": "progressbar"
 },
 "dataSources": {
 "primary": "ds_search_2"
 },
 "title": "Orca Functions Failure Count"
 },
 "viz_table_1": {
 "type": "splunk.table",
 "options": {
 "drilldown": "row",
 "rowNumbers": true
 },
 "dataSources": {
 "primary": "ds_search_3"
 },
 "title": "Top Errors from Spinnaker Orca Service (System Scheduler)"
 }
 },
 "dataSources": {
 "ds_search_1": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker clouddriver error| timechart count by event.content.context.exception.details.error limit=10 usenull=0",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_2": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker \"event.details.type\"=\"orca:*:failed\" | stats count by event.details.type | sort -count | rename event.details.type as \"Orca Functions\"",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_3": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker \"event.details.type\"=\"orca:*:failed\"| top limit=20 \"event.content.context.exception.details.errors{}\" | rename event.content.context.exception.details.errors{} as \"Spinnaker Platform Errors\"",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 }
 },
 "defaults": {
 "dataSources": {
 "ds.search": {
 "options": {
 "queryParameters": {}
 }
 }
 }
 },
 "inputs": {
 "input_1": {
 "type": "input.timerange",
 "title": "Select Time",
 "options": {
 "token": "field1",
 "defaultValue": "-7d@h,now"
 }
 }
 },
 "layout": {
 "type": "absolute",
 "options": {
 "submitButton": false,
 "display": "auto-scale",
 "height": 500
 },
 "structure": [
 {
 "item": "viz_chart_1",
 "type": "block",
 "position": {
 "x": 0,
 "y": 0,
 "w": 600,
 "h": 250
 }
 },
 {
 "item": "viz_chart_2",
 "type": "block",
 "position": {
 "x": 600,
 "y": 0,
 "w": 600,
 "h": 250
 }
 },
 {
 "item": "viz_table_1",
 "type": "block",
 "position": {
 "x": 0,
 "y": 250,
 "w": 1200,
 "h": 250
 }
 }
 ],
 "globalInputs": [
 "input_1"
 ]
 },
 "description": "",
 "title": "Spinnaker Platform"
}

Continuous verification

Show me the JSON

{
 "visualizations": {
 "viz_chart_1": {
 "type": "viz.area",
 "options": {
 "axisTitleX.text": "Time Series",
 "drilldown": "all",
 "legend.placement": "bottom",
 "refresh.display": "progressbar"
 },
 "dataSources": {
 "primary": "ds_search_3"
 },
 "title": "Log Verification - App Logs with \"Error\" in the event"
 },
 "viz_chart_2": {
 "type": "viz.column",
 "options": {
 "axisTitleX.text": "Time Series",
 "drilldown": "all",
 "refresh.display": "progressbar"
 },
 "dataSources": {
 "primary": "ds_search_4"
 },
 "title": "Log Verification - App Logs with \"Exception\" in the event"
 },
 "viz_chart_4": {
 "type": "viz.area",
 "options": {
 "drilldown": "none"
 },
 "dataSources": {
 "primary": "ds_search_7"
 },
 "title": "Automated Rollback by Application (MTTR) - Splunk Webhooks to Spinnaker"
 }
 },
 "dataSources": {
 "ds_search_1": {
 "type": "ds.search",
 "options": {
 "query": "index=* | fields index | dedup index | table index",
 "queryParameters": {
 "earliest": "0"
 }
 }
 },
 "ds_search_2": {
 "type": "ds.search",
 "options": {
 "query": "index=$field2$ | fields sourcetype | dedup sourcetype | table sourcetype",
 "queryParameters": {
 "earliest": "0"
 }
 }
 },
 "ds_search_3": {
 "type": "ds.search",
 "options": {
 "query": "index=$field2$ sourcetype=$field3$ | fields index, sourcetype, source | timechart count as events by source",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_4": {
 "type": "ds.search",
 "options": {
 "query": "index=$field2$ sourcetype=$field3$ exception | timechart count as events by source",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_5": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker status=5* OR status=4* | fields status, index, sourcetype | timechart count by status",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_6": {
 "type": "ds.search",
 "options": {
 "query": "index=_internal sourcetype=scheduler search_type=scheduled alert_actions!=\"\" app=\"TA-Spinnaker\" | table _time, savedsearch_name, app, alert_actions, run_time",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_7": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker \"event.content.execution.trigger.payload.search_name\"=Rollback | timechart count by event.details.application",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 }
 },
 "defaults": {
 "dataSources": {
 "ds.search": {
 "options": {
 "queryParameters": {}
 }
 }
 }
 },
 "inputs": {
 "input_1": {
 "type": "input.timerange",
 "title": "Select your time range",
 "options": {
 "token": "field1",
 "defaultValue": "-30d@d,now"
 }
 },
 "input_2": {
 "type": "input.dropdown",
 "title": "Find Application Index",
 "options": {
 "token": "field2",
 "defaultValue": "*",
 "items": [
 {
 "value": "*",
 "label": "All"
 }
 ]
 },
 "encoding": {
 "value": "primary.index",
 "label": "primary.index"
 },
 "dataSources": {
 "primary": "ds_search_1"
 }
 },
 "input_3": {
 "type": "input.dropdown",
 "title": "Find Application SourceType",
 "options": {
 "token": "field3",
 "defaultValue": "*",
 "items": [
 {
 "value": "*",
 "label": "All"
 }
 ]
 },
 "encoding": {
 "value": "primary.sourcetype",
 "label": "primary.sourcetype"
 },
 "dataSources": {
 "primary": "ds_search_2"
 }
 }
 },
 "layout": {
 "type": "absolute",
 "options": {
 "submitButton": false,
 "display": "auto-scale",
 "height": 500
 },
 "structure": [
 {
 "item": "viz_chart_1",
 "type": "block",
 "position": {
 "x": 0,
 "y": 0,
 "w": 650,
 "h": 250
 }
 },
 {
 "item": "viz_chart_2",
 "type": "block",
 "position": {
 "x": 650,
 "y": 0,
 "w": 550,
 "h": 250
 }
 },
 {
 "item": "viz_chart_4",
 "type": "block",
 "position": {
 "x": 0,
 "y": 250,
 "w": 1200,
 "h": 250
 }
 }
 ],
 "globalInputs": [
 "input_1",
 "input_2",
 "input_3"
 ]
 },
 "description": "",
 "title": "Continuous Verification"
}

Compliance center

Show me the JSON

{
 "visualizations": {
 "viz_chart_1": {
 "type": "viz.bar",
 "options": {
 "drilldown": "none",
 "refresh.display": "progressbar"
 },
 "dataSources": {
 "primary": "ds_search_1"
 },
 "title": "Top Spinnaker Deployment Accounts"
 },
 "viz_table_1": {
 "type": "splunk.table",
 "options": {
 "drilldown": "none",
 "refresh.display": "progressbar"
 },
 "dataSources": {
 "primary": "ds_search_2"
 },
 "title": "Manual Judgements - Approvals for Production Deployments"
 },
 "viz_chart_2": {
 "type": "viz.area",
 "options": {
 "drilldown": "none"
 },
 "dataSources": {
 "primary": "ds_search_3"
 },
 "title": "Gate API sessions by Authenticated Users"
 },
 "viz_table_2": {
 "type": "splunk.table",
 "options": {
 "drilldown": "none"
 },
 "dataSources": {
 "primary": "ds_search_4"
 },
 "title": "Deployments to Namespace Environment - by Deployment Account"
 }
 },
 "dataSources": {
 "ds_search_1": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker \"event.details.type\"=\"orca:pipeline:starting\" | timechart count by event.content.execution.stages{}.context.account limit=10 usenull=f",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_2": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker | stats count by event.content.execution.authentication.user, event.details.application | sort -count | stats list(event.details.application) as \"Application\", list(count) as count, sum(count) as total by event.content.execution.authentication.user | sort -total | rename event.content.execution.authentication.user as \"Authentication Propagated\"",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_3": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker \"event.details.source\"=gate event.content.user=!anonymous | timechart count by event.content.user",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 },
 "ds_search_4": {
 "type": "ds.search",
 "options": {
 "query": "index=Spinnaker | stats count by event.content.execution.stages{}.context.account, event.content.context.kato.tasks{}.resultObjects{}.manifests{}.metadata.namespace | sort -count | stats list(event.content.context.kato.tasks{}.resultObjects{}.manifests{}.metadata.namespace) as \"Deployment Namespaces\", list(count) as count, sum(count) as total by event.content.execution.stages{}.context.account | sort -total | rename event.content.execution.stages{}.context.account as \"Spinnaker Deployment Accounts\"",
 "queryParameters": {
 "earliest": "$field1.earliest$",
 "latest": "$field1.latest$"
 }
 }
 }
 },
 "defaults": {
 "dataSources": {
 "ds.search": {
 "options": {
 "queryParameters": {}
 }
 }
 }
 },
 "inputs": {
 "input_1": {
 "type": "input.timerange",
 "title": "Select Time",
 "options": {
 "token": "field1",
 "defaultValue": "-30d@d,now"
 }
 }
 },
 "layout": {
 "type": "absolute",
 "options": {
 "submitButton": false,
 "display": "auto-scale",
 "height": 500
 },
 "structure": [
 {
 "item": "viz_chart_1",
 "type": "block",
 "position": {
 "x": 0,
 "y": 0,
 "w": 600,
 "h": 250
 }
 },
 {
 "item": "viz_table_1",
 "type": "block",
 "position": {
 "x": 600,
 "y": 0,
 "w": 600,
 "h": 250
 }
 },
 {
 "item": "viz_chart_2",
 "type": "block",
 "position": {
 "x": 0,
 "y": 250,
 "w": 600,
 "h": 250
 }
 },
 {
 "item": "viz_table_2",
 "type": "block",
 "position": {
 "x": 600,
 "y": 250,
 "w": 600,
 "h": 250
 }
 }
 ],
 "globalInputs": [
 "input_1"
 ]
 },
 "description": "",
 "title": "Compliance Center"
}

Continuous-Deployment: Integrate a Sumo Logic Dashboard in Armory Continuous Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Advantages to integrating Sumo Logic

Integrating Spinnaker and Sumo Logic provides you with the ability to monitor the health and productivity of your end-to-end software delivery process through live dashboards. You gain at-a-glance visibility and longitudinal trends in usage and pipeline deployments across all dev, staging, and production environments.

This application has been developed and is supported by Armory, Inc. In case of technical questions, please contact Armory for support.

Log types

The Spinnaker Sumo Logic App uses echo logs that output event information from Spinnaker to Sumo Logic. Events include the following:

Calls made to gate
Pipeline events, which includes stages and tasks

Sample log message

{"details":{"source":"gate","type":"gate:session","created":"1567639483521","organization":null,"project":null,"application":null,"_content_id":null,"attributes":null,"requestHeaders":{}},"content":{"headers":{"x-request-id":"50cc1dcafa77a7fcf85612678cca01a6","sec-fetch-mode":"cors","referer":"https://spinnaker.se.armory.io/","sec-fetch-site":"same-site","x-forwarded-proto":"https","accept-language":"en-...

Sumo Logic truncates messages greater than 64kb in length, and may therefore drop some event messages. Message lengths often depend on the size of your pipeline (i.e. number of stages).

Query sample

The following sample query generates a dashboard to monitor pipelines starting:

_sourceCategory="dev/sales-demo-cluster/echo" and _collector="Spinnaker Instance" orca pipeline starting
| json field=_raw "content.execution.status","content.execution.application","details.type","content.execution.name","content.execution.trigger.type","content.execution.authentication.user" as status,application,logType,pipeline, triggerType, user nodrop
| where logType = "orca:pipeline:starting"
| timeslice by 1d
| count by _timeslice, pipeline
| transpose row _timeslice column pipeline

Collect logs for Spinnaker

Collection process overview

Note: This assumes you have already installed Spinnaker and have access to the configuration files to make changes to Spinnaker.

Collection Step 1: Retrieve SumoLogic URL for HTTP logs**

Configure an HTTP Source. And get a copy of the endpoint URL from your SumoLogic console.

Collection Step 2: Modify Spinnaker configuration to export logs to SumoLogic**

Add to the SpinnakerService manifest the following snippet:

apiVersion: spinnaker.armory.io/v1alpha2
kind: SpinnakerService
metadata:
 name: spinnaker
spec:
 spinnakerConfig:
 profiles:
 echo:
 rest:
 enabled: true
 endpoints:
 - wrap: false
 url: [insert your custom HTTP endpoint] #e.g. https://endpoint1.collection.us1.sumologic.com/...

Collection Step 3: Apply changes**

Run kubectl -n <spinnaker namespace> apply -f <SpinnakerService manifest>, and wait for the services to restart. Check SumoLogic console for ingestion of data.

Install the Spinnaker app and view the Dashboards

Go to Sumo Logic App Catalog and search for “Spinnaker” by Armory.

Dashboard filters

The Spinnaker Pipelines dashboard has a set of filters that you can apply to the entire dashboard as shown in the following example. Click the funnel icon in the top dashboard menu bar to display a scrollable list of filters that are applied across the entire dashboard.

NOTE: You can use filters to drill down and examine the data on a granular level by application and pipeline.

Each panel has a set of filters that are applied to the results for that panel only, as shown in the following example. Click the funnel icon in the top panel menu bar to display a list of panel-specific filters.

Spinnaker Overview dashboard

The Spinnaker Overview provides the health and usage of your Spinnaker instance at a quick glance, including applications deployed, clusters targeted, pipelines run, and users in spinnaker. Use this dashboard to:

Quickly see number of active apps, clusters, pipelines, and users.
See activity levels apps, clusters, pipelines and users

Spinnaker Pipelines dashboard

This is a dashboard that contains pipeline execution history and metrics, including most common errors. Use this dashboard to:

See pipeline execution history and compare the last 14 days to the 14 days that preceded them
Understand which pipelines are executing and how often
See which stages/tasks are failing most often
Uncover the most common errors in your pipelines
See how many pipelines have failed and completed over time

Spinnaker User Activity dashboard

This is a dashboard that contains user activity, manual judgments, rollbacks, canary, and trigger metrics. Use this dashboard to:

See how many manual judgments, rollbacks, and canaries have occurred over time
See which users are most active and usage trends over time
Understand how pipelines are getting triggered

Sumo Logic collector limitations

Log messages greater than 64kb get truncated by the Sumo Logic collector. This means that there are potential events that are dropped and not displayed in the dashboards. Long pipelines composed of many stages or triggered by other pipelines can make messages longer.

Continuous-Deployment: Monitor Armory Continuous Deployment with Prometheus

Mon, 01 Jan 0001 00:00:00 +0000

Overview

Armory recommends monitoring the health of Armory Continuous Deployment in every production instance. This document describes how to set up a basic Prometheus and Grafana stack as well as enable monitoring for the Armory Continuous Deployment services.

Additional Prometheus and Grafana configuration is necessary to make them production-grade, and this configuration is not a part of this document. Also note that monitoring the Pipelines-as-Code service (Dinghy) and the Terraform Integration service (Terraformer) are not discussed on this page.

Before you begin

You are familiar with Prometheus and Grafana
Armory Continuous Deployment is deployed in the spinnaker namespace
Prometheus and Grafana are deployed in the monitoring namespace
You have configured monitoring using the Observability Plugin.

Use `kube-prometheus` to create a monitoring stack

You can skip this section if you already have a monitoring stack.

A quick and easy way to configure a cluster monitoring solution is to use kube-prometheus. This project creates a monitoring stack that includes cluster monitoring with Prometheus and dashboards with Grafana.

To create the stack, follow the kube-prometheus quick start instructions beginning with the Compatibility Matrix section.

After you complete the instructions, you have pods running in the monitoring namespace:

% kubectl get pods --namespace monitoring

NAME READY STATUS RESTARTS AGE
alertmanager-main-0 2/2 Running 0 44s
alertmanager-main-1 2/2 Running 0 44s
alertmanager-main-2 2/2 Running 0 44s
grafana-77978cbbdc-x5rsq 1/1 Running 0 40s
kube-state-metrics-7f6d7b46b4-crzx2 3/3 Running 0 40s
node-exporter-nrc88 2/2 Running 0 41s
prometheus-adapter-68698bc948-bl7p8 1/1 Running 0 40s
prometheus-k8s-0 3/3 Running 1 39s
prometheus-k8s-1 3/3 Running 1 39s
prometheus-operator-6685db5c6-qfpbj 1/1 Running 0 106s

Access the Prometheus web interface by using the kubectl port-forward command. If you want to expose this interface for others to use, create an ingress service. Make sure you nable security controls that follow Prometheus best practices.

% kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090 &

Navigate to http://localhost:9090/targets.

Grant Prometheus RBAC permissions

There are two steps to configure Prometheus to monitor Armory Continuous Deployment:

Add permissions for Prometheus to talk to the Spinnaker namespace
Configure Prometheus to discover the Armory Continuous Deployment endpoints

Add permissions for Prometheus by applying the following configuration to your cluster. You can learn more about this process on the Prometheus Operator homepage.

Example config:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
 # name can be either prometheus or prometheus-k8s depending on the version of the prometheus-operator
 name: prometheus
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 # name can be either prometheus or prometheus-k8s depending on the version of the prometheus-operator
 name: prometheus
subjects:
 - kind: ServiceAccount
 # name can be either prometheus or prometheus-k8s depending on the version of the prometheus-operator
 name: prometheus-k8s
 namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
 # name can be either prometheus or prometheus-k8s depending on the version of the prometheus-operator
 name: prometheus
rules:
- apiGroups: [""]
 resources:
 - nodes
 - nodes/metrics
 - services
 - endpoints
 - pods
 verbs: ["get", "list", "watch"]
- apiGroups: [""]
 resources:
 - configmaps
 verbs: ["get"]
- nonResourceURLs: ["/metrics"]
 verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
 namespace: monitoring
 # name can be either prometheus or prometheus-k8s depending on the version of the prometheus-operator
 name: prometheus-k8s

Add the ServiceMonitor

Prometheus Operator uses a “ServiceMonitor” to add targets that get scraped for monitoring. The following example config shows how to monitor pods that are using the Observability Plugin to expose the aop-prometheus endpoint. Note that the example contains both the exclusion of certain services (such as Redis) and changes to the Gate endpoint to show you different options.

These are examples of potential configurations. Use them as a starting point. Armory recommends that you understand how they operate and find services. Adapt them to your environment.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 labels:
 app: spin
 # This label is here to match the prometheus operator serviceMonitorSelector attribute
 # prometheus.prometheusSpec.serviceMonitorSelector. For more information, see
 # https://github.com/helm/charts/tree/master/stable/prometheus-operator
 release: prometheus-operator
 name: spinnaker-all-metrics
 namespace: spinnaker
spec:
 endpoints:
 - interval: 10s
 path: /aop-prometheus
 selector:
 matchExpressions:
 - key: cluster
 operator: NotIn
 values:
 - spin-gate
 - spin-gate-api
 - spin-gate-custom
 - spin-deck
 - spin-deck-custom
 - spin-redis
 - spin-terraformer
 - spin-dinghy
 matchLabels:
 app: spin

The example excludes Gate, the API service since Gate restricts access to the endpoints unless authenticated (excluding health).

The following example is for a service monitor for Gate on a different path and using TLS.

Once these are applied, you can port forward prometheus and validate that prometheus has discovered and scraped targets as appropriate.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 name: spinnaker-internal-metrics
 namespace: spinnaker
 labels:
 app: spin
 # This label is here to match the prometheus operator serviceMonitorSelector attribute
 # prometheus.prometheusSpec.serviceMonitorSelector
 # https://github.com/helm/charts/tree/master/stable/prometheus-operator
 release: prometheus-operator
spec:
 selector:
 matchLabels:
 cluster: spin-gate
 endpoints:
 - interval: 10s
 path: "/api/v1/aop-prometheus"
 # If Prometheus returns the error "http: server gave HTTP response to HTTPS client" then
 # replace scheme with targetPort:
 # Note that "port" is string only. "targetPort" is integer or string.
 # For example, targetPort: 8084
 scheme: "https"
 tlsConfig:
 insecureSkipVerify: true

Check for Armory Continuous Deployment targets in Prometheus

After applying these changes, you should be able to see Armory Continuous Deployment targets in Prometheus. It may take 3 to 5 minutes for this to show up depending on where Prometheus is in its config polling interval.

Access Grafana

Configure port forwarding for Grafana:

$ kubectl --namespace monitoring port-forward svc/grafana 3000

Access the Grafana web interface via http://localhost:3000 and use the default Grafana username and password of admin:admin.

Add Armory dashboards to Grafana

Armory provides some sample dashboards (in JSON format) that you can import into Grafana as a starting point for metrics to graph for monitoring. Armory has additional dashboards that are available to Armory customers. You can skip this section if you are a Grafana expert.

To import the sample dashboards, perform the following steps:

Git clone this repo to your local workstation: (https://github.com/uneeq-oss/spinnaker-mixin)
Access the Grafana web interface (as shown above)
Navigate to Dashboards then Manage
Click on the Import button
Upload the one or more of the sample dashboard files from the repo you cloned

After importing the dashboards, you can explore graphs for each service by clicking on Dashboards > Manage > Spinnaker Kubernetes Details.

Available metrics by service

Disclaimer: the following tables may not contain every available metric for each service.

Clouddriver

Metric Name	Base Unit	Description
amazonClientProvider_rateLimitDelayMillis
authorization
aws_request_clientExecuteTime	milliseconds
aws_request_credentialsRequestTime	milliseconds
aws_request_httpClientReceiveResponseTime	milliseconds
aws_request_httpClientSendRequestTime	milliseconds
aws_request_httpRequestTime	milliseconds
aws_request_requestCount
aws_request_requestMarshallTime	milliseconds
aws_request_requestSigningTime	milliseconds
aws_request_responseProcessingTime	milliseconds
aws_request_retryPauseTime	milliseconds
aws_request_throttling
awsSdkClientSupplier_averageLoadPenalty
awsSdkClientSupplier_hitCount
awsSdkClientSupplier_loadExceptionCount
awsSdkClientSupplier_missRate
cats_sqlCache_evict_deleteOperations
cats_sqlCache_evict_itemCount
cats_sqlCache_evict_itemsDeleted
cats_sqlCache_get_itemCount
cats_sqlCache_get_relationshipsRequested
cats_sqlCache_get_requestedSize
cats_sqlCache_get_selectOperations
cats_sqlCache_merge_deleteOperations
cats_sqlCache_merge_itemCount
cats_sqlCache_merge_itemsStored
cats_sqlCache_merge_relationshipCount
cats_sqlCache_merge_relationshipsStored
cats_sqlCache_merge_selectOperations
cats_sqlCache_merge_writeOperations
cf_okhttp_requests	milliseconds	Timer of OkHttp operation
controller_invocations
controller_invocations_contentLength
controller_invocations_contentLength_summary
executionTime	milliseconds
health_kubernetes_errors
http_server_requests	milliseconds
jvm_buffer_count	buffers	An estimate of the number of buffers in the pool
jvm_gc_pause	milliseconds	Time spent in GC pause
jvm_memory_committed	bytes	The amount of memory in bytes that is committed for the Java virtual machine to use
jvm_memory_max	bytes	The maximum amount of memory in bytes that can be used for memory management
jvm_threads_daemon	threads	The current number of live daemon threads
jvm_threads_peak	threads	The peak live thread count since the Java virtual machine started or peak was reset
jvm_threads_states	threads	The current number of threads having BLOCKED state
kubernetes_api	milliseconds
logback_events	events	Number of debug level events that made it to the logs
onDemand_cache	milliseconds
onDemand_count
onDemand_error
onDemand_evict	milliseconds
onDemand_read	milliseconds
onDemand_store	milliseconds
onDemand_total	milliseconds
onDemand_transform	milliseconds
operations	milliseconds
orchestrations	milliseconds
process_files_max	files	The maximum file descriptor count
reservedInstances_surplusByAccountClassic
reservedInstances_surplusByAccountVpc
reservedInstances_surplusOverall
resilience4j_retry_calls		The number of failed calls after a retry attempt
sql_cacheCleanupAgent_dataTypeCleanupDuration	milliseconds
sql_cacheCleanupAgent_dataTypeRecordsDeleted
sql_healthProvider_invocations
sql_taskCleanupAgent_deleted
sql_taskCleanupAgent_timing	milliseconds
system_load_average_1m		The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
tasks
tasks
tomcat_sessions_active_current	sessions
tomcat_sessions_expired	sessions
tomcat_sessions_rejected	sessions

Echo

Metric Name	Base Unit	Description
aws_request_httpClientGetConnectionTime	milliseconds
controller_invocations
controller_invocations_contentLength
controller_invocations_contentLength_summary
echo_events_processed
echo_triggers_sync_executionTimeMillis	milliseconds
fiat_enabled
fiat_getPermission
fiat_legacyFallback_enabled
fiat_permissionsCache_evictions
fiat_permissionsCache_evictions-weight
fiat_permissionsCache_hits
fiat_permissionsCache_loads	milliseconds
fiat_permissionsCache_loads-failure
fiat_permissionsCache_loads-success
fiat_permissionsCache_misses
front50_lastPoll
front50_requests
http_server_requests	milliseconds
jvm_buffer_count	buffers	An estimate of the number of buffers in the pool
jvm_buffer_memory_used	bytes	An estimate of the memory that the Java virtual machine is using for this buffer pool
jvm_buffer_total_capacity	bytes	An estimate of the total capacity of the buffers in this pool
jvm_classes_loaded	classes	The number of classes that are currently loaded in the Java virtual machine
jvm_classes_unloaded	classes	The total number of classes unloaded since the Java virtual machine has started execution
jvm_gc_allocationRate
jvm_gc_live_data_size	bytes	Size of old generation memory pool after a full GC
jvm_gc_liveDataSize
jvm_gc_max_data_size	bytes	Max size of old generation memory pool
jvm_gc_maxDataSize
jvm_gc_memory_allocated	bytes	Incremented for an increase in the size of the young generation memory pool after one GC to before the next
jvm_gc_memory_promoted	bytes	Count of positive increases in the size of the old generation memory pool before GC to after GC
jvm_gc_pause	milliseconds	Time spent in GC pause
jvm_gc_promotionRate
jvm_memory_committed	bytes	The amount of memory in bytes that is committed for the Java virtual machine to use
jvm_memory_max	bytes	The maximum amount of memory in bytes that can be used for memory management
jvm_memory_used	bytes	The amount of used memory
jvm_threads_daemon	threads	The current number of live daemon threads
jvm_threads_live	threads	The current number of live threads including both daemon and non-daemon threads
jvm_threads_peak	threads	The peak live thread count since the Java virtual machine started or peak was reset
jvm_threads_states	threads	The current number of threads having NEW state
logback_events	events	Number of info level events that made it to the logs
okhttp_requests	milliseconds
orca_requests
orca_trigger_success
pipelines_triggered
process_cpu_usage		The recent cpu usage for the Java Virtual Machine process
process_files_max	files	The maximum file descriptor count
process_files_open	files	The open file descriptor count
process_start_time	milliseconds	Start time of the process since unix epoch_
process_uptime	milliseconds	The uptime of the Java virtual machine
quietPeriod_tests
resilience4j_circuitbreaker_buffered_calls		The number of buffered failed calls stored in the ring buffer
resilience4j_circuitbreaker_calls	milliseconds	Total number of calls which failed but the exception was ignored
resilience4j_circuitbreaker_failure_rate		The failure rate of the circuit breaker
resilience4j_circuitbreaker_slow_call_rate		The slow call of the circuit breaker
resilience4j_circuitbreaker_state		The states of the circuit breaker
system_cpu_count		The number of processors available to the Java virtual machine
system_load_average_1m		The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
tomcat_sessions_active_current	sessions
tomcat_sessions_active_max	sessions
tomcat_sessions_alive_max	milliseconds
tomcat_sessions_created	sessions
tomcat_sessions_expired	sessions
tomcat_sessions_rejected	sessions

Fiat

Metric Name	Base Unit	Description
controller_invocations
controller_invocations_contentLength
controller_invocations_contentLength_summary
fiat_getUserPermission
fiat_userRoles_syncAnonymous	milliseconds
fiat_userRoles_syncCount
fiat_userRoles_syncTime	milliseconds
fiat_userRoles_syncUsers	milliseconds
http_server_requests	milliseconds
jvm_buffer_count	buffers	An estimate of the number of buffers in the pool
jvm_buffer_memory_used	bytes	An estimate of the memory that the Java virtual machine is using for this buffer pool
jvm_buffer_total_capacity	bytes	An estimate of the total capacity of the buffers in this pool
jvm_classes_loaded	classes	The number of classes that are currently loaded in the Java virtual machine
jvm_classes_unloaded	classes	The total number of classes unloaded since the Java virtual machine has started execution
jvm_gc_allocationRate
jvm_gc_live_data_size	bytes	Size of old generation memory pool after a full GC
jvm_gc_liveDataSize
jvm_gc_max_data_size	bytes	Max size of old generation memory pool
jvm_gc_maxDataSize
jvm_gc_memory_allocated	bytes	Incremented for an increase in the size of the young generation memory pool after one GC to before the next
jvm_gc_memory_promoted	bytes	Count of positive increases in the size of the old generation memory pool before GC to after GC
jvm_gc_pause	milliseconds	Time spent in GC pause
jvm_gc_promotionRate
jvm_memory_committed	bytes	The amount of memory in bytes that is committed for the Java virtual machine to use
jvm_memory_max	bytes	The maximum amount of memory in bytes that can be used for memory management
jvm_memory_used	bytes	The amount of used memory
jvm_threads_daemon	threads	The current number of live daemon threads
jvm_threads_live	threads	The current number of live threads including both daemon and non-daemon threads
jvm_threads_peak	threads	The peak live thread count since the Java virtual machine started or peak was reset
jvm_threads_states	threads	The current number of threads having TERMINATED state
kork_lock_acquire
kork_lock_acquire_duration
kork_lock_heartbeat
kork_lock_release
logback_events	events	Number of debug level events that made it to the logs
okhttp_requests	milliseconds
permissionsRepository_get1_invocations
permissionsRepository_get1_timing
permissionsRepository_getAllById_invocations
permissionsRepository_getAllById_timing
permissionsRepository_put1_invocations
permissionsRepository_put1_timing
permissionsRepository_putAllById1_invocations
permissionsRepository_putAllById1_timing
process_cpu_usage		The recent cpu usage for the Java Virtual Machine process
process_files_max	files	The maximum file descriptor count
process_files_open	files	The open file descriptor count
process_start_time	milliseconds	Start time of the process since unix epoch_
process_uptime	milliseconds	The uptime of the Java virtual machine
redis_command_invocation_del
redis_command_invocation_eval
redis_command_invocation_get
redis_command_invocation_hgetAll
redis_command_invocation_hmset
redis_command_invocation_hscan
redis_command_invocation_pipelined
redis_command_invocation_rename
redis_command_invocation_sadd
redis_command_invocation_set
redis_command_invocation_sismember
redis_command_invocation_srem
redis_command_invocation_sscan
redis_command_invocation_time
redis_command_latency_del
redis_command_latency_eval	milliseconds
redis_command_latency_get	milliseconds
redis_command_latency_get
redis_command_latency_hgetAll
redis_command_latency_hmset
redis_command_latency_hscan
redis_command_latency_pipelined
redis_command_latency_rename
redis_command_latency_sadd
redis_command_latency_set
redis_command_latency_sismember
redis_command_latency_srem
redis_command_latency_sscan
redis_command_latency_time
redis_command_payloadSize_eval
redis_command_payloadSize_eval_summary
redis_command_payloadSize_sadd
redis_command_payloadSize_sadd_summary
redis_command_payloadSize_set
redis_command_payloadSize_set_summary
resilience4j_circuitbreaker_buffered_calls		The number of buffered failed calls stored in the ring buffer
resilience4j_circuitbreaker_calls	milliseconds
resilience4j_circuitbreaker_failure_rate		The failure rate of the circuit breaker
resilience4j_circuitbreaker_slow_call_rate		The slow call of the circuit breaker
resilience4j_circuitbreaker_state		The states of the circuit breaker
resilience4j_retry_calls		The number of failed calls after a retry attempt
system_cpu_count		The number of processors available to the Java virtual machine
system_cpu_usage		The recent cpu usage for the whole system
system_load_average_1m		The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
tomcat_sessions_active_current	sessions
tomcat_sessions_active_max	sessions
tomcat_sessions_alive_max	milliseconds
tomcat_sessions_created	sessions
tomcat_sessions_expired	sessions
tomcat_sessions_rejected	sessions

Front50

Metric Name	Base Unit	Description
aws_request_clientExecuteTime	milliseconds
aws_request_credentialsRequestTime	milliseconds
aws_request_httpClientGetConnectionTime	milliseconds
aws_request_httpClientReceiveResponseTime	milliseconds
aws_request_httpClientSendRequestTime	milliseconds
aws_request_httpRequestTime	milliseconds
aws_request_requestCount
aws_request_requestSigningTime	milliseconds
aws_request_responseProcessingTime	milliseconds
aws_request_retryPauseTime	milliseconds
controller_invocations
controller_invocations_contentLength
controller_invocations_contentLength_summary
fiat_enabled
fiat_getPermission
fiat_legacyFallback_enabled
fiat_permissionsCache_evictions
fiat_permissionsCache_evictions-weight
fiat_permissionsCache_hits
fiat_permissionsCache_loads	milliseconds
fiat_permissionsCache_loads-failure
fiat_permissionsCache_loads-success
fiat_permissionsCache_misses
http_server_requests	milliseconds
jvm_buffer_count	buffers	An estimate of the number of buffers in the pool
jvm_buffer_memory_used	bytes	An estimate of the memory that the Java virtual machine is using for this buffer pool
jvm_buffer_total_capacity	bytes	An estimate of the total capacity of the buffers in this pool
jvm_classes_loaded	classes	The number of classes that are currently loaded in the Java virtual machine
jvm_classes_unloaded	classes	The total number of classes unloaded since the Java virtual machine has started execution
jvm_gc_allocationRate
jvm_gc_live_data_size	bytes	Size of old generation memory pool after a full GC
jvm_gc_liveDataSize
jvm_gc_max_data_size	bytes	Max size of old generation memory pool
jvm_gc_maxDataSize
jvm_gc_memory_allocated	bytes	Incremented for an increase in the size of the young generation memory pool after one GC to before the next
jvm_gc_memory_promoted	bytes	Count of positive increases in the size of the old generation memory pool before GC to after GC
jvm_gc_pause	milliseconds	Time spent in GC pause
jvm_gc_promotionRate
jvm_memory_committed	bytes	The amount of memory in bytes that is committed for the Java virtual machine to use
jvm_memory_max	bytes	The maximum amount of memory in bytes that can be used for memory management
jvm_memory_used	bytes	The amount of used memory
jvm_threads_daemon	threads	The current number of live daemon threads
jvm_threads_live	threads	The current number of live threads including both daemon and non-daemon threads
jvm_threads_peak	threads	The peak live thread count since the Java virtual machine started or peak was reset
jvm_threads_states	threads	The current number of threads having WAITING state
logback_events	events	Number of error level events that made it to the logs
okhttp_requests	milliseconds
process_cpu_usage		The recent cpu usage for the Java Virtual Machine process
process_files_max	files	The maximum file descriptor count
process_files_open	files	The open file descriptor count
process_start_time	milliseconds	Start time of the process since unix epoch_
process_uptime	milliseconds	The uptime of the Java virtual machine
resilience4j_circuitbreaker_buffered_calls
resilience4j_circuitbreaker_calls	milliseconds
resilience4j_circuitbreaker_failure_rate		The failure rate of the circuit breaker
resilience4j_circuitbreaker_slow_call_rate		The slow call of the circuit breaker
resilience4j_circuitbreaker_slow_calls		The number of slow failed calls which were slower than a certain threshold
resilience4j_circuitbreaker_state		The states of the circuit breaker
storageServiceSupport_autoRefreshTime	milliseconds
storageServiceSupport_cacheAge
storageServiceSupport_cacheRefreshTime	milliseconds
storageServiceSupport_cacheSize
storageServiceSupport_mismatchedIds
storageServiceSupport_numAdded
storageServiceSupport_numRemoved
storageServiceSupport_numUpdated
storageServiceSupport_scheduledRefreshTime	milliseconds
system_cpu_count		The number of processors available to the Java virtual machine
system_cpu_usage		The recent cpu usage for the whole system
system_load_average_1m		The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
tomcat_sessions_active_current	sessions
tomcat_sessions_active_max	sessions
tomcat_sessions_alive_max	milliseconds
tomcat_sessions_created	sessions
tomcat_sessions_expired	sessions
tomcat_sessions_rejected	sessions

Gate

Metric Name	Base Unit	Description
controller_invocations
controller_invocations_contentLength
controller_invocations_contentLength_summary
fiat_enabled
fiat_getPermission
fiat_legacyFallback_enabled
fiat_login
fiat_permissionsCache_evictions
fiat_permissionsCache_evictions-weight
fiat_permissionsCache_hits
fiat_permissionsCache_loads	milliseconds
fiat_permissionsCache_loads-failure
fiat_permissionsCache_loads-success
fiat_permissionsCache_misses
http_server_requests	milliseconds
http_server_requests	milliseconds
http_server_requests	milliseconds
jvm_buffer_count	buffers	An estimate of the number of buffers in the pool
jvm_buffer_memory_used	bytes	An estimate of the memory that the Java virtual machine is using for this buffer pool
jvm_buffer_total_capacity	bytes	An estimate of the total capacity of the buffers in this pool
jvm_classes_loaded	classes	The number of classes that are currently loaded in the Java virtual machine
jvm_classes_unloaded	classes	The total number of classes unloaded since the Java virtual machine has started execution
jvm_gc_allocationRate
jvm_gc_live_data_size	bytes	Size of old generation memory pool after a full GC
jvm_gc_liveDataSize
jvm_gc_max_data_size	bytes	Max size of old generation memory pool
jvm_gc_maxDataSize
jvm_gc_memory_allocated	bytes	Incremented for an increase in the size of the young generation memory pool after one GC to before the next
jvm_gc_memory_promoted	bytes	Count of positive increases in the size of the old generation memory pool before GC to after GC
jvm_gc_pause	milliseconds	Time spent in GC pause
jvm_gc_promotionRate
jvm_memory_committed	bytes	The amount of memory in bytes that is committed for the Java virtual machine to use
jvm_memory_max	bytes	The maximum amount of memory in bytes that can be used for memory management
jvm_memory_used	bytes	The amount of used memory
jvm_threads_daemon	threads	The current number of live daemon threads
jvm_threads_live	threads	The current number of live threads including both daemon and non-daemon threads
jvm_threads_peak	threads	The peak live thread count since the Java virtual machine started or peak was reset
jvm_threads_peak	threads	The peak live thread count since the Java virtual machine started or peak was reset
jvm_threads_states	threads	The current number of threads having RUNNABLE state
logback_events	events	Number of error level events that made it to the logs
okhttp_requests	milliseconds
plugins_deckAssets_hits
plugins_deckCache_downloadDuration	milliseconds
plugins_deckCache_hits
plugins_deckCache_misses
plugins_deckCache_refreshDuration	milliseconds
plugins_deckCache_versions
process_cpu_usage		The recent cpu usage for the Java Virtual Machine process
process_files_max	files	The maximum file descriptor count
process_files_open	files	The open file descriptor count
process_start_time	milliseconds	Start time of the process since unix epoch_
process_uptime	milliseconds	The uptime of the Java virtual machine
system_cpu_count		The number of processors available to the Java virtual machine
system_cpu_usage		The recent cpu usage for the whole system
system_load_average_1m		The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
tomcat_sessions_active_current	sessions
tomcat_sessions_active_current	sessions
tomcat_sessions_active_max	sessions
tomcat_sessions_alive_max	milliseconds
tomcat_sessions_created	sessions
tomcat_sessions_expired	sessions
tomcat_sessions_rejected	sessions

Igor

Metric Name	Base Unit	Description
controller_invocations
controller_invocations_contentLength
controller_invocations_contentLength_summary
fiat_enabled
fiat_getPermission
fiat_legacyFallback_enabled
fiat_permissionsCache_evictions
fiat_permissionsCache_evictions-weight
fiat_permissionsCache_hits
fiat_permissionsCache_loads	milliseconds
fiat_permissionsCache_loads-failure
fiat_permissionsCache_loads-success
fiat_permissionsCache_misses
http_server_requests	milliseconds
jvm_buffer_count	buffers	An estimate of the number of buffers in the pool
jvm_buffer_memory_used	bytes	An estimate of the memory that the Java virtual machine is using for this buffer pool
jvm_classes_loaded	classes	The number of classes that are currently loaded in the Java virtual machine
jvm_classes_unloaded	classes	The total number of classes unloaded since the Java virtual machine has started execution
jvm_gc_allocationRate
jvm_gc_live_data_size	bytes	Size of old generation memory pool after a full GC
jvm_gc_liveDataSize
jvm_gc_max_data_size	bytes	Max size of old generation memory pool
jvm_gc_maxDataSize
jvm_gc_memory_allocated	bytes	Incremented for an increase in the size of the young generation memory pool after one GC to before the next
jvm_gc_pause	milliseconds	Time spent in GC pause
jvm_gc_promotionRate
jvm_memory_committed	bytes	The amount of memory in bytes that is committed for the Java virtual machine to use
jvm_memory_max	bytes	The maximum amount of memory in bytes that can be used for memory management
jvm_memory_used	bytes	The amount of used memory
jvm_threads_daemon	threads	The current number of live daemon threads
jvm_threads_live	threads	The current number of live threads including both daemon and non-daemon threads
jvm_threads_peak	threads	The peak live thread count since the Java virtual machine started or peak was reset
jvm_threads_states	threads	The current number of threads having NEW state
logback_events	events
okhttp_requests	milliseconds
pollingMonitor_docker_retrieveImagesByAccount	milliseconds
pollingMonitor_jenkins_retrieveProjects	milliseconds
pollingMonitor_pollTiming	milliseconds
process_cpu_usage		The recent cpu usage for the Java Virtual Machine process
process_files_max	files	The maximum file descriptor count
process_files_open	files	The open file descriptor count
process_start_time	milliseconds	Start time of the process since unix epoch_
process_uptime	milliseconds	The uptime of the Java virtual machine
resilience4j_circuitbreaker_buffered_calls		The number of buffered failed calls stored in the ring buffer
resilience4j_circuitbreaker_calls		Total number of not permitted calls
resilience4j_circuitbreaker_failure_rate		The failure rate of the circuit breaker
resilience4j_circuitbreaker_slow_call_rate		The slow call of the circuit breaker
resilience4j_circuitbreaker_state		The states of the circuit breaker
system_cpu_count		The number of processors available to the Java virtual machine
system_cpu_usage		The recent cpu usage for the whole system
system_load_average_1m		The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
tomcat_sessions_active_current	sessions
tomcat_sessions_alive_max	milliseconds
tomcat_sessions_created	sessions
tomcat_sessions_expired	sessions
tomcat_sessions_rejected	sessions

Kayenta

Metric Name	Base Unit	Description
canary_pipelines_initiated
canary_telemetry_query
controller_invocations	milliseconds
controller_invocations_contentLength
controller_invocations_contentLength_summary
executions_active
executions_completed
executions_started
http_server_requests	milliseconds
jvm_gc_allocationRate
jvm_gc_liveDataSize
jvm_gc_maxDataSize
jvm_gc_pause	milliseconds
jvm_gc_promotionRate
okhttp_requests	milliseconds
orca_task_result
queue_acknowledged_messages
queue_depth
queue_duplicate_messages
queue_last_poll_age
queue_last_retry_check_age
queue_message_lag	milliseconds
queue_orphaned_messages
queue_pushed_messages
queue_ready_depth
queue_unacked_depth
redis_command_invocation_exists
redis_command_invocation_hdel
redis_command_invocation_hget
redis_command_invocation_hgetAll
redis_command_invocation_hmset
redis_command_invocation_hset
redis_command_invocation_multi
redis_command_invocation_sadd
redis_command_invocation_srem
redis_command_invocation_zadd
redis_command_latency_exists
redis_command_latency_exists
redis_command_latency_hdel
redis_command_latency_hget
redis_command_latency_hgetAll
redis_command_latency_hmset	milliseconds
redis_command_latency_hset
redis_command_latency_multi
redis_command_latency_sadd
redis_command_latency_srem
redis_command_latency_zadd
redis_command_payloadSize_hmset
redis_command_payloadSize_hmset_summary
redis_command_payloadSize_hset
redis_command_payloadSize_hset_summary
redis_command_payloadSize_sadd
redis_command_payloadSize_sadd_summary
redis_command_payloadSize_srem
redis_command_payloadSize_srem_summary
redis_connectionPool_maxIdle
redis_connectionPool_minIdle
redis_connectionPool_numActive
redis_connectionPool_numIdle
redis_connectionPool_numWaiters
redis_executionRepository_store1_invocations
redis_executionRepository_store1_timing	milliseconds
redis_executionRepository_storeStage1_invocations
redis_executionRepository_storeStage1_timing
redis_executionRepository_updateStatus1_invocations
redis_executionRepository_updateStatus1_timing	milliseconds
retrieveById_redis_executionRepository_invocations
retrieveById_redis_executionRepository_timing
stage_invocations
stage_invocations_duration
task_completions_duration	milliseconds
task_completions_duration_withType	milliseconds
task_invocations_duration	milliseconds
task_invocations_duration_withType	milliseconds
threadpool_activeCount
threadpool_blockingQueueSize
threadpool_corePoolSize
threadpool_maximumPoolSize
threadpool_poolSize
tomcat_sessions_active_current	sessions
tomcat_sessions_active_max	sessions
tomcat_sessions_alive_max	milliseconds
tomcat_sessions_created	sessions
tomcat_sessions_expired	sessions
tomcat_sessions_rejected	sessions

Orca

Metric Name	Base Unit	Description
aws_request_httpClientGetConnectionTime	milliseconds
controller_invocations
controller_invocations_contentLength
controller_invocations_contentLength_summary
executions_active
executions_completed
executions_started
executions_totalTime	milliseconds
fiat_enabled
fiat_getPermission
fiat_legacyFallback_enabled
fiat_permissionsCache_loads	milliseconds
fiat_permissionsCache_loads-failure
http_server_requests	milliseconds
jdbc_connections_active
jdbc_connections_idle
jdbc_connections_max
jvm_gc_allocationRate
jvm_gc_pause	milliseconds
jvm_gc_promotionRate
mpt_requests
okhttp_requests	milliseconds
orca_task_result
queue_acknowledged_messages
queue_depth
queue_duplicate_messages
queue_last_poll_age
queue_message_notfound
queue_orphaned_messages
queue_pushed_messages
queue_retried_messages
queue_unacked_depth
redis_connectionPool_maxIdle
redis_connectionPool_numActive
redis_connectionPool_numIdle
resilience4j_retry_calls		The number of successful calls after a retry attempt
retrieveById_sql_executions_invocations
retrieveById_sql_executions_timing
sql_executions_addStage1_timing
sql_executions_cancel4_invocations
sql_executions_cancel4_timing
sql_executions_countActiveExecutions_invocations
sql_executions_countActiveExecutions_timing
sql_executions_handlesPartition1_invocations
sql_executions_handlesPartition1_timing	milliseconds
sql_executions_retrieveByCorrelationId2_timing
sql_executions_retrieveOrchestrationsForApplication3_timing
sql_executions_store1_timing
sql_executions_storeStage1_invocations
sql_executions_storeStage1_timing
sql_executions_updateStatus1_invocations
sql_executions_updateStatus1_timing
sql_healthProvider_invocations
sql_pool_default_connectionAcquiredTiming	milliseconds
sql_queueActivator_invocations
stage_invocations
stage_invocations_duration
task_completions_duration	milliseconds
task_completions_duration_withType	milliseconds
task_invocations_duration	milliseconds
task_invocations_duration_withType	milliseconds
tasks_serverGroupCacheForceRefresh
threadpool_activeCount
threadpool_blockingQueueSize
threadpool_corePoolSize
threadpool_corePoolSize
threadpool_maximumPoolSize
threadpool_poolSize
tomcat_sessions_active_current	sessions
tomcat_sessions_active_max	sessions
tomcat_sessions_alive_max	milliseconds
tomcat_sessions_rejected	sessions

Rosco

Metric Name	Base Unit	Description
bakesActive
bakesCompleted	milliseconds
controller_invocations
controller_invocations_contentLength
controller_invocations_contentLength_summary
http_server_requests	milliseconds
jvm_buffer_count	buffers	An estimate of the number of buffers in the pool
jvm_buffer_memory_used	bytes	An estimate of the memory that the Java virtual machine is using for this buffer pool
jvm_buffer_total_capacity	bytes	An estimate of the total capacity of the buffers in this pool
jvm_classes_loaded	classes	The number of classes that are currently loaded in the Java virtual machine
jvm_classes_unloaded	classes	The total number of classes unloaded since the Java virtual machine has started execution
jvm_gc_allocationRate
jvm_gc_live_data_size	bytes	Size of old generation memory pool after a full GC
jvm_gc_liveDataSize
jvm_gc_max_data_size	bytes	Max size of old generation memory pool
jvm_gc_maxDataSize
jvm_gc_memory_allocated	bytes	Incremented for an increase in the size of the young generation memory pool after one GC to before the next
jvm_gc_memory_promoted	bytes	Count of positive increases in the size of the old generation memory pool before GC to after GC
jvm_gc_pause	milliseconds	Time spent in GC pause
jvm_gc_promotionRate
jvm_memory_committed	bytes	The amount of memory in bytes that is committed for the Java virtual machine to use
jvm_memory_max	bytes	The maximum amount of memory in bytes that can be used for memory management
jvm_memory_used	bytes	The amount of used memory
jvm_threads_daemon	threads	The current number of live daemon threads
jvm_threads_live	threads	The current number of live threads including both daemon and non-daemon threads
jvm_threads_peak	threads	The peak live thread count since the Java virtual machine started or peak was reset
jvm_threads_states	threads
logback_events	events
okhttp_requests	milliseconds
process_cpu_usage		The recent cpu usage for the Java Virtual Machine process
process_files_max	files	The maximum file descriptor count
process_files_open	files	The open file descriptor count
process_start_time	milliseconds	Start time of the process since unix epoch_
process_uptime	milliseconds	The uptime of the Java virtual machine
system_cpu_count		The number of processors available to the Java virtual machine
system_cpu_usage		The recent cpu usage for the whole system
system_load_average_1m		The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
tomcat_sessions_active_current	sessions
tomcat_sessions_active_max	sessions
tomcat_sessions_alive_max	milliseconds
tomcat_sessions_created	sessions
tomcat_sessions_expired	sessions
tomcat_sessions_rejected	sessions

Armory Docs – Observability

Continuous-Deployment: Armory Continuous Deployment Monitoring Indicators

Common Metrics - All Services

Service Level Indicators (SLI)

Controller latency

Outgoing requests latency

5xx errors sent

5xx errors received

Hystrix fallbacks

JVM memory usage

Configurations affecting performance

OkHttp client timeout

JVM memory requests/limits

Clouddriver Metrics

Average caching agent time

Number of caching agents

Kubernetes, kubectl latency

AWS failed requests

SQL blocked connections

Redis latency

Configurations affecting performance

checkPermissionsOnStartup

liveManifestCalls

cacheThreads

metrics

replicas

cpu/memory requests/limits

onDemandClusterThreshold

max-concurrent-agents

jobs.local.timeout-minutes

Continuous-Deployment: Configure Monitoring

Configure monitoring using the Observability Plugin

Caution

Install the plugin

Continuous-Deployment: Configure Support Diagnostics for Armory Continuous Deployment

Advantages to enabling Armory Diagnostics

Turning on Armory Diagnostics

Migrating Diagnostics to Operator from Halyard

What data gets sent to Armory

How Armory uses the data

A Note on Private Data Exposure Risks

Continuous-Deployment: Enable Logging in Armory Continuous Deployment

Metrics and logging overview

Before you begin

Enable logging data

Verify data flow to your aggregator

Creating dashboards

Splunk dashboard examples

Developer insights

Software deployments

Armory Continuous Deployment platform

Continuous verification

Compliance center

Continuous-Deployment: Integrate a Sumo Logic Dashboard in Armory Continuous Deployment

Advantages to integrating Sumo Logic

Log types

Sample log message

Query sample

Collect logs for Spinnaker

Collection process overview

Collection Step 1: Retrieve SumoLogic URL for HTTP logs**

Collection Step 2: Modify Spinnaker configuration to export logs to SumoLogic**

Collection Step 3: Apply changes**

Install the Spinnaker app and view the Dashboards

Dashboard filters

Spinnaker Overview dashboard

Spinnaker Pipelines dashboard

Spinnaker User Activity dashboard

Sumo Logic collector limitations

Continuous-Deployment: Monitor Armory Continuous Deployment with Prometheus

Overview

Before you begin

Use kube-prometheus to create a monitoring stack

Grant Prometheus RBAC permissions

Add the ServiceMonitor

Check for Armory Continuous Deployment targets in Prometheus

Access Grafana

Add Armory dashboards to Grafana

Available metrics by service

Clouddriver

Use `kube-prometheus` to create a monitoring stack