High availability (HA) refers to a system or application that offers high operational availability. This means the entire site or application will not be down if one server goes down due to traffic overload or other issues. HA represents that the application remains available with no interruption. You'll achieve high availability when an application continues to operate, even when one or more underlying components fail.
HA isn't just a safety net; it's your ace in the sleeve against service disruptions. Progress Chef Automate HA is your reliability guarantee, weaving efficiency and productivity into the fabric of your system. Redundancy and failover are its building blocks, tackling service and zone failures head-on.
Prometheus is a time-series monitoring tool for managing various system resources and applications. It provides a multidimensional data model, the ability to query the collected data and detailed reporting and data visualization through Grafana.
Prometheus, an open-source tool excels at scraping metrics from HTTP endpoints, making it a go-to for harvesting data from a variety of targets. Platforms for infrastructure (like Kubernetes), business applications and services are all supported targets (e.g., database management systems). Prometheus also works as a flexible metrics collection and alerting tool with its companion Alert manager service.
Features of Prometheus:
As a Chef Automate HA customer, leveraging Prometheus becomes essential to receive metrics regarding the health and availability of the solution. Whether you're in the cloud with AWS Managed Services or grounded in on-premises environments, weaving Prometheus into your AHA deployment is key for comprehensive monitoring, alerting and crafting personalized dashboards for insightful visualizations.
When using AWS Managed Services, several benefits can be achieved. This includes automatic updates on AWS status within the Events Explorer, obtaining CloudWatch metrics for EC2 hosts without the need for Agent installation, tagging EC2 hosts with EC2-specific details, visibility into EC2 scheduled maintenance events and the ability to collect CloudWatch metrics and events from a wide range of AWS products.
By default, Prometheus exporter is enabled to collect metrics on the server where it is installed. With the help of various exporters, metrics can be collected from other resources like web servers, containers, databases, custom applications and other third-party systems.
Prometheus is configured via command-line flags and a configuration file. While the command-line flags configure immutable system parameters (such as storage locations, amount of data to keep on disk and in memory, etc.), the configuration file defines everything related to scraping jobs and their instances and which rule files to load.
Before you can install Prometheus exporters on any Chef Automate nodes, you must do the following:
| Exporters | Firewall Ports |
| Node | 9100 |
| Postgres | 9101 |
| Nginx | 9113 |
| Blackbox | 9115 |
| OpenSearch | 9200 |
| HTTPS | 443 |
o Connect to the EC2 instance using SSH and enter the below command. This will create a Linux user account, Prometheus and exporter.
$ sudo useradd --no-create-home --shell /bin/false exporter
o Add users and system directories.
These user accounts are created for the sole purpose of management and, therefore, do not require additional user services or permissions beyond the scope of this setup. In this procedure, you also create directories for storing and managing the files, service settings and data that Prometheus uses to monitor resources.
Steps:
o Run the following command to extract the contents of the downloaded Node Exporter files.
$ tar -xvf nodeexporter-1.3.1.linux-amd64.tar.gz
o Enter the following command to copy the node_exporter file from the ./node_exporter* subdirectory to the /usr/local/bin programs directory.
$ sudo cp -p ./nodeexporter-1.3.1.linux-amd64/nodeexporter /usr/local/bin
o Enter the following command to enable Node Exporter to start when the instance is booted.
$ sudo systemctl enable nodeexporter
o Enter the following command to check the status of the Prometheus service.
$ sudo systemctl status prometheus
Verify that these steps are repeated on all the following servers.
Steps:
Confirm these steps are repeated on all the following servers.
Confirm you are running the steps on Postgres nodes.
Steps:
Double-check to see if these steps are repeated on all the following servers.
Steps:
o Enter the following command to create an environment file for the exporter.
$ mkdir /opt/blackboxexporter
$ sudo vi /opt/blackboxexporter/blackbox.yml
o Add the following content to the yaml file. Update the following parameters for your environment.
o Enter the following command to enable Blackbox Exporter to start when the instance is booted.
$ sudo systemctl enable blackbox_exporter
Verify these steps are repeated on all of the following servers:
Steps:
o Run the following command(s) one by one to extract the contents of the downloaded Exporter files.
$ tar -xvf nginx-prometheus-exporter0.11.0linux_amd64.tar.gz
o Enter the following command to change the ownership of the file to the exporter user that you created.
$ sudo chown exporter:exporter /usr/local/bin/nginx-prometheus-exporter
o Enter the following command to enable Nginx Exporter to start when the instance is booted.
$ sudo systemctl enable nginx-prometheus-exporter
Verify these steps are repeated on all the following servers:
More information on agent installations can be found here.
To monitor the Chef Automate HA infrastructure, a Prometheus server must be installed, and exporters must be installed and configured on the infrastructure nodes to get the possible configured metrics and other information that can be sent to the monitoring tool on the server side to track the health of the overall infrastructure.
Steps:
o Run the following commands one by one to extract the contents of the downloaded Prometheus and Node Exporter files.
$ tar -xvf prometheus-2.37.0.linux-amd64.tar.gz
$ tar -xvf nodeexporter-1.3.1.linux-amd64.tar.gz
o Enter the following command to create a backup copy of the prometheus.yml file before you open and edit it.
$ sudo cp /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml.backup
o Enter the following command to start the Prometheus service.
$ sudo -u prometheus /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries
o Enter the following command to enable Prometheus to start when the instance is booted.
$ sudo systemctl enable prometheus
More information on Download, Installation and Configuration can be found here.
The dashboard allows you to analyze data from across your entire system in a single pane of glass. It lets your team immediately benefit from dynamic views without requiring query language or coding.
o This article provides step-by-step assistance on how to install Grafana
Type of Dashboards:
Screenboards: Screenboards are designed for ad-hoc analysis and troubleshooting. They allow you to create a custom layout of widgets and panels to visualize your data.
Timeboards: Timeboards are designed for monitoring and alerting. They show a historical view of your data over time and can be used to set alerts and notifications.
Automate HA Recommendation:
This article provides detailed guidance to create a dashboard in Grafana.
Steps to create a dashboard for Chef Automate HA component health dashboard. This dashboard will provide the following:
Prerequisites:
Add four new rows and modify the title.
Importing a Dashboard:
The Prometheus community has contributed to the various dashboards and the following dashboards may be configured to monitor Chef Automate HA implementation.
The following process explains how to import existing dashboards in Grafana.
More information on Setting up a Dashboard Configuration can be found here.
The following section documents the metrics collected by various exporters for Chef Managed Automate HA implementation. Similar metrics may be collected from AWS-hosted deployments. These metrics provide guidance to use and build monitoring rules and dashboards based on these metrics.
The following metrics are recommended to monitor Chef Automate HA implementation.
System Metrics:
| Component | Metrics Experience | |
| CPU Usage |
| |
| CPU Steal | (avg(irate(node_cpu_seconds_total{mode="steal"}[5m]) * 100) by(instance,job))> 20 | |
| System Memory |
| |
| Disk Utilization | 100-(node_filesystem_avail_bytes{mountpoint="/"}/node_filesystem_size_bytes{mountpoint="/"}*100) > 85 | |
| Disk Utilization | 100-(node_filesystem_avail_bytes{mountpoint="/hab"}/node_filesystem_size_bytes{mountpoint="/hab"}*100) > 90 | |
| Disk Utilization | 100 - (node_filesystem_avail_bytes{mountpoint="/hab"}/node_filesystem_size_bytes{mountpoint="/hab"}*100) > 85 | |
| Disk Utilization | 100 - (node_filesystem_avail_bytes{mountpoint="/hab"}/node_filesystem_size_bytes{mountpoint="/hab"}*100) > 90 | |
| Disk Utilization | 100 - (node_filesystem_avail_bytes{mountpoint="/tmp"}/node_filesystem_size_bytes{mountpoint="/tmp"}*100) > 85 | |
| Disk Utilization | 100 - (node_filesystem_avail_bytes{mountpoint="/tmp"}/node_filesystem_size_bytes{mountpoint="/tmp"}*100) > 90 | |
| Host Monitoring | Up==0 |
More information on Metrics and AlertManager can be found here.
The Node exporter provides system-level metrics, e.g. CPU, disk, memory, etc. Refer to the node exporter documentation to learn full capabilities and the node exporter service configuration file for the chef-managed Automate HA deployment.
o All Chef Automate Frontend Servers
o All Chef Infra Frontend Servers
o Any other Server to Monitoring Load Balancers
o Chef-Sever-URL - Monitors elastic load balancer for Chef Infra frontend servers
o Chef-Automate-URL - Monitors elastic load balancer for Chef Automate frontend servers.
o Chef-Server-services.* - Monitors all services running on each Chef Infra frontend server.
o Automate-services.* - Monitors all services on each Chef Automate frontend server.
o $ curl -LO https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz
o $ tar -xvf alertmanager-0.25.0.linux-amd64.tar.gz
o $ mv alertmanager-0.25.0.linux-amd64/alertmanager /usr/local/bin
Perform the following steps to configure the alert manager:
$ mkdir /etc/alertmanager/
$ vi /etc/alertmanager/alertmanager.yml
Based on Alert integration to Slack, MS teams, and pager-duty, add the following under the receiver section.
Refer to the Prometheus alert manager configuration documentation for detailed options.
Slack Integration:
The following steps guide preparing Slack receivers for the alert manager to send alerts:
More information on Setting up Slack Integration can be found here.
The following steps guide preparing PagerDuty receivers for the alert manager to send alerts.
o Click on New Service
o Specify Service Name
More information on Setting up PagerDuty Integration can be found here.
The following steps guide preparing MS Teams receivers for the alert manager to send alerts.
o Create a channel in MS-team where you want to send alerts
o Click on connectors (in channel options), then search for the ‘incoming webhook’ connector
More information on Setting up Microsoft Teams Integration can be found here.
Regardless of how your organization is leveraging Progress Chef, the solution's availability and performance should be monitored with the utmost urgency. Leveraging Chef Automate HA, coupled with a monitoring solution such as Prometheus, will give organizations the confidence that the solution is always available and performing at its peak to deliver the services to scale and grow the business.
Reference: