How Prometheus works

By periodically capturing the status of monitored components through HTTP, any component can be connected to Prometheus monitoring as long as it provides the corresponding HTTP interface and conforms to the data format defined by Prometheus. Prometheus Server is responsible for fetching metrics (metrics) data on targets on a regular basis, and each target needs to expose an HTTP service interface for Prometheus fetching. This method of calling the monitored object to obtain monitoring data is called Pull. The Pull approach reflects Prometheus's unique design philosophy, which is different from most monitoring using the Push approach.
Advantage

• A powerful multidimensional data model
• Flexible and powerful query statement (PromQL): In the same query statement, multiple metrics can be multiplied, added, joined, scored bits and so on.
• Easy to manage: Prometheus server is a separate binary file that works directly locally and does not depend on distributed storage.
• Efficient: Only 3.5 bytes per sample point on average, and one Prometheus server can handle millions of metrics.
• Collecting time series data using pull mode not only facilitates native testing but also avoids bad metrics pushed by problematic servers.
• Time series data can be pushed to Prometheus server using push gateway
• Monitoring targets can be obtained through service discovery or static configuration.
• There are many visual graphical interfaces.
• Easy to scale.
  Characteristic
• Multidimensional data model: Time series data identified by measure name and key-value pairs.
• PromSQL: A flexible query language that can use multidimensional data to complete complex queries.
• Instead of relying on distributed storage, a single server node can work directly.
• Time series data is collected by pull based on HTTP.
• Push time series data is supported by the PushGateway component.
• Discover targets through service discovery or static configuration.
• grafana is supported in a variety of graphics modes and dashboards.
  Framework
  From this schema diagram, you can also see that the main modules of Prometheus include Server, Exporters, Pushgateway, PromQL, Alertmanager, WebUI, etc. It roughly uses the logic:
• Prometheus server periodically pulls data from statically configured targets or from targets discovered by services.
• When the newly pulled data is larger than the configured memory cache, Prometheus persists the data to disk (if remote storage is used to persist to the cloud).
• Prometheus can configure rules, then periodically query data, and when a condition triggers, it pushes alert to the configured aletmanager.
• When Alertmanager receives a warning, it can configure, aggregate, de-weigh, de-noise, and finally send a warning.
• You can query and aggregate data using API s, Prometheus Console, or Grafana.
  

Deploy Prometheus

Download promethues

 wget https://github.com/prometheus/prometheus/releases/download/v2.31.1/prometheus-2.31.1.linux-amd64.tar.gz

Unzip and rename

[root@master ~]# tar xf prometheus-2.31.1.linux-amd64.tar.gz -C /usr/local/
[root@master ~]# cd /usr/local/
[root@master local]# ls
bin    include  libexec                        share
etc    lib      prometheus-2.31.1.linux-amd64  src
games  lib64    sbin
[root@master local]# mv prometheus-2.31.1.linux-amd64/ prometheus
[root@master local]# ls
bin  games    lib    libexec     sbin   src
etc  include  lib64  prometheus  share
[root@master local]# useradd -M -s /sbin/nologin prometheus
[root@master local]# mkdir -p /data/prometheus
[root@master local]# chown -R prometheus.prometheus /usr/local/prometheus /data/prometheus

Configure service file

[root@master ~]# vim /usr/lib/systemd/system/prometheus.service
[Unit]
Description=The Prometheus Server
After=network.target

[Service]
Restart=on-failure
ExecStart=/usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml
RestartSec=15s

[Install]
WantedBy=multi-user.target

[root@master ~]#  systemctl daemon-reload

Start Services

[root@master ~]# systemctl restart prometheus
[root@master ~]# ss -anlt
State  Recv-Q Send-Q  Local Address:Port   Peer Address:Port 
LISTEN 0      128           0.0.0.0:22          0.0.0.0:*    
LISTEN 0      128                 *:9090              *:*    
LISTEN 0      128              [::]:22             [::]:*    

Test Access