Prometheus

Gotchas

Staleness

When queries are run, timestamps at which to sample data are selected
independently of the actual present time series data. This is mainly to support
cases like aggregation (, , and so on), where multiple aggregated
time series do not exactly align in time. Because of their independence,
Prometheus needs to assign a value at those timestamps for each relevant time
series. It does so by simply taking the newest sample before this timestamp.

If a target scrape or rule evaluation no longer returns a sample for a time
series that was previously present, that time series will be marked as stale.
If a target is removed, its previously returned time series will be marked as
stale soon afterwards.

If a query is evaluated at a sampling timestamp after a time series is marked
stale, then no value is returned for that time series. If new samples are
subsequently ingested for that time series, they will be returned as normal.

If no sample is found (by default) 5 minutes before a sampling timestamp,
no value is returned for that time series at this point in time. This
effectively means that time series «disappear» from graphs at times where their
latest collected sample is older than 5 minutes or after they are marked stale.

Staleness will not be marked for time series that have timestamps included in
their scrapes. Only the 5 minute threshold will be applied in that case.

Avoiding slow queries and overloads

If a query needs to operate on a very large amount of data, graphing it might
time out or overload the server or browser. Thus, when constructing queries
over unknown data, always start building the query in the tabular view of
Prometheus’s expression browser until the result set seems reasonable
(hundreds, not thousands, of time series at most). Only when you have filtered
or aggregated your data sufficiently, switch to graph mode. If the expression
still takes too long to graph ad-hoc, pre-record it via a .

This is especially relevant for Prometheus’s query language, where a bare
metric name selector like could expand to thousands
of time series with different labels. Also keep in mind that expressions which
aggregate over many time series will generate load on the server even if the
output is only a small number of time series. This is similar to how it would
be slow to sum all values of a column in a relational database, even if the
output value is only a single number.

This documentation is . Please help improve it by filing issues or pull requests.

Установка

TL;DR: вот роль для Ansible, писалась под CentOS 7, но после минимальных правок (замена firewalld на iptables) должна работать на любой системе с systemd — https://github.com/UnitedTraders/ansible-postgresql-exporter

Архитектура решения

Я не буду подробно рассказывать про сам Prometheus, материалов про него достаточно, но пройдусь по общей архитектуре и экспортеру.

Prometheus использует pull модель сбора метрик: у него есть список экспортеров и он опрашивает их по HTTP, собирая с них список метрик и кладя их к себе в хранилище.

Экспортер — это агент, который занимается сбором метрик непосредственно с сущности (сервера в целом, или конкретного приложения), которую надо мониторить. У Prometheus богатые возможности для инструментации, поэтому экспортеры есть для большинства популярных приложений, и написать свой в случае надобности не представляет особого труда.

postgres_exporter работает следующим образом: он подключается к PostgreSQL, выполняет запросы к служебным таблицам и выставляет результаты в специальном формате с помощью внутреннего HTTP-сервера для забора их Prometheus’ом. Важный момент: помимо большого набора дефолтных запросов, можно определить свои и собирать любые данные, которые можно получить с помощью SQL, включая какие-нибудь бизнес-метрики.

Таким образом, настройка postgres_exporter’а сводится к трем действиям:

• Установить postgres_exporter на сервер, который мы хотим мониторить;
• Написать запросы (если нужно) для мониторинга своих параметров;
• Показать серверу Prometheus, откуда забирать метрики.

Установка экспортера

Экспортер написан на Go, так что все банально:

• Качаем нужный бинарник из https://github.com/wrouesnel/postgres_exporter/releases
• Определяем переменную окружения в env-файле
• Создаем systemd юнит, типа такого:

• Создаем файл с кастомными запросами для своих метрик (см. ниже);
• Запускаем сервис.

Настройка своих метрик

По умолчанию, postgres_exporter не умеет собирать данные по запросам. Но в PostgreSQL есть очень полезное расширение , которое именно этим и занимается. Установка pg_stat_statements сводится к трем простым шагам:

• Установить contrib-модули для PostgreSQL;
• Добавить в postgresql.conf параметр ;
• Создать расширение в самом постгресе: .

Т.к

нам важно было собирать в первую очередь время выполнения запросов, то файл с метриками получился вот такой:. Тут есть один подводный камень: записи из pg_stat_statements необходимо агрегировать, т.к

в этой таблице может существовать несколько записей с одинаковым сочетанием query id, database id и query. Наличие таких записей приводит к падению postgres_exporter, т.к. он формирует названия метрик, исходя из этих данных, а они должны быть уникальными

Тут есть один подводный камень: записи из pg_stat_statements необходимо агрегировать, т.к. в этой таблице может существовать несколько записей с одинаковым сочетанием query id, database id и query. Наличие таких записей приводит к падению postgres_exporter, т.к. он формирует названия метрик, исходя из этих данных, а они должны быть уникальными.

Для упрощения, я не стал добавлять метрики по чтению/записи (shared_blks_written и т.д., добавляются они по аналогии). Также, повторюсь, подобные запросы можно делать к любой таблице, а не только к pg_stat_statements.

Примеры запросов в Prometheus

С вышеуказанным конфигом, экспортер будет генерировать значительное количество метрик — по 5 штук на каждый вид запроса. Агрегировать мы их будем на стороне Prometheus.

Примеры запросов (сразу с графановскими переменными для templating):

• — среднее время запроса за последнюю минуту по всему серверу
• — среднее время запроса за последнюю минуту по выбранной базе
• — количество запросов за минуту к базе
• — top-20 самых частых запросов к базе за последние 10 минут
• — top-20 самых продолжительных запросов к базе

В графане это выглядит примерно вот так (ссылка на json):

Configure rules for aggregating scraped data into new time series

Though not a problem in our example, queries that aggregate over thousands of
time series can get slow when computed ad-hoc. To make this more efficient,
Prometheus allows you to prerecord expressions into completely new persisted
time series via configured recording rules. Let’s say we are interested in
recording the per-second rate of cpu time () averaged
over all cpus per instance (but preserving the , and
dimensions) as measured over a window of 5 minutes. We could write this as:

Try graphing this expression.

To record the time series resulting from this expression into a new metric
called , create a file
with the following recording rule and save it as :

To make Prometheus pick up this new rule, add a statement in your . The config should now
look like this:

Restart Prometheus with the new configuration and verify that a new time series
with the metric name
is now available by querying it through the expression browser or graphing it.

This documentation is . Please help improve it by filing issues or pull requests.

Учреждения среднего и среднего специального образования

Россия  

• Альметьевский политехнический техникум
• Вечерняя (сменная) общеобразовательная школа №90, г. Москва
• Екатеринбургский торгово-экономический техникум
• Йошкар-Олинский медицинский колледж
• Каменск-Уральский политехнический колледж (алюминиевый техникум)
• Колледж автоматизации и информационных технологий № 20 г. Москвы
• Курганский государственный колледж
• Лицей №130
• Лицей №1535, г. Москва
• Медицинское училище №22 Департамента здравоохранения г. Москвы
• Межрегиональный правовой колледж Сибирского федерального университета
• Московский издательско-полиграфический колледж им. Ивана Федорова
• Московский колледж железнодорожного транспорта МПС РФ
• Нижневартовский государственный социально-гуманитарный колледж
• Омский механико-технологический техникум
• Рыбинский авиационный колледж (Ярославская обл., г. Рыбинск)
• Самарский государственный профессионально-педагогический колледж
• Санкт-Петербургский колледж информатизации и управления
• Средняя школа № 1254 с углубленным изучением информатики, г. Москва
• Якутский колледж телекоммуникаций и информационных технологий
• Ярославский техникум железнодорожного транспорта МПС РФ

irate()

calculates the per-second instant rate of increase of
the time series in the range vector. This is based on the last two data points.
Breaks in monotonicity (such as counter resets due to target restarts) are
automatically adjusted for.

The following example expression returns the per-second rate of HTTP requests
looking up to 5 minutes back for the two most recent data points, per time
series in the range vector:

should only be used when graphing volatile, fast-moving counters.
Use for alerts and slow-moving counters, as brief changes
in the rate can reset the clause and graphs consisting entirely of rare
spikes are hard to read.

Note that when combining with an
(e.g. )
or a function aggregating over time (any function ending in ),
always take a first, then aggregate. Otherwise cannot detect
counter resets when your target restarts.

Third-party exporters

Some of these exporters are maintained as part of the official Prometheus GitHub organization,
those are marked as official, others are externally contributed and maintained.

We encourage the creation of more exporters but cannot vet all of them for
best practices.
Commonly, those exporters are hosted outside of the Prometheus GitHub
organization.

The exporter default
port
wiki page has become another catalog of exporters, and may include exporters
not listed here due to overlapping functionality or still being in development.

The JMX exporter can export from a
wide variety of JVM-based applications, for example Kafka and
Cassandra.

Databases

• Aerospike exporter
• ClickHouse exporter
• Consul exporter (official)
• Couchbase exporter
• CouchDB exporter
• Druid Exporter
• ElasticSearch exporter
• EventStore exporter
• KDB+ exporter
• Memcached exporter (official)
• MongoDB exporter
• MSSQL server exporter
• MySQL router exporter
• MySQL server exporter (official)
• OpenTSDB Exporter
• Oracle DB Exporter
• PgBouncer exporter
• PostgreSQL exporter
• Presto exporter
• ProxySQL exporter
• RavenDB exporter
• Redis exporter
• RethinkDB exporter
• SQL exporter
• Tarantool metric library
• Twemproxy

Hardware related

• apcupsd exporter
• BIG-IP exporter
• Collins exporter
• Dell Hardware OMSA exporter
• Fortigate exporter
• IBM Z HMC exporter
• IoT Edison exporter
• IPMI exporter
• knxd exporter
• Modbus exporter
• Netgear Cable Modem Exporter
• Netgear Router exporter
• Node/system metrics exporter (official)
• NVIDIA GPU exporter
• ProSAFE exporter
• Ubiquiti UniFi exporter
• Waveplus Radon Sensor Exporter
• Windows exporter

Messaging systems

• Beanstalkd exporter
• EMQ exporter
• Gearman exporter
• IBM MQ exporter
• Kafka exporter
• NATS exporter
• NSQ exporter
• Mirth Connect exporter
• MQTT blackbox exporter
• MQTT2Prometheus
• RabbitMQ exporter
• RabbitMQ Management Plugin exporter
• RocketMQ exporter
• Solace exporter

Storage

• Ceph exporter
• Ceph RADOSGW exporter
• Gluster exporter
• Hadoop HDFS FSImage exporter
• Lustre exporter
• ScaleIO exporter

HTTP

• Apache exporter
• HAProxy exporter (official)
• Nginx metric library
• Nginx VTS exporter
• Passenger exporter
• Squid exporter
• Tinyproxy exporter
• Varnish exporter
• WebDriver exporter

APIs

• AWS ECS exporter
• AWS Health exporter
• AWS SQS exporter
• Azure Health exporter
• BigBlueButton
• Cloudflare exporter
• Cryptowat exporter
• DigitalOcean exporter
• Docker Cloud exporter
• Docker Hub exporter
• GitHub exporter
• Gmail exporter
• InstaClustr exporter
• Mozilla Observatory exporter
• OpenWeatherMap exporter
• Pagespeed exporter
• Rancher exporter
• Speedtest exporter
• Tankerkönig API Exporter

Other monitoring systems

• Akamai Cloudmonitor exporter
• Alibaba Cloudmonitor exporter
• AWS CloudWatch exporter (official)
• Azure Monitor exporter
• Cloud Foundry Firehose exporter
• Collectd exporter (official)
• Google Stackdriver exporter
• Graphite exporter (official)
• Heka dashboard exporter
• Heka exporter
• Huawei Cloudeye exporter
• InfluxDB exporter (official)
• ITM exporter
• JavaMelody exporter
• JMX exporter (official)
• Munin exporter
• Nagios / Naemon exporter
• New Relic exporter
• NRPE exporter
• Osquery exporter
• OTC CloudEye exporter
• Pingdom exporter
• scollector exporter
• Sensu exporter
• site24x7_exporter
• SNMP exporter (official)
• StatsD exporter (official)
• TencentCloud monitor exporter
• ThousandEyes exporter

Miscellaneous

• ACT Fibernet Exporter
• BIND exporter
• BIND query exporter
• Bitcoind exporter
• Blackbox exporter (official)
• BOSH exporter
• cAdvisor
• Cachet exporter
• ccache exporter
• c-lightning exporter
• DHCPD leases exporter
• Dovecot exporter
• Dnsmasq exporter
• eBPF exporter
• Ethereum Client exporter
• JFrog Artifactory Exporter
• Hostapd Exporter
• IRCd exporter
• Linux HA ClusterLabs exporter
• JMeter plugin
• JSON exporter
• Kannel exporter
• Kemp LoadBalancer exporter
• Kibana Exporter
• kube-state-metrics
• Locust Exporter
• Meteor JS web framework exporter
• Minecraft exporter module
• Nomad exporter
• nftables exporter
• OpenStack exporter
• OpenStack blackbox exporter
• oVirt exporter
• Pact Broker exporter
• PHP-FPM exporter
• PowerDNS exporter
• Process exporter
• rTorrent exporter
• Rundeck exporter
• SABnzbd exporter
• Script exporter
• Shield exporter
• Smokeping prober
• SMTP/Maildir MDA blackbox prober
• SoftEther exporter
• Teamspeak3 exporter
• Transmission exporter
• Unbound exporter
• WireGuard exporter
• Xen exporter

When implementing a new Prometheus exporter, please follow the
guidelines on writing exporters
Please also consider consulting the . We are
happy to give advice on how to make your exporter as useful and consistent as
possible.

Expression language data types

In Prometheus’s expression language, an expression or sub-expression can
evaluate to one of four types:

• Instant vector — a set of time series containing a single sample for each time series, all sharing the same timestamp
• Range vector — a set of time series containing a range of data points over time for each time series
• Scalar — a simple numeric floating point value
• String — a simple string value; currently unused

Depending on the use-case (e.g. when graphing vs. displaying the output of an
expression), only some of these types are legal as the result from a
user-specified expression. For example, an expression that returns an instant
vector is the only type that can be directly graphed.

Overview

The Prometheus operator includes, but is not limited to, the following features:

• Kubernetes Custom Resources: Use Kubernetes custom resources to deploy and manage Prometheus, Alertmanager,
  and related components.

• Simplified Deployment Configuration: Configure the fundamentals of Prometheus like versions, persistence,
  retention policies, and replicas from a native Kubernetes resource.

• Prometheus Target Configuration: Automatically generate monitoring target configurations based
  on familiar Kubernetes label queries; no need to learn a Prometheus specific configuration language.

Implementation

Why are all sample values 64-bit floats? I want integers.

We restrained ourselves to 64-bit floats to simplify the design. The
IEEE 754 double-precision binary floating-point
format
supports integer precision for values up to 253. Supporting
native 64 bit integers would (only) help if you need integer precision
above 253 but below 263. In principle, support
for different sample value types (including some kind of big integer,
supporting even more than 64 bit) could be implemented, but it is not
a priority right now. A counter, even if incremented one million times per
second, will only run into precision issues after over 285 years.

Why don’t the Prometheus server components support TLS or authentication? Can I add those?

Note: The Prometheus team has changed their stance on this during its development summit on
August 11, 2018, and support for TLS and authentication in serving endpoints is now on the
.
This document will be updated once code changes have been made.

While TLS and authentication are frequently requested features, we have
intentionally not implemented them in any of Prometheus’s server-side
components. There are so many different options and parameters for both (10+
options for TLS alone) that we have decided to focus on building the best
monitoring system possible rather than supporting fully generic TLS and
authentication solutions in every server component.

If you need TLS or authentication, we recommend putting a reverse proxy in
front of Prometheus. See, for example Adding Basic Auth to Prometheus with
Nginx.

This applies only to inbound connections. Prometheus does support
, and other
Prometheus components that create outbound connections have similar support.

This documentation is . Please help improve it by filing issues or pull requests.

rate()

calculates the per-second average rate of increase of the
time series in the range vector. Breaks in monotonicity (such as counter
resets due to target restarts) are automatically adjusted for. Also, the
calculation extrapolates to the ends of the time range, allowing for missed
scrapes or imperfect alignment of scrape cycles with the range’s time period.

The following example expression returns the per-second rate of HTTP requests as measured
over the last 5 minutes, per time series in the range vector:

should only be used with counters. It is best suited for alerting,
and for graphing of slow-moving counters.

Note that when combining with an aggregation operator (e.g. )
or a function aggregating over time (any function ending in ),
always take a first, then aggregate. Otherwise cannot detect
counter resets when your target restarts.

When does it fit?

Prometheus works well for recording any purely numeric time series. It fits
both machine-centric monitoring as well as monitoring of highly dynamic
service-oriented architectures. In a world of microservices, its support for
multi-dimensional data collection and querying is a particular strength.

Prometheus is designed for reliability, to be the system you go to
during an outage to allow you to quickly diagnose problems. Each Prometheus
server is standalone, not depending on network storage or other remote services.
You can rely on it when other parts of your infrastructure are broken, and
you do not need to setup extensive infrastructure to use it.

What is Prometheus?

Prometheus is an open-source systems
monitoring and alerting toolkit originally built at
SoundCloud. Since its inception in 2012, many
companies and organizations have adopted Prometheus, and the project has a very
active developer and user community. It is now a standalone open source project
and maintained independently of any company. To emphasize this, and to clarify
the project’s governance structure, Prometheus joined the
Cloud Native Computing Foundation in 2016
as the second hosted project, after Kubernetes.

For more elaborate overviews of Prometheus, see the resources linked from the
media section.

Features

Prometheus’s main features are:

• a multi-dimensional data model with time series data identified by metric name and key/value pairs
• PromQL, a flexible query language
  to leverage this dimensionality
• no reliance on distributed storage; single server nodes are autonomous
• time series collection happens via a pull model over HTTP
• pushing time series is supported via an intermediary gateway
• targets are discovered via service discovery or static configuration
• multiple modes of graphing and dashboarding support

Components

The Prometheus ecosystem consists of multiple components, many of which are
optional:

• the main Prometheus server which scrapes and stores time series data
• client libraries for instrumenting application code
• a push gateway for supporting short-lived jobs
• special-purpose exporters for services like HAProxy, StatsD, Graphite, etc.
• an alertmanager to handle alerts
• various support tools

Most Prometheus components are written in Go, making
them easy to build and deploy as static binaries.

Architecture

This diagram illustrates the architecture of Prometheus and some of
its ecosystem components:

Prometheus scrapes metrics from instrumented jobs, either directly or via an
intermediary push gateway for short-lived jobs. It stores all scraped samples
locally and runs rules over this data to either aggregate and record new time
series from existing data or generate alerts. Grafana or
other API consumers can be used to visualize the collected data.

Removal

To remove the operator and Prometheus, first delete any custom resources you created in each namespace. The
operator will automatically shut down and remove Prometheus and Alertmanager pods, and associated ConfigMaps.

for n in $(kubectl get namespaces -o jsonpath={..metadata.name}); do
  kubectl delete --all --namespace=$n prometheus,servicemonitor,podmonitor,alertmanager
done

After a couple of minutes you can go ahead and remove the operator itself.

kubectl delete -f bundle.yaml

The operator automatically creates services in each namespace where you created a Prometheus or Alertmanager resources,
and defines three custom resource definitions. You can clean these up now.

for n in $(kubectl get namespaces -o jsonpath={..metadata.name}); do
  kubectl delete --ignore-not-found --namespace=$n service prometheus-operated alertmanager-operated
done

kubectl delete --ignore-not-found customresourcedefinitions \
  prometheuses.monitoring.coreos.com \
  servicemonitors.monitoring.coreos.com \
  podmonitors.monitoring.coreos.com \
  alertmanagers.monitoring.coreos.com \
  prometheusrules.monitoring.coreos.com

Установка

1. Ubuntu 16.04 x64 (VM)

2. Ubuntu 14.04 x64 (VM)

3. Mac OS

Сложный путь

Установка производится на Ubuntu 14.04 x64. Процесс установки расписан ниже, а также есть скринкаст, который поможет разобраться, если что-то пошло не так.

Установка Prometheus на Ubuntu 14.04 x64:

Для начала скачаем архив для установка Prometheus:

$ cd ~/Downloads
$ wget "https://github.com/prometheus/prometheus/releases/download/0.15.1/prometheus-0.15.1.linux-amd64.tar.gz"

Далее выполним саму установку. Для этого разархивируем тарбол с Prometheus:

$ mkdir -p ~/Prometheus/server
$ cd ~/Prometheus/server
$ tar -xvzf ~/Downloads/prometheus-0.15.1.linux-amd64.tar.gz
$ ./prometheus -version
prometheus, version 0.15.1 (branch: master, revision: 64349aa)
  build user:       julius@julius-thinkpad
  build date:       20150727-17:56:00
  go version:       1.4.2

Поставим экспортер node_exporter, о котором говорилось ранее:

$ mkdir -p ~/Prometheus/node_exporter
$ cd ~/Prometheus/node_exporter
$ wget https://github.com/prometheus/node_exporter/releases/download/0.11.0/node_exporter-0.11.0.linux-amd64.tar.gz -O ~/Downloads/node_exporter-0.11.0.linux-amd64.tar.gz
$ tar -xvzf ~/Downloads/node_exporter-0.11.0.linux-amd64.tar.gz

Чтобы удобнее было запускать экспортер –– создадим симлинк на бинарник node_exporter в /usr/bin:

$ sudo ln -s ~/Prometheus/node_exporter/node_exporter /usr/bin

Пропишем конфигурационный файл экспортера. Нам необходимо отредактировать следующий файл: /etc/init/node_exporter.conf, и вставить туда следующий код :

start on startup

script
   /usr/bin/node_exporter
end script

После проделанного выше экспортер уже доступен для работы. Запустим его.

$ sudo service node_exporter start

$ curl http://localhost:9100/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.00023853100000000002
go_gc_duration_seconds{quantile="0.25"} 0.00023998700000000002
go_gc_duration_seconds{quantile="0.5"} 0.00028122

Логи выше говорят о том, что все прошло успешно. Теперь перейдем к запуску сервера Prometheus. Нам будет нужно отредактировать файл ~/Prometheus/server/prometheus.yml и вставить туда следующий код:

scrape_configs:
  - job_name: "node"
    scrape_interval: "15s"
    target_groups:
    - targets: 'localhost:9100'

$ cd ~/Prometheus/server
$ vim ~/Prometheus/server/prometheus.yml
$ nohup ./prometheus > prometheus.log 2>&1 &
$ tail ~/Prometheus/server/prometheus.log
INFO Starting target manager...         file=targetmanager.go line=75
INFO Listening on :9090                 file=web.go line=118

Простой путь

Попробуем несколько вариантов.

Первый способ

Привяжем локальный конфиг Prometheus к образу и запустим:

$ docker run -p 9090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Второй способ

Можно собрать образ, какой нужен именно вам. Для этого нужно сделать свой Dockerfile и использовать Prometheus.yml, содержимое которых указано ниже.

# Dockerfile
FROM prom/prometheus
ADD prometheus.yml /etc/prometheus/

# TODO: some steps (optional)

# Prometheus.yml
global:
  scrape_interval:     15s
  evaluation_interval: 15s
  external_labels:
      monitor: 'docker-host-alpha'

rule_files:
  - "alert.rules"

scrape_configs:
  - job_name: 'nodeexporter'
    scrape_interval: 5s
    static_configs:
      - targets: 'nodeexporter:9100'

  - job_name: 'cadvisor'
    scrape_interval: 5s
    static_configs:
      - targets: 'cadvisor:8080'

  - job_name: 'prometheus'
    scrape_interval: 10s
    static_configs:
      - targets: 'localhost:9090'

  - job_name: 'pushgateway'
    scrape_interval: 10s
    honor_labels: true
static_configs:
      - targets: 'pushgateway:9091'

alerting:
  alertmanagers:
  - scheme: http
    static_configs:
    - targets: 
      - 'alertmanager:9093'

Теперь перейдем к самому процессу. Для начала создадим папку, где будут лежать конфигурационные файлы.

$ mkdir -p /path/to/Prometheus/
$ cd /path/to/Prometheus/

Далее создаем два файла и вставляем в них содержимое, которое представлено выше.

$ vim /path/to/Prometheus/Dockerfile
$ vim /path/to/Prometheus/prometheus.yml

Теперь производим сборку образа и запуск контейнера:

$ docker build -t your-name . 
$ docker run -p 9090:9090 your-name

Проверка работоспособности

Рисунок 3 –– Графический интерфейс Prometheus

Чтобы проверить работоспособность сервера, можно воспользоваться двумя вариантами. На рисунке 3 представлен графический интерфейс GUI.

2. Проверить утилитой curl, если вы это делаете локально на удаленной машине, а порт наружу не пробросили.

Prometheus Operator: что он делает?

1. — определяет инсталляцию (кластер) Prometheus;
2. — определяет, как мониторить набор сервисов (т.е. собирать их метрики);
3. — определяет кластер Alertmanager’ов (мы ими не пользуемся, поскольку отправляем метрики напрямую в свою систему уведомлений, которая принимает, агрегирует и ранжирует данные из множества источников — в том числе, интегрируется со Slack и Telegram).

1. StatefulSet (с самим Prometheus);
2. Secret с (конфиг Prometheus) и (конфиг для ).

ConfigMaps

Что в поде с Prometheus?

1. — сам Prometheus;
2. — обвязка, которая следит за изменениями и при необходимости вызывает reload конфигурации Prometheus (специальным HTTP-запросом — см. подробнее ниже), а также следит за ConfigMaps с правилами (они указаны в — см. подробнее ниже) и по необходимости скачивает их и перезапускает Prometheus.

(volumes)

1. — примонтированный секрет (два файла: и ). Подключён в оба контейнера;
2. — , который наполняет , а читает . Подключён в оба контейнера, но в — в режиме только для чтения;
3. — данные Prometheus. Подмонтирован только в .

Корпорации, крупные компании

Россия  

• KPMG International – в лице российского подразделения ЗАО «КПМГ»
• ОТП Банк (Россия), ОАО
• OTP Bank, Омский филиал «ОПСБ», ОАО
• Stins Coman Group, группа компаний (в лице «Академии Информационных Систем»)
• Softline (в лице НО АНО «Софтлайн эдьюкейшн»)
• Аптечная сеть Живика, ООО
• АВТОМИР, Группа компаний
• Академкнига/Учебник
• «Аргументы и факты», ЗАО
• ЕЭС России, РАО (в лице Корпоративного образовательного и научного центра ЕЭС)
• Ингосстрах, ОАО
• ИнтерЭлектроКомплект (международная электротехническая компания)
• Иркутская нефтяная компания, ОАО
• Иркутскэнерго, ОАО (в лице Учебного центра «Иркутскэнерго»)
• Иркутскэнерго, ОАО (в лице Корпоративного
  Учебно-Исследовательского Центра ИР ГТУ)
• КБ ОТКРЫТИЕ, ЗАО
• КОМУС, ТЦ
• КРЕДО-ДИАЛОГ, ООО, г. Москва
• Кросс-индустриальные производственные системы, ООО
• ЛУКОЙЛ
• Межрегионэнергосбыт, ОАО
• НОРНИКЕЛЬ ГРК Быстринское, АО
• Пермская ГРЭС, ОАО (в лице Центра подготовки и тренажа)
• Полиметалл УК, ОАО
• Полюс Строй, ООО
• «Пять-пятьдесят пять», группа компаний
• РН Банк, АО
• «Российский Алюминий, Объединенная компания»
• Самарский металлургический завод, ОАО
• Систематика, ООО
• Уралсвязьинформ, ОАО
• Финансовое Агентство по Сбору Платежей (ФАСП), г.Москва
• «ЦентрТелеком» (Верхневолжский филиал), ОАО
• Цифровой Центр ИОН
• Магнитогорский металлургический комбинат, Центр подготовки кадров «Персонал»
• ЦВ «ПРОТЕК»
• Экомир, группа компаний
• Южно-уральская Горноперерабатывающая Компания, ООО

Украина  

• «АСКА», Украинская акционерная страховая компания
• «КВИЗА-ТРЭЙД»
• «Киевстар Дж.Эс.Эм», ЗАО
• «Оранта», Национальная Акционерная Страховая Компания
• «ПриватБанк», ЗАО
• «УКРАВТОЗАПЧАСТЬ», Производственно-торговый холдинг
• «СЕБ Банк», ОАО

increase()

calculates the increase in the
time series in the range vector. Breaks in monotonicity (such as counter
resets due to target restarts) are automatically adjusted for. The
increase is extrapolated to cover the full time range as specified
in the range vector selector, so that it is possible to get a
non-integer result even if a counter increases only by integer
increments.

The following example expression returns the number of HTTP requests as measured
over the last 5 minutes, per time series in the range vector:

should only be used with counters. It is syntactic sugar
for multiplied by the number of seconds under the specified
time range window, and should be used primarily for human readability.
Use in recording rules so that increases are tracked consistently
on a per-second basis.

Configure Prometheus to monitor the sample targets

Now we will configure Prometheus to scrape these new targets. Let’s group all
three endpoints into one job called . However, imagine that the
first two endpoints are production targets, while the third one represents a
canary instance. To model this in Prometheus, we can add several groups of
endpoints to a single job, adding extra labels to each group of targets. In
this example, we will add the label to the first group of
targets, while adding to the second.

To achieve this, add the following job definition to the
section in your and restart your Prometheus instance:

Go to the expression browser and verify that Prometheus now has information
about time series that these example endpoints expose, such as .

Binary operators

Prometheus’s query language supports basic logical and arithmetic operators.
For operations between two instant vectors, the
can be modified.

Arithmetic binary operators

The following binary arithmetic operators exist in Prometheus:

• (addition)
• (subtraction)
• (multiplication)
• (division)
• (modulo)
• (power/exponentiation)

Binary arithmetic operators are defined between scalar/scalar, vector/scalar,
and vector/vector value pairs.

Between two scalars, the behavior is obvious: they evaluate to another
scalar that is the result of the operator applied to both scalar operands.

Between an instant vector and a scalar, the operator is applied to the
value of every data sample in the vector. E.g. if a time series instant vector
is multiplied by 2, the result is another vector in which every sample value of
the original vector is multiplied by 2.

Between two instant vectors, a binary arithmetic operator is applied to
each entry in the left-hand side vector and its
in the right-hand vector. The result is propagated into the result vector with the
grouping labels becoming the output label set. The metric name is dropped. Entries
for which no matching entry in the right-hand vector can be found are not part of
the result.

Comparison binary operators

The following binary comparison operators exist in Prometheus:

• (equal)
• (not-equal)
• (greater-than)
• (less-than)
• (greater-or-equal)
• (less-or-equal)

Comparison operators are defined between scalar/scalar, vector/scalar,
and vector/vector value pairs. By default they filter. Their behavior can be
modified by providing after the operator, which will return or
for the value rather than filtering.

Between two scalars, the modifier must be provided and these
operators result in another scalar that is either () or
(), depending on the comparison result.

Between an instant vector and a scalar, these operators are applied to the
value of every data sample in the vector, and vector elements between which the
comparison result is get dropped from the result vector. If the
modifier is provided, vector elements that would be dropped instead have the value
and vector elements that would be kept have the value .

Between two instant vectors, these operators behave as a filter by default,
applied to matching entries. Vector elements for which the expression is not
true or which do not find a match on the other side of the expression get
dropped from the result, while the others are propagated into a result vector
with the grouping labels becoming the output label set.
If the modifier is provided, vector elements that would have been
dropped instead have the value and vector elements that would be kept have
the value , with the grouping labels again becoming the output label set.

Logical/set binary operators

These logical/set binary operators are only defined between instant vectors:

• (intersection)
• (union)
• (complement)

results in a vector consisting of the elements of
for which there are elements in with exactly matching
label sets. Other elements are dropped. The metric name and values are carried
over from the left-hand side vector.

results in a vector that contains all original elements
(label sets + values) of and additionally all elements of
which do not have matching label sets in .

results in a vector consisting of the elements of
for which there are no elements in with exactly matching
label sets. All matching elements in both vectors are dropped.