У себя в компании я часто сталкиваюсь, что нужно поднять какой-то сервис, чтобы "общупать" его досконально. Хотя PCшники у нас довольно мощные, но большую часть ресурсов съедают PyCharm и Chrome, а на виртуалки с экспериментами очень часто остаётся совсем мало.
Поэтому мы завели у себя небольшую стойку с парой-тройкой серверов для экспериментов и локального Gitlab'а. Но что-то пошло не так и очень захотелось поиграться с чем-то новым.
Disclaimer
Я далеко не журналист, поэтому прошу заранее простить меня за возможные ошибки в тексте. Хотя я и пробежался по тексту, но всё равно мог что-то упустить. Всегда открыт к исправлениям в личном сообщении.
Помимо всего прочего "о вкусах не спорят" и я ничего не рекламирую, а просто хочу поделиться тем, как мы решили нашу задачу. Ну и конечно же всё это можно было сделать иначе, но мне захотелось именно так.
Лирика
В своё время, так как у наших заказчиков часто стоит OpenStack, мы его так же успешно развернули, чтобы совместить боль и слёзы тестирование одних продуктов с другими.
Но проблема в том, что офис != ДЦ и нам повезло снимать офис в месте, где стабильно каждые 2-3 месяца вырубают свет, притом делают это ночью, пока никто не видит. А вы пробовали поднимать кластер OpenStack после сбоя по питанию? Да и в целом продукт явно заточен не на наши нужды (хоть и очень удобно поднимать по 5-10 нод или разворачивать ansible'м целую инфраструктуру).
Ещё очень огорчало поедание ресурсов просто работающего кластера (фактически 2-3 сервера по 4CPU/16GbRAM чисто под работу пустого кластера) и необходимость держать контроллер отдельно от compute-ноды, чтобы не мешать работе (AIO отъедает в простое 35Гб ОЗУ — это как-то многовато для сервера в офисе).
Конечно можно было бы всё вынести в AWS/DO/Azure/GCE, но если есть 3-4 сервера, то почему бы их не использовать, ведь так или иначе уже имеющееся оборудование выйдет дешевле "облака".
Оборудование
Собственно об железе:
- Шкаф напольный KRAULER KRC6832
- 4U HITACHI HA8000/RS440(AG) 4x Intel Xeon X7460(6C/6T)/160GB/4x Memory Raiser/4x 2.5" HDD 1TB SATA + 4x 2.5" HDD 0.5TB SATA/LSI RAID 0-1-5-6/6xGLAN/2xPSU 1570W (не знаю, зачем там такие БП, если больше 500Вт он никогда не отъедал под полной нагрузкой)
- ИБП APC Smart-UPS 3000VA RM 2U
- Mikrotik Cloud Router Switch 125-24G-1S-RM (это не ядро офисной инфраструктуры — чисто под стойку)
- SSD WD Blue 500Gb SATA
Помимо всего прочего, есть:
- Сервер 1U Fujitsu RX100S6 Intel Core i3-540/16 GB DDR-3 ECC/2x 3.5" HDD 147Gb SAS/LSI Raid 0-1/2xGlan/PSU 250W (на котором крутится Gitlab CE)
- Сервер 1U Nec Express 5800/R110c c Intel Xeon X3430 на борту и характеристиками как в п.1 (на котором крутится Gitlab CI runner).
- Пара ИБП на 1200VA.
Последние 2 сервера были успешно оккупированы после того, как очередной внезапный ночной сбой по питанию привёл Ceph в OpenStack'е в негодность и было принято вынести их на независимые ноды. Впрочем, ничего не мешает их потом перенести в VM.
HITACHI вообще 2 шт, но мы всё вгрузили в одну машину и позже я поясню почему.
Я так подробно расписываю, просто, чтобы было понятно принятое решение.
Муки выбора
Основная проблема с OpenStack была в том, что AIO был слишком прожорливым и конкурировал с виртуалками, а размазывание по хостам… Короче говоря, ванильный OpenStack как бы нам ни нравился, но не подходил под наши нужды и возможности.
Логично было бы предположить, что может стоит перейти на VMWare ESXi (как многие товарищи и предлагали). Но не лежала душа у меня к этому продукту...
Нужно было то, что можно будет хорошо закастомизировать под наши нужды (где-то в пересечении дёшево, производительно и гибко), а так же, чтобы этим было удобно пользоваться простым разработчикам, которые не хотят ничего настраивать, а только делать "фыр-фыр-фыр" кодить и ставить эксперименты.
Выбор пал на Proxmox VE, потому что:
- Это debian, пусть и на "стероидах", а значит будет проще разобраться + кастомизировать.
- Есть как KVM, так и простые LXC контейнеры, которые гораздо экономичнее по ресурсам.
- Может в вполне себе бесплатной редакции расширяться в кластер.
- Дофигаллион различных хранилищ (lvm thin, Ceph, GlusterFS, NFS и т.д.)
- 100% поддержка того оборудования на которое будем ставить.
Настройка Proxmox
Публикация далее описывает то, как я настраивал инфраструктуру только сервера с Proxmox практически в живом режиме. Перенос серверов Gitlab'а никак не описывается, потому что к этой теме никак не относится.
Сеть и маршрутизатор
Есть некоторое legacy после OpenStack по нарезке сети, в частности публичная сеть и настройки OSPF на Mikrotik'ах. Хотелось следующего:
- Обеспечить изоляцию внутренней сети сервера от публичной сети, в которую будут ходить виртуалки.
- Использовать Mikrotik как основной маршрутизатор, DNS и ntp для стойки.
- Использовать все 6 интерфейсов сервера как для доступа, так и для выхода виртуалок "в свет".
Буду описывать как это сделать "с нуля":
- Загоняем все интерфейсы в bridge (считаем, что настройка происходит с 2го порта):
Разверни меня
/interface bridge
add name=bridge1 protocol-mode=none
/interface bridge port
add bridge=bridge1 interface=ether1 trusted=yes
add bridge=bridge1 interface=ether2
add bridge=bridge1 interface=ether3
add bridge=bridge1 interface=ether4
add bridge=bridge1 interface=ether5
add bridge=bridge1 interface=ether6
add bridge=bridge1 interface=ether7
add bridge=bridge1 interface=ether8
add bridge=bridge1 interface=ether9
add bridge=bridge1 interface=ether10
add bridge=bridge1 interface=ether11
add bridge=bridge1 interface=ether12
add bridge=bridge1 interface=ether13
add bridge=bridge1 interface=ether14
add bridge=bridge1 interface=ether15
add bridge=bridge1 interface=ether16
add bridge=bridge1 interface=ether17
add bridge=bridge1 interface=ether18
add bridge=bridge1 interface=ether19
add bridge=bridge1 interface=ether20
add bridge=bridge1 interface=ether21
add bridge=bridge1 interface=ether22
add bridge=bridge1 interface=ether23
add bridge=bridge1 interface=ether24
add bridge=bridge1 interface=sfp1- Настраиваем OSPF микротиков по сети 10.20.0.0/24 (правилами firewall'а потом запретим ходить во внутреннюю сеть):
Разверни меня
/ip address
add address=10.20.0.254/24 interface=bridge1 network=10.20.0.0
/ip route
add check-gateway=ping distance=1 gateway=10.20.0.1
/routing ospf network
add area=local network=10.20.0.0/24
/routing ospf instance
set [ find default=yes ] redistribute-connected=as-type-2 \
redistribute-static=as-type-2 router-id=10.20.0.254- Настраиваем VLAN100 для публичной сети + DHCP сервер:
Разверни меня
/interface vlan
add interface=bridge1 name=public_rack vlan-id=100
/ip address
add address=172.16.0.1/16 interface=public_rack network=172.16.0.0
/ip pool
add name=public_rack_dhcp_pool1 ranges=172.16.253.1-172.16.253.254
add name=public_rack_dhcp_pool0 next-pool=public_rack_dhcp_pool1 ranges=\
172.16.254.1-172.16.254.254
/ip dhcp-server
add add-arp=yes address-pool=public_rack_dhcp_pool0 disabled=no interface=\
public_rack lease-time=1h name=public_rack_dhcp
/ip dhcp-server network
add address=172.16.0.0/16 dns-server=172.16.0.1 domain=domain.lan\
gateway=172.16.0.1 ntp-server=172.16.0.1
/ip dns
set allow-remote-requests=yes cache-size=4096KiB max-concurrent-queries=400 \
max-concurrent-tcp-sessions=80 query-server-timeout=10s \
query-total-timeout=1m40s servers=10.20.0.1- Настраиваем правила Firewall так, чтобы доступ был только к следующему маршрутизатору:
Разверни меня
/ip firewall filter
add action=drop dst-address=10.20.0.0/24 in-interface=public_rack
add action=drop dst-address=10.20.0.0/24 src-address=172.16.0.0/24 in-interface=bridge1
add action=accept dst-address=10.20.0.1 in-interface=public_rackНельзя сказать, что это прям очень правильное правило, но если кто из знатоков подскажет как правильно его написать, то буду премного благодарен. Вообще нам изоляция нужна только на L2, но на L3 привожу больше для ознакомления.
Настройка сервера
Сервер будет настроен так, чтобы получить максимальную производительность от текущих дисков. Однако мы не забываем, что всё может случиться и ночью свет может резко на час-другой пропасть пока все спят.
- Устанавливаем в корзину все диски, кроме SSD.
- SSD подключаем через салазки вместо привода.
- Подключаем все порты LAN к Mikrotik.
- Настраиваем RAID с writethrough cache:
- 4x 1Тб в RAID 10
- 4x 500Гб в RAID 10
- Делаем оба массива загрузочными.
- Устанавливаем Proxmox VE 6 на первый массив и первый интерфейс (брать отсюда).
- Назначаем адрес 10.20.0.10/24
- По завершению установки перезагружаемся.
Так как мы ничего не хотим покупать, то следует отключить всё, что связано с подпиской:
- Удаляем popup при входе в GUI (действительно для версий >=5.1):
sed -i.bak "s/data.status !== 'Active'/false/g" /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js && systemctl restart pveproxy.service - Переключаемся на репозиторий для экономных людей заменой в файле строки:
`/etc/apt/sources.list.d/pve-enterprise.list`
# deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise
deb http://download.proxmox.com/debian/pve buster pve-no-subscriptionНа этом пока закончим основную настройку.
Магия сетевых интерфейсов
Вообще настройка Proxmox Bridge для работы достаточно простая. Но я не я, если не удалю гланды ректально сделаю что-нибудь экстравагантное. Мне очень хотелось, чтобы работал OVSBridge и на OVSBond. Но проблема в том, что мне так же не хотелось бы, чтобы пользователи каждый раз назначали VLAN при создании VM/LXC. Поэтому порядок настройки следующий для всех 6ти интерфейсов:
- Устанавливаем пакет Open vSwitch:
apt install openvswitch-switch -y - Редактируем файл
/etc/network/interfacesследующим образом:
Разверни меня
# Назначаем OVSBond именно на vlan-интерфейсы
allow-vmbr0 bond0
iface bond0 inet manual
ovs_bonds enp5s0f0.100 enp5s0f1.100 enp6s0f0.100 enp6s0f1.100 enp8s0f0.100 enp8s0f1.100
ovs_type OVSBond
ovs_bridge vmbr0
ovs_options vlan_mode=native-untagged bond_mode=balance-slb
auto lo
iface lo inet loopback
iface enp5s0f0 inet manual
iface enp5s0f1 inet manual
iface enp6s0f0 inet manual
iface enp6s0f1 inet manual
iface enp8s0f0 inet manual
iface enp8s0f1 inet manual
# Создаём для OVSBond vlan-интерфейсы
auto enp5s0f0.100
iface enp5s0f0.100 inet manual
auto enp5s0f1.100
iface enp5s0f1.100 inet manual
auto enp6s0f0.100
iface enp6s0f0.100 inet manual
auto enp6s0f1.100
iface enp6s0f1.100 inet manual
auto enp8s0f0.100
iface enp8s0f0.100 inet manual
auto enp8s0f1.100
iface enp8s0f1.100 inet manual
# Создаём классический bond для "внутренней кухни".
# Это не обязательно, но я бы отделил эту сеть для кластера и Ceph (в будущем).
auto bond1
iface bond1 inet static
address 10.20.0.10
netmask 24
bond-slaves enp5s0f0 enp5s0f1 enp6s0f0 enp6s0f1 enp8s0f0 enp8s0f1
bond-miimon 100
bond-mode balance-xor
bond-xmit-hash-policy layer2+3
# Мост в публичную сеть с VLAN100
allow-ovs vmbr0
iface vmbr0 inet static
address 172.16.0.2
netmask 16
gateway 172.16.0.1
ovs_type OVSBridge
ovs_ports bond0
Настройка LVM + lvmcache
Прежде всего удаляем старый pve/data, чтобы не запутаться с meta-разделом и т.д.
lvremove pve/dataДобавляем второй раздел к нашему volume group:
pvcreate /dev/sdb
vgextend pve /dev/sdbСоздаём новый раздел как нас учит документация Proxmox + небольшой хак для оптимизации snapshot'ов:
lvcreate -L 2.4T -n data pve
lvconvert --type thin-pool -c 64K --poolmetadatasize 15.81G pve/dataТеперь нам нужно подключить lvmcache, но в целях оптимизации ресурса SSD, я специально выделил раздел на нём размером 241Гб. Этого объёма вполне хватит для кеша, а оставшаяся часть диска будет ресурсом для убитых блоков. Правда это в теории, потому что лично мне до конца не понятно работает ли это. В любом случае, кеш в 465Гб потом будет вдвое дольше и сложнее удалять (например, чтобы заменить диск или расширить массив).
vgextend pve /dev/sdc1
lvcreate --type cache-pool -L240G pve /dev/sdc1
lvconvert --type cache --cachepool pve/lvol1 pve/dataUPD: есть один глюк с dm_cache, из-за которого нужно проделать ещё одну магию:
apt install thin-provisioning-tools -y
echo "dm_cache" >> /etc/initramfs-tools/modules
echo "dm_cache_mq" >> /etc/initramfs-tools/modules
echo "dm_persistent_data" >> /etc/initramfs-tools/modules
echo "dm_bufio" >> /etc/initramfs-tools/modules
update-initramfs -k `uname -r` -u -t
update-grubПоначалу всё работало, но потом почему-то перестало. Видимо с обновлением удалили эти модули.
Финальные штрихи
Обычно я перед перезагрузкой делаю, чтобы быть уверенным, что всё обновлено и ядро самое свежее:
apt update && apt dist-upgrade -y && apt autoremove -y
pveam updateК слову, в 6ой версии ядро 5.0, а на подходе (на момент публикации) вроде как 5.3.7.
Всё, теперь отправляем систему в ребут. Если мы всё правильно сделали, то после перезагрузки наш сервер будет доступен извне по обоим адресам. Так же можно будет загрузить свежие шаблоны контейнеров и радоваться жизни.
Делаем красивые графики
Ну мы же не просто админы, а DevOps'ы (что это вообще значит?), поэтому нам нужно знать, когда какая-то из виртуалок взбесится и будет творить непотребства.
Контейнер с InfluxDB и Grafana
Вообще все эти действия проще всего сделать в GUI, но на меня внезапно напала агрессивная форма лени, чтобы делать скриншоты, поэтому опишу командами.
Смотрим доступные нам шаблоны:
root@cloud:~# pveam available --section system
system alpine-3.10-default_20190626_amd64.tar.xz
system alpine-3.9-default_20190224_amd64.tar.xz
system archlinux-base_20190924-1_amd64.tar.gz
system centos-6-default_20191016_amd64.tar.xz
system centos-7-default_20190926_amd64.tar.xz
system centos-8-default_20191016_amd64.tar.xz
system debian-10.0-standard_10.0-1_amd64.tar.gz
system debian-8.0-standard_8.11-1_amd64.tar.gz
system debian-9.0-standard_9.7-1_amd64.tar.gz
system fedora-29-default_20181126_amd64.tar.xz
system fedora-30-default_20190718_amd64.tar.xz
system gentoo-current-default_20190718_amd64.tar.xz
system opensuse-15.0-default_20180907_amd64.tar.xz
system opensuse-15.1-default_20190719_amd64.tar.xz
system ubuntu-16.04-standard_16.04.5-1_amd64.tar.gz
system ubuntu-18.04-standard_18.04.1-1_amd64.tar.gz
system ubuntu-19.04-standard_19.04-1_amd64.tar.gz
system ubuntu-19.10-standard_19.10-1_amd64.tar.gzДумаю для начала можно загрузить ubuntu-18.04-standard_18.04.1-1_amd64.tar.gz:
root@cloud:~# pveam download local ubuntu-18.04-standard_18.04.1-1_amd64.tar.gz
root@cloud:~# pveam list local
NAME SIZE
local:vztmpl/ubuntu-18.04-standard_18.04.1-1_amd64.tar.gz 203.54MBСоздадим контейнер для InfluxDB и Grafana:
root@cloud:~# pct create 999 local:vztmpl/ubuntu-18.04-standard_18.04.1-1_amd64.tar.gz --cores 2 --memory 2048 --password hostpassword --onboot 1 --net0 name=eth0,bridge=vmbr0,gw=172.16.0.1,ip=172.16.1.101/24 --storage local-lvm --hostname monitПодключаемся к терминалу нашей виртуалки по ssh root@172.16.1.101 или через pct console 999.
Устанавливаем пакеты InfluxDB и Grafana:
apt update && apt install curl wget gnupg software-properties-common -y
add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
echo "deb https://repos.influxdata.com/ubuntu bionic stable" | tee /etc/apt/sources.list.d/influxdb.list
curl -sL https://repos.influxdata.com/influxdb.key | apt-key add -
wget -q -O - https://packages.grafana.com/gpg.key | apt-key add -
apt update && apt install influxdb grafana -y
systemctl daemon-reload
systemctl enable grafana-server
systemctl start grafana-server
systemctl enable influxdb
systemctl start influxdbНастраиваем InfluxDB для работы. Нам для этого нужно будет сперва создать базу:
influx
> CREATE DATABASE proxmox
> quitДалее, нам необходимо настроить UDP-порт, на который Proxmox будет слать запросы:
`/etc/influxdb/influxdb.conf`
# Находим эту секцию, а остальное оставляем как было.
[[udp]]
enabled = true
bind-address = "0.0.0.0:8089"
database = "proxmox"
batch-size = 1000
batch-timeout = "1s"… и теперь перезапустим influx: systemctl restart influxdb
Настройка Proxmox и Dashboard
Тут всё предельно просто. На сервере с proxmox'ом нужно создать файл следующего содержания:
`/etc/pve/status.cfg`
influxdb: influx1
server 172.16.1.101
port 8089
Файла там по умолчанию нет.
По какой-то мне неведомой причине метрики пошли не сразу, поэтому пришлось вызвать systemctl restart pvestatd на сервере проксмокса.
Теперь нам нужно всё это визуализировать. Заходим в нашу grafana по адресу http://172.16.1.101:3000/ (admin:admin), меняем пароль.
Нашёл в дебрях сайта графаны, что можно прикрутить красивый дашборд для визуализации Proxmox, но его пришлось немного подпилить напильником исправить, чтобы учесть все наши реалии.
- Заходим в Configuration->Data Sources->Add data source, выбираем InfluxDB.
- Ставим галочку
default. - Заполняем URL:
http://localhost:8086. - Заполняем Database:
proxmox. - Сохраняем.
Теперь нам нужно импортировать Dashboard:
- Заходим в Create->Import.
- Вставляем туда:
Содержимое борда
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"description": "Display data from the Proxmox host and hosted VM's and Containers.",
"editable": false,
"gnetId": 10048,
"graphTooltip": 0,
"id": 2,
"iteration": 1574232555436,
"links": [],
"panels": [
{
"collapsed": false,
"datasource": null,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"id": 43,
"panels": [],
"title": "Summary",
"type": "row"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "InfluxDB",
"format": "none",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 3,
"w": 2,
"x": 0,
"y": 1
},
"id": 58,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"groupBy": [],
"measurement": "cpustat",
"orderByTime": "ASC",
"policy": "default",
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"cpus"
],
"type": "field"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": "",
"timeFrom": null,
"timeShift": null,
"title": "Cores",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "InfluxDB",
"format": "bytes",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 3,
"w": 3,
"x": 2,
"y": 1
},
"id": 60,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"groupBy": [],
"measurement": "memory",
"orderByTime": "ASC",
"policy": "default",
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"memtotal"
],
"type": "field"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": "",
"timeFrom": null,
"timeShift": null,
"title": "Total Memory",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "InfluxDB",
"format": "bytes",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 3,
"w": 3,
"x": 5,
"y": 1
},
"id": 62,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"groupBy": [],
"measurement": "memory",
"orderByTime": "ASC",
"policy": "default",
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"swaptotal"
],
"type": "field"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": "",
"timeFrom": null,
"timeShift": null,
"title": "Total Swap",
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#73BF69",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "InfluxDB",
"decimals": 2,
"description": "",
"format": "percentunit",
"gauge": {
"maxValue": 1,
"minValue": 0,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 6,
"w": 4,
"x": 8,
"y": 1
},
"id": 2,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "cpustat",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT mean(\"cpu\") FROM \"cpustat\" WHERE (\"host\" =~ /^$server$/) AND $timeFilter GROUP BY time($__interval) fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"cpu"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": "0.5,0.75",
"timeFrom": null,
"timeShift": null,
"title": "Server CPU",
"transparent": true,
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#73BF69",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "InfluxDB",
"decimals": 2,
"description": "",
"format": "short",
"gauge": {
"maxValue": 10,
"minValue": 0,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 6,
"w": 4,
"x": 12,
"y": 1
},
"id": 9,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "cpustat",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT mean(\"avg1\") FROM \"cpustat\" WHERE (\"host\" =~ /^$server$/) AND $timeFilter GROUP BY time($__interval) fill(null)\n",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"avg1"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": "6,8",
"timeFrom": null,
"timeShift": null,
"title": "Load Average (1 Minute)",
"transparent": true,
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#73BF69",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "InfluxDB",
"decimals": 2,
"description": "",
"format": "percentunit",
"gauge": {
"maxValue": 1,
"minValue": 0,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 6,
"w": 4,
"x": 16,
"y": 1
},
"id": 10,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "cpustat",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT mean(\"wait\") FROM \"cpustat\" WHERE (\"host\" =~ /^$server$/) AND $timeFilter GROUP BY time($__interval) fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"wait"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": "0.5,0.75",
"timeFrom": null,
"timeShift": null,
"title": "I/O Wait",
"transparent": true,
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#73BF69",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "InfluxDB",
"decimals": 2,
"description": "",
"format": "gbytes",
"gauge": {
"maxValue": 160,
"minValue": 0,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 6,
"w": 4,
"x": 20,
"y": 1
},
"id": 11,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": true
},
"tableColumn": "",
"targets": [
{
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "memory",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT mean(\"memused\") / 1073741824 FROM \"memory\" WHERE (\"host\" =~ /^$server$/) AND $timeFilter GROUP BY time($__interval) fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"memused"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
"/ 1073741824"
],
"type": "math"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": "96,140",
"timeFrom": null,
"timeShift": null,
"title": "Used Memory",
"transparent": true,
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"bgColor": null,
"clockType": "24 hour",
"countdownSettings": {
"customFormat": null,
"endCountdownTime": "2019-04-12T19:20:00.000Z",
"endText": "00:00:00"
},
"datasource": "InfluxDB",
"dateSettings": {
"dateFormat": "YYYY-MM-DD",
"fontSize": "20px",
"fontWeight": "normal",
"showDate": true
},
"gridPos": {
"h": 3,
"w": 8,
"x": 0,
"y": 4
},
"id": 20,
"links": [],
"mode": "time",
"offsetFromUtc": null,
"offsetFromUtcMinutes": null,
"options": {},
"refreshSettings": {
"syncWithDashboard": false
},
"targets": [
{
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"orderByTime": "ASC",
"policy": "default",
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"value"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": []
}
],
"timeFrom": null,
"timeSettings": {
"customFormat": "HH:mm:ss",
"fontSize": "40px",
"fontWeight": "normal"
},
"timeShift": null,
"timezone": null,
"timezoneSettings": {
"fontSize": "12px",
"fontWeight": "normal",
"showTimezone": false,
"zoneFormat": "offsetAbbv"
},
"title": "Time",
"transparent": true,
"type": "grafana-clock-panel"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "InfluxDB",
"format": "dateTimeAsIso",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 6,
"w": 6,
"x": 0,
"y": 7
},
"id": 24,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "cpustat",
"orderByTime": "ASC",
"policy": "default",
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"cpu"
],
"type": "field"
},
{
"params": [],
"type": "last"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": "",
"timeFrom": null,
"timeShift": null,
"title": "Last Update",
"transparent": true,
"type": "singlestat",
"valueFontSize": "100%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "last_time"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "InfluxDB",
"decimals": 1,
"format": "dtdurations",
"gauge": {
"maxValue": 100,
"minValue": 0,
"show": false,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 6,
"w": 6,
"x": 6,
"y": 7
},
"id": 16,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"groupBy": [],
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"uptime"
],
"type": "field"
},
{
"params": [],
"type": "last"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": "",
"timeFrom": null,
"timeShift": null,
"title": "Host Uptime",
"transparent": true,
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"columns": [],
"datasource": "InfluxDB",
"fontSize": "100%",
"gridPos": {
"h": 6,
"w": 6,
"x": 12,
"y": 7
},
"id": 7,
"interval": "",
"links": [],
"options": {},
"pageSize": null,
"scroll": true,
"showHeader": true,
"sort": {
"col": 1,
"desc": false
},
"styles": [
{
"alias": "Time",
"colorMode": null,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"dateFormat": "YYYY-MM-DD HH:mm:ss",
"decimals": 2,
"mappingType": 1,
"pattern": "Time",
"thresholds": [],
"type": "hidden",
"unit": "short"
},
{
"alias": "Uptime",
"colorMode": null,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"dateFormat": "YYYY-MM-DD HH:mm:ss",
"decimals": 1,
"mappingType": 1,
"pattern": "uptime",
"rangeMaps": [
{
"from": "",
"text": "",
"to": ""
}
],
"thresholds": [],
"type": "number",
"unit": "dtdurations",
"valueMaps": [
{
"text": "Offline",
"value": "0"
}
]
},
{
"alias": "VM",
"colorMode": null,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"dateFormat": "YYYY-MM-DD HH:mm:ss",
"decimals": 2,
"mappingType": 1,
"pattern": "host",
"sanitize": true,
"thresholds": [],
"type": "string",
"unit": "short",
"valueMaps": []
}
],
"targets": [
{
"groupBy": [
{
"params": [
"host"
],
"type": "tag"
}
],
"limit": "1",
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT last(uptime) AS \"uptime\" FROM \"system\" WHERE (\"object\" = 'qemu') AND (\"nodename\" =~ /^$server$/) AND time > (now() - 10m) AND uptime > 0 GROUP BY \"host\" limit 1",
"rawQuery": true,
"refId": "A",
"resultFormat": "table",
"select": [
[
{
"params": [
"uptime"
],
"type": "field"
},
{
"params": [
"uptime"
],
"type": "alias"
}
]
],
"tags": [
{
"key": "object",
"operator": "=",
"value": "qemu"
}
]
}
],
"timeFrom": null,
"timeShift": null,
"title": "Running VMs",
"transform": "table",
"transparent": true,
"type": "table"
},
{
"columns": [],
"datasource": "InfluxDB",
"fontSize": "100%",
"gridPos": {
"h": 5,
"w": 6,
"x": 18,
"y": 7
},
"id": 8,
"interval": "",
"links": [],
"options": {},
"pageSize": null,
"scroll": true,
"showHeader": true,
"sort": {
"col": 1,
"desc": false
},
"styles": [
{
"alias": "Time",
"colorMode": null,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"dateFormat": "YYYY-MM-DD HH:mm:ss",
"decimals": 2,
"mappingType": 1,
"pattern": "Time",
"thresholds": [],
"type": "hidden",
"unit": "short"
},
{
"alias": "Uptime",
"colorMode": null,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"dateFormat": "YYYY-MM-DD HH:mm:ss",
"decimals": 1,
"mappingType": 1,
"pattern": "uptime",
"thresholds": [],
"type": "number",
"unit": "dthms"
},
{
"alias": "LXC",
"colorMode": null,
"colors": [
"rgba(245, 54, 54, 0.9)",
"rgba(237, 129, 40, 0.89)",
"rgba(50, 172, 45, 0.97)"
],
"dateFormat": "YYYY-MM-DD HH:mm:ss",
"decimals": 2,
"mappingType": 1,
"pattern": "host",
"sanitize": true,
"thresholds": [],
"type": "string",
"unit": "short",
"valueMaps": []
}
],
"targets": [
{
"groupBy": [
{
"params": [
"host"
],
"type": "tag"
}
],
"limit": "1",
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT last(uptime) AS \"uptime\" FROM \"system\" WHERE (\"object\" = 'lxc') AND (\"nodename\" =~ /^$server$/) AND time > (now() - 10m) AND uptime > 0 GROUP BY \"host\" limit 1",
"rawQuery": true,
"refId": "A",
"resultFormat": "table",
"select": [
[
{
"params": [
"uptime"
],
"type": "field"
},
{
"params": [
"uptime"
],
"type": "alias"
}
]
],
"tags": [
{
"key": "object",
"operator": "=",
"value": "qemu"
}
]
}
],
"timeFrom": null,
"timeShift": null,
"title": "Running LXCs",
"transform": "table",
"transparent": true,
"type": "table"
},
{
"collapsed": false,
"datasource": null,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 13
},
"id": 26,
"panels": [],
"title": "Memory And CPU Usage",
"type": "row"
},
{
"aliasColors": {
"I/O Wait": "dark-blue",
"Load Average (1 Min)": "dark-red"
},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 7,
"w": 12,
"x": 0,
"y": 14
},
"id": 22,
"interval": "",
"legend": {
"alignAsTable": false,
"avg": true,
"current": true,
"max": false,
"min": false,
"rightSide": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [
{
"alias": "Load Average (1 Min)",
"yaxis": 2
}
],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "CPU Usage",
"groupBy": [
{
"params": [
"1m"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "cpustat",
"orderByTime": "ASC",
"policy": "default",
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"cpu"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
"* 100"
],
"type": "math"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
}
]
},
{
"alias": "I/O Wait",
"groupBy": [
{
"params": [
"1m"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "cpustat",
"orderByTime": "ASC",
"policy": "default",
"refId": "B",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"wait"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
" * 100"
],
"type": "math"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
}
]
},
{
"alias": "Load Average (1 Min)",
"groupBy": [
{
"params": [
"1m"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "cpustat",
"orderByTime": "ASC",
"policy": "default",
"refId": "C",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"avg1"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Host CPU Usage",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": null,
"format": "percent",
"label": "Percent",
"logBase": 1,
"max": null,
"min": "0",
"show": true
},
{
"decimals": null,
"format": "short",
"label": "Load",
"logBase": 1,
"max": "10",
"min": "0",
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 7,
"w": 12,
"x": 12,
"y": 14
},
"id": 4,
"interval": "",
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"max": true,
"min": true,
"rightSide": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "Memory Use",
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "memory",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT mean(\"memused\")/mean(\"memtotal\") * 100 FROM \"memory\" WHERE (\"host\" =~ /^$server$/) AND $timeFilter GROUP BY time($__interval) fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"memused"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "Host Memory Usage",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": null,
"format": "percent",
"label": null,
"logBase": 1,
"max": "100",
"min": "0",
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"decimals": 2,
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 12,
"x": 0,
"y": 21
},
"id": 28,
"interval": "",
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": false,
"hideZero": true,
"max": false,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "$tag_host",
"groupBy": [
{
"params": [
"1m"
],
"type": "time"
},
{
"params": [
"host"
],
"type": "tag"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT last(\"cpu\") FROM \"system\" WHERE (\"object\" = 'qemu') AND $timeFilter GROUP BY time($__interval) fill(null)",
"rawQuery": false,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"cpu"
],
"type": "field"
},
{
"params": [],
"type": "last"
}
]
],
"tags": [
{
"key": "object",
"operator": "=",
"value": "qemu"
},
{
"condition": "AND",
"key": "nodename",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "VM CPU Usage",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": 2,
"format": "percentunit",
"label": "",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 12,
"x": 12,
"y": 21
},
"id": 31,
"interval": "",
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideZero": true,
"max": false,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "$tag_host",
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"host"
],
"type": "tag"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT (mean(\"mem\")/last(maxmem))*100 FROM \"system\" WHERE (\"object\" = 'qemu') AND (\"nodename\" =~ /^$server$/) AND $timeFilter GROUP BY time($__interval), \"host\" fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"mem"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": [
{
"key": "object",
"operator": "=",
"value": "qemu"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "VMs Memory Usage",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "percent",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"decimals": 2,
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 12,
"x": 0,
"y": 27
},
"id": 29,
"interval": "",
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": false,
"hideZero": true,
"max": false,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "$tag_host",
"groupBy": [
{
"params": [
"1m"
],
"type": "time"
},
{
"params": [
"host"
],
"type": "tag"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT last(\"cpu\") FROM \"system\" WHERE (\"object\" = 'lxc' AND \"nodename\" =~ /^$server$/) AND $timeFilter GROUP BY time(1m), \"host\" fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"cpu"
],
"type": "field"
},
{
"params": [],
"type": "last"
}
]
],
"tags": [
{
"key": "object",
"operator": "=",
"value": "lxc"
},
{
"condition": "AND",
"key": "nodename",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "LXC CPU Usage",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": 2,
"format": "percentunit",
"label": "",
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 12,
"x": 12,
"y": 27
},
"id": 32,
"interval": "",
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideZero": true,
"max": false,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "$tag_host",
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"host"
],
"type": "tag"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT (mean(\"mem\")/last(maxmem))*100 FROM \"system\" WHERE (\"object\" = 'lxc') AND (\"nodename\" =~ /^$server$/) AND $timeFilter GROUP BY time($__interval), \"host\" fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"mem"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": [
{
"key": "object",
"operator": "=",
"value": "qemu"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "LXC Memory Usage",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "percent",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"collapsed": false,
"datasource": null,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 33
},
"id": 34,
"panels": [],
"title": "Disk I/O",
"type": "row"
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 12,
"x": 0,
"y": 34
},
"id": 36,
"interval": "",
"legend": {
"alignAsTable": true,
"avg": false,
"current": true,
"hideZero": true,
"max": true,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "$tag_host",
"groupBy": [
{
"params": [
"1m"
],
"type": "time"
},
{
"params": [
"host"
],
"type": "tag"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT abs(derivative(mean(\"diskread\"), 1s)) FROM \"system\" WHERE (\"object\" = 'qemu' AND \"nodename\" =~ /^$server$/) AND $timeFilter GROUP BY time(1m), \"host\" fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"diskread"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
"1s"
],
"type": "derivative"
}
]
],
"tags": [
{
"key": "object",
"operator": "=",
"value": "qemu"
},
{
"condition": "AND",
"key": "nodename",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "VMs I/O Read",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "Bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 12,
"x": 12,
"y": 34
},
"id": 37,
"interval": "",
"legend": {
"alignAsTable": true,
"avg": false,
"current": true,
"hideZero": true,
"max": true,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "$tag_host",
"groupBy": [
{
"params": [
"1m"
],
"type": "time"
},
{
"params": [
"host"
],
"type": "tag"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"hide": false,
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT abs(derivative(mean(\"diskwrite\"), 1s)) FROM \"system\" WHERE (\"object\" = 'qemu' AND \"nodename\" =~ /^$server$/) AND $timeFilter GROUP BY time(1m), \"host\" fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"diskwrite"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
"1s"
],
"type": "derivative"
}
]
],
"tags": [
{
"key": "object",
"operator": "=",
"value": "qemu"
},
{
"condition": "AND",
"key": "nodename",
"operator": "=",
"value": "dianna"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "VMs I/O Write",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "Bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 12,
"x": 0,
"y": 40
},
"id": 38,
"interval": "",
"legend": {
"alignAsTable": true,
"avg": false,
"current": true,
"hideZero": true,
"max": true,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "$tag_host",
"groupBy": [
{
"params": [
"1m"
],
"type": "time"
},
{
"params": [
"host"
],
"type": "tag"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT abs(derivative(mean(\"diskread\"), 1s)) FROM \"system\" WHERE (\"object\" = 'lxc' AND \"nodename\" =~ /^$server$/) AND $timeFilter GROUP BY time(1m), \"host\" fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"diskread"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
"1s"
],
"type": "derivative"
}
]
],
"tags": [
{
"key": "object",
"operator": "=",
"value": "lxc"
},
{
"condition": "AND",
"key": "nodename",
"operator": "=~",
"value": "/^$server$/"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "LXCs I/O Read",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"decimals": null,
"format": "Bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": "0",
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": 0
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 12,
"x": 12,
"y": 40
},
"id": 39,
"interval": "",
"legend": {
"alignAsTable": true,
"avg": false,
"current": true,
"hideZero": true,
"max": true,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 2,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "$tag_host",
"groupBy": [
{
"params": [
"1m"
],
"type": "time"
},
{
"params": [
"host"
],
"type": "tag"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT abs(derivative(mean(\"diskwrite\"), 1s)) FROM \"system\" WHERE (\"object\" = 'lxc' AND \"nodename\" =~ /^$server$/) AND $timeFilter GROUP BY time(1m), \"host\" fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"diskwrite"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
"1s"
],
"type": "derivative"
}
]
],
"tags": [
{
"key": "object",
"operator": "=",
"value": "lxc"
},
{
"condition": "AND",
"key": "nodename",
"operator": "=",
"value": "dianna"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "LXCs I/O Write",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "Bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"collapsed": false,
"datasource": null,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 46
},
"id": 41,
"panels": [],
"title": "Network Traffic",
"type": "row"
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 12,
"x": 0,
"y": 47
},
"id": 45,
"interval": "",
"legend": {
"avg": true,
"current": true,
"max": false,
"min": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "Rx-$tag_instance",
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "nics",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT abs(derivative(mean(\"receive\"), 10s)) AS \"Rx\" FROM \"nics\" WHERE (\"host\" =~ /^$server$/ AND \"instance\" =~ /bond[01]/) AND $timeFilter GROUP BY time($__interval), \"instance\" fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"receive"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
"10s"
],
"type": "non_negative_derivative"
},
{
"params": [
"Rx"
],
"type": "alias"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
},
{
"condition": "AND",
"key": "instance",
"operator": "=",
"value": "enp2s0"
}
]
},
{
"alias": "Tx-$tag_instance",
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "nics",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT abs(derivative(mean(\"transmit\"), 10s)) FROM \"nics\" WHERE (\"host\" =~ /^$server$/ AND \"instance\" =~ /bond[01]/) AND $timeFilter GROUP BY time($__interval), \"instance\" fill(null)",
"rawQuery": true,
"refId": "B",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"transmit"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
"10s"
],
"type": "non_negative_derivative"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
},
{
"condition": "AND",
"key": "instance",
"operator": "=",
"value": "enp2s0"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "bonds rx-tx",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 12,
"x": 12,
"y": 47
},
"id": 46,
"interval": "",
"legend": {
"avg": true,
"current": true,
"max": false,
"min": false,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "Rx",
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "nics",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT abs(derivative(mean(\"receive\"), 10s)) FROM \"nics\" WHERE (\"host\" =~ /^$server$/ AND \"instance\" = 'vmbr0') AND $timeFilter GROUP BY time($__interval) fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"receive"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
"10s"
],
"type": "non_negative_derivative"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
},
{
"condition": "AND",
"key": "instance",
"operator": "=",
"value": "vmbr0"
}
]
},
{
"alias": "Tx",
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "nics",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT abs(derivative(mean(\"transmit\"), 10s)) FROM \"nics\" WHERE (\"host\" =~ /^$server$/ AND \"instance\" = 'vmbr0') AND $timeFilter GROUP BY time($__interval) fill(null)",
"rawQuery": true,
"refId": "B",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"transmit"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
"10s"
],
"type": "non_negative_derivative"
}
]
],
"tags": [
{
"key": "host",
"operator": "=~",
"value": "/^$server$/"
},
{
"condition": "AND",
"key": "instance",
"operator": "=",
"value": "vmbr0"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "vmbr0",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 12,
"x": 0,
"y": 53
},
"id": 47,
"interval": "",
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": false,
"hideZero": true,
"max": false,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "$tag_host",
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"host"
],
"type": "tag"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT abs(derivative(mean(\"netin\"), 10s)) FROM \"system\" WHERE (\"nodename\" =~ /^$server$/ AND \"object\" = 'qemu') AND $timeFilter GROUP BY time($__interval), \"host\" fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"netin"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
"10s"
],
"type": "non_negative_derivative"
}
]
],
"tags": [
{
"key": "nodename",
"operator": "=~",
"value": "/^$server$/"
},
{
"condition": "AND",
"key": "object",
"operator": "=",
"value": "qemu"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "VMs Traffic In",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 12,
"x": 12,
"y": 53
},
"id": 49,
"interval": "",
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": false,
"hideZero": true,
"max": false,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "$tag_host",
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"host"
],
"type": "tag"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT abs(derivative(mean(\"netout\"), 10s)) FROM \"system\" WHERE (\"nodename\" =~ /^$server$/ AND \"object\" = 'qemu') AND $timeFilter GROUP BY time($__interval), \"host\" fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"netout"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
"10s"
],
"type": "non_negative_derivative"
}
]
],
"tags": [
{
"key": "nodename",
"operator": "=~",
"value": "/^$server$/"
},
{
"condition": "AND",
"key": "object",
"operator": "=",
"value": "qemu"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "VMs Traffic Out",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 12,
"x": 0,
"y": 59
},
"id": 48,
"interval": "",
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": false,
"hideZero": true,
"max": false,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "$tag_host",
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"host"
],
"type": "tag"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT abs(derivative(mean(\"netin\"), 10s)) FROM \"system\" WHERE (\"nodename\" =~ /^$server$/ AND \"object\" = 'lxc') AND $timeFilter GROUP BY time($__interval), \"host\" fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"netin"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
"10s"
],
"type": "non_negative_derivative"
}
]
],
"tags": [
{
"key": "nodename",
"operator": "=~",
"value": "/^$server$/"
},
{
"condition": "AND",
"key": "object",
"operator": "=",
"value": "lxc"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "LCXs Traffic In",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"aliasColors": {},
"bars": false,
"dashLength": 10,
"dashes": false,
"datasource": "InfluxDB",
"fill": 1,
"fillGradient": 0,
"gridPos": {
"h": 6,
"w": 12,
"x": 12,
"y": 59
},
"id": 50,
"interval": "",
"legend": {
"alignAsTable": true,
"avg": true,
"current": true,
"hideEmpty": false,
"hideZero": true,
"max": false,
"min": false,
"rightSide": true,
"show": true,
"total": false,
"values": true
},
"lines": true,
"linewidth": 1,
"links": [],
"nullPointMode": "null",
"options": {
"dataLinks": []
},
"percentage": false,
"pointradius": 2,
"points": false,
"renderer": "flot",
"seriesOverrides": [],
"spaceLength": 10,
"stack": false,
"steppedLine": false,
"targets": [
{
"alias": "$tag_host",
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"host"
],
"type": "tag"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"measurement": "system",
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT abs(derivative(mean(\"netout\"), 10s)) FROM \"system\" WHERE (\"nodename\" =~ /^$server$/ AND \"object\" = 'lxc') AND $timeFilter GROUP BY time($__interval), \"host\" fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"netout"
],
"type": "field"
},
{
"params": [],
"type": "mean"
},
{
"params": [
"10s"
],
"type": "non_negative_derivative"
}
]
],
"tags": [
{
"key": "nodename",
"operator": "=~",
"value": "/^$server$/"
},
{
"condition": "AND",
"key": "object",
"operator": "=",
"value": "lxc"
}
]
}
],
"thresholds": [],
"timeFrom": null,
"timeRegions": [],
"timeShift": null,
"title": "LXCs Traffic Out",
"tooltip": {
"shared": true,
"sort": 0,
"value_type": "individual"
},
"type": "graph",
"xaxis": {
"buckets": null,
"mode": "time",
"name": null,
"show": true,
"values": []
},
"yaxes": [
{
"format": "bps",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": true
},
{
"format": "short",
"label": null,
"logBase": 1,
"max": null,
"min": null,
"show": false
}
],
"yaxis": {
"align": false,
"alignLevel": null
}
},
{
"collapsed": false,
"datasource": null,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 65
},
"id": 52,
"panels": [],
"title": "Storage Pools",
"type": "row"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "InfluxDB",
"format": "percentunit",
"gauge": {
"maxValue": 1,
"minValue": 0,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 7,
"w": 4,
"x": 6,
"y": 66
},
"id": 54,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"pluginVersion": "6.1.3",
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT last(\"used\") / last(\"total\") FROM \"system\" WHERE (\"host\" = 'local-lvm' AND \"nodename\" =~ /^$server$/) AND $timeFilter GROUP BY time($__interval) fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"value"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": []
}
],
"thresholds": "0.75,0.875",
"timeFrom": null,
"timeShift": null,
"title": "local-lvm Usage",
"transparent": true,
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "InfluxDB",
"format": "percentunit",
"gauge": {
"maxValue": 1,
"minValue": 0,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 7,
"w": 4,
"x": 10,
"y": 66
},
"id": 63,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"pluginVersion": "6.1.3",
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT last(\"used\") / last(\"total\") FROM \"system\" WHERE (\"host\" = 'local-lvm-test' AND \"nodename\" =~ /^$server$/) AND $timeFilter GROUP BY time($__interval) fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"value"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": []
}
],
"thresholds": "0.75,0.875",
"timeFrom": null,
"timeShift": null,
"title": "local-lvm-test Usage",
"transparent": true,
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
},
{
"cacheTimeout": null,
"colorBackground": false,
"colorValue": false,
"colors": [
"#299c46",
"rgba(237, 129, 40, 0.89)",
"#d44a3a"
],
"datasource": "InfluxDB",
"format": "percentunit",
"gauge": {
"maxValue": 1,
"minValue": 0,
"show": true,
"thresholdLabels": false,
"thresholdMarkers": true
},
"gridPos": {
"h": 7,
"w": 4,
"x": 15,
"y": 66
},
"id": 55,
"interval": null,
"links": [],
"mappingType": 1,
"mappingTypes": [
{
"name": "value to text",
"value": 1
},
{
"name": "range to text",
"value": 2
}
],
"maxDataPoints": 100,
"nullPointMode": "connected",
"nullText": null,
"options": {},
"pluginVersion": "6.1.3",
"postfix": "",
"postfixFontSize": "50%",
"prefix": "",
"prefixFontSize": "50%",
"rangeMaps": [
{
"from": "null",
"text": "N/A",
"to": "null"
}
],
"sparkline": {
"fillColor": "rgba(31, 118, 189, 0.18)",
"full": false,
"lineColor": "rgb(31, 120, 193)",
"show": false
},
"tableColumn": "",
"targets": [
{
"groupBy": [
{
"params": [
"$__interval"
],
"type": "time"
},
{
"params": [
"null"
],
"type": "fill"
}
],
"orderByTime": "ASC",
"policy": "default",
"query": "SELECT last(\"used\") / last(\"total\") FROM \"system\" WHERE (\"host\" = 'local' AND \"nodename\" =~ /^$server$/) AND $timeFilter GROUP BY time($__interval) fill(null)",
"rawQuery": true,
"refId": "A",
"resultFormat": "time_series",
"select": [
[
{
"params": [
"value"
],
"type": "field"
},
{
"params": [],
"type": "mean"
}
]
],
"tags": []
}
],
"thresholds": "0.75,0.875",
"timeFrom": null,
"timeShift": null,
"title": "local Usage",
"transparent": true,
"type": "singlestat",
"valueFontSize": "80%",
"valueMaps": [
{
"op": "=",
"text": "N/A",
"value": "null"
}
],
"valueName": "current"
}
],
"refresh": "10s",
"schemaVersion": 20,
"style": "dark",
"tags": [
"ProxMox",
"Storage",
"Hypervisor"
],
"templating": {
"list": [
{
"allValue": null,
"current": {
"text": "cloud",
"value": "cloud"
},
"datasource": "InfluxDB",
"definition": "SHOW TAG VALUES FROM system WITH KEY=host WHERE object='nodes'",
"hide": 0,
"includeAll": false,
"label": "Server",
"multi": true,
"name": "server",
"options": [],
"query": "SHOW TAG VALUES FROM system WITH KEY=host WHERE object='nodes'",
"refresh": 1,
"regex": "",
"skipUrlSync": false,
"sort": 5,
"tagValuesQuery": "",
"tags": [],
"tagsQuery": "",
"type": "query",
"useTags": false
}
]
},
"time": {
"from": "now-24h",
"to": "now"
},
"timepicker": {
"refresh_intervals": [
"5s",
"10s",
"30s",
"1m",
"5m",
"15m",
"30m",
"1h",
"2h",
"1d"
],
"time_options": [
"5m",
"15m",
"1h",
"6h",
"12h",
"24h",
"2d",
"7d",
"30d"
]
},
"timezone": "",
"title": "Proxmox",
"uid": "kxQQuHRZk",
"version": 29
}- Заполняем наше хранилище и делаем Import.
В общем-то всё. Для удобства, я дополнительно туда же подключил Dashboard'ы из Gitlab. Описывать этот процесс нет смысла, надо только иметь в виду, что Prometheus должен быть доступен с этого контейнера, его необходимо подключить как Datastore в Grafana и при экспорте дашбордов указывать, что они external, чтобы выбрать нужный datastore.
Несколько скриншотов
Со скриншотами из свежей Grafana косяк вышел, поэтому простите за качество.
Комментарии (5)
chemtech
13.11.2019 18:29+1Чем разворачиваете ВМ?
Смотрели в сторону github.com/Telmate/terraform-provider-proxmox?
onegreyonewhite Автор
14.11.2019 11:27+1Чем разворачиваете ВМ?
Пока ничем — используем кубик для "визуального тестирования" приложения. В остальных случаях тыкнуть мышкой для клонирования шаблона не проблема.
Смотрели в сторону github.com/Telmate/terraform-provider-proxmox?
Теперь точно посмотрим. Интересный проект. Раньше пользовались Ansible, потому что куча модулей для OpenStack, но в принципе ничего не мешает добавить слой terraform'а, который так же может вызывать Ansible на крайний случай (или наоборот)./
Спасибо за наводку.
werter78
Спасибо за статью.
Немного от себя. В Вашем случае я бы пользовал Software ZFS Raid 10 — надежно, не зависит от железа, есть компрессия (и дедупликация — можно и Redhat VDO прикрутить). Тем более, что PVE умеет ZFS из коробки.
Зы. Моя небольшая вики по PVE (и не только) forum.netgate.com/topic/120102/proxmox-ceph-zfs-pfsense-и-все-все-все
onegreyonewhite Автор
ZFS любит память… очень сильно. Меня просто замучила бы амфибиотропная асфиксия. Одна из причин, почему решил не заморачиваться. Ко всему, не проверял на ситуациях, когда резко пропадает питание на сервере — восстановится ли всё как надо?
Но самая главная причина была в том, что sas-провода только до контроллера дотягивались, а за новыми лень было ехать. Вот так вот решаются судьбы.
Но вообще есть в планах сервер развернуть на ZFS, чтобы "обкатать", так сказать.
werter_l
У меня неск-ко Proxmox VE со всего лишь 8 Гб ОЗУ на каждом — проблем нет.
ZFS ARC гибко настраивается. Просто умерьте его аппетиты. Плюс можно\нужно настроить L2ARC на SSD.
## Optimise ZFS arc size
if [ "$(command -v zfs)" != "" ]; then
RAM_SIZE_GB=$(( $(vmstat -s | grep -i «total memory» | xargs | cut -d" " -f 1) / 1024 / 1000))
if [[ RAM_SIZE_GB -lt 16 ]]; then
# 1GB/1GB
MY_ZFS_ARC_MIN=1073741824
MY_ZFS_ARC_MAX=1073741824
else
MY_ZFS_ARC_MIN=$((RAM_SIZE_GB * 1073741824 / 16))
MY_ZFS_ARC_MAX=$((RAM_SIZE_GB * 1073741824 / 8))
fi
# Enforce the minimum, incase of a faulty vmstat
if [[ MY_ZFS_ARC_MIN -lt 1073741824 ]]; then
MY_ZFS_ARC_MIN=1073741824
fi
if [[ MY_ZFS_ARC_MAX -lt 1073741824 ]]; then
MY_ZFS_ARC_MAX=1073741824
fi
cat < /etc/modprobe.d/zfs.conf
# eXtremeSHOK.com ZFS tuning
# Use 1/16 RAM for MAX cache, 1/8 RAM for MIN cache, or 1GB
options zfs zfs_arc_min=$MY_ZFS_ARC_MIN
options zfs zfs_arc_max=$MY_ZFS_ARC_MAX
# use the prefetch method
options zfs l2arc_noprefetch=0
# max write speed to l2arc
# tradeoff between write/read and durability of ssd (?)
# default: 8 * 1024 * 1024
# setting here: 500 * 1024 * 1024
options zfs l2arc_write_max=524288000
Отсюда github.com/extremeshok/xshok-proxmox/blob/master/install-post.sh