Zabbix

unformatted & untested notes

Installing Zabbix 3.x on Ubuntu LTS with PostgreSQL

Setting up PostgreSQL

As root do

su postgres
psql

Enter

psql -U <the username you just created>

=> create database zabbix;
CREATE DATABASE
=> \q

cd /usr/share/doc/zabbix-server-pgsql
gunzip create.sql.gz
psql -U <username> zabbix < create.sql

username/password for DB needs adding in both UI conf and in /etc/zabbix/zabbix-server.conf

installing Zabbix server on Ubuntu 14.04

# wget http://repo.zabbix.com/zabbix/3.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.0-1+trusty_all.deb
# dpkg -i zabbix-release_3.0-1+trusty_all.deb
# apt-get update
# apt-get install zabbix-agent zabbix-server-pgsql


wget http://repo.zabbix.com/zabbix/3.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.0-1+trusty_all.deb && dpkg -i zabbix-release_3.0-1+trusty_all.deb && apt-get update && apt-get install zabbix-agent zabbix-server-pgsql

Then edit conf file to suit environment

Using auto-discovery

Zabbix auto-discovery helps find both devices you want to monitor and rogue devices. Discovery can be be through a number of techniques and used to add the discovered devices to the monitoring.

Basic using ICMP ping

The simplest discovery method is to use ICMP ping. Go to the Configuration menu and select Discovery, then click New Discovery Rule. Give it a suitably descriptive name e.g. "branch office 1 workstations", add an IP range to search through, then click New next to Checks. From the Check type drop down list choose ICMP ping, then click Add in that section, and then the Add button at the bottom of the page.

Note if adding multiple discovery rules that they run consecutively, so if any are for a large subnet then it will take a long while to run through every IP address before the next rule is tried.

enabling auto-registration of devices

Auto registration combines well with discovery to streamline registration of new devices with your Zabbix server. To enable use the web UI and go to Configuration > Actions and use the Event Source dropdown box to select Auto registration. Then click Create Action, name it "Auto register action" then in Operations click New, select Operation type as Add to host group and choose the host group as Discovered hosts.

NB can make different actions for auto registration using Conditions - particularly useful for pushing all end user devices into their own group by metadata in their agent conf file.

installing agent on Ubuntu 14.04

# wget http://repo.zabbix.com/zabbix/3.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.0-1+trusty_all.deb
# dpkg -i zabbix-release_3.0-1+trusty_all.deb
# apt-get update
# apt-get install zabbix-agent

wget http://repo.zabbix.com/zabbix/3.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.0-1+trusty_all.deb && dpkg -i zabbix-release_3.0-1+trusty_all.deb && apt-get update
apt-get install zabbix-agent

Then edit conf file to suit environment

vi /etc/zabbix/zabbix_agentd.conf

and set to zabbix server IP

Server=ip.add.re.ss
ServerActive=ip.add.re.ss

And comment out

Hostname=

and restart zabbix-agent

# service zabbix-agent restart

TO BE CONSIDERED:

• making repo per site with zabbix-agent bundled with working conf file (added benefit is repo could have builds for many OSes)
• or adding DNS records based on source IP, and having conf file same for all e.g. zabbixproxy.newro.co, zabbixserver.newro.co, then DNS records point to correct internal IP, then use single repo for all customer sites

installing agent through salt on Ubuntu 16.04 target server

On salt master server use the following commands(where '*' means all minions of the salt master server):

sudo salt '*' cmd.run 'wget http://repo.zabbix.com/zabbix/3.4/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.4-1+xenial_all.deb'
sudo salt '*' cmd.run 'dpkg -i zabbix-release_3.4-1+xenial_all.deb'

Where 3.4 is the version of zabbix and xenial is the version of ubuntu on the minion server. Feel free to change them to fit your needs.

sudo salt '*' cmd.run 'apt-get update'
sudo salt '*' cmd.run 'apt-get -y install zabbix-agent'

Next we have to edit /etc/zabbix/zabbix_agentd.conf, you can use this command to see the uncommented lines of the file:

sudo salt '*' cmd.run 'grep "^[^#;]" /etc/zabbix/zabbix_agentd.conf'

Now we have to set Server=<zabix-server ip>, ServerActive=<zabbix-server ip> and comment Hostname= line:

sudo salt '*' cmd.run 'sed -i 's/Server=127.0.0.1/Server=<zabbix-server ip>/g' /etc/zabbix/zabbix_agentd.conf' #this command replaces "Server=127.0.0.1" with "Server=<zabbix-server ip>"
sudo salt '*' cmd.run 'sed -i 's/ServerActive=127.0.0.1/ServerActive=<zabbix-server ip>/g' /etc/zabbix/zabbix_agentd.conf'
sudo salt '*' cmd.run 'sed -i 's/Hostname=/#Hostname=/g' /etc/zabbix/zabbix_agentd.conf'

And now restart zabbix agent:

sudo salt '*' cmd.run 'service zabbix-agent restart'
sudo salt '*' cmd.run 'systemctl enable zabbix-agent.service'

Info When adding the agent on zabbix Gui the Hostname must correspond otherwise zabbix server will not receive data from the agents

Install zabbix proxy

Info Install zabbix-proxy on different server than zabbix-server

Update the repos for the latest packeges

# wget http://repo.zabbix.com/zabbix/<version>/ubuntu/pool/main/z/zabbix-release/zabbix-release_<version>+<ubuntu version>_all.deb
# sudo dpkg -i zabbix-release_<version>+<ubuntu-version>_all.deb
# sudo apt-get update

Info

Use a different database from zabbix sever

Install database

apt-get install postgresql postgresql-contrib

Create the base DB & allow web app access

Set up the database and user,

    # su postgres
    # psql -d template1
     
    CREATE USER user_db_zabbix_proxy CREATEDB;
    CREATE DATABASE zabbix_proxy OWNER user_db_zabbix_proxy;
    \q
     
    # exit

Notice: If the server does not work without a password or it requires you can add it:

   su postgres
   psql -d template1
   ALTER USER user_db_zabbix_proxy WITH PASSWORD 'type_your_password';
   \q

   #exit

Edit PostgreSQL settings to allow local connections for yourdbuser to the nextcloud database (on all DB servers):

# vi /etc/postgresql/<version number>/main/pg_hba.conf
   
# "local" is for Unix domain socket connections only
local   all             user_db_zabbix_proxy             trust
local   all             all                              peer

Then restart postgresql

service postgresql restart

Install zabbix proxy with desired database

apt-get install zabbix-proxy-pgsql

apt-get upgrade

Creating proxy database

zcat /usr/share/doc/zabbix-proxy-pgsql/schema.sql.gz | psql -U user_db_zabbix_proxy zabbix_proxy

Start service zabbix-proxy

systemctl start zabbix-proxy
systemctl enable zabbix-proxy

Edit the configuration file

# vi /etc/zabbix/zabbix_proxy.conf
ProxyMode=0
Server=94.130.33.45
Hostname=compendium_zbxProxy   ###same name will be used when added on the web interface
DBHost=localhost
DBName=zabbix_proxy
DBUser=user_db_zabbix_proxy
DBPassword=password_db

Restart zabbix-proxy

systemctl restart zabbix-proxy

Info If it is needed to be installed over WAN create port forwarding rule on both server on for ports 10050 and 10051 Notice: The installation must be on the host.

iptables -t nat -I PREROUTING -p tcp -d <host-ip> --dport 10050 -j DNAT --to-destination <zabbix-proxy-ip>:10050
iptables -t nat -I PREROUTING -p tcp -d <host-ip> --dport 10051 -j DNAT --to-destination <zabbix-proxy-ip>:10051 
sudo iptables -I FORWARD -m state -d <zabbix-proxy-network>/24  --state NEW,RELATED,ESTABLISHED -j ACCEPT

adding kvm specific monitoring

requires python-libvirt (probably already installed)

https://github.com/bushvin/zabbix-kvm-res

UserParameter=custom.kvm.pool.discover,/usr/local/bin/zabbix-kvm-res.py --resource pool --action list UserParameter=custom.kvm.pool.size[*],/usr/local/bin/zabbix-kvm-res.py --resource pool --action $1 --pool $2 UserParameter=custom.kvm.pool.active[*],/usr/local/bin/zabbix-kvm-res.py --resource pool --action active --pool $1 UserParameter=custom.kvm.pool.uuid[*],/usr/local/bin/zabbix-kvm-res.py --resource pool --action UUID --pool $1 UserParameter=custom.kvm.net.discover,/usr/local/bin/zabbix-kvm-res.py --resource net --action list UserParameter=custom.kvm.net.active[*],/usr/local/bin/zabbix-kvm-res.py --resource net --action active --net $1 UserParameter=custom.kvm.net.uuid[*],/usr/local/bin/zabbix-kvm-res.py --resource net --action UUID --net $1 UserParameter=custom.kvm.domain.discover,/usr/local/bin/zabbix-kvm-res.py --resource domain --action list UserParameter=custom.kvm.domain.active[*],/usr/local/bin/zabbix-kvm-res.py --resource domain --action active --domain $1 UserParameter=custom.kvm.domain.uuid[*],/usr/local/bin/zabbix-kvm-res.py --resource domain --action UUID --domain $1

zabbix agent

http://docswiki.newro.co/index.php/KVM_monitoring_xml

Monitoring ISCSI volume for read only

Create a file with the following script:

#!/bin/bash

if [ $# -eq 1 ]; then
        file=$1
else
        echo "Wrong number of parameters!"
        exit 1;
fi

touch $file &> /dev/null
if [ $? -eq 0 ]; then
        rm $file &> /dev/null
        echo "1"
else
        echo "0"
fi

Add a line like this in /etc/zabbix/zabbix_agentd.conf:

UserParameter=custom.checkro[*],/path/to/script/file $1

And restart the agent

service zabbix-agent restart

To call the script from zabbix GUI, create an Item and in the Key field put "custom.checkro[/path/to/iscsivolume/test-file]".

User Parameter for zabbix-agent

For all Linux servers

UserParameter=ServiceStatus[*],service  status | grep Active
UserParameter=CoreNum,grep processor /proc/cpuinfo | wc -l
UserParameter=system.sw.debianupdates,apt-get dist-upgrade -s  |sed -n 's/^\([0-9]\+\) upgraded.*/\1/p'

Only for physical servers, not on VMs

UserParameter=CPUtemp,cat /sys/class/thermal/thermal_zone0/temp

RabbitMQ

UserParameter=AliveRabbit,ps -aux | grep [r]abbitMqDaemon.py | wc -l

Reset password for user on GUI

Zabbix reset password

Setting Host Zabbix Proxy and Group

To set up the proxy, first create the Group Host, if it doesn't exists. Afterwards create the Hosts, with zabbix-proxy and http proxy. After it's done go to Zabbix -> Administration -> Proxies and add up your proxy and proxy hosts which it will supervise. If it works it should be displayed available.

Check in /etc/zabbix/zabbix_proxy.conf the following:

Hostname -> must be the same in the file and on the monitoring zabbix website.

Server ip -> must be the server ip on which the zabbix website is hosted.

Update zabbix-agent with salt

This is a salt recipe for upgrading zabbix-agent on multiple servers:

create-upgrade-zabbix-agent:
  file.managed:
    - name: /tmp/upgrade-zabbix-agent.sh
    - mode: 755
    - contents: |
            #!/bin/bash
            codename=`lsb_release -c | cut -f2`
            wget https://repo.zabbix.com/zabbix/4.2/ubuntu/pool/main/z/zabbix-release/zabbix-release_4.2-1+"$codename"_all.deb
            yes | dpkg -i zabbix-release_4.2-1+"$codename"_all.deb
            apt-get update
            apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install --only-upgrade zabbix-agent
            systemctl restart zabbix-agent
            rm zabbix-release_4.2-1+"$codename"_all.deb

run-upgrade-zabbix-agent:
  cmd.run:
    - name: /tmp/upgrade-zabbix-agent.sh

delete-upgrade-script:
  cmd.run:
    - name: rm /tmp/upgrade-zabbix-agent.sh

POSTGRESQL_DB_FREE_SPACE

POSTGRESQL_DB_FREE_SPACE[1]

CHECKUPS for backups

Steps to reproduce:

1. Create an item using vfs.file.time[<directory or file>, modify] it will monitorize the folder or file if it changed it's date or if a newer sub-file or sub-folder changed it's date. If it's a backup check for the backup date, if the time it's the creation date of the imported folder and not the current date, you should monitor the whole folder and not the "current" which is the symlink, it will check for new files or folders inside.

2. At unit use unixtime to convert from epoch to unix date and time.

3. For schedule checkups use Scheduling and set update interval to 0 so it will select the schedule checkup.

4. For trigger use the item created and use fuzzytime as function and set it up at fuzzytime requested time in seconds 86400 is 24h for checkup ( just complete the requested fields) and set the value equal to 0, it will check if the file changed in the last previous 24h if not it will send you a message with a PROBLEM state.

5. Do tests to be sure everything works ok. Done.

Dependency triggers

To set dependency triggers set the requested trigger to be dependent on what you would like and if that trigger occurs the dependent trigger won't be triggered. The serie of triggers need to have a distance of time between them so it won't have an conflict when it request the data from the first to the last trigger.

MYSQL monitoring

1.Import this template on your zabbix WEB UI

2. Create a new user in your MYSQL DB and grant the required privileges:

CREATE USER 'zabbix'@'localhost' IDENTIFIED WITH mysql_native_password BY '<password>';
GRANT USAGE,REPLICATION CLIENT,PROCESS,SHOW DATABASES,SHOW VIEW ON *.* TO 'zabbix'@'%';

3.Create a new file on /etc/zabbix/ called .my.cnf and paste:

[mysql]
user=zabbix
password=zabb345monitor!87
[mysqladmin]
user=zabbix
password=zabb345monitor!87

4. On /etc/zabbix/zabbix_agentd.d create a new file called userparameter_mysql.conf anp paste:

# Flexible parameter to determine database or table size. On the frontend side, use keys like mysql.size[zabbix,history,data].
# Key syntax is mysql.size[<database>,<table>,<type>]. # Database may be a database name or "all". Default is "all". # Table may be a table name or "all". Default is "all". # Returns value in bytes.
UserParameter=mysql.size[*],bash -c 'echo "select sum($(case "$3" in both|"") echo "data_length+index_length";; data|index) echo "$3_length";; free) echo "data_free";; esac)) from information_schema.tables$([[ "$1" = "all" || ! "$1" ]] || echo " where table_schema=\"$1\"")$([[ "$2" = "all" || ! "$2" ]] || echo "and table_name=\"$2\"");" | /usr/bin/mysql --defaults-extra-file=/etc/zabbix/.my.cnf -N'

# MySQL version and status
UserParameter=mysql.ping,mysqladmin --defaults-extra-file=/etc/zabbix/.my.cnf ping | grep -c alive
UserParameter=mysql.version,mysql --defaults-extra-file=/etc/zabbix/.my.cnf -V

# Flexible parameter to grab global variables. On the frontend side, use keys like mysql.status[Com_insert].
UserParameter=mysql.extended_status[*],/usr/bin/mysqladmin --defaults-extra-file=/etc/zabbix/.my.cnf extended-status variables |awk {'print $$2"| "$$4'} | grep "$1|" | awk {'print $$2'}

# Replication monitoring
UserParameter=mysql.slave_lagging,/usr/bin/mysql --defaults-extra-file=/etc/zabbix/.my.cnf -Bse "show slave status\\G" | grep Seconds_Behind_Master | awk '{print $$2}' | sed -e 's/^NULL$/-1/; s/![0-9]+/-1/' | awk '{if($2 ~ /d/) {print 100} else {print $2}}'

UserParameter=mysql.slave_status[*],/usr/bin/mysql --defaults-extra-file=/etc/zabbix/.my.cnf -e "show slave status\\G"|grep -w $1 | awk '{print $$2}'

UserParameter=mysql.slave_status2[*],/usr/bin/mysql --defaults-extra-file=/etc/zabbix/.my.cnf -e "show slave status\\G"|grep -i $1 | awk '{print $$2}' 1>> /dev/null 2>> /dev/null && echo yes || echo no