Zabbix
unformatted & untested notes
Installing Zabbix 3.x on Ubuntu LTS with PostgreSQL
Setting up PostgreSQL
As root do
su postgres psql
Enter
psql -U <the username you just created> => create database zabbix; CREATE DATABASE => \q cd /usr/share/doc/zabbix-server-pgsql gunzip create.sql.gz psql -U <username> zabbix < create.sql
username/password for DB needs adding in both UI conf and in /etc/zabbix/zabbix-server.conf
installing Zabbix server on Ubuntu 14.04
# wget http://repo.zabbix.com/zabbix/3.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.0-1+trusty_all.deb # dpkg -i zabbix-release_3.0-1+trusty_all.deb # apt-get update # apt-get install zabbix-agent zabbix-server-pgsql wget http://repo.zabbix.com/zabbix/3.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.0-1+trusty_all.deb && dpkg -i zabbix-release_3.0-1+trusty_all.deb && apt-get update && apt-get install zabbix-agent zabbix-server-pgsql
Then edit conf file to suit environment
Using auto-discovery
Zabbix auto-discovery helps find both devices you want to monitor and rogue devices. Discovery can be be through a number of techniques and used to add the discovered devices to the monitoring.
Basic using ICMP ping
The simplest discovery method is to use ICMP ping. Go to the Configuration menu and select Discovery, then click New Discovery Rule. Give it a suitably descriptive name e.g. "branch office 1 workstations", add an IP range to search through, then click New next to Checks. From the Check type drop down list choose ICMP ping, then click Add in that section, and then the Add button at the bottom of the page.
Note if adding multiple discovery rules that they run consecutively, so if any are for a large subnet then it will take a long while to run through every IP address before the next rule is tried.
enabling auto-registration of devices
Auto registration combines well with discovery to streamline registration of new devices with your Zabbix server. To enable use the web UI and go to Configuration > Actions and use the Event Source dropdown box to select Auto registration. Then click Create Action, name it "Auto register action" then in Operations click New, select Operation type as Add to host group and choose the host group as Discovered hosts.
NB can make different actions for auto registration using Conditions - particularly useful for pushing all end user devices into their own group by metadata in their agent conf file.
installing agent on Ubuntu 14.04
# wget http://repo.zabbix.com/zabbix/3.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.0-1+trusty_all.deb # dpkg -i zabbix-release_3.0-1+trusty_all.deb # apt-get update # apt-get install zabbix-agent wget http://repo.zabbix.com/zabbix/3.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.0-1+trusty_all.deb && dpkg -i zabbix-release_3.0-1+trusty_all.deb && apt-get update apt-get install zabbix-agent
Then edit conf file to suit environment
vi /etc/zabbix/zabbix_agentd.conf
and set to zabbix server IP
Server=ip.add.re.ss ServerActive=ip.add.re.ss
And comment out
Hostname=
and restart zabbix-agent
# service zabbix-agent restart
TO BE CONSIDERED:
- making repo per site with zabbix-agent bundled with working conf file (added benefit is repo could have builds for many OSes)
- or adding DNS records based on source IP, and having conf file same for all e.g. zabbixproxy.newro.co, zabbixserver.newro.co, then DNS records point to correct internal IP, then use single repo for all customer sites
installing agent through salt on Ubuntu 16.04 target server
On salt master server use the following commands(where '*' means all minions of the salt master server):
sudo salt '*' cmd.run 'wget http://repo.zabbix.com/zabbix/3.4/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.4-1+xenial_all.deb' sudo salt '*' cmd.run 'dpkg -i zabbix-release_3.4-1+xenial_all.deb'
Where 3.4 is the version of zabbix and xenial is the version of ubuntu on the minion server. Feel free to change them to fit your needs.
sudo salt '*' cmd.run 'apt-get update' sudo salt '*' cmd.run 'apt-get -y install zabbix-agent'
Next we have to edit /etc/zabbix/zabbix_agentd.conf, you can use this command to see the uncommented lines of the file:
sudo salt '*' cmd.run 'grep "^[^#;]" /etc/zabbix/zabbix_agentd.conf'
Now we have to set Server=<zabix-server ip>, ServerActive=<zabbix-server ip> and comment Hostname= line:
sudo salt '*' cmd.run 'sed -i 's/Server=127.0.0.1/Server=<zabbix-server ip>/g' /etc/zabbix/zabbix_agentd.conf' #this command replaces "Server=127.0.0.1" with "Server=<zabbix-server ip>" sudo salt '*' cmd.run 'sed -i 's/ServerActive=127.0.0.1/ServerActive=<zabbix-server ip>/g' /etc/zabbix/zabbix_agentd.conf' sudo salt '*' cmd.run 'sed -i 's/Hostname=/#Hostname=/g' /etc/zabbix/zabbix_agentd.conf'
And now restart zabbix agent:
sudo salt '*' cmd.run 'service zabbix-agent restart' sudo salt '*' cmd.run 'systemctl enable zabbix-agent.service'
Info When adding the agent on zabbix Gui the Hostname must correspond otherwise zabbix server will not receive data from the agents
Install zabbix proxy
Info Install zabbix-proxy on different server than zabbix-server
Update the repos for the latest packeges
# wget http://repo.zabbix.com/zabbix/<version>/ubuntu/pool/main/z/zabbix-release/zabbix-release_<version>+<ubuntu version>_all.deb # sudo dpkg -i zabbix-release_<version>+<ubuntu-version>_all.deb # sudo apt-get update
Info
Use a different database from zabbix sever
Install database
apt-get install postgresql postgresql-contrib
Create the base DB & allow web app access
Set up the database and user,
# su postgres
# psql -d template1
CREATE USER user_db_zabbix_proxy CREATEDB;
CREATE DATABASE zabbix_proxy OWNER user_db_zabbix_proxy;
\q
# exit
Notice: If the server does not work without a password or it requires you can add it:
su postgres psql -d template1 ALTER USER user_db_zabbix_proxy WITH PASSWORD 'type_your_password'; \q #exit
Edit PostgreSQL settings to allow local connections for yourdbuser to the nextcloud database (on all DB servers):
# vi /etc/postgresql/<version number>/main/pg_hba.conf # "local" is for Unix domain socket connections only local all user_db_zabbix_proxy trust local all all peer
Then restart postgresql
service postgresql restart
Install zabbix proxy with desired database
apt-get install zabbix-proxy-pgsql apt-get upgrade
Creating proxy database
zcat /usr/share/doc/zabbix-proxy-pgsql/schema.sql.gz | psql -U user_db_zabbix_proxy zabbix_proxy
Note: If it doesn't work check https://repo.zabbix.com/zabbix/<version>/debian/pool/main/z/zabbix/ and install it manually.
Start service zabbix-proxy
service zabbix-proxy start systemctl enable zabbix-proxy.service
Edit the configuration file
# vi /etc/zabbix/zabbix_proxy.conf DBHost=localhost DBName=zabbix_proxy DBUser=user_db_zabbix_proxy DBPassword=password_db
Notice: If something pops up wrong you can comment the DBHost and see if it works.
Info
If it is needed to be installed over WAN create port forwarding rule on both server on for ports 10050 and 10051
Notice: The installation must be on the host.
sudo iptables -t nat -I PREROUTING -p tcp -d 94.130.33.44 --dport 80 -j DNAT --to-destination 172.16.10.12:80 sudo iptables -I FORWARD -m state -d 172.16.10.0/24 --state NEW,RELATED,ESTABLISHED -j ACCEPT
adding kvm specific monitoring
requires python-libvirt (probably already installed)
https://github.com/bushvin/zabbix-kvm-res
UserParameter=custom.kvm.pool.discover,/usr/local/bin/zabbix-kvm-res.py --resource pool --action list
UserParameter=custom.kvm.pool.size[*],/usr/local/bin/zabbix-kvm-res.py --resource pool --action $1 --pool $2
UserParameter=custom.kvm.pool.active[*],/usr/local/bin/zabbix-kvm-res.py --resource pool --action active --pool $1
UserParameter=custom.kvm.pool.uuid[*],/usr/local/bin/zabbix-kvm-res.py --resource pool --action UUID --pool $1
UserParameter=custom.kvm.net.discover,/usr/local/bin/zabbix-kvm-res.py --resource net --action list
UserParameter=custom.kvm.net.active[*],/usr/local/bin/zabbix-kvm-res.py --resource net --action active --net $1
UserParameter=custom.kvm.net.uuid[*],/usr/local/bin/zabbix-kvm-res.py --resource net --action UUID --net $1
UserParameter=custom.kvm.domain.discover,/usr/local/bin/zabbix-kvm-res.py --resource domain --action list
UserParameter=custom.kvm.domain.active[*],/usr/local/bin/zabbix-kvm-res.py --resource domain --action active --domain $1
UserParameter=custom.kvm.domain.uuid[*],/usr/local/bin/zabbix-kvm-res.py --resource domain --action UUID --domain $1
zabbix agent
http://docswiki.newro.co/index.php/KVM_monitoring_xml
Monitoring ISCSI volume for read only
Create a file with the following script:
#!/bin/bash
if [ $# -eq 1 ]; then
file=$1
else
echo "Wrong number of parameters!"
exit 1;
fi
touch $file &> /dev/null
if [ $? -eq 0 ]; then
rm $file &> /dev/null
echo "1"
else
echo "0"
fi
Add a line like this in /etc/zabbix/zabbix_agentd.conf:
UserParameter=custom.checkro[*],/path/to/script/file $1
And restart the agent
service zabbix-agent restart
To call the script from zabbix GUI, create an Item and in the Key field put "custom.checkro[/path/to/iscsivolume/test-file]".
User Parameter for zabbix-agent
For all Linux servers
UserParameter=ServiceStatus[*],service status | grep Active UserParameter=CoreNum,grep processor /proc/cpuinfo | wc -l UserParameter=system.sw.debianupdates,apt-get dist-upgrade -s |sed -n 's/^\([0-9]\+\) upgraded.*/\1/p'
Only for physical servers, not on VMs
UserParameter=CPUtemp,cat /sys/class/thermal/thermal_zone0/temp
RabbitMQ
UserParameter=AliveRabbit,ps -aux | grep [r]abbitMqDaemon.py | wc -l
Reset password for user on GUI
Setting Host Zabbix Proxy and Group
To set up the proxy, first create the Group Host, if it doesn't exists. Afterwards create the Hosts, with zabbix-proxy and http proxy. After it's done go to Zabbix -> Administration -> Proxies and add up your proxy and proxy hosts which it will supervise. If it works it should be displayed available.
Check in /etc/zabbix/zabbix_proxy.conf the following:
Hostname -> must be the same in the file and on the monitoring zabbix website.
Server ip -> must be the server ip on which the zabbix website is hosted.
Update zabbix-agent with salt
This is a salt recipe for upgrading zabbix-agent on multiple servers:
create-upgrade-zabbix-agent:
file.managed:
- name: /tmp/upgrade-zabbix-agent.sh
- mode: 755
- contents: |
#!/bin/bash
codename=`lsb_release -c | cut -f2`
wget https://repo.zabbix.com/zabbix/4.2/ubuntu/pool/main/z/zabbix-release/zabbix-release_4.2-1+"$codename"_all.deb
yes | dpkg -i zabbix-release_4.2-1+"$codename"_all.deb
apt-get update
apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install --only-upgrade zabbix-agent
systemctl restart zabbix-agent
rm zabbix-release_4.2-1+"$codename"_all.deb
run-upgrade-zabbix-agent:
cmd.run:
- name: /tmp/upgrade-zabbix-agent.sh
delete-upgrade-script:
cmd.run:
- name: rm /tmp/upgrade-zabbix-agent.sh
POSTGRESQL_DB_FREE_SPACE
POSTGRESQL_DB_FREE_SPACE[1]
CHECKUPS for backups
Steps to reproduce:
1. Create an item using vfs.file.time[<directory or file>, modify] it will monitorize the folder or file if it changed it's date or if a newer sub-file or sub-folder changed it's date. If it's a backup check for the backup date, if the time it's the creation date of the imported folder and not the current date, you should monitor the whole folder and not the "current" which is the symlink, it will check for new files or folders inside.
2. At unit use unixtime to convert from epoch to unix date and time.
3. For schedule checkups use Scheduling and set update interval to 0 so it will select the schedule checkup.
4. For trigger use the item created and use fuzzytime as function and set it up at fuzzytime requested time in seconds 86400 is 24h for checkup ( just complete the requested fields) and set the value equal to 0, it will check if the file changed in the last previous 24h if not it will send you a message with a PROBLEM state.
5. Do tests to be sure everything works ok. Done.
Dependency triggers
To set dependency triggers set the requested trigger to be dependent on what you would like and if that trigger occurs the dependent trigger won't be triggered. The serie of triggers need to have a distance of time between them so it won't have an conflict when it request the data from the first to the last trigger.
