Icinga Check Scripts
====================

Various check scripts for use in icinga.

All scripts support option `--help`, so call `./scriptname --help` for
more information.


load_per_cpu.sh
---------------

Icinga plugin to check load average per CPU. Shows 1min, 5min 15min
average per CPU. Different to check_load, the load is divided by the
number of CPU units and therefore normalized. So the same warning and
critical levels fit for any server.

If a server's load is often high, you can find tipps in [Isolating Linux High System Load](https://www.tummy.com/articles/isolating-heavy-load/), in short words:

 - see the load average in `uptime`
 - check system logs with `dmesg`
 - see `vmstat`:
    - high `WA` (wait) column: CPU is often waiting, you probably have high disk load
       - check `swap` → `si` and `so`, if they are often much above 0, you are out of memory
          - check memory usage of processes with `ps awwlx --sort=vsz`
          - solution: add more memory
    - if `cpu` → `ID` (idle) is around 0, your CPU is overloaded
       - if `SY` (system) is high, there could be large directories, e.g. mail spam
       - if `SY` (system) is high, it could be the firewall iptables
       - if `US` (user-space) is high, check with `top` for CPU consuming processes
    - if `io` → `bi` and `bo` (in/out) are high, check with `iostat` or `sudo iotop`
 

mountpoint.sh
-------------

Icinga plugin to check whether a given mountpoint is
available. Detects problems, e.g. with glusterfs: In case of a
problem, there is an error message such as "socket not
connected". Called with path to check.

diskio.sh
---------

Icinga plugin to check whether a given path is writable and checks the
performance of the given path. Detects problems with the filesystem or
specific io performance. Writes a test file using `dd` and reports
write speed. It is recommended to use a subdirectory tmp for writing
test files.

gluster-geo-replication.sh
--------------------------

Icinga plugin to check gluster geo replication. Returns OK, if no
connection is Faulty and one connection is Active. Returns CRITICAL,
if one connection is Faulty- Returns WARNING, if no connection is
Active.


Installation
============

Install
-------

Checkout to a path, e.g. `/opt/icinga-checks`:

    cd /opt
    sudo git co https://mrw.sh/admin-scripts/icinga-checks.git

Update
------

    cd /opt/icinga-checks
    sudo git pull

Configure
---------

Go to [Icinga Director](https://mrw.sh/docker/icingaweb2), in `Commands` add one:
 - Command type: `Plugin Check Command`
 - Command name: `load per cpu`
 - Command: `check_by_ssh`

After adding the command, add the following parameters:

    -C /opt/icinga-checks/load_per_cpu.sh
    -H $host.name$
    -oStrictHostKeyChecking=no

The last argument `-oStrictHostKeyChecking=no` is a key only ith no value.

To check mountpoint of `/var/volumes` use argument name `-C` with value `/opt/icinga-checks/mountpount.sh /var/volumes`.

Then create a new Service Template, give it a name. e.g. `load per cpu` and assign the check command just created.

Finally add a service, again you can use the same name and import the template just created, then assign the hosts to check. I use a host template for all linux servers, so the command is run in each of them.

Don't forget to deploy the activity log in Icinga Director.