I’ve been talking a lot about Zabbix lately (mostly because it’s a lot of what is on my plate at work)  and I keep finding little gotcha points that I understand but are very frustrating.  Last week (and into this week) I’ve been setting up trigger dependency.  Here’s an example, just as a hypothetical situation.

You have an end device, like an Access Point or Security Camera (we’ll call it E, for Endpoint).  This device E is sitting behind a switch in an IDF (we’ll call it IS, for IDF Switch).  This switch is sitting behind another switch in an MDF (we’ll call it MS, for MDF Switch).  This switch is behind your Core Switch (we’ll call it CS).  So your layout is:

CS -> MS -> IS -> E.

Any one of the “->” in this layout is a point where the chain can break. If MS goes dark for any reason, then IS and E both go down (by definition) since CS cannot communicate with either of them.  E is therefore dependent on IS which is dependent on MS which is dependent on CS. In Zabbix, we can setup a similar dependency chain.  This is important because if you don’t setup the dependency chain you’ll get an email about all the devices behind a switch going down.  Continuing our example about MS going down, you’d get an alert about MS, IS, and E all being down.  If you have a lot of monitored end points (like I do) then you’ll get a LOT of email.  I mean, a lot.  We lost power in the district over the weekend.  It was well over 100 emails.  Not very helpful.

So we setup a dependency chain.  The endpoint devices (Access Points, Security Cameras, ESXi Hosts, Servers, etc) all depend on MDF or IDF switches.  The IDF switch stacks depend on the MDF switch stack.  The MDF switch stack in the building depends on the Core switch stack at the Tech office.  If the MDF switch stack goes down, we get an alert about the MDF switch stack and that’s it.  None of the IDF or endpoint devices.  It’s working well so far.

Now that we’ve set the background let’s talk about this post about Zabbix.  Zabbix sucks at setting trigger dependency en masse.  And by suck at it, I mean, they don’t do it.  At all.  Which is surprising!  You can set templates up and apply templates in bulk.  You can make a bunch of changes to individual hosts without difficulty (bulk edit tool).  Just not triggers.  It was very frustrating.  I have 500+ devices in Zabbix.  It takes a lot of clicks per device to set the dependency.  So, we did it the manual way: via SQL commands. 😀

1. (Optional) Install and Configure PHPMyAdmin for your Zabbix box.  We’ll uninstall it when we’re done.  It makes exploring and looking through the database just a bunch more simple.  You can always do it via the MySQL command line, but honestly why when you can make your life SO much easier.  The steps to do this are well documented across the internet but if you’re feeling lazy then here’s a link that includes steps for setting up access restrictions which I highly recommend if your site is internet-facing.
2. Collect the info you need.
   1. What is the name of the trigger you want to depend on some other trigger?  We’ll call that host trigger, HT.
   2. What is the name of the trigger you want HT to depend on? We’ll call that PT (parent trigger).
   3. What is the name(s) of the hosts you want to reference HT to PT?  In our naming scheme it was easy to collect batches of data because our naming conventions are universal.  Cameras are always <building>_<number>.  Access points are always <building>-AP-<room>.  We’ll call this HN.
   4. What is the name(s) of the hosts you want to use to get the PT?  We’ll call this PN.
   5. What is the table that links hosts, triggers, monitoring items?  … Take your time with this one.  It isn’t straight-forward.
      …
      …
      Give up?  It’s a table called functions.  Yeah.  Took me a while to find it and wrap my head around it.
3. So we need a query that will get us two sets of hosts: PN and PT, then HN to HT.  The query is the same, just with modified parameters:

   1. SELECT a.host, c.triggerid
      FROM triggers d, functions c, items b, hosts a
      WHERE d.triggerid=c.triggerid
      AND c.itemid=b.itemid
      AND b.hostid=a.hostid
      AND a.host LIKE “<PN or HN>”
      AND b.name=”<PT or HT>”
      ORDER BY `a`.`host` ASC LIMIT 0,100;

   2. More specifically, in my case to get a school-specific list of Security Cameras referencing the ICMP Ping trigger, I used:

      1. SELECT a.host, c.triggerid
         FROM triggers d, functions c, items b, hosts a
         WHERE d.triggerid=c.triggerid
         AND c.itemid=b.itemid
         AND b.hostid=a.hostid
         AND a.host LIKE “CMS\_%”
         AND b.name=”ICMP Ping”
         ORDER BY `a`.`host` ASC LIMIT 0,100;

   3. Furthermore, to get a list of school-specific switches referencing the ICMP Ping trigger, I used:

      1. SELECT a.host, c.triggerid
         FROM triggers d, functions c, items b, hosts a
         WHERE d.triggerid=c.triggerid
         AND c.itemid=b.itemid
         AND b.hostid=a.hostid
         AND a.host LIKE “cms.%.3%”
         AND b.name=”ICMP Ping”
         ORDER BY `a`.`host` ASC LIMIT 0,100;

      2. This got me a list of switches named cms.<list of strings>.3<list of strings> matching things like cms.mdf.3850 and cms.lowerlevel.3750.  Yay the power of wildcards!
4. Now you need to match HT to PT and assign each pairing a unique ID code.  The ID code is based on the highest triggerdepid field in the trigger_depends table.  For example, if the highest value of triggerdepid is 300, your first pair will be 301.
5. Now you need to insert the new rows into the trigger_depends table.  Let’s say we have HT with values 1500, 1503, 1506 all depending on a PT of 750, and another HT with values 1755, 1758, 1801 all depending on a different PT of 1500.  Your resulting query would be:

   1. INSERT INTO trigger_depends (triggerdepid,triggerid_down,triggerid_up)
      VALUES
      (301,1500,750),(302,1503,750),
      (303,1506,750),(304,1755,1500),
      (305,1758,1500),(306,1801,1500);

6. You’ll now need to update the ids table field of trigger_depends and set it to the highest value of triggerdepid.

   1. UPDATE ids SET trigger_depends=”306″;

7. Then you’re done.  Obviously, this only makes sense to do if you’re doing a lot of dependency sets.  Once I got the method down I wrote all the dependency sets in about 8 hours, as opposed to the week I thought it was going to take me.

That being said: there’s easier ways I’m sure.  You could script it entirely if you know PT and HT.  It’s not entirely straightforward though (and involves joins I think).  When I come up with it I’ll post it as a follow-up example.

-M

4