by ·0 Comments

A few months ago I successfully deployed and configured an Ubuntu 12.04 LTS Server Edition for the purposes of installing Nagios and doing on-site monitoring for key servers.  Yesterday I did a bunch of security and hotfix updates to the server, since it was VERY behind (talking >20 security updates and what not) on a lot of packages.  Following a reboot (I know, I know, not necessary but I’m still firmly rooted in the land of Windows where updates and reboots go hand in hand; for shame) I noticed a surge of alerts from Nagios and was thoroughly annoyed to see that the load on the server was consistently way too high – I was getting Warning / Critical alerts nearly every 15 minutes.  Uh oh!

***** Nagios *****

Notification Type: PROBLEM

Service: Current Load
Host: localhost
Host Alias: localhost
Address: 127.0.0.1
State: CRITICAL

Date/Time: Wed Dec 4 23:48:58 EST 2013

Additional Info:

CRITICAL – load average: 5.89, 4.95, 4.04

Status Details: https://nagios/nagios/cgi-bin/extinfo.cgi?type=1&host=localhost

This continued for a day until I got fed up with it.  I told Nagios to stop paying attention to that service and to ignore it completely.  I basically shit-canned the project and ignored it for about 24 hours until I realized it was going to be a slow day today.  So I SSH into the box and load up top, and I wait.  I wait and I wait and I wait.  Every 15 minutes I would see a surge of spawned processes for ping, check_snmp, and a few other ones.  I started to get a clear picture of what was going on.

The reboot of Nagios reset the next check period to be the same for all devices, which was a big problem.  We have 360+ devices in our monitoring scheme, and of that nearly 760+ services being monitored (with more coming in the future, I need to setup Windows device monitoring for our DCs and file servers).  They were all trying to run at the same time.  All 760.  This isn’t a ridiculously beefy server we’re talking about here.

srv-nagios: Virtual Machine Details
VMware Virtual Platform
Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
1GB RAM
20GB SCSI Virtual HDD

So after sitting on it a bit I decided to do some research and found that I could limit the number of simultaneous checks in Nagios.  This is of course noted in the documentation.  I must have glossed over it.

 

Maximum Concurrent Service Checks

 

Format: max_concurrent_checks=<max_checks>
Example: max_concurrent_checks=20

This option allows you to specify the maximum number of service checks that can be run in parallel at any given time. Specifying a value of 1 for this variable essentially prevents any service checks from being run in parallel. Specifying a value of 0 (the default) does not place any restrictions on the number of concurrent checks. You’ll have to modify this value based on the system resources you have available on the machine that runs Nagios, as it directly affects the maximum load that will be imposed on the system (processor utilization, memory, etc.). More information on how to estimate how many concurrent checks you should allow can be found here.

Our setting was of course 0, meaning that Nagios tried to run as many checks as it wanted at the same time.  After thinking on it a bit, I figured out what I wanted to do.  We have 760 someodd checks.  We have 15 minute intervals.  I did 760/15 and it came out to be about 51 checks per minute.  I started there.  I set max_concurrent_checks to 51 and BAM load immediately dropped down to a more stable level.

OK – load average: 0.00, 0.13, 0.19

As I add more devices and services to Nagios I will tweak the value.  Checking 60 things at a time should be easy enough to handle, it works out to about 1 a second and since the vast majority of my checks and services are pings and simple snmp queries it shouldn’t be too bad.

Here’s hoping.

A happy Nagios is a happy Mike.

Nagios

by ·1 Comment

This morning I woke up feeling pretty great.  It’s the day before Thanksgiving, which is my all-time favorite holiday of all time.  None of this pretentious marketing and monetization (excluding Black Friday and how it’s creeping into Thanksgiving) that comes with major holidays.  Just family, food, and relaxation.

I had to go to one of the K-3 schools this morning to help a teacher with a projector, and as part of the process I had to climb onto a chair and reach my hands above my head to grab the projector cables and pull it down.  In the process, my shirt rode up a bit and you could see my stomach and sides (which are nothing to be thrilled about, but still far better than I used to be at 4XL and now a L/XL).  Soon after, I heard lots of kids giggling, I turned around to get off the chair and saw a few of them pointing.  A few others were whispering things like “gross” and “disgusting” while pointing at me.  And it hit me.  They’re talking about me here.

After almost 2 years of working at losing weight and getting in shape, it doesn’t matter.

And now I just feel completely broken.  Like all my work was for absolutely nothing.

I just want to disappear.

So just remember: your words and actions can destroy how someone is doing.  Be mindful.

by ·2 Comments

A very cool feature. Almost enough to make me get one.

Almost.

News: Microsoft Confirms PS4 Can Be Connected To Xbox One | MegaGames.

If it had backward compatibility, I would have bought it already.

But it doesn’t.

For whatever reason.

(the reason being, that way they can make you buy the Xbox One version of your old games for full price, thus double charging you for content you already own)

by ·2 Comments

Switch to e-books was an unmitigated disaster, says school principal – Independent.ie.

—

This just in, poorly informed people making tech decisions without guidance nor the correct kind of knowledge will invariably make bad decisions and then blame the tech instead of the poor management.

Can we really trust a person who refers to hard disk drive storage as memory to make the right call?

An unmitigated disaster, perhaps, but not because of the tech.  A poorly planned out roll out with poor tech management for sure.  But the fault does not lie squarely on the shoulders of the technology.