We’ve recently switched our process monitor on our Poll Everywhere servers from the God process monitor (read about my frustrations with god and why we switched) to monit. So far things have been running great, but I ran into a few "gotchya's" while setting up monit.

Below are a few tips and tricks that I wish had appeared in Google while I was configuring monit on all of our servers:

Monit from the command line won’t work with HTTPS enabled

Actually it does work; I titled this section as-is so that others may Google and fix this problem faster. Here is what my httpd file looked like:

set httpd port 4567
         ssl enable
         pemfile /etc/monit/certs/monit.pem
         allow crypt /etc/monit/htpasswd

Then from terminal I kept running into this problem:

~# monit status
monit: cannot read status from the monit daemon

It turns out monit had no way to authenticate to the HTTP server because it didn’t have access to a clear-text password (the ones in my htpasswd file were encrypted). I just added a new user with a clear-text password to my httpd directive:

set httpd port 4567
        ssl enable
        pemfile /etc/monit/certs/monit.pem
        allow crypt /etc/monit/htpasswd
        allow monit:some-big-random-string-of-chars

When monit fires up, it reads the password from this config file and uses those credentials to control the HTTP interface.

Passing environmental variables to daemons in Monit

If you’re like us, you need to pass the RAILS_ENV=production (and maybe a few other environmental variables) into thin when booting Rails. My first swing at the problem looked like:

check process lemonade_stand_thin_9001
    with pidfile /var/run/thin.9001.pid
    start = "RAILS_ENV=production thin --port 9001 
     \ --pid /var/run/thin.9001.pid --daemonize start"

Unfortunately that didn’t work until I discovered:

check process lemonade_stand_thin_9001
    with pidfile /var/run/thin.9001.pid
    start = "/usr/bin/env RAILS_ENV=production thin --port 9001
     \ --pid /var/run/thin.9001.pid --daemonize start"

Monit XML Format

Monit exposes status information in an XML format that you could consume and use in your own apps. To get at this data format the mont url as:

http://yourserver.net:2812/_status?format=xml

I may consolidate the web interfaces on all of our machines into one console with a Sinatra app that I have yet to write. M/Monit is also available as a commercial product that consolidates monits for you starting at approximately $200.

So far monit has been pretty solid. Its already improved our uptime by quickly restarting processes that mess up and notifying us about the situation. Aside from the small pains I had getting the monit console working with HTTPS, the downside of monit is the verbosity of the configuration files when setting up a cluster:

check process lemonade_stand_thin_9000
    with pidfile /var/run/thin.9000.pid
    start = "/usr/bin/env RAILS_ENV=production thin --port 9000  
     \ --rackup /home/lemonade_stand/web/current/config.ru 
     \ --log /home/lemonade_stand/web/current/log/thin.log 
     \ --pid /var/run/thin.9000.pid --daemonize start" 
    stop = "/usr/bin/env thin --pid /var/run/thin.9000.pid stop" 
    if totalmem > 190.0 MB for 3 cycles then alert
    if totalmem > 200.0 MB for 3 cycles then restart
    if 9 restarts within 9 cycles then timeout
    group lemonade_stand
    
  check process lemonade_stand_thin_9001
    with pidfile /var/run/thin.9001.pid
    start = "/usr/bin/env RAILS_ENV=production thin --port 9001 
     \ --rackup /home/lemonade_stand/web/current/config.ru 
     \ --log /home/lemonade_stand/web/current/log/thin.log 
     \ --pid /var/run/thin.9001.pid --daemonize start" 
    stop = "/usr/bin/env thin --pid /var/run/thin.9001.pid stop" 
    if totalmem > 190.0 MB for 3 cycles then alert
    if totalmem > 200.0 MB for 3 cycles then restart
    if 9 restarts within 9 cycles then timeout
    group lemonade_stand
    
  # ... and so on ...

That is a whole lot of copy and pasting! This is an area where God has an advantage because you could iterate through an array and instanciate a God::Watch block through each pass; however, it wouldn’t be that hard to implement some sort of templating system with ERB that could compile monit configurations like this:

<% (9000..9011).each do |port| %>
check process lemonade_stand_thin_<%= port %>
    with pidfile /var/run/thin.<%= port %>.pid
    start = "/usr/bin/env RAILS_ENV=production thin --port <%= port %> 
     \ --rackup /home/lemonade_stand/web/current/config.ru 
     \ --log /home/lemonade_stand/web/current/log/thin.log 
     \ --pid /var/run/thin.<%= port %>.pid --daemonize start" 
    stop = "/usr/bin/env thin --pid /var/run/thin.<%= port %>.pid stop" 
    if totalmem > 190.0 MB for 3 cycles then alert
    if totalmem > 200.0 MB for 3 cycles then restart
    if 9 restarts within 9 cycles then timeout
    group lemonade_stand
<% end %>

Permalink | Leave a comment  »

First, let me get a few things off my chest about God (the process monitor, I'm not being religious here)

"God - Process Monitoring Done Horribly Wrong"
"God (like monit, but doesn't work)"

Ok, I'm here to say that I hate the process monitor called God. Why? Because contrary to what Tom Preston-Werner says, it doesn't work and its not awesome. Where do I start?

Permalink | Leave a comment  »