All plugins return 0 for stats

[dogenpunk]

I'm current running a Rails 3.0.4 application with Passenger 3.0.7 on FreeBSD 8.2 with Munin 1.4.4. After installing this gem and moving the symlinks to /usr/local/etc I get nothing but 0 return values for every plugin except munin_passenger_status. I have installed the application specific plugins, which also return 0 for all statistics. When I run request-log-analyzer itself I get statistics as expected.

[dogenpunk]

Also, when running the app specific plugins directly I get 'yes' when run with autoconf and an error when run with debug:

Filepath unspecified. Exiting

I believe this is may be due to /proc/meminfo being specific to Lunixes.

[wakemaster39]

For the rails plugins at least the issue has to do with the logger format. The request-log-analyzer gem does its best to try and figure out which log format it is reading, but has issues determining between rails 2 and 3. It defaults to rails 2 I believe, which makes it run into errors in reading the log and you get out zero's. To fix this, be sure to set "env.log_format rails3" in the plugin-conf.d folder for EACH of the files that use this gem.

Also note that by default the generated files specify the binary location for ruby, and passeneger-memory stats, etc to be in the "/usr/local/bin/" folder, while if you install these gems as root, they will default to the "/usr/bin" folder. The defaults are the safer and better choice if you can make them install there, but you can edit all the config files to remove the local part and this should get your plugins running.

[dogenpunk]

Ok, thanks. Setting the log format worked for the request-log-analyzer plugins. The only one I seem to have an issue with now is the munin_passenger_memory_stats plugin. It's still returning 0 for memory.value. Is this to be expected? When I run

ruby munin_passenger_memory_stats debug

I do get the expected output from passenger-memory-stats.

[wakemaster39]

Can you paste the conf file for this? It is likely an issue there since running it from ruby is working.

[dogenpunk]

Here it is:

[munin_passenger_memory_stats] user root command /usr/local/bin/ruby %c env.passenger_status /usr/local/bin/passenger-status env.apache_pid_file /var/run/httpd.pid env.passenger_memory_stats /usr/local/bin/passenger-memory-stats env.graph_category Passenger

I've confirmed each path. Ruby is located in /usr/local/bin as I had to manually upgrade from 1.8 to 1.9.

[wakemaster39]

I believe the issue is that you don't have quotes on your variables. env.passenger_status /usr/local/bin/passenger-status env.apache_pid_file /var/run/httpd.pid env.passenger_memory_stats /usr/local/bin/passenger-memory-stat

These should be formatted like env.passenger_status '/usr/local/bin/passenger-status' env.apache_pid_file '/var/run/httpd.pid' env.passenger_memory_stats '/usr/local/bin/passenger-memory-stat'

Its the only issue that I can see that differ from my commands.

[dogenpunk]

I gave this a try and I still get the same results. Also, none of the other plugins have the values quoted in this manner. I've forked the repo and will look at the code for this to see if I can't find anything. FYI, munin_passenger_status is returning actual numbers (it's config variables aren't quoted). munin_passenger_queue is returning 0's also, but I'm thinking that this actual data as the server is under a pretty light load.

*** Update *** In an irb session I ran passenger-memory-stats and I do get a return value as expected from this program. However, the regex on line 37 of lib/munin/plugins/passenger_memory_stats.rb

stats =~ /RSS:\s*([\d\.]+)\s*MB\Z/m
memory = ($1.to_f * 1024 * 1024).to_i

This is looking for the string 'RSS: ' correct? This token doesn't appear in the output of passenger-memory-stats on FreeBSD. Here's the output I receive when running it from the command line:

---------- Apache processes ----------

PID PPID VMSize Resident Name

1691 1 79.2 MB 8.2 MB /usr/local/sbin/httpd -k start 1700 1691 79.2 MB 8.2 MB /usr/local/sbin/httpd -k start 1701 1691 79.2 MB 8.2 MB /usr/local/sbin/httpd -k start 1702 1691 79.2 MB 8.2 MB /usr/local/sbin/httpd -k start 1703 1691 79.2 MB 8.2 MB /usr/local/sbin/httpd -k start 1704 1691 79.2 MB 8.2 MB /usr/local/sbin/httpd -k start 43565 1 76.5 MB 8.1 MB /usr/local/sbin/httpd -DNOHTTPACCEPT 90229 43565 76.5 MB 8.5 MB /usr/local/sbin/httpd -DNOHTTPACCEPT 90265 43565 76.5 MB 8.4 MB /usr/local/sbin/httpd -DNOHTTPACCEPT 90266 43565 76.5 MB 8.4 MB /usr/local/sbin/httpd -DNOHTTPACCEPT 90268 43565 76.5 MB 8.2 MB /usr/local/sbin/httpd -DNOHTTPACCEPT 90272 43565 76.5 MB 8.2 MB /usr/local/sbin/httpd -DNOHTTPACCEPT 90273 43565 76.5 MB 8.2 MB /usr/local/sbin/httpd -DNOHTTPACCEPT 90281 43565 76.5 MB 8.2 MB /usr/local/sbin/httpd -DNOHTTPACCEPT 90295 43565 76.5 MB 8.2 MB /usr/local/sbin/httpd -DNOHTTPACCEPT 90296 43565 76.5 MB 8.2 MB /usr/local/sbin/httpd -DNOHTTPACCEPT 90297 43565 76.5 MB 8.2 MB /usr/local/sbin/httpd -DNOHTTPACCEPT 90298 43565 76.5 MB 8.1 MB /usr/local/sbin/httpd -DNOHTTPACCEPT 90301 43565 76.5 MB 8.1 MB /usr/local/sbin/httpd -DNOHTTPACCEPT 90307 43565 76.5 MB 8.1 MB /usr/local/sbin/httpd -DNOHTTPACCEPT 90308 43565 76.5 MB 8.1 MB /usr/local/sbin/httpd -DNOHTTPACCEPT

-------- Nginx processes ---------

----- Passenger processes ------

PID VMSize Resident Name

1692 15.2 MB 3.1 MB PassengerWatchdog 1694 17.5 MB 3.7 MB PassengerHelperAgent 1696 29.4 MB 9.5 MB Passenger spawn server 1697 24.4 MB 6.0 MB PassengerLoggingAgent 84462 190.6 MB 131.7 MB Rack: /usr/local/docs/arc/current 88558 161.2 MB 107.1 MB Rack: /usr/local/docs/arc/current 90245 148.0 MB 98.4 MB Passenger ApplicationSpawner: /usr/local/docs/arc/current 90253 159.2 MB 105.3 MB Rack: /usr/local/docs/arc/current 94888 15.2 MB 3.2 MB PassengerWatchdog 94890 21.1 MB 7.5 MB PassengerHelperAgent 94892 29.4 MB 10.6 MB Passenger spawn server 94893 24.4 MB 6.3 MB PassengerLoggingAgent

[wakemaster39]

Yes RSS is correct, it must be an issue with FreeBSD and not reporting the RSS. On my linux boxes, you get an output like this. -------- Apache processes ---------

PID PPID VMSize Private Name

2229 1 90.8 MB 0.2 MB /usr/sbin/apache2 -k start 2259 2229 90.8 MB 0.3 MB /usr/sbin/apache2 -k start 2260 2229 90.8 MB 0.3 MB /usr/sbin/apache2 -k start 2261 2229 90.8 MB 0.3 MB /usr/sbin/apache2 -k start 2262 2229 90.8 MB 0.3 MB /usr/sbin/apache2 -k start 2263 2229 90.8 MB 0.3 MB /usr/sbin/apache2 -k start 2392 2229 90.8 MB 0.3 MB /usr/sbin/apache2 -k start 2408 2229 90.8 MB 0.3 MB /usr/sbin/apache2 -k start 2409 2229 90.8 MB 0.2 MB /usr/sbin/apache2 -k start

Processes: 9

Total private dirty RSS: 2.19 MB

That RSS is what it is looking for, though I am not sure why it isn't there on FreeBSD must have to do with memory management differences between the two kernels. I suppose it would be possible to do a look and add up all the VMSize memory, but I feel this is an issue with the passenger gem and not the munin plugin, though that is only my opinion.

[dogenpunk]

Checking the Passenger source, there's a call to #platform_provides_dirty_rss_information? that basically just checks to see if ruby_platform =~ /linux/. I'll look into seeing if there isn't a different way to determine this, but I'm guessing that I may be out of luck. Would you be interested in me updating the code to tally these numbers on non-linux platforms?

_Update_ I need to start checking the docs before I make these comments. From the Passenger User Guide:

Private dirty RSS reporting only works on Linux. Unfortunately other operating systems don’t provide facilities for determining processes' private dirty RSS. On non-Linux systems, the Resident Set Size is reported instead.

So I'm guessing that these guys know what they're talking about. Would updating this plugin to report Resident Set Size on non-linux OSes be alright with you? If so, I'll close this and submit a pull request when I have it done.

Thanks for your help!

[barttenbrinke]

Wow, this issue actually resolved itself :)! Thanks Wakemaster39! I'll add a note to the docs referring to the passenger user guide. The rails 3 log format specification is allready in the docs, but as this is the second ticket about this I think i'll put env.log_format rails3 in as default. As this is probably what people expect.

As for the plugin update.. pull requests are allways welcome. It really depends on 1) the amount of code it will take and 2) if it is possible at all (given the Passenger docs comment).