GCViz seems to be completely tied to Netflix's system structure

[richievos]

I was interested in setting up GCViz to do some garbage collection analysis, and in doing so have found it seems to be completely tied to internal Netflix configurations. Examples:

• It has a bunch of assumptions on paths being a certain way (search for /mnt/logs or /apps/apache/htdocs/AdminGCViz/)
• netflix environment variables (/etc/profile.d/netflix_environment.sh)
• netflix S3 config (s3://netflix.bulkdata.), ...

In general, it seems like this would require some pretty significant changes (or significant amounts of documentation) to be generally usable.

Is there any plans to tweak it to be a general purpose tool versus a Netflix one? If not, it seems like a disclaimer may be warranted.

[mooreb]

/etc/profile.d/netflix_environment.sh is used to determine if the environment is a netflix environment.

/apps/apache/htdocs/AdminGCViz/ is not required; you can put the index and generate files anywhere you like on your web server.

If that file can't be found, the gcviz is put into non-netflix mode.

There is the hardcoding of /apps/tomcat/logs/gc.log and the archived files under /apps/tomcat/logs/archive/* which is an issue.

I'm not sure how to generally address these points; I did write in: https://github.com/Netflix/gcviz that the gc log file location was required with: -Xloggc:/apps/tomcat/logs/gc.log and I also wrote: "I wrote gcviz to address challenges we face inside Netflix."

It works for us but if you want to submit patches that make it work for both of us then I will certainly consider those patches.

b

[rwonly]

hi Brian, I am trying this great tool on my Ubuntu 12.04, jdk1.6, python2.7, but got the following errors: Traceback (most recent call last): File "/apps/apache/htdocs/AdminGCViz/visualize-gc.py", line 167, in fig.autofmt_xdate() File "/usr/lib/pymodules/python2.7/matplotlib/figure.py", line 329, in autofmt_xdate for label in ax.get_xticklabels(): File "/usr/lib/pymodules/python2.7/matplotlib/axes.py", line 2530, in get_xticklabels self.xaxis.get_ticklabels(minor=minor)) File "/usr/lib/pymodules/python2.7/matplotlib/axis.py", line 1109, in get_ticklabels return self.get_majorticklabels() File "/usr/lib/pymodules/python2.7/matplotlib/axis.py", line 1093, in get_majorticklabels ticks = self.get_major_ticks() File "/usr/lib/pymodules/python2.7/matplotlib/axis.py", line 1191, in get_major_ticks numticks = len(self.get_major_locator()()) File "/usr/lib/pymodules/python2.7/matplotlib/dates.py", line 749, in call self.refresh() File "/usr/lib/pymodules/python2.7/matplotlib/dates.py", line 758, in refresh dmin, dmax = self.viewlim_to_dt() File "/usr/lib/pymodules/python2.7/matplotlib/dates.py", line 530, in viewlim_to_dt return num2date(vmin, self.tz), num2date(vmax, self.tz) File "/usr/lib/pymodules/python2.7/matplotlib/dates.py", line 289, in num2date if not cbook.iterable(x): return _from_ordinalf(x, tz) File "/usr/lib/pymodules/python2.7/matplotlib/dates.py", line 203, in _from_ordinalf dt = datetime.datetime.fromordinal(ix) ValueError: ordinal must be >= 1

I googled but got no matches. Did you meet this issue and any clue about that?

-Rex

[mooreb]

Hi Rex,

I am sorry to see that you're having trouble.

Which version of matplotlib is installed? At netflix we use 1.1.0.

You can check for yourself by examining the output of:

import matplotlib help(matplotlib) ...

VERSION 1.1.0

This call ( fig.autofmt_xdate() ) doesn't have any data dependencies so it seems to me that it's possible that matplotlib is either misinstalled or there's a forward/backward compatibility problem.

Hoping this helps and finds you well,

b

[rwonly]

Hi, Brian, I found this is due to the gc log was not correctly parsed, so there is no data to show. Because the regular expression used in the perl scripts to match the date time is hard coded to "+0000", but the log generated in my local machine is +0800. So no lucky. Anyway, finally I got the codes worked and generated 2 png pics. I am not sure if my pics display enough information. So attach two, and could you show me yours?

-Rex

[mooreb]

Hi rwonly,

congrats on getting the PNGs! That indicates that you are most of the way there.

Please look at your gc logs and see if you have more than five events over the six second period (between 5:51:54 and 5:52:00) there. Did you expect more than six seconds of activity?

If the gc logs have more activity than that then please look for rejects/rejected files in the gc reports directory. Some of these are: gcdotlog_extract_sizes_rejected_lines gcdotlog_extract_time_rejected_lines

If the rejected files are large then there may still be a problem with the parser.

Hoping this helps and finds you well,

b