Jobs visualization

[jamesmontalvo3]

Create special page that visualizes job queue. For example, paste the following into JS console (modifying the variables at the top as required):

// --------- modify below as req'd --------

// var wiki = "demo";
var wiki = window.location.href.split('/')[3];
// var fqdn = "example.com";
var fqdn = window.location.href.split('/')[2];
var refreshSeconds = 10;

// ----------- no changes below -----------

$("#content").html('<div style="width:100%;"><canvas id="canvas"></canvas></div>');
$("canvas").css({
	"-moz-user-select": "none",
	"-webkit-user-select": "none",
	"-ms-user-select": "none"
});

function getRemainingJobs ( callback ) {
	$.get(
		"https://" + fqdn + "/" + wiki + "/api.php?action=query&meta=siteinfo&format=json&siprop=statistics",
		{},
		callback
	);
}

$.getScript("https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.13.0/moment.min.js", function () {
	$.getScript("https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.6.0/Chart.min.js", function () {
		getRemainingJobs( function ( mwApiResponse ) {

			var now = new Date();
			var timeFormat = 'HH:mm:ss';
			var config = {
				type: 'line',
				data: {
					labels: [
						new Date(), 
					],
					datasets: [{
						label: "Jobs",
						// backgroundColor: window.chartColors.red,
						// borderColor: window.chartColors.red,
						data: [ mwApiResponse.query.statistics.jobs ],
						fill: false,
					}]
				},
				options: {
					responsive: true,
					title:{
						display:true,
						text:'Jobs Chart'
					},
					tooltips: {
						mode: 'index',
						intersect: false,
					},
					hover: {
						mode: 'nearest',
						intersect: true
					},
					scales: {
						xAxes: [{
							type: "time",
							time: {
								format: timeFormat,
								// round: 'day'
								tooltipFormat: 'll HH:mm:ss'
							},
							scaleLabel: {
								display: true,
								labelString: 'Date'
							}
						}],
						yAxes: [{
							// display: true,
							scaleLabel: {
								display: true,
								labelString: 'Jobs'
							}
						}]
					}
				}
			};

			var ctx = document.getElementById("canvas").getContext("2d");
			window.myLine = new Chart(ctx, config);

			setInterval( function () {
				getRemainingJobs( function ( mwApiResponse ) {
					var now = new Date();
					config.data.labels.push( now )
					var newData = {
						x: now,
						y: mwApiResponse.query.statistics.jobs
					};
					console.log( newData );
					config.data.datasets[0].data.push( newData );
					window.myLine.update();
				});
			}, refreshSeconds * 1000 );

		
		});
	} );
} );

[jamesmontalvo3]

Consider method to output raw data. This will work currently:

var dataset = myLine.config.data.datasets[0].data;
var output = "";
for (var i = 1; i < dataset.length; i++ ) {
    output += dataset[i].x.toLocaleString() + "\t" + dataset[i].y + "\n";
}
console.log( output );

[jamesmontalvo3]

Also maybe add an item to personal URLs for admins, showing how many jobs are pending. Might be a good thing to be able to keep an eye on. Maybe make it optional, though. Or only show it if it's over some threshold.

[jamesmontalvo3]

Here's some analysis of a particular run of jobs that just completed. I think the meza job runner is too conservative. For one, the 30 seconds max per wiki, when running the script every 180 seconds, means that if only one wiki has jobs then it can only run at 1/6 speed. Since the order of the wikis is random, it should be okay to allow a wiki to use the whole time (or at least most of it). However, we have to be careful of one wiki taking the max time minus 1 second, then the following wiki starting (since we haven't exceeded max time) and taking it's max time. This would result in a big overlap of two run-jobs sessions running. Then again, perhaps the max-CPU setting would prevent that.

[jamesmontalvo3]

The same change was made here, but with different job-runner specs, and no explicit running of runJobs.php (just what the cron job does). Specs:

• cron runs every three minutes (same as previous)
• max time for a wiki = 3 minutes
• max time for script = 3 minutes, though this could be exceeded if the last wiki to run starts at 2:59 and runs for its full three minutes.
• max jobs per wiki = effectively infinite
• max CPU = 200% (same as previous, saturate 2 CPUs)

This worked a lot better, of course. However, CPU was pretty high throughout. Might need to knock the max-CPU down to 180% or so just to keep spikes away from saturation of all 4 CPUs (in this particular setup) and drastically affecting response times.

[freephile]

Nice feature! LGTM

[jamesmontalvo3]

@freephile thanks. @krisfield said he is going to extension-ify it.