Proposal: Serve results via a webserver (from PR #779 discussion)

[sarahec]

Prior discussion

What if the fix for headless servers is to write the page and test results (with unique names) to an external directory where they can be served via a web server? You can pass the result file name in a url parameter.

This would enable running batches of tests and reviewing them at leisure. It also opens the door to comparing the results of changes.

Emery:

Right now, the solution for headless servers is to output a JSON file (e.g., --json --outfile foo.json, or --no-browser which outputs profile.json) and load that file from scalene --viewer. I didn't quite follow the rest of your proposal, or at least how it would work in a common enough use case -- connection to some machine on Amazon or Azure or Google Cloud, where hosting a webserver is doable but maybe fraught?

Unpacking the idea:

1. A web server often has a static files directory (e.g. .html, .js,, .css).
2. The viewer page and its supporting files don't have to be assembled by scalene, but can live as static files.
3. We still have to get the data into the viewer page, perhaps as a URL parameter. (Modern browsers also have a built-in file picker the page can use.)
4. On a headless server, the setup can include a web server that sees that directory. Locally, in Python, the built-in web server module also supports static directories.
5. Now the browser launch becomes open("https://${host}:${port}/viewer.html?profile=${profile-path}") . It works the same way locally or on the development server. Locally could use a temp dir and fixed name by default.
6. You can support batch operation, etc. by writing the profile file as profile{$timestamp}.json or, optionally, profile-$appname-$timestamp.json with the prefix coming from a command-line flag.
7. This assumes you have a default port number that can also be changed with a command-line flag.
8. If you're feeling wild and crazy, perhaps the app could use an optional configuration file.
9. Extreme option: allow gzipping the JSON files. The browser will decompress these automatically on load.

Granted, it's a lot of little changes -- changing page generation, how the page gets its data, updating the server, documentation, etc. -- so it could qualify as a major release. Having a series of runs available enables comparisons between them, taking multiple runs and averaging results, etc.

P.S. --outfile foo.json, --no-browser, and scalene --viewer could remain grandfathered in.

[sarahec]

Take #691 into consideration. Rename? Overwrite?

[sarahec]

re:#691 if an outfile name is supplied, pass it to the browser (as is being done now.)

[emeryberger]

Right now, the Scalene profile page is generated from a template with Jinja - the reason for this is that I want it to create a self-contained HTML file that is easily shareable. Is this doable in your proposal?

[sarahec]

TL;dr yes, it's easy

The HTML page is capable of loading and displaying the profile at run time, you could also merge profiling data and other inline content to generate a page.

I see three main use cases for shipping profile data files with an external viewer:

1. A lab server -- people run their tests on the remote server and it writes a series of profiles to a specified directory. A web server on the remote hosts the html page and people can load their profiles by path. This is especially good for overnight runs such as profiling machine learning code.
2. Parameterized tests. I recently tested some ML code through a range of settings (memory, number of threads, etc.) and recorded the results. Being able to generate a set of profiles and compare them in the viewer (page, app, whatever) would be awesome.
3. Test-and-iterate. I might start with one profile, make code changes, and run another profile. Then compare the results.

In any case, if you have a viewer page that can fetch a profile (as now), it's still possible to have a tool embed the data in a page to generate a new shareable page.