Serve static files from web server

[wardi]

We should serve static files with a web server, not a Python wsgi process. Even better if we can serve them from a different machine or service.

[adamamyl]

Big +1 here (have active interest in this)

[davidread]

Good call. Also, using putting them on another, cookieless domain is enabled ( https://developers.google.com/speed/docs/best-practices/request )

Currently the static files live in ckan/public and plugins call p.toolkit.add_public_directory() with the path. I guess it's doable to add these paths to the config for nginx (or whatever) to get it to serve these directories by listing them manually. However it is a bit of manual chore and breaks the DRY principal - is there a better way?

[nigelbabu]

How would we get this to work with extensions? Get fanstatic to dump all the static files into one location and then serve that with a web server?

[rossjones]

Perhaps a plugin interface for defining your 'media', so that it gets pulled into the correct place at startup? Perhaps at plugin installation so that it can be minified etc?

[rossjones]

In an effort to get this discussion underway again and as there was a lot of discussion about this in #95, - I think in the end there was a preference from the anti-fanstatic club for:

1. A single location for serving static content (when using paster or for httpd in production) such as /static, but was unclear how plugins would get their static files into that location (as David pointed out above)
2. Clarification for plugin authors that their templates should have something like:

{% block styles %}
    <link rel="stylesheet".....
  {{super()}}
{% endblock style %}

and something similar for JS in their templates to include their own resources.

Apart from the question of how plugin static files get put into the main static folder this seems reasonably simple.

There's also the question of how static files are minified and/or merged, if at all. Perhaps whatever is putting the plugins static files into the /static folder might be responsible for that?

@rshk is that a fair summary? If we were to try and get this into 2.4 might we be able to remove fanstatic in 2.5?

[davidread]

I'm not well acquainted with CKAN's use of fanstatic, but from reading about it there might be hope:

1. You don't have to get fanstatic to serve the assets via WSGI (as we do now). You can configure fanstatic to 'compile' assets during a request, if the source asset has changed. This is the lazy option for production. You also have the option to run fanstatic-compile on the command-line to write a bundle. Either way, you get the compiled asset on disk to serve using nginx. See: http://www.fanstatic.org/en/latest/compilers.html
2. Rather than a different JS and CSS bundle for each page, we want site-wide bundles. From the description of options 'rollup' and 'eager_superseder' they may cover this. See: http://www.fanstatic.org/en/latest/configuration.html

So maybe with a bit of hacking we might get what we want, without the disruption of changing from fanstatic, with the convenience of not needing to do a new bundling step on deploy or when installing an extension.

[wardi]

How about we add a ckan.static_url option to the config file. If present then url_for_static would prefix static URLs with that domain and/or partial path, and the fanstatic middleware that mixes static files in with page urls would be disabled. Users would need to run fanstatic-compile configure their web server to serve the resulting files from that path.

We would have to update the example http://docs.ckan.org/en/latest/theming/static-files.html to encourage use of url_for_static instead of hard-coding paths in templates.

[davidread]

I've collect together the threads and put together this anti-fanstatic proposal:

1. CKAN-core and every extension store all static files (images, css & js) in a standard named directory: /ckanext//static/ (and sub-directories). The understanding is that the CSS & JS will be bundled and loaded site-wide. However if any of those CSS & JS files are too large for site-wise bundling, then some config can be used to tell CKAN not to bundle them.
2. When running paster in debug mode, WSGI serves all the static directories. If you run in production mode then you must call a paster function to 'collect' the assets into a central location and you use apache/nginx/CDN/whatever to serve that. The 'collect' also makes a site-wide bundle each of JS and CSS, excluding those particular ones configured. (Under the hood we might use django pipeline, or generate a gruntfile and run grunt.)
3. A template that refers to an asset (e.g. an image or for loading a JS/CSS file not in the bundle) uses a Jinja function to allow the static serve location be moved (e.g. to a CDN):

{% block styles %}
    <link rel="stylesheet" src="{% static "myextension/bigscript.js" %}" />
  {{super()}}
{% endblock style %}

Compared to now, this proposal would add the new "collect" step to CKAN deployment or when installing an extension, plus a bit of nginx/apache config. We could make that optional by asking WSGI to serve the files in-place in production. But since WSGI serving completely unbundled files is a further step back compared to fanstatic, we'd have slower CKANs running, and that just doesn't look good for the CKAN community. So I think we should bite the bullet and accept this step as mandatory.

For advanced users, or for CKAN in the future, the 'collect' step could also be used to process images for different screen resolutions and mobile, let alone doing less/sass & coffeescript. That would mean that the 'collect' step would be needed during development as well, so developers would have to remember to run it each time or keep 'grunt watch' running in the background. But it's not necessarily needed now.

This proposal follows closely the Django conventions https://docs.djangoproject.com/en/dev/howto/static-files/ (aside from the bundling, which you'd use an extensino for). Django ways are familiar to hordes of developers, and gives the proposal some weight.

@amercader said in #95 :

The most important thing is that extensions with static files should just work, reasonably well without any custom tweaking. They should have at least the option to use minified code and be able bundle at least their own files. The big difficulty here is that the extension need some way of expressing not just what static files they have, but also which templates/routes they need to load those static files in as well as where they need to be placed in those templates. At the moment this is done with Resource tags, and those tags sort out dependency order too for javascript. Any new attempt needs to address this issue.

This proposal only needs configuration for the case where an asset is not bundled. It does 'just work'.

You just have to accept that you'd run the 'collect assets' command on deploy. Asset pipelines are pretty common practice on the web now, and growing in importance with responsive & hi-res images etc.

I wonder if fanstatic is unnecessarily too clever and complicated. I don't see it as the direction that the vast majority of sites are going down.

Having a site-wide bundle for JS (or one for top and one for bottom) reduces complexity of JS dependencies - I just don't see this being an issue at all. We might need some config to set the ordering, so that JQuery comes before other bits. But are you suggesting that there are some libraries you simply can't put together in the same bundle?

[davidread]

@wardi sounds good in principle

[KrzysztofMadejski]

Have there been any followup on this?