/tmp should be a volume in the per-grain storage (so it can be larger in size) but retain the semantics of being empty when the app starts

[paulproteus]

Right now, it's easy for apps to use a lot of space in /tmp, and then run into a lack of disk space on /tmp.

Proposal: Use a directory in the grain storage, so all the following is true:

• If apps decide to use a lot of space on /tmp, the app is using resources that we already know how to limit (namely the grain storage directory). (AKA: simple implementation for us)
• Apps can make big files and this is likely to work for them (AKA: low cognitive load for app maintainers)

There are some alternatives, but I think they're noticeably worse because they don't have the above two properties.

UPDATE:

• We should clear /tmp for the grain on app start, or app shut-down, or something like that, so that we retain the important property that /tmp is empty on app start. (I don't have an opinion personally on specifically when or how we do the cleaning of /tmp.)

[paulproteus]

FWIW @zarvox linked me to http://0pointer.de/blog/projects/tmp.html while thinking about this.

[zarvox]

I'm actually -1 on this. I think it moves us in a direction that is less likely to be well-supported in the future, and moves us further from what distributions are likely to provide by default moving forward. Additionally, it adds a durability requirement that app developers may rely on, which makes revising this decision later much more difficult.

Multiple distributions have already made /tmp a tmpfs. https://fedoraproject.org/wiki/Features/tmp-on-tmpfs Most software will successfully deal with this out-of-the-box, so I posit that most apps won't actually have any problem here. Said another way: apps which decide to use a lot of space on /tmp will already hit problems on other much more widely used platforms, so I don't think property 1 is particularly convincing.

/tmp should have the "automatically clears between runs" semantics, which we'd have to either add manually (rm -rf by supervisor on run? rm -rf when we kill the grain, to cost the user less storage?) or choose not to provide, which can result in higher cognitive load.

Finally, we already have to account for RAM as well as disk usage, so I'm not convinced that the implementation is actually any simpler. On the upside, it will cost our users less if we use RAM than disk space, since it won't still be taken up when running.

[paulproteus]

I agree that /tmp should be empty on grain start. I fixed the original issue to add that. Thanks for catching the importance of that!

The rest I'll try to respond to later, e.g. tomorrow.

[paulproteus]

Conclusion from a conversation between @kentonv @paulproteus + @zarvox - for now, we'll make sure MySQL has a happy enough life in vagrant-spk by setting its temporary directory to /var/tmp.

@kentonv and @paulproteus think that if there are any other apps/etc. that need similar patching, then the platform should change in accordance with this ticket. @zarvox is not yet fully convinced, but is open to that as a possibility if that time comes.

[FiloSottile]

The small /tmp makes PIWIK pretty much unusable for blog.filippo.io every day I have a spike. So +1 on a bigger /tmp, and is there a workaround in the meantime?

[zarvox]

I've updated the Piwik package to put MySQL's tmpdir on /var/tmp, so applying updates should sort things out for Piwik.

[zenhack]

Since @ocdtrekkie brought this up in another issue: It would be nice to make it so that apps aren't limited to such a small /tmp; "/tmp is too small" continues to be one of the more common gotchas in packaging stuff for Sandstorm (mysql isn't alone in not liking this).

That said, I'm wary of doing more work at grain startup time, as this is a place where many apps already have poor performance, in a place that really affects user experience. I don't want to make it worse, and if forced to choose I'd rather have app packagers continue to have to work around this than have their users have to deal with slower apps.

Is my memory correct that the reason it is so small is because we don't have a good way of telling the kernel who should be charged for the memory it takes up?

[xet7]

@zenhack

It would be so super cool if there can be increased size of /tmp and I would not need to make any /tmp related code changes to Sandstorm WeKan !!!!