xp icon indicating copy to clipboard operation
xp copied to clipboard

Create dump straight to remote location

Open gbbirkisson opened this issue 4 years ago • 11 comments

There is this huge issue of creating dumps when the disk space is to low.

Be able to create system dumps that are streamed straight to remote storage, using i.e.:

  • S3 compatible object storage
  • SFTP

gbbirkisson avatar Feb 18 '21 16:02 gbbirkisson

Is it so low disk that even #7312 can't help?

Also, could probably be done via OS mounts (s3fs, sshfs and similar)

rymsha avatar Feb 18 '21 17:02 rymsha

Yeah this is one of those things you don't really plan for. The disk is almost full, so you cant create a dump, but you want to do it to migrate the data or do a backup. You don't want to restart because its a production server. OS mounts are a possibility in some cases but a nuisance in others (K8s). We are having this scenario with some customers now.

Being able to point to a remote location would save us a lot of headaches.

gbbirkisson avatar Feb 19 '21 08:02 gbbirkisson

There is a security concern: If dump can be made to virtually any location when it is easier to steal this dump, by specifying location "to outside world" So, this issue is covered by #7654 but not fixed just yet due to https://github.com/enonic/xp/issues/7654#issuecomment-855813301

rymsha avatar Dec 12 '21 14:12 rymsha

This of course depends on how you facilitate this remote location. I would for example argue that S3 buckets are safe.

gbbirkisson avatar Dec 13 '21 09:12 gbbirkisson

I don't see how S3 is safer than anything else, if this S3 bucket is owned by a thief

rymsha avatar Dec 13 '21 09:12 rymsha

I obviously assume that you would never configure XP to upload a dump to a random location on the internet. You would point to a location you own. I don't quite get your argument here. What is the scenario you are thinking about?

gbbirkisson avatar Dec 13 '21 09:12 gbbirkisson

Ah. we are talking about the same things here. It is just me not understand. :)

The issue is blocked by #7654, anyway. dump location is not configurable (yet).

rymsha avatar Dec 13 '21 09:12 rymsha

Is it still important, BTW? Dumps are not recommended for backups anymore.

rymsha avatar Dec 13 '21 09:12 rymsha

It is mostly useful in migration processes AFAIK, unless we find a way to upgrade without dump/load. So problem is if the current disk is not big enough and similar. So - I guess it is still relevant.

sigdestad avatar Dec 13 '21 09:12 sigdestad

I read it as - "not important" then. This feature will not land on 6.x . So we are talking about 7.x to 8.x upgrade - 8.0 does not exist.
Upgrade is done "once" And usually allows some compromises, including disk re-mount, restart, etc. Implementing a feature (with extra library dependencies, like sftp, s3) that is needed only once it 5 years - is a waste of time.

rymsha avatar Dec 13 '21 10:12 rymsha

I guess we can look into this as part of XP8. Ability to perform upgrades in a smoother way seems to be relevant anyhow

sigdestad avatar Dec 13 '21 11:12 sigdestad