SHMACK icon indicating copy to clipboard operation
SHMACK copied to clipboard

Make data exchange workspace with cluster work again / improve it

Open mspringma opened this issue 8 years ago • 1 comments

Right now, sync'ing local files from workspace with cluster fails.

The good news: Mesosphere did improve default security setup on DC/OS 1.7 compared to prior releases. The bad news: You have to access any node in the cluster via a proxy - in particular also for SSH.

For interactive SSH sessions, that's no big deal as "dcos node ssh --master-proxy --leader" as used in ssh-into-dcos-mater.sh is already available and in use.

But for the scripts using rsync, some reworking would be needed to make them work again (infrastructure tests relying on that currently fail). It may be possible to solve this issue - but probably we should switch to a different solution altogether, e.g. using S3 - which could also help providing non-volatile storage for big inout and result data.

mspringma avatar Jun 04 '16 15:06 mspringma

Workaround for non-critical data is described in https://github.com/Zuehlke/hackzurich-sensordataanalysis/blob/master/S3ForSparkSubmit.md

mspringma avatar Sep 16 '16 07:09 mspringma