SHMACK
SHMACK copied to clipboard
Make data exchange workspace with cluster work again / improve it
Right now, sync'ing local files from workspace with cluster fails.
The good news: Mesosphere did improve default security setup on DC/OS 1.7 compared to prior releases. The bad news: You have to access any node in the cluster via a proxy - in particular also for SSH.
For interactive SSH sessions, that's no big deal as "dcos node ssh --master-proxy --leader" as used in ssh-into-dcos-mater.sh is already available and in use.
But for the scripts using rsync, some reworking would be needed to make them work again (infrastructure tests relying on that currently fail). It may be possible to solve this issue - but probably we should switch to a different solution altogether, e.g. using S3 - which could also help providing non-volatile storage for big inout and result data.
Workaround for non-critical data is described in https://github.com/Zuehlke/hackzurich-sensordataanalysis/blob/master/S3ForSparkSubmit.md