cloudserver
cloudserver copied to clipboard
Published
20 hours ago •
scality
scality
Create directories from environment variables
Description
I'm using the scality/s3server image and I find the behavior of the S3METADATAPATH and S3DATAPATH environment variables not following common patterns for container images.
Steps to reproduce the issue
$ docker run --rm -v export:/export -e S3DATAPATH=/export/localData -e S3METADATAPATH=/export/localMetaData scality/s3server
Actual result
Output:
npm info it worked if it ends with ok
npm info using [email protected]
npm info using [email protected]
npm info lifecycle [email protected]~prestart: [email protected]
npm info lifecycle [email protected]~start: [email protected]
> [email protected] start /usr/src/app
> npm-run-all --parallel start_dmd start_s3server
npm info it worked if it ends with ok
npm info using [email protected]
npm info using [email protected]
npm info it worked if it ends with ok
npm info using [email protected]
npm info using [email protected]
npm info lifecycle [email protected]~prestart_dmd: [email protected]
npm info lifecycle [email protected]~prestart_s3server: [email protected]
npm info lifecycle [email protected]~start_dmd: [email protected]
npm info lifecycle [email protected]~start_s3server: [email protected]
> [email protected] start_dmd /usr/src/app
> npm-run-all --parallel start_mdserver start_dataserver
> [email protected] start_s3server /usr/src/app
> node index.js
npm info it worked if it ends with ok
npm info using [email protected]
npm info using [email protected]
npm info it worked if it ends with ok
npm info using [email protected]
npm info using [email protected]
npm info lifecycle [email protected]~prestart_mdserver: [email protected]
npm info lifecycle [email protected]~start_mdserver: [email protected]
> [email protected] start_mdserver /usr/src/app
> node mdserver.js
npm info lifecycle [email protected]~prestart_dataserver: [email protected]
npm info lifecycle [email protected]~start_dataserver: [email protected]
> [email protected] start_dataserver /usr/src/app
> node dataserver.js
{"name":"S3","time":1505845693588,"level":"warn","message":"scality kms unavailable. Using file kms backend unless mem specified.","hostname":"fb0c354091d0","pid":49}
{"name":"MetadataFileClient","time":1505845693599,"level":"info","message":"connecting to metadata service at http://localhost:9990/MDFile/metadata","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693773,"workerId":1,"workerPid":102,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693779,"workerId":2,"workerPid":108,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693796,"workerId":3,"workerPid":109,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693822,"workerId":4,"workerPid":120,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693831,"workerId":5,"workerPid":126,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693843,"workerId":6,"workerPid":132,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693854,"workerId":7,"workerPid":133,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693880,"workerId":8,"workerPid":139,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693901,"workerId":9,"workerPid":150,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693914,"workerId":10,"workerPid":156,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
fs.js:249
binding.access(pathModule._makeLong(path), mode);
^
Error: ENOENT: no such file or directory, access '/export/localMetaData'
at Error (native)
at Object.fs.accessSync (fs.js:249:11)
at MetadataFileServer.startServer (/usr/src/app/node_modules/arsenal/lib/storage/metadata/file/MetadataFileServer.js:124:12)
at Object.<anonymous> (/usr/src/app/mdserver.js:17:14)
at Module._compile (module.js:570:32)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.runMain (module.js:604:10)
npm info lifecycle [email protected]~start_mdserver: Failed to exec start_mdserver script
npm ERR! Linux 4.9.41-moby
npm ERR! argv "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/bin/npm-cli.js" "run" "start_mdserver"
npm ERR! node v6.11.2
npm ERR! npm v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! [email protected] start_mdserver: `node mdserver.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start_mdserver script 'node mdserver.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the s3 package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node mdserver.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs s3
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls s3
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! /usr/src/app/npm-debug.log
{"name":"DataFileStore","time":1505845694281,"error":{"errno":-2,"code":"ENOENT","syscall":"access","path":"/export/localData"},"level":"error","message":"Data path is not readable or writable","hostname":"fb0c354091d0","pid":92}
{"name":"S3","time":1505845694282,"error":{"errno":-2,"code":"ENOENT","syscall":"access","path":"/export/localData"},"level":"error","message":"Error initializing REST data server","hostname":"fb0c354091d0","pid":92}
npm info lifecycle [email protected]~poststart_dataserver: [email protected]
npm info ok
ERROR: "start_mdserver" exited with 1.
npm info lifecycle [email protected]~start_dmd: Failed to exec start_dmd script
npm ERR! Linux 4.9.41-moby
npm ERR! argv "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/bin/npm-cli.js" "run" "start_dmd"
npm ERR! node v6.11.2
npm ERR! npm v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! [email protected] start_dmd: `npm-run-all --parallel start_mdserver start_dataserver`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start_dmd script 'npm-run-all --parallel start_mdserver start_dataserver'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the s3 package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! npm-run-all --parallel start_mdserver start_dataserver
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs s3
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls s3
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! /usr/src/app/npm-debug.log
ERROR: "start_dmd" exited with 1.
npm info lifecycle [email protected]~start: Failed to exec start script
npm ERR! Linux 4.9.41-moby
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "start"
npm ERR! node v6.11.2
npm ERR! npm v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! [email protected] start: `npm-run-all --parallel start_dmd start_s3server`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script 'npm-run-all --parallel start_dmd start_s3server'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the s3 package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! npm-run-all --parallel start_dmd start_s3server
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs s3
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls s3
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! /usr/src/app/npm-debug.log
Expected result
Docker entrypoint script creating the non-existing directories mentioned in the output:
Error: ENOENT: no such file or directory, access '/export/localMetaData'
Additional information: (Node.js version, Docker version, etc)
$ docker version
Client:
Version: 17.06.2-ce
API version: 1.30
Go version: go1.8.3
Git commit: cec0b72
Built: Tue Sep 5 20:12:06 2017
OS/Arch: darwin/amd64
Server:
Version: 17.06.2-ce
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: cec0b72
Built: Tue Sep 5 19:59:19 2017
OS/Arch: linux/amd64
Experimental: false
Feature Request
Create the directories specified in the environment variables if they are set and are not already created.
Use case
The common patterns found with other container images is to be smarter about these things and not force the user to customize the entrypoint.
Thanks for the input! Since these directories will contain persistent data, our thought was that if people did not want to go with the default localData/localMetadata, they would want to set up the directories themselves, but we can revisit that approach.
Thank you for your reply. Since localData and localMetaData are shipped empty, they should be created by the entrypoint script, taking the environment variables into account.
$ docker run --rm -it scality/s3server bash -c "ls -al local*"
localData:
total 12
drwxr-xr-x 2 root root 4096 Sep 21 20:38 .
drwxr-xr-x 1 root root 4096 Aug 15 19:52 ..
-rw-r--r-- 1 root root 71 Aug 15 19:51 .gitignore
localMetadata:
total 12
drwxr-xr-x 2 root root 4096 Sep 21 20:38 .
drwxr-xr-x 1 root root 4096 Aug 15 19:52 ..
-rw-r--r-- 1 root root 71 Aug 15 19:51 .gitignore