aiming to close spot-23

[natedogs911]

QA and Review and advise on any changes please, This will require some doc changes.

known issues:

• spot-nfdump must be fixed upstream and version number changed in spot-ingest/install.sh

TBD:

• error handling needs to happen in all scripts. need an error handling function
• would like to keep installs down to 2 scripts, looking for feedback on better ways to do this
• need a better solution then spot-ml/post_build.sh to install in /opt/spot/bin
• add launch scripts for each component in /opt/spot/bin (i.e. start ingest, run OA etc)

Discussion:

should the assumption be made that all components are being landed on the same server? This would greatly simplify things as in spot-setup you could just run a script to source each components scripts. feedback would be great.

EDIT: really there is only the one known issue, nfdump is in the spot-nfdump repo that i'm pulling in and must be fixed there.

[natedogs911]

aims to close the #spot-23 jira with pr #24

[rabarona]

Spot-23: https://issues.apache.org/jira/browse/SPOT-23 (for some reason # is not linking this PR with the Jira issue).

Hey, @natedogs911 would you paste the discussion part in the JIRA issue? Also, are the known issues something happening in the code you are pushing or it's exactly what your code is fixing?

[natedogs911]

I edited my comments, nfdump is the known issue, the rest are things that i want to improve/add

[natedogs911]

Ready for review and merge...

[raypanduro]

+1

[natedogs911]

RE: @rabarona

1. README changes, Yes the web needs to be updated.

2. a little bit of both, i'd like to catch errors and notify the user "installation failed at line 22, with 'foo'". it's fairly trivial and can easily be added with more iterations.

3. I didn't change the workflow for the conf file, although i can add something like this:

# copy spot.conf if [[ -f ./spot.conf ]]; then cp ./spot.conf /etc/spot.conf log_cmd "Copied spot.conf into /etc/" else log_cmd "Please run \cp ./spot.conf.template ./spot.conf` and customize before running ./install.sh" exit 1 fi`

and change the spot.conf file to spot.conf.template

4. I think a more elegant solution is needed then just sourcing each individual installation script. this is somewhat related to my answer to question 5.

5. Should the assumption be made that all components will be run from the same node? if so then, (assuming i make the above change)

a. from the ./spot-setup run cp spot.conf.template spot.conf and edit the configurations for your specific cluster

b. run sudo ./install.sh

c. per ./spot-setup/README.md run ./hdfs_setup.sh

d. from ./spot-ingest run sudo ./install.sh

e. while in ./spot-ml run sudo ./install.sh

f. while in ./spot-oa run sudo ./install.sh

If you want to run a particular component such as spot-ingest on another node, simply copy /etc/spot.conf to that node along with the spot-ingest folder and run the ./install.sh for that component.

[natedogs911]

how about to simplify the 6 step process outlined above we can add an --install-all option to the spot-setup/install.sh script which executes the script for each component?