A tutorial under distributed environment without Kubernetes

[Zongshun96]

Hi, Will/is there a tutorial for deployment in distributed environment without Kubernetes? Are the operator container deployment are designed to be managed by Kubernetes?

Thanks,

[balis]

Hi, other infrastructures are also supported (e.g. IaaS clouds and cloud functions on a few major cloud providers). You can also run master components (hyperflow engine) from your local machine instead of containers. However, Kubernetes support is currently most mature and well documented. We'll be adding tutorials for other infrastructures in the near future.

[Zongshun96]

Hi, can you point me to the code or existing doc about deployment on IaaS clouds? I hope to learn how you were able to schedule/deploy different operators. It doesn't seem to be done in the engine.(please correct me...)

Thank you!

[balis]

It was done with a Master-Worker model using this executor and a RabbitMQ as a job queue: https://github.com/hyperflow-wms/hyperflow-amqp-executor Here is an old tutorial that involved a lot of manual steps and needs updating: https://github.com/hyperflow-wms/hyperflow/wiki/TutorialAMQP

I guess you could try this tutorial with the following adjustments (not tested, it will likely blow up):

• Install Hyperflow from the master branch (or latest release tag)
• Omit the steps that download Montage data and generate the workflow file -- just use the zip archives from the montage-workflow repository.
• If you want to configure more nodes, you can start RabbitMQ and Redis on some master node and the AMQP executor and Montage software on one or more worker nodes
• On multiple nodes, you'll need to configure NFS or Amazon S3 storage.
• Run the workflow with HF_VAR_function=amqpCommand hflow run workflow.json

[Zongshun96]

Thank you! I was able to run on a single ubuntu 16.04 node following your steps, while I need to add

/home/cc/hyperflow/montage-workflow/software/Montage_v3.3_patched_4/bin/

to may path and link

libc.musl-x86_64.so.1

to be able to run montage software. To do that I was following this issue https://github.com/alexander-akhmetov/python-telegram/issues/3

[Zongshun96]

One more question, it seems that I always need to setup executor on node(s) manually to run the workflow, right? Can you point me if there is a scheduling mechanism to setup executor remotely for IaaS node in this project? That can save me a lot of time reading.😊

Thanks,

[balis]

Thank you! I was able to run on a single ubuntu 16.04 node following your steps, while I need to add

/home/cc/hyperflow/montage-workflow/software/Montage_v3.3_patched_4/bin/

to may path and link

libc.musl-x86_64.so.1

to be able to run montage software. To do that I was following this issue alexander-akhmetov/python-telegram#3

Great! You could also build Montage yourself from the sources.

[balis]

One more question, it seems that I always need to setup executor on node(s) manually to run the workflow, right? Can you point me if there is a scheduling mechanism to setup executor remotely for IaaS node in this project? That can save me a lot of time reading.😊

Thanks,

We used to use chef for automation, there's some old code (recipes) here: https://github.com/malawski/hyperflow-deployment/tree/master/cookbooks/workflows/recipes I guess you could try to adapt some of these recipes, but they're pretty old.

In the future we'll be probably replacing the AMQP/RabbitMQ executor with something Redis-based and look into some automation solution. However, IaaS is definitely not the priority now.