aztk
aztk copied to clipboard
Azure
AZTK powered by Azure Batch: On-demand, Dockerized, Spark Jobs on Azure
If we were to create a Virtual Network, and restrict inbound access only from a bastion host, will aztk cluster be able operate without any issues? For example, for HDInsight...
As part of the strategy to support all Azure Data systems, we should make sure that BlobFuse works out-of-the-box for customers who want to mount a file system backed by...
Currently, this is always set to 1. Might be useful to expose this as a parameter and/or set to max possible value by default. https://github.com/Azure/aztk/blob/f7c1cb51729ce5347ebd7a732d5b735142c1332c/aztk/client.py#L87
https://github.com/Azure/aztk/blob/f7c1cb51729ce5347ebd7a732d5b735142c1332c/docker-image/gpu/spark1.6.3/Dockerfile#L78 This should be set on base, so it is propagated down to all other images.
I have to manually delete a task via the portal each time it completes (successfully or otherwise). Is there an option to delete a task automatically on termination? Otherwise, if...
the default on jupyter is to have the metastore created on the node – this might be a problem to retain metadata since the storage is ephemeral 
Despite everything I have tried, I still cannot figure out how to configure logging for my Spark application. The `log4j.properties` file is always ignored. ### log4j.properties in the resources If...
users should be allowed to specify flags and arguments when submitting apps. `aztk spark cluster submit --id --name /path/to/program.py -flag1 arg1 -flag2 arg2` In the above example, `-flag1 arg1 -flag2...
From the examples directory: `aztk spark cluster submit --id --name PythonPi pi.py` causes: ``` (snip) raise AzureHttpError(message, http_error.status) azure.common.AzureHttpError: The specifed resource name contains invalid characters. InvalidResourceNameThe specifed resource name...