Conda support

This is a reboot of #94 targetting the dev branch. Still WIP, discussion can start after some documentation is provided.

Coverage decreased (-2.2%) to 74.816% when pulling ea78346eb9a49fd9b5d43e7f81277612e2c99fad on mathieu1:conda_support into 1202044ee13879e9638a6b3ee2d06b7b19ab5260 on Miserlou:dev.

Coverage decreased (-2.2%) to 74.847% when pulling 618aa8e31682d492a59eee43f7617c941f4a4415 on mathieu1:conda_support into 1202044ee13879e9638a6b3ee2d06b7b19ab5260 on Miserlou:dev.

Coverage decreased (-2.3%) to 74.724% when pulling 2916e9e26f950dcd4d33f8d95eea3bd6297d097c on mathieu1:conda_support into 1202044ee13879e9638a6b3ee2d06b7b19ab5260 on Miserlou:dev.

Ok, the discussion can start. Here's the current status:

• [x] Implement condasupport, including full content of conda environment (not only python packages).
• [x] Make a conda package for zappa and for each of its dependencies (uploaded to my channel on anaconda.org). The current version of this branch is packaged under version 0.17.7.dev.
• [x] Create a repository (see mathieu1/zappa_conda) which serves both to test the current branch and can serve as an initial documentation for now, to help decide on the next steps.
• [ ] Document how to use zappa with conda on other platforms than linux-64 (and discuss limitations, e.g. Windows doesn't have symlinks)
• [ ] Work further on minifying conda packages. On this topic, available conda packages of numpy depend on large numerical optimization libraries (MKL, >100MB! or openblas, >10MB). This contributes to making it hard to fit other packages that depend on numpy such as pandas, scipy, scikit-learn, (although each of them is large on its own). A "noblas" version of numpy is underway, see conda-forge/numpy-feedstock#2

To start the discussion, here are a few questions/issues:

• Given the advantages listed in Miserlou/Zappa#94 and the fact that it does not change any behavior for non-conda users, does this PR look viable for inclusion in Zappa? Should I explain/motivate further?
• I am still unsure how to proceed with testing. The best I have in that direction is the above-mentioned repository, but it's hard to automate this testing from within zappa.

Coverage decreased (-2.6%) to 74.45% when pulling 75b98f673d4120fdf01c0c5912bb1e071dcbf2fd on mathieu1:conda_support into 1202044ee13879e9638a6b3ee2d06b7b19ab5260 on Miserlou:dev.

This is really cool progress! Sorry I haven't been very responsive on this thread.

As far as testing goes I'd recommend mocking and building out units vs trying to figure out how to make some AWS deployment for it.

Edit: testing packaging/unpackaging locally would probably be a good idea too.

It might be useful to warn the end user if there are large packages in their conda environment, too. The limits can be found here: http://docs.aws.amazon.com/lambda/latest/dg/limits.html . Note that there's a compressed as well as an uncompressed limitation.

Tried out the dev branch and did some tinkering. Could not get conda support to work for me. I'm going to stick with elastic beanstalk for now. Definitely, be back in the future when I have more time.

hello @mathieu1, At the target, would the approach you are exploring/developing allow to develop within a conda environement on whatever Conda supported system (linux/windows/MacOS, 32/64 bit, ...) and then to package in a zip file the equivalent environment for the AWS image running behind AWS lambda ? With the ability to choose "light" version to fit within AWS limits ?

@mathieu1 @Miserlou this is super exciting.

Coverage decreased (-1.1%) to 77.094% when pulling 35618b0192b7f037e46f8d7a6b7a35e239b5e004 on mathieu1:conda_support into bee26014a30883b949b8e532726a0a8f42ee59bc on Miserlou:dev.

Hey everyone. I'm the lead developer on the conda project right now. Just wanted to chime in and say I think this is awesome. I'll try to spend more time with this in the coming days. If there are any specific questions or things you are stuck on, please ask! Tagging @msarahan here too, who's the tech lead of the conda-build project.

Cool! Hey @kalefranz and @msarahan ! Thanks for stopping by, I would love to have your support on this awesome but neglected PR.

Please also feel free to stop by the Slack channel for discussion: https://slack.zappa.io/

+1 here. I think the conda support for Zappa is going to be critical for those of us who want to deploy existing applications which rely on scientific computing libraries (numpy, tensorflow et al) which are already running in these environments. It's impossible to move them to a virtuaenv format given conda ships with platform compliant versions of the lower level C libraries that speed up their python wrappers.

Zappa does support hundreds of those libraries already, however conda support would certainly improve that number. See more here: https://blog.zappa.io/posts/zappa-adds-support-for-manylinux-wheels

+1

+1 for conda support!

+1 from me too. @Miserlou What needs to be done here to merge? Get the coverage to 100%?

Coverage increased (+1.6%) to 79.786% when pulling fd85ccaaff589a6b07167f29d0e1e64ef0195ded on mathieu1:conda_support into bee26014a30883b949b8e532726a0a8f42ee59bc on Miserlou:dev.

Coverage increased (+1.6%) to 79.786% when pulling 03e8d95b242233d66caebf17c1e8bc703eb8c5ed on mathieu1:conda_support into bee26014a30883b949b8e532726a0a8f42ee59bc on Miserlou:dev.

Coverage increased (+1.6%) to 79.786% when pulling 03e8d95b242233d66caebf17c1e8bc703eb8c5ed on mathieu1:conda_support into bee26014a30883b949b8e532726a0a8f42ee59bc on Miserlou:dev.

Coverage increased (+1.6%) to 79.786% when pulling 03e8d95b242233d66caebf17c1e8bc703eb8c5ed on mathieu1:conda_support into bee26014a30883b949b8e532726a0a8f42ee59bc on Miserlou:dev.

It basically needs somebody from conda or a heavy conda user to say "okay, this is ready, this is how it works", provide functional test coverage, and ideally maintain the code.

Is it ready?

+1 and big applause for this. @kalefranz and @msarahan can you help @Miserlou out? I love conda, but wouldn't dare to call the shot on this.

Coverage decreased (-1.04%) to 79.786% when pulling f8421eca6557ec069d96543c47459d324633cf2b on mathieu1:conda_support into 309417c8fbb7ddbb96cdfd27a980af65ac7357da on Miserlou:master.

I'm going to try to review this code and actually work with it myself with lambda either today or tomorrow.

@Miserlou, Is there one or two specific functional tests you can point me to that would be good models for a conda functional test?

Quick question... Would the following be an accurate docstring for create_lambda_zip()?

def create_lambda_zip(self, prefix='lambda_package', handler_file=None, minify=True,
                      exclude=None, use_precompiled_packages=True, include=None, venv=None,
                      exclude_conda_packages=()):
    """
    Create a Lambda-ready zip file of the current virtualenvironment and working directory.

    Args:
        prefix (str): name of the current project
        handler_file (str): path to an AWS Lambda handler file for this project
            http://docs.aws.amazon.com/lambda/latest/dg/python-programming-model-handler-types.html
        minify (bool): if True, apply file excludes supplied via configuration
        exclude (Optional[Sequence]): glob patterns for files that should be excluded
            from the zip
        use_precompiled_packages (bool): use packages from the lambda_packages repo
            and/or manylinux wheels
        include (?): unused?
        venv (str): path to virtualenv prefix  (I don't see any place this is actually invoked by a caller)
        exclude_conda_packages (Sequence): glob patterns for files that should be
            excluded from the zip

    Returns:
        str: absolute path to a Lambda-ready zip file
    """

Thanks everyone for the outstanding support!

The twin repository providing an example zappa deployment in conda mode is now functional again :tada: and picks up the latest version of zappa (I've updated all necessary conda packages in my channel. Previous attempts at using this might have led to downloading older versions of zappa such as 0.17 or 0.22, which don't really work out of the box with the latest conda).

As a side note, this PR now targets master and is rebased accordingly (it seems dev is not the branch to use to contribute anymore).

@kalefranz thanks for having a look. Enlightenment from you (or @msarahan or anyone else) on the following fairly technical points would be greatly appreciated:

1. As @Miserlou mentions, this PR could use some tests. However, to test the core functionality of this PR, conda itself needs to be available to zappa both in a (possibly non-root) conda environment (for normal use) and to a virtualenv-installed zappa (for testing purposes). If I understand correctly, this means I should use conda as a python library as opposed to invoking it from the command line, right? AFAICT the current way to make conda available as a python library inside a non-root conda environment is by pip-installing conda (which equally works in a virtualenv). However, requiring user to pip install conda after conda installing zappa is not only inelegant but also beaks some reproducibility benefits of conda (libconda is another option but I can't tell if it's future-proof). @kalefranz, if this is making sense, do you have an idea which setup would be the best to implement tests?

2. This PR currently completely disregards an important technical benefit of conda: handling of hard-coded paths in files/binaries. Now I have yet to find a package (useful in a zappa context) for which this actually is a problem, but some conda guru might be able to point me to one. The gist of this issue is that since the conda packages are meant to be zipped up together and shipped on lambda, one should in fact use the absolute path corresponding to the lambda unzipping location, which is presumably different from the location where the local conda environment is installed. Is there a way to replace the conda placeholder with a prefix which is different from the local install location? If not possible currently, do you think such a non-standard feature would have a chance to ever make it in conda?

3. Partly answering a question of @sdementen, one less pressing issue for this PR is that for now, the conda-mode of zappa only works on a linux-64 machine (since the binaries are simply moved around, zipped up and shipped to lambda). There's currently a warning when one is trying to use zappa in a conda environment from another architecture. Now there's never going to be a perfect solution to make zappa work in all situations in a cross-platform way (whether by using wheels or conda or anything), but here are some ideas that would work in many cases.

   1. Loop through all the packages in the environment, and look for the same package in the same channel but changing the architecture to linux-64. (PRO : simple-minded. Allows for disregarding packages which don't have a linux-64 equivalent. CON : might miss necessary linux-64-specific dependencies.)
   2. List all conda packages and their versions, and try to resolve this using conda but on the linux-64 architecture. (PRO : straightforward way to come up with a coherent environment. CON : will fail whenever some package doesn't have a linux-64 equivalent, e.g. vs runtime, ...)
   3. Look for an environment.yml file (optionally specified in the zappa configuration) and resolve that for the linux-64 architecture. (PRO : flexible way to describe the desired environment in a cross-platform way. CON: the user must have an environment.yml file.)

   Am I missing other options? My preferred option is iii, but I could understand how casual users might enjoy the comfort of i, ii or a smart combination of the two (when it would work). Any thoughts?

@kalefranz IMHO The docstring looks good, except that exclude_conda_packages should read: list of conda packages to exclude from the zip. No glob pattern, just regular conda package names. The default is currently not to ship any of ['pip','python','readline','sqlite','wheel', 'boto3', 'botocore'] (as found in zappa/cli.py, also mentioned in the readme). @mathom suggested adding tk and openssl to this list. Any other suggestions?

So @mathieu1 has written a lot here while I've been writing code :)

Please check out some ideas I've sketched out here.

I haven't read all of the above yet, but I think it might start to address some of the problems that others might have already stumbled onto.

The new make_temp_project_from_conda() method in this code isn't actually used anywhere yet. If you want to go ahead and use it, I'd incorporate it somehow in create_lambda_zip(), and maybe rearrange create_lambda_zip() slightly, inverting the two sections "# First, do the project.." and "# Then, do the site-packages.." so that site-packages/conda-env comes first.

Ok, I think the code I wrote is actually a solution to your point 3.

For your point 1, the code I wrote just assumes that the conda executable is on PATH somewhere. It doesn't need to be accessible to import into the current python process. I think this is the easiest solution for now. We can get fancier later. One note: conda environments and virtualenv DO NOT play nicely together right now. It's really unfortunate. It's sort of virtualenv's fault, but conda's problem, since a lot of people come to conda after using virtualenv. Some background on the issue is here: https://github.com/conda-forge/staged-recipes/issues/1139

For your point 2, that's a really good point. And one that totally slipped my consideration until you brought it up. If we know the exact absolute path of where the environment will end up in its lambda execution context, and that path never changes, then it won't be hard to do those path rewrites. Most of the logic is in https://github.com/conda/conda/blob/master/conda/core/portability.py.

However, it might not be necessary to do those rewrites for many/most packages. Conda-build does its best to replace absolute paths in linked .so libraries with relative paths via the $ORIGIN variable (see here for some background).

tk should absolutely be ignored. openssl, I'm not so sure. What version of openssl does AWS Linux currently run? If it's a 1.0.2* series, then it might be ok.

@kalefranz Thanks a lot for the great suggestions! I will look into incorporating this as soon as I can.

Regarding point 1: @miserlou, would that be ok to have some tests requiring miniconda to be installed and on the path?

Regarding point 2: indeed, maybe this should only be addressed when stumbling upon an actual problem.

Regarding point 3: This is certainly a first solution. I read somewhere that conda might be aware of which packages were explicitly installed and which are "mere dependencies picked up by the solver". @kalefranz Do you know if this is indeed the case, and if it is accessible from the cli? It would then be even nicer to only feed the former packages to conda install.

@kalefranz Yet another remark : I can definitely picture your solution to work very well on macOS, but I expect difficulties on Windows with all kinds of links in conda packages, as well as permission issues (seems that Lambda is very picky about permissions). Maybe we'll just see once I incorporated your solution. An elaborate solution could involve manipulating zip files without ever extracting to the file system.

@mathieu1 - yes, absolutely, I think that half of the problem with testing this will be around the tooling. I don't think it'll be too hard, but it'll mean that we'll have to make some big chops to our Travis YAML.

I'm trying to get zappa and conda to work with flask-ask on OSX so I've been following this thread pretty closely.

So far I've been able to successfully get @mathieu1's example repo working correctly using the most up to date conda_support branch (with a few small tweaks). 🎉

But I'm having trouble getting my flask-ask example working because of the pyopenssl issues mentioned above. Here's the error I get: ImportError: /var/task/cryptography/hazmat/bindings/_openssl.so: invalid ELF header

From what I see here and other things I've read, this is an issue with the OSX binary that is getting zipped not being compatible with the Linux instance on lambda, but nothing I've tried seems to fix it.

I'm using an environment.yml file (have tried both with and without pyopenssl in dependencies), and have tried adding 'openssl' to exclude_conda_packages as suggested, but am still receiving the same error.

Not sure if this is still in the works, or what the best way to solve it would be, but wanted to provide some early feedback on this awesome feature. Let me know if I can provide any more detail.

@cscanlin This is definitely in the works, and your problems should hopefully evaporate very soon. I'm working on integrating multi-platform support at the very moment... (again, thanks @kalefranz for his awesome help on this)

Coverage decreased (-63.4%) to 17.472% when pulling 272936b33f5f1d8bfb4ab77e8d0a7aeb58ac442e on mathieu1:conda_support into f34beb51c2d7d0932769ccd54498010f47a36041 on Miserlou:master.

Coverage decreased (-63.4%) to 17.472% when pulling 272936b33f5f1d8bfb4ab77e8d0a7aeb58ac442e on mathieu1:conda_support into f34beb51c2d7d0932769ccd54498010f47a36041 on Miserlou:master.

Coverage decreased (-63.4%) to 17.465% when pulling 42a84cd7b65a47168295d01c5f203d151fefbcd6 on mathieu1:conda_support into f34beb51c2d7d0932769ccd54498010f47a36041 on Miserlou:master.

Coverage decreased (-0.9%) to 79.956% when pulling a0c1a6d695fb5d4d22ef231f68ed9badd3fd59ae on mathieu1:conda_support into f34beb51c2d7d0932769ccd54498010f47a36041 on Miserlou:master.

Coverage increased (+0.03%) to 82.74% when pulling a28ff109cb966f1cce0a09f2907954abd371e8e6 on mathieu1:conda_support into 8c2b1b4614256c2dc9df0d572a691dcb4a479f48 on Miserlou:master.

Coverage decreased (-65.1%) to 17.567% when pulling e9aa540d00a1132e4df060fdbfd0d1de1cd911a3 on mathieu1:conda_support into 8c2b1b4614256c2dc9df0d572a691dcb4a479f48 on Miserlou:master.

Coverage decreased (-65.1%) to 17.567% when pulling e9aa540d00a1132e4df060fdbfd0d1de1cd911a3 on mathieu1:conda_support into 8c2b1b4614256c2dc9df0d572a691dcb4a479f48 on Miserlou:master.

Coverage increased (+0.4%) to 83.092% when pulling 5b3664c45840ed9b708d9951b959416c103533b2 on mathieu1:conda_support into 8c2b1b4614256c2dc9df0d572a691dcb4a479f48 on Miserlou:master.

Status update:

• I integrated @kalefranz' code and rearranged things in a much cleaner fashion: there used to be lots of ugly copy-pasta for virtualenv and conda modes (with slight variations). Now, the workflow to create the package zip for conda mode is:
  • clone the current conda environment in a temp location (forcing the use of linux binaries if applicable)
  • strip from it the excluded conda packages (e.g. python, ...)
  • then, pretend this temp location is your virtual env and continue in virtual env mode.
• I have set up Travis to run all tests in virtual env mode and in conda mode as well.

This is reaching a mergeable state :smile:

I definitely plan to add a bit of documentation about conda mode (can be in this PR or another).

A remark about testing: the workflow is slightly simpler for linux-64 than for other architectures (it's faster and it handles pip packages as well, which is not the case currently for other archs). This means that part of the code is currently not tested through travis, although I don't see a huge added value in duplicating the build on macOS for example just to get the confirmation that this small piece of code works as intended. I have run the tests locally on mac and they pass.

Next steps include getting all dependencies of zappa on conda-forge for all architectures, as well as zappa itself for the first release that will include this PR.

+1 for conda support

~@kalefranz @msarahan It seems the code to create linux-64 environments on other platforms is not quite working as intended. In essence, the suggestion was to use e.g. CONDA_SUBDIR=linux-64 conda install --mkdir --prefix=/path/to/env python=2.7 zappa. However, CONDA_SUBDIR doesn't seem to have an effect: running the above on macOS creates an environment with osx-64 binaries. Am I missing something?~

~The only way I know of getting conda to download (say) linux binaries from macOS through the cli is by conda-installing an explicit environment.yml. Are there any other ways you would know about? If there is, people in that thread would also be interested: https://github.com/conda/conda/issues/1150.~

I found what the issue was, I was stuck with an earlier version of conda from conda-forge. One step closer towards multi-platform support...

Coverage increased (+0.4%) to 83.092% when pulling d8e488ec4bfeff17e5bfb0fd595b369eb1c9e3da on mathieu1:conda_support into 8c2b1b4614256c2dc9df0d572a691dcb4a479f48 on Miserlou:master.

Coverage increased (+0.4%) to 83.092% when pulling d8e488ec4bfeff17e5bfb0fd595b369eb1c9e3da on mathieu1:conda_support into 8c2b1b4614256c2dc9df0d572a691dcb4a479f48 on Miserlou:master.

Coverage increased (+0.4%) to 83.092% when pulling d8e488ec4bfeff17e5bfb0fd595b369eb1c9e3da on mathieu1:conda_support into 8c2b1b4614256c2dc9df0d572a691dcb4a479f48 on Miserlou:master.

Coverage increased (+0.4%) to 83.092% when pulling d8e488ec4bfeff17e5bfb0fd595b369eb1c9e3da on mathieu1:conda_support into 8c2b1b4614256c2dc9df0d572a691dcb4a479f48 on Miserlou:master.

Sorry about the conflicts now, hopefully it's nothing significant.

Is this ready?

Coverage decreased (-9.8%) to 2.646% when pulling a5b1976d5f762d42166cafa2882db838231ec2ea on mathieu1:conda_support into 9c2d06d6c57f816c004a3c88f1a37f73808b7c4d on Miserlou:master.

Coverage increased (+2.2%) to 14.674% when pulling 066741dc870dae472f2fa691df972a9be93a8797 on mathieu1:conda_support into 9c2d06d6c57f816c004a3c88f1a37f73808b7c4d on Miserlou:master.

Coverage increased (+2.2%) to 14.674% when pulling 066741dc870dae472f2fa691df972a9be93a8797 on mathieu1:conda_support into 9c2d06d6c57f816c004a3c88f1a37f73808b7c4d on Miserlou:master.

Coverage increased (+66.8%) to 79.29% when pulling de4becc5a5a71c28b7dcdefe061b214cab18ec51 on mathieu1:conda_support into 9c2d06d6c57f816c004a3c88f1a37f73808b7c4d on Miserlou:master.

+1 for conda support. Is this something that is still be worked on?

Not actively that I'm aware of. We'd need a Zappa + Conda user to step up to be the maintainer of that subsystem. A lot of the need for this has been eclipsed by our support of manylinux wheels during packaging.

+1 :)

+1! :) Conda + Flask + Serverless = Ultimate Batteries Included Web Apps

+1 for conda support

+1 for conda support

+1 for conda support

+1 for conda support

Just ran into this myself. I was looking into helping with python 3 support as well. Was is the current state of automating builds? I know this was building talked about for Windows.

I use conda exclusively... would really like to use Zappa instead of Serverless. Status?

+1 for conda support

+1 for conda support

+1

I've checked out https://github.com/mathieu1/zappa_conda I have python3.6 installed via conda Python 3.6.3 |Anaconda, Inc.. Changing the environment.yml file to require python=3.6 results in:

  - python 3.6*
  - zappa -> base58 ==0.2.4 -> python 2.7*
Use "conda info <package>" to see the dependencies for each package.```

+1 for conda support

+1 for conda support

+1 for conda support

+1

PLEASE STOP COMMENTING "+1". You're asking for the thread to be locked.

It adds nothing to the conversation, and annoys the project maintainers and others who have interest in this feature.

If you want to help, there is plenty of information here to help get you started, and even a bug report on some of the existing code would be more helpful than "+1".

If you just want to express interest, use the "thumbs up" reaction instead.

Sorry if it's annoying, but the mainpage includes the line:

Conda users should comment here.

Maybe there should be links to the stuff you're describing instead?

Fair enough, but the stuff I'm describing is here. The branch this PR references is functional for basic conda environments, and you can use it as you see fit. It can be a little buggy though, so it's just not ready to be merged.

I guess more specifically my suggestion would be for someone to try out this branch if they're interested, and then report back with error messages if it doesn't work. This way we can start moving forward on fixing the remaining bugs so this can actually be merged.

That said, this feature doesn't really have a "sponsor" at the moment to help drive this forward. So anybody who really needs it is encouraged to step up!

Oh, neat! Assumed it would take a deep knowledge of Lambda and/or conda in order to help out here. If just giving it a go and posting error messages would be useful, that's a horse of a different color! :)

👍 Check out this repo linked above for a minimum example of a working deploy too: https://github.com/mathieu1/zappa_conda

This is awesome! My team is very interested in implementing this. What still needs to be done to get Zappa working completely with anaconda?

Here's what I got after trying to do the "Hello World" example:

Creating zappa-conda-dev-ZappaLambdaExecutionRole IAM Role..
Creating zappa-permissions policy on zappa-conda-dev-ZappaLambdaExecutionRole IAM Role.
Packaging project as zip..
Traceback (most recent call last):
  File "/home/matt/anaconda2/envs/zappa_conda_env/bin/zappa", line 6, in <module>
    sys.exit(zappa.cli.handle())
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/zappa/cli.py", line 2197, in handle
    cli.on_exit()
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/zappa/cli.py", line 1945, in on_exit
    self.remove_uploaded_zip()
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/zappa/cli.py", line 1936, in remove_uploaded_zip
    self.zappa.remove_from_s3(self.handler_path, self.s3_bucket_name)
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/zappa/zappa.py", line 775, in remove_from_s3
    self.s3_client.delete_object(Bucket=bucket_name, Key=file_name)
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/botocore/client.py", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/botocore/client.py", line 517, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/botocore/client.py", line 572, in _convert_to_request_dict
    api_params, operation_model)
  File "/home/matt/anaconda2/envs/zappa_conda_env/lib/python2.7/site-packages/botocore/validate.py", line 270, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter Key, value: None, type: <type 'NoneType'>, valid types: <type 'basestring'>

+1 for Conda, deploying some machine learning models would be easier

+1 for conda support

@Miserlou and @mathieu1 any updates on readiness of conda support? Thread seems to have gone quite.

bump

• 1 for support

This is something I've wanted to work on for a long time, but I just don't have the bandwidth given other priorities. However, I'm at PyCON now and through the full sprint week next week, and if anyone has questions about conda or wants pointers, I'm happy to chat.

bump

Comparing this PR's implementation for conda support with the current packaging implementation in zappa/core.py, it looks like a workaround could be:

1. Activate your conda environment.
2. Run export VIRTUAL_ENV=$CONDA_PREFIX.
3. Try to zappa update dev.

I imagine you would need to run these commands on Linux or a lambda-compatible Docker container for it to work though. Also, you might want to disable use_precompiled_packages.

Anyone care to give the above a try?

bump

I can confirm that the above workaround works fine for conda environments.

The only remaining issue is distributing shared libraries under $VIRTUAL_ENV/lib. That's an issue that applies to both conda and virtualenv's however, and a workaround for that is described in #1260.

@lsorber I tried as you said, like this:

cd /some/zappa/demo1
source activate zappa
export VIRTUAL_ENV=/somepath/conda/anaconda3/envs/zappa
zappa deploy dev

it seem worked first, but finally got these error :

Downloading and installing dependencies..
 - sqlite==python36: Using precompiled lambda package
Packaging project as zip.
Traceback (most recent call last):
  File "/some/zappa/demo1/handler_venv/lib/python3.6/site-packages/botocore/vendored/requests/packages/urllib3/util/ssl_.py", line 267, in ssl_wrap_socket
FileNotFoundError: [Errno 2] No such file or directory

update1: I try again， it said Connection timed out, maybe the problem of network, I will try later.

update2: now I got this error:

Deploying API Gateway..
Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.

thanks for help~

update: I do like as this https://developer.amazon.com/zh/blogs/alexa/post/8e8ad73a-99e9-4c0f-a7b3-60f92287b0bf/new-alexa-tutorial-deploy-flask-ask-skills-to-aws-lambda-with-zappa ,it works！ and then I replace virtualenv with conda， it works also！ try it just~

@mathieu1 is this thread died or it is still WIP?

@kalefranz I love the approach of just including an environment.yml file and letting Zappa install the packages from that. (You proposed this way back in https://github.com/Miserlou/Zappa/pull/108#issuecomment-271469645).

The current Zappa approach of using the packages in your active environment causes unpredictable problems for us. Some devs end up with different packages in there working env that are deployed by Zappa, but aren't actually needed by the app. Also both our continuous integration server and our containerized version of the app need to install a virtualenv and activate it before running zappa update. It would be slick if it was just enough to have a environment.yml file (or Pipfile.lock, requirements.txt, etc. for that matter)