python-twitter isn't compatible with google app engine

[abloch]

python-twitter uses local files (tempfile.gettempdir) for caching purposes. due to the file access restriction in google app engine this library isn't compatible and can't be run in gae.

please find the restriction here: https://developers.google.com/appengine/docs/python/runtime#Pure_Python

I'd like to open a new branch which is gae-friendly. the idea is to re-implement the cache mechanism based on tempfile.TemporaryFile and prefixes. other issues with gaw might happen as well and to be treated in that branch.

before I do so, I'd like to have your opinion first please.

thanks in advance Akiva

[bear]

Sounds like you have a great idea, the only thing you may run into is that I'm working on the v1.1 API support - but I can do that work without even thinking about caching and then add that in later.

The other thing I would like to do is see if we can move to using the Python Requests library - can that be made GAE friendly?

All of this would be made easier if I add you to the project - would you mind if I make you a committer?

[abloch]

1. I noticed that the api v1.1 support is on a separate branch. I suggest doing the GAE support on another branch so that GAE doesn't disturb anything and make the merge once everything's completed.
2. requests looks like it's on the best way to be GAE-friendly (please see here: https://github.com/kennethreitz/requests/issues/498) I guess using it is safe, and if any problem happens it will be minor and easy to solve.
3. being a committer to python-twitter is a great honor and I'd be happy to take it. I don't want to be too optimistic about my spare time right now (I'm very busy because of happy familial reasons :-) but for the long run I'll definitely be happy to assist here and then.

[bear]

Oh definitely it would be on another branch :) that just makes sense - agreed

cool, switching to requests I can do on the v1.1 api branch

I'm glad to have help and you've shown you have interest - what time you can give is perfect :)

(and to be brutally honest, you can't do any worse than I did when I first got started ;)

[bear]

ok, i've created the branch "gae" and added you to the contributor's list

and don't feel pressured or anything, any work you do is more than I could do by myself

[abloch]

thanks a lot!

[IntegersOfK]

It has been a few years since this was discussed, but as a developer using Google App Engine I use this python-twitter library and I wanted to quickly share how I got it to work for people like me who stumbled across this issue.

Paste the python-twitter module folder (the twitter folder inside, not the whole root of this repository) directly into your App Engine project folder (where you put app.yaml).

My main issue was the requests library, which Google now offers a "monkeypatch" so the library will run through their urlfetch function instead of using the standard requests.

My fix was to add these lines after the requests import to the top of api.py:

...
import requests
import requests_toolbelt.adapters.appengine
requests_toolbelt.adapters.appengine.monkeypatch()
...

Now all the requests use App Engine's urlfetch, which deals with the fact that your App Engine instance is behind a bunch of datacentre routing/firewalls and returns everything as you would expect.

I believe you also need to include SSL in your app.yaml:

libraries:
- name: ssl
  version: latest

And finally, I was getting an error with the caching but instead of figuring out how to get that to work with App Engine's caching, I manually disabled python-twitter caching by forcing None at the SetCache method again in api.py. Your use case might not need this part (I'm not an expert on caching strategies).

I don't know if it makes sense to reformat the python-twitter models into something that'll work more tightly with datastore, but for my usecase I just made new datastore models with the properties I wanted to keep from the content returned by the library's api.

So for this library, I would say a good App Engine branch todo list would look like this, with number 1 being the only truly breaking issue to fix:

1. Either reformat the requests to use urlfetch directly, or add the monkeystrap described above.
2. Integrate App Engine's memcache.
3. Define the Twitter models as NDB datastore models.

[bear]

we could offer a cache class that by default is nothing but a wrapper around the requests module and then people on GAE can pass it in to the API?

[IntegersOfK]

That sounds like a good track, but I'm new to this library so I'm going to have to familiarize myself with these issues before I would be able to make any radical changes. I made a pull request #383 for the main issue in case that's helpful? I'm happy to take another look at the app engine memcache to see if we can't get that integrated some day, but again it's a bit of a learning curve so you might be faster to pick that up more directly.