Running Dataflow jobs from Google App Engine

This directory contains two different examples that show how you can run Cloud Dataflow pipelines from App Engine apps, as a replacement for the older GAE Python MapReduce libraries, as well as do much more.

The examples show how to periodically launch a Python Dataflow pipeline from GAE, to analyze data stored in Cloud Datastore; in this case, tweets from Twitter.

The example in sdk_launch shows how to launch Dataflow jobs via the Dataflow SDK. This requires the use of an App Engine Flex service to launch the pipeline.

The example in job_template_launch shows how to launch Dataflow jobs via job Templates. This can be done using only App Engine Standard. Prior to deploying the app, you create a pipeline template (via your local command line, in this example) for the app to use.

Contributions

Contributions are not currently accepted. This is not an official Google product.