appengine

appengine copied to clipboard

tl;dr: provide a simple XML GCS client for use with app engine

dev_appserver.py provides a simple emulator for GCS. It emulates the XML API.

Currently, we recommend people use the cloud.google.com/go/storage package for interacting with GCS on App Engine. Unfortunately, it only can talk the JSON version of the API and it has no idea how to deal with dev_appserver.

Python and Java maintain a separate version of the storage client library specifically for this purpose: https://github.com/GoogleCloudPlatform/appengine-gcs-client. It is completely separate from the full GCS client version (eg. the google-cloud-python library et al.)

The python demo for GCS on App Engine is much simpler than the equivalent Go version. For example, it doesn't do anything with ACLs etc. Importantly, this demo works in both dev (using dev_appserver.py) and with the production service.

I propose that we construct a Go-equivalent of the simple GCS library, and update our demo to use it. It probably should live here in this repo, and we could take this opportunity to clean up the blobstore package too?

@adamtanner @broady @rakyll

An alternate approach would be to provide a translation layer that the google-cloud-go library could use to talk to dev_appserver. That seems difficult and fraught, but would reduce the number of API surfaces we provide.

Python and Java maintain a separate version of the storage client library specifically for this purpose: https://github.com/GoogleCloudPlatform/appengine-gcs-client.

I would like to hear some insights from the maintainers of these clients. A separate client is obviously not ideal, confusing, hard to document and hard to develop against. I doubt the current usage of the gcs clients is at a satisfactory level to rationalize the man-hour spent on the development and the maintenance.

The translation layer would be nice to have even thought it is fairly complicated but would be more aligned with the emulator support we have for datastore and is a good solution rather than providing a hack around a problem.

Depending on how simple the XML GCS support in dev_appserver is, would another option be to add equivalent JSON support to dev_appserver and have a host override env variable in the AppEngine packages like, I think, Datastore has? That seems like the easiest and maybe more correct approach. On Tue, Sep 13, 2016 at 10:50 AM Jaana Burcu Dogan [email protected] wrote:

Python and Java maintain a separate version of the storage client library specifically for this purpose: https://github.com/GoogleCloudPlatform/appengine-gcs-client.

I would like to hear some insights from the maintainers of these clients. A separate client is obviously not ideal, confusing, hard to document and hard to develop against. I doubt the current usage of the gcs clients is at a satisfactory level to rationalize the man-hour spent on the development and the maintenance.

The translation layer would be nice to have even thought it is fairly complicated but would be more aligned with the emulator support we have for datastore and is a good solution rather than providing a hack around a problem.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/appengine/issues/21#issuecomment-246765307, or mute the thread https://github.com/notifications/unsubscribe-auth/AABrbu0c3vBFqzb7gxaY_dFivnVtui5Xks5qpuJ2gaJpZM4J7KEk .

@adamtanner, do you think we'd have much luck getting this fixed in dev_appserver (in a reasonable timeframe)? That does sounds like a good option, since we'd only need a little logic in the storage package to detect it was running in dev. It would enable other languages similarly.

I'll talk to @raggi about it and see if we can get it prioritized. It seems like we'd just need to port https://github.com/GoogleCloudPlatform/appengine-gcs-client/blob/master/python/src/cloudstorage/cloudstorage_api.py to use JSON instead of XML and it will probably just work.

@ajessup

Has there been any movement on this? The README here notes that both file and blobstore are mostly deprecated, but actually getting GCS to work in the dev environment essentially involves either going through an oauth flow or downloading service credentials to your machine and passing them to dev_appserver. It'd be great to have a working solution for dev in the long term.