Query Support

[derekperkins]

Do you have any intention of supporting Queries via GetAll or Run? I can see how that would be much more challenging than Get/Put/Delete.

[jongillham]

It would be an incredibly complex (and fun) challenge which I certainly want to do but I expect development will not occur until I have reached 100% unit test coverage and am happy the bug reports are drying up.

[derekperkins]

Sounds great. We should be pushing nds to production within the next week or so. I'm excited to see what it does to our datastore reads. I'll stay active with any bug reports and try to help with unit testing.

Any idea how many people are using nds or on what scale?

[jongillham]

I don't expect many people are using this package right now. I'm not sure how to get the word out without spamming people.

I have been using it in production for several months, but not with PropertyLoadSaver or anything other than integers and strings as property values. I have been pleasantly surprised at how much it has decreased the cost of my app with regards to datastore calls!

I'm now looking forward to seeing the nds call times being reduced by 4x with the fix for the latest bug you uncovered! Hopefully that will decrease the number of GAE instances I use significantly.

[derekperkins]

I tried to give you a boost since I can shamelessly promote nds without feeling spammy. :) https://groups.google.com/forum/#!topic/google-appengine-go/BjtbQOdsdEo

[jongillham]

That's very kind of you! :blush:

[stephanos]

FYI I created hrd, a library with a similar purpose. It offers a bunch more features but is still more or less alpha status. I'll investigate if I could use nds internally and build the other features on top of it. Or maybe you can steal a few ideas from it (I stole most from gaelic and goon myself :)).

[jongillham]

hrd looks pretty feature rich! The main reason for using nds in your package is if you want strong cache consistency. Note that it is not possible to have strong cache consistency if you use memory caching as well.

Otherwise feel free to use nds. All you need to do is replace your datastore.Put_, Get_, Delete* and RunInTransaction calls with the nds.* equivalents.

If you don't care about strong cache consistency then it will be more performant to roll your own cache. App Engine requires four calls to the memcache API to save to memcache and ensure cache consistency compared to the one memcache.Set if you don't care about this.

[kidoman]

Have been wondering about a good way for adding query support to nds.

One pattern I follow when doing queries on entities:

• Do a KeysOnly() query and get all the keys involved in the query
• Invoke a nds.GetMulti on the keys and get my slice of structs

How are others doing it?

[jongillham]

I do the same thing. It's simple, cheap and works well. There have been several discussions about this with regards to Python's NDB for example here: https://github.com/googlecloudplatform/datastore-ndb-python/issues/118.

My basic premise is that if the query is likely to return more than a few entities then I do a KeysOnly() query followed by nds.GetMulti as you suggest. Otherwise I just get the entity directly from the query.

[tim-lebedkov]

what do you think about this:

1. write the following function func getAllByQuery(memcacheKey string, q Query) ([]*Entity, []*Key, error) { keys := getKeysFromMemCache(memcacheKey) if keys == nil { // execute query and fill the keys } return nds.GetMulti(keys), keys, nil }

2. write another function that computes the hash of a query: func queryHash(q Query) string { ..... }

3. combine both functions func GetAllByQuery(q Query) ([]*Entity, []*Key, error) { h := queryHash(q) return getAllByQuery(h, q) }

[jongillham]

That is certainly a valid solution although it will not work with transactions easily. I should probably close this issue as I don't think much more work will be done on this package as Google appears to be moving us all over to the https://godoc.org/cloud.google.com/go/datastore API. If/when they add memcache to that then all this code will be converted over.

[tim-lebedkov]

Why do you think that Google is moving us to https://godoc.org/cloud.google.com/go/datastore?

https://cloud.google.com/appengine/docs/standard/go/datastore/ explicitely says "You cannot use the Cloud Datastore client library (cloud.google.com/go/datastore) with Go applications in the App Engine standard environment."

[jongillham]

I think it's the only thing that makes sense in order to reduce the duplication of effort having multiple almost identical APIs to maintain.

[tim-lebedkov]

So it's only you speculating and there is no evidence at the moment, right? In this case it would take at least 2 years IMHO even if Google would decide today to switch to the new API. Also NDS could be ported to the new API. In my opinion there are no significant differences between the two APIs.

Do you plan to make any changes to NDS? Bugfixes? New features?

[jongillham]

Yes it's complete speculation by me.

I'll port NDS to the new API if/when the new API gives access to memcache. There will have to be two versions if the new API doesn't also support App Engine Standard.

As for bug fixes, NDS is used in several large production systems so we are always looking for bugs and their subsequent fixes.

With regards to new features, I am generally reticent about adding anything that can be recreated by the client easily. With regards to this issue of query support. It appears to be very hard/impossible to do whilst maintaining the same data consistency features that the datastore gives.