Support for PFADD , PFCOUNT

[justinrmiller]

Support for PFADD , PFCOUNT from spark - This is the work of @nirmalc, I've simply created a new branch, fixed a few tests, and provided an easier to merge branch off of master.

[nirmalc]

Thanks @justinrmiller

[fe2s]

Hi Justin, Thanks for the PR! Some unit tests failing on Travis, could you please take a look?

[justinrmiller]

@fe2s Looking at the errors in Travis, I'm not sure those are caused by this change. Did I miss something in the review?

The following appear to be failing, but I didn't alter them:

• RedisKVRDD *** FAILED ***
• groupKeysByNode

[fe2s]

@justinrmiller

The issue is that HLL datastructure is implemented in Redis as a String:

127.0.0.1:6379> PFADD hll-test a a b
(integer) 1
127.0.0.1:6379> type hll-test
string
127.0.0.1:6379> get hll-test
"HYLL\x01\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00q\xa6\x84K\xfb\x80BZ"

RedisKVRDD reads all KVs (strings) from a database (using pattern *). Since extra HLL strings are inserted in beforeAll() method, the actual number of strings doesn't match the expected number. I think it can be fixed by writing word count strings (sc.toRedisKV(wcnts)) with a prefix and also prefix HLL keys, so we can read them separately.

[justinrmiller]

@fe2s ahh that makes sense, OK let me try to get that going when I get home. Thanks!

[justinrmiller]

Ok I think that should take care of the tests, I did notice some exceptions getting thrown but I've seen those exceptions thrown in other PRs so I'm not sure if that's something introduced by this PR. Is there any documentation on setting up the test environment? I kind of had to piece one together locally.

[fe2s]

@justinrmiller to run all the tests just execute make test in terminal If you'd like to debug individual test in IDE you can start redis test instances with make start and then run any test from IDE. There are tests that throw exceptions intentionally, those are intercepted and shouldn't result in failing tests. I will add a page to the documentation with some details how to setup a dev environment and run tests.

[justinrmiller]

Ah yeah I was just running mike test. I'll try running make start to get the Redis instances going next time.

[fe2s]

Hi @justinrmiller,

Would you have some time to address the review comments? If not, I can take over from here if you'd like.

[justinrmiller]

My wife and I just finished moving back to LA. I should be able to take a look at it over the next few days. Thanks.