spark-redis
spark-redis copied to clipboard
RedisLabs
How to implement toRedisHASH(keyMaps: RDD[(String, Map[String, String])], ttl: Int = 0)
The current implementation of toRedisHASH seems write one key per time. Now, I wanna write a RDD [(String, Map[String, String])], is there possible? or any suggestions?
This is interesting as well. Why isn't this the RDD returned from fromRedisHash?? You lose the key-context with the current API. You cannot tell which hashes belong to which keys, without embedding the keys themselves in the hash.
Or am I missing something fundamental?
current the implementation of save multi map to redis
val wc = sc.parallelize(List(
("hello", "1")
))
sc.toRedisHASH(wc, "map1")
sc.toRedisHASH(wc, "map2")
sc.toRedisHASH(wc, "map3")
what if I have an rdd like
val wcBatch = sc.parallelize(List(
("map1", "hello", "1"),
("map2", "hello", "1"),
("map3", "hello", "1")
))
How can I batch save multi map into redis with single method call?
this batch style also work for list, zset and so on..
PS: If I use pure redis api, I can do this job with pipeline
val rdd = sc.parallelize(List(
("map1", "key1", "value1"),
("map1", "key2", "value2"),
("map1", "key3", "value3"),
("map2", "key1", "value1")
))
rdd.foreachPartition{iter =>
val redis = new Jedis("localhost",6379,400000)
val ppl = redis.pipelined()
iter.foreach{row =>
val mapKey = row._1
val key = row._2
val value = row._3
ppl.hmset(mapKey, Map(key -> value))
}
ppl.sync()
}
