play-plugins icon indicating copy to clipboard operation
play-plugins copied to clipboard

String Redis cache value is limited to length < 2^16

Open ildar15 opened this issue 6 years ago • 2 comments

String Redis cache is using DataInputStream / DataOutputStream readUTF / write UTF for string serialization. This is stepping on Java's inherent limitation of these functions, namely, they use 2 byte header and cannot process strings longer than 2^16 bytes see for example https://www.drillio.com/en/2009/java-encoded-string-too-long-64kb-limit/ I created a test case which exercises that limitation: simply using string of 70K bytes will fail string test.

ildar15 avatar Jan 11 '19 14:01 ildar15

This issue can be fixed in two way:

  1. the easy way to allow large strings into the cache is simply use String as an Object. The drawback of this approach is that other tools besides Java (Scala) will not be able to access redis value directly.
  2. another approach will be just to avoid using writeUTF and convert String to byte array directly. I tested number

Number 1 certainly works and I can submit patch for that solution. I will be testing second approach, which is more favorable (flexible) in my opinion. One can certainly try to support both provided we have correct default.

ildar15 avatar Jan 11 '19 14:01 ildar15

I forked the project here

https://github.com/ildar15/play-plugins

and implemented the solution to this problem. The writeUTF was eliminated, and the value is stored as text, without length header. The result is transparent to the client, but will invalidate existing string caches. You will need to skip the first two bytes of the returned result if you need to read old cache values.

ildar15 avatar Jan 27 '19 02:01 ildar15