datasplash
datasplash copied to clipboard
Issues with boolean serialization
The test that illustrates this issue is the following (and adapted version of the first map-test
)
(testing "bad map-fn serialization"
(let [p (sut/make-pipeline [])
input (-> [{:a 1} {:b 2} {:c 3}]
(sut/generate-input p))
my-bool false
rslt (sut/map (fn [x]
(if my-bool
x
{:random :value}))
{:name :map-w-sys
:initialize-fn (fn [] {:init 10})}
input)]
(is (str/starts-with? (.getName rslt) "map-w-sys"))
(is (-> (PAssert/that rslt)
(.containsInAnyOrder [{:a 1 }
{:b 2 }
{:c 3 }])))
(sut/wait-pipeline-result (sut/run-pipeline p))))
This test currently passes on master (it shouldn't). The problem seems to be the my-bool
serialization. Replacing it with a literal value of false
in the map fn makes the test fail (as expected).
I think the problem has to do with the anonymous fn serialization. my-bool
is out scope when deserialized and executed on some worker. The question is how we should deal with it.
user> (import '(java.io ObjectOutputStream ObjectInputStream FileOutputStream FileInputStream))
user> (with-open [wrtr (ObjectOutputStream. (FileOutputStream. "tmpfile"))
rdr (ObjectInputStream. (FileInputStream. "tmpfile"))]
(.writeObject wrtr false)
(let [bool (.readObject rdr)]
(println (= false bool))
(println (if bool :then :else))))
true
:then
;; => nil
I'm pretty sure it's this exact issue:
user> (if (Boolean. false) :then :else)
;; => :then
user> (= false (Boolean. false))
;; => true
So not much we can do, patch Clojure or patch the Serializable implementation for Boolean...