datasplash icon indicating copy to clipboard operation
datasplash copied to clipboard

Issues with boolean serialization

Open FiV0 opened this issue 2 years ago • 1 comments

The test that illustrates this issue is the following (and adapted version of the first map-test)

(testing "bad map-fn serialization"
    (let [p (sut/make-pipeline [])
          input (-> [{:a 1} {:b 2} {:c 3}]
                    (sut/generate-input p))
          my-bool false
          rslt (sut/map (fn [x]
                          (if my-bool
                            x
                            {:random :value}))
                        {:name :map-w-sys
                         :initialize-fn (fn [] {:init 10})}
                        input)]

      (is (str/starts-with? (.getName rslt) "map-w-sys"))
      (is (-> (PAssert/that rslt)
              (.containsInAnyOrder [{:a 1 }
                                    {:b 2 }
                                    {:c 3 }])))

      (sut/wait-pipeline-result (sut/run-pipeline p))))

This test currently passes on master (it shouldn't). The problem seems to be the my-bool serialization. Replacing it with a literal value of false in the map fn makes the test fail (as expected).

I think the problem has to do with the anonymous fn serialization. my-bool is out scope when deserialized and executed on some worker. The question is how we should deal with it.

FiV0 avatar Nov 09 '22 13:11 FiV0

user> (import '(java.io ObjectOutputStream ObjectInputStream FileOutputStream FileInputStream))
user> (with-open [wrtr (ObjectOutputStream. (FileOutputStream. "tmpfile")) 
                  rdr (ObjectInputStream. (FileInputStream. "tmpfile"))]
        (.writeObject wrtr false) 
        (let [bool (.readObject rdr)]
          (println (= false bool))
          (println (if bool :then :else))))
true
:then
;; => nil

I'm pretty sure it's this exact issue:

user> (if (Boolean. false) :then :else)
;; => :then
user> (= false (Boolean. false))
;; => true

So not much we can do, patch Clojure or patch the Serializable implementation for Boolean...

RolT avatar Nov 29 '22 22:11 RolT