How to read a captain proto message (.bin file) without Premature's EOFs in Python?

[brando90]

I'm trying to read a captian proto .bin message (written in packed):

dag_file = dag_api_capnp.Dag.read_packed(dependency_file, traversal_limit_in_words=2 ** 64 - 1)

but it yields the following error:

Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 1, in Exec
    def Exec(exp, global_vars, local_vars=None):
  File "<input>", line 1, in <module>
  File "capnp/lib/capnp.pyx", line 3013, in capnp.lib.capnp._StructModule.read_packed
  File "capnp/lib/capnp.pyx", line 3600, in capnp.lib.capnp._PackedFdMessageReader.__init__
capnp.lib.capnp.KjException: kj/io.c++:53: failed: expected n >= minBytes; Premature EOF
stack: 122b411e3 122b43536 122a48a97 108ca7d40 1229aa340 122a5a3d7 108c0fcf0 108c15396 108d64cf4 108d62abe 10a9f7f4c 108d5520a 108d4f728 108c6ef33 108d64cf4 108d62a04 10a9f7f4c 108d5520a 108c1117d 108d64cf4 108d62a04 10a9f7f4c 108c110c1 108d64cf4 108d616f9 10a9f7f4c 108c110c1 108d64cf4 108d61662 10a9f7f4c 108c110c1

Why is this error occurring? I assume I am not reading the packed message properly (or something like that). How do I open the contents of this file properly in captain proto for python?

I read the following links but they didn't help:

1. https://stackoverflow.com/questions/46690956/reading-writing-cap-n-proto-messages-partially
2. https://stackoverflow.com/questions/54950431/how-to-set-the-capn-proto-rpc-message-traversal-limit
3. https://github.com/capnproto/capnproto/issues/545
4. https://jparyani.github.io/pycapnp/capnp.html#capnp._StructModule.read_packed

but they didn't help to much. It seems it's pretty hard to find anything online with the exact error message: expected n >= minBytes; Premature EOF except for link 4 that didn't seem too helpful (and doesn't mention the word python).

---

crossposted: https://stackoverflow.com/questions/67374563/how-to-read-a-captain-proto-message-bin-file-without-prematures-eofs

[brando90]

can the error:

capnp.lib.capnp.KjException: kj/io.c++:53: failed: expected n >= minBytes; Premature EOF
stack: 122b411e3 122b43536 122a48a97 108ca7d40 1229aa340 122a5a3d7 108c0fcf0 108c15396 108d64cf4 108d62abe 10a9f7f4c 108d5520a 108d4f728 108c6ef33 108d64cf4 108d62a04 10a9f7f4c 108d5520a 108c1117d 108d64cf4 108d62a04 10a9f7f4c 108c110c1 108d64cf4 108d616f9 10a9f7f4c 108c110c1 108d64cf4 108d61662 10a9f7f4c 108c110c1

be explained better? What does it mean?

Currently it seems my bug went away for now (not sure if it was making sure the right captain proto file was there or the fact I read all files with read only once).

[haata]

I believe this message is coming from kj/libcapnp (not Python). My guess is that input .bin file was unexpectedly short for the parser.

[brando90]

I believe this message is coming from kj/libcapnp (not Python). My guess is that input .bin file was unexpectedly short for the parser.

could this be because the wrong captain proto schema is being used?

[kentonv]

The error means that the file was truncated before the end of the message, OR that the file is corrupted in such a way that the parser thinks the message should be longer.

could this be because the wrong captain proto schema is being used?

No, wrong-schema wouldn't produce this.

However, you could get the error if you serialized using "packed" format and then deserialized using regular format, or vice versa.