kiko
kiko copied to clipboard
Python 3: Kiko writes empty files
Hi there, I've been having trouble saving .kiko files correctly with python 3. As far as I can tell, when a kiko file is saved the data files that get added to the underlying archive are truncated at zero bytes. You can reproduce this with the following code snippet:
import os
import tempfile
from kiko.io.kikofile import KikoFile
handle, kiko_path = tempfile.mkstemp(suffix=".kiko")
os.close(handle)
kiko_file = KikoFile(kiko_path)
kiko_file.set_data({"test": "data"})
kiko_file.save()
kiko_file.parse()
When I do this I get an error that looks something like:
Traceback (most recent call last):
File "/home/ben.hawkyard/Documents/scripts/testing/kiko_metadata_testing.py", line 11, in <module>
kiko_file.parse()
File "/opt/rez/rez_package_cache/kiko/1.0.0/a99e/a/python/kiko/io/kikofile.py", line 145, in parse
self._metadata = json.load(v)
^^^^^^^^^^^^
File "/opt/rez/rez_package_cache/python/3.11.8/3e7f/a/lib/python3.11/json/__init__.py", line 293, in load
return loads(fp.read(),
^^^^^^^^^^^^^^^^
File "/opt/rez/rez_package_cache/python/3.11.8/3e7f/a/lib/python3.11/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/rez/rez_package_cache/python/3.11.8/3e7f/a/lib/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/rez/rez_package_cache/python/3.11.8/3e7f/a/lib/python3.11/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I think this is caused by this bit of code in io/kikofile.py:
@staticmethod
def _add_to_tar(tar_file, name, f_obj):
info = tarfile.TarInfo(name=name)
if sys.version_info.major == 2:
info.size = len(f_obj.buf)
info.time = time.time()
else:
info.size = f_obj.tell()
info.mtime = time.time()
tar_file.addfile(tarinfo=info, fileobj=f_obj)
@classmethod
def _add_to_tar_from_dict(cls, tar_file, name, data):
io = StringIO()
json.dump(data, io)
io.seek(0)
cls._add_to_tar(tar_file, name, io)
calling io.seek(0)
in _add_to_tar_from_dict
means that f_obj.tell()
in _add_to_tar
returns zero and this causes us to write empty files to our kiko file when adding dicts to it.
I've done some testing and I think this could be fixed by changing the else branch in _add_to_tar
:
else:
f_obj = BytesIO(f_obj.read().encode("utf-8")) # TarFile expects file objects to be opened in binary mode.
info.size = f_obj.seek(0, os.SEEK_END)
f_obj.seek(0)
info.mtime = time.time()
I'm happy to put in a pull request for this if there's interest. I can also see that mottosse fixes this in their pull request for kiko here: https://github.com/Toolchefs/kiko/pull/6. Is this likely to be accepted, do you think?