kiko icon indicating copy to clipboard operation
kiko copied to clipboard

Python 3: Kiko writes empty files

Open ben-hawkyard-absolute opened this issue 2 months ago • 0 comments

Hi there, I've been having trouble saving .kiko files correctly with python 3. As far as I can tell, when a kiko file is saved the data files that get added to the underlying archive are truncated at zero bytes. You can reproduce this with the following code snippet:

import os
import tempfile
from kiko.io.kikofile import KikoFile


handle, kiko_path = tempfile.mkstemp(suffix=".kiko")
os.close(handle)
kiko_file = KikoFile(kiko_path)
kiko_file.set_data({"test": "data"})
kiko_file.save()
kiko_file.parse()

When I do this I get an error that looks something like:

Traceback (most recent call last):
  File "/home/ben.hawkyard/Documents/scripts/testing/kiko_metadata_testing.py", line 11, in <module>
    kiko_file.parse()
  File "/opt/rez/rez_package_cache/kiko/1.0.0/a99e/a/python/kiko/io/kikofile.py", line 145, in parse
    self._metadata = json.load(v)
                     ^^^^^^^^^^^^
  File "/opt/rez/rez_package_cache/python/3.11.8/3e7f/a/lib/python3.11/json/__init__.py", line 293, in load
    return loads(fp.read(),
           ^^^^^^^^^^^^^^^^
  File "/opt/rez/rez_package_cache/python/3.11.8/3e7f/a/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rez/rez_package_cache/python/3.11.8/3e7f/a/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rez/rez_package_cache/python/3.11.8/3e7f/a/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I think this is caused by this bit of code in io/kikofile.py:

    @staticmethod
    def _add_to_tar(tar_file, name, f_obj):
        info = tarfile.TarInfo(name=name)
        if sys.version_info.major == 2:
            info.size = len(f_obj.buf)
            info.time = time.time()
        else:
            info.size = f_obj.tell()
            info.mtime = time.time()
        tar_file.addfile(tarinfo=info, fileobj=f_obj)

    @classmethod
    def _add_to_tar_from_dict(cls, tar_file, name, data):
        io = StringIO()
        json.dump(data, io)
        io.seek(0)
        cls._add_to_tar(tar_file, name, io)

calling io.seek(0) in _add_to_tar_from_dict means that f_obj.tell() in _add_to_tar returns zero and this causes us to write empty files to our kiko file when adding dicts to it.

I've done some testing and I think this could be fixed by changing the else branch in _add_to_tar:

        else:
            f_obj = BytesIO(f_obj.read().encode("utf-8"))  # TarFile expects file objects to be opened in binary mode.
            info.size = f_obj.seek(0, os.SEEK_END)
            f_obj.seek(0)
            info.mtime = time.time()

I'm happy to put in a pull request for this if there's interest. I can also see that mottosse fixes this in their pull request for kiko here: https://github.com/Toolchefs/kiko/pull/6. Is this likely to be accepted, do you think?

ben-hawkyard-absolute avatar Apr 29 '24 13:04 ben-hawkyard-absolute