kapitan icon indicating copy to clipboard operation
kapitan copied to clipboard

Published 20 hours ago • verified publisher icon kapicorp

Kapitan yaml serialization error

Open jkrzemin opened this issue 3 years ago • 3 comments

Describe the bug/feature Nested lists seems to break compile step.

To Reproduce Steps to reproduce the behavior: kadet component

from kapitan.inputs import kadet

inv = kadet.inventory()

name = "other_component"
labels = kadet.BaseObj.from_dict({"app": name})


def main():
    output = kadet.BaseObj()
    output.root['my-custom-resource'] = inv.parameters.custom_resource
    return output

class

parameters:
  custom_resource:
    this:
      is:
        going:
          to:
            blow:
              up:
              - - name: oh
                  value: kapitan
              - - name: my
                  value: kapitan
  kapitan:
    compile:
      - output_path: custom_resource
        input_type: kadet
        output_type: yaml
        input_paths:
          - components/custom_resource

Target

classes:
  - my_component

parameters:
  kapitan:
    vars:
      target: my_target

Kapitan compile invocation breaks in following matter:

Unknown (Non-Kapitan) Error occurred
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.9.0_5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.9/site-packages/kapitan/targets.py", line 465, in compile_target
    input_compiler.compile_obj(comp_obj, ext_vars, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/kapitan/inputs/base.py", line 54, in compile_obj
    self.compile_input_path(input_path, comp_obj, ext_vars, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/kapitan/inputs/base.py", line 69, in compile_input_path
    self.compile_file(
  File "/usr/local/lib/python3.9/site-packages/kapitan/inputs/kadet.py", line 157, in compile_file
    fp.write_yaml(item_value)
  File "/usr/local/lib/python3.9/site-packages/kapitan/inputs/base.py", line 125, in write_yaml
    yaml.dump(obj, stream=self.fp, indent=indent, Dumper=PrettyDumper, default_flow_style=False)
  File "/usr/local/lib/python3.9/site-packages/yaml/__init__.py", line 290, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
  File "/usr/local/lib/python3.9/site-packages/yaml/__init__.py", line 278, in dump_all
    dumper.represent(data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 27, in represent
    node = self.represent_data(data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 199, in represent_list
    return self.represent_sequence('tag:yaml.org,2002:seq', data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 92, in represent_sequence
    node_item = self.represent_data(item)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 199, in represent_list
    return self.represent_sequence('tag:yaml.org,2002:seq', data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 92, in represent_sequence
    node_item = self.represent_data(item)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 58, in represent_data
    node = self.yaml_representers[None](self, data)
  File "/usr/local/lib/python3.9/site-packages/yaml/representer.py", line 231, in represent_undefined
    raise RepresenterError("cannot represent an object", data)
yaml.representer.RepresenterError: ('cannot represent an object', {'name': 'oh', 'value': 'kapitan'})
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/kapitan/targets.py", line 136, in compile_targets
    [p.get() for p in pool.imap_unordered(worker, target_objs) if p]
  File "/usr/local/lib/python3.9/site-packages/kapitan/targets.py", line 136, in <listcomp>
    [p.get() for p in pool.imap_unordered(worker, target_objs) if p]
  File "/usr/local/Cellar/[email protected]/3.9.0_5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 870, in next
    raise value
yaml.representer.RepresenterError: ('cannot represent an object', {'name': 'oh', 'value': 'kapitan'})


('cannot represent an object', {'name': 'oh', 'value': 'kapitan'})

Expected behavior Successful compile step and YAML output

this:
      is:
        going:
          to:
            blow:
              up:
              - - name: oh
                  value: kapitan
              - - name: my
                  value: kapitan

** If it's a bug (please complete the following information):**

  • python --version: 3.9.0
  • pip3 --version: 20.3.1
  • Are you using pyenv or virtualenv? both

Additional context Add any other context about the problem here.

jkrzemin avatar Sep 01 '21 09:09 jkrzemin

Apparently it's due addict.Dict behavior - it's not resolving recursively all references.

Python 3.8.1 (default, Jun 22 2020, 13:42:41)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from addict import addict
>>> d =
KeyboardInterrupt
>>> from addict import Dict
>>> from addict import addict
KeyboardInterrupt
>>> d = Dict({'this': {'is': {'going': {'to': {'blow': {'up': [[{'name': 'oh', 'value': 'kapitan'}], [{'name': 'my', 'value': 'kapitan'}]]}}}}}})
>>> d
{'this': {'is': {'going': {'to': {'blow': {'up': [[{'name': 'oh', 'value': 'kapitan'}], [{'name': 'my', 'value': 'kapitan'}]]}}}}}}
>>> type(d['this']['is']['going']['to']['blow']['up'][0][0])
<class 'addict.addict.Dict'>

jkrzemin avatar Sep 07 '21 12:09 jkrzemin

@jkrzemin good find! We are planning to update kapitan's kadet input type with the standalone kadet version that doesn't run addict anymore. We should make this example part of the tests.

ramaro avatar Oct 30 '21 09:10 ramaro