python
python copied to clipboard
kubernetes-client
Convert JSON or Dict to client objects
See also https://github.com/kubernetes-client/python/issues/340#issuecomment-526249075 (was closed for inactivity, NOT resolved) See also #63
Motivation It is sometimes useful to be able to load Python objects from JSON, for example to read it from a config file. It should be possible to get Python objects from JSON, to pass around the code in a type-safe manner, for modification and ultimately sending through the client.
Feature request There should be functions to convert JSON to objects and objects to JSON. This code already exists (it's required to talk to the API server), it should be exposed.
Workaround? Right now, you can put this raw JSON into Python objects' fields, and the API client will apparently accept this, however if you use such mixed objects your code will probably break since their attribute names are different (API uses PascalCase, Python objects use snake_case). Even if taking care to handle that, you will reach the point where you have to handle things coming from the API client (Python objects using snake_case) differently from what you load from config files (JSON using PascalCase). Really messy.
Right now I am left considering dropping this lib altogether and issuing requests directly, since the point of the wrapper is to expose safe typed Python classes instead of the raw JSON, but this doesn't work if you use any kind of config file.
Current status
It is somewhat possible to convert from a Python object to JSON using client.api_client.sanitize_for_serialization(), however converting from JSON to Python is completely impossible; the deserialize() method that exists expects a requests.Response, not JSON. Calling client.api_client._ApiClient__deserialize() works, but that is a private method, and you need to pass it the kind read from the JSON yourself.
Apologies for duplicating #340 but I cannot re-open it myself.
@yliaog sure, I'll give it a shot. i think the deserializer() might have been auto generated by swagger codegen. So where would this be appropriate? utils?
Yes that looks like generated code. I think utils should be fine, but somewhere in the base repo may work as well since we don’t necessarily require e2e tests on this one (do we?)
I've been doing some tests and using a similar approach to utils.create_from_yaml to infer the object response_type to use in the already generated codegen deserialize function.
Using this approach is possible to deserialize single objects and a list of multiple items to a list of individual namespaced objects; however this not covered all the different deserialize options we have. For example we have:
- list_xxx_for_all_namespaces
- list_namespaced_xxx
My question is is there a need to deserialize to these type of objects? Or deserializing to single objects covers the use case? One complication of this is when the is a List with multiple kind of objects (something like json output of kubectl get all -o json)
@remram44 will the proposed solution work for you? Anyone else has any opinions or suggestions ?
I'm not sure what you mean, what kind of lists wouldn't be deserializable?
Is really more about the object we output and not the input. And almost every object has a list object as well.
For example:
pods = core_v1.list_pod_for_all_namespaces()
type(pods)
kubernetes.client.models.v1_pod_list.V1PodList
type(pods.items[0])
kubernetes.client.models.v1_pod.V1Pod
So in the proposed solution we would output a native python list/array that has individually deserialize objects.
Something like this:
objs = deserialize.load_from_json('/Users/user/all.ns.data.json')
for obj in objs:
print(type(obj))
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_pod.V1Pod'>
<class 'kubernetes.client.models.v1_service.V1Service'>
<class 'kubernetes.client.models.v1_service.V1Service'>
<class 'kubernetes.client.models.v1_service.V1Service'>
<class 'kubernetes.client.models.v1_service.V1Service'>
<class 'kubernetes.client.models.v1_daemon_set.V1DaemonSet'>
<class 'kubernetes.client.models.v1_deployment.V1Deployment'>
<class 'kubernetes.client.models.v1_deployment.V1Deployment'>
<class 'kubernetes.client.models.v1_replica_set.V1ReplicaSet'>
<class 'kubernetes.client.models.v1_replica_set.V1ReplicaSet'>
<class 'kubernetes.client.models.v1_replica_set.V1ReplicaSet'>
type(objs)
list
The alternative is to separate the list items by kind and return another native python list but instead of individual objects return k8s list objects that would be accesible via the each obj.items
kubernetes.client.models.v1_pod_list.V1PodList
kubernetes.client.models.v1_service_list.V1ServiceList
etc
The reason I mentioned this is that they way we would iterate over the items would be different. I'm inclined to take the first approach since it's less complicated, but was wondering if anyone needs to deserialize to these other k8s List objects.
I see there is a PR already open for this functionality. I'm not sure if the discussion is better had here or on the PR directly. There has been talk over on the apache-airflow repo of how useful the private deserialize methods would be exposed as public. It seems to me that the way the private method approaches this is the simplest, leaving it to the caller to also pass in the kind. Are there not ambiguities when trying to infer the kind? For example if I have a json { "name": "source" } this could be inferred as a V1SecretEnvSource or a V1ConfigMapEnvSource
@davlum if I understand correctly, you would like to pass the object type directly; perhaps we can pass it along and not infer it, basically skip this line: https://github.com/chekolyn/python/blob/json_deserialize/kubernetes/utils/deserialize.py#L252 and pass the value directly.
Additionally, we look for more than just the kind to infer an object. We look for the api version and api group.
Do you have any json/yaml examples I can test with?
Yeah, I'm essentially looking for the functionality of __deserialize_model(self, data, klass): but made public. Because you have to pass the model and the models include the API version number, the API version is explicit as well.
__deserialize_model({ "name": "source" }, V1ConfigMapEnvSource)
@davlum i think a klaas/response_type str parameter in here https://github.com/chekolyn/python/blob/json_deserialize/kubernetes/utils/deserialize.py#L238 would do pass the klass to the __deserialize_model via this line: https://github.com/kubernetes-client/python/blob/ca4f31198e9748f1075af0e02abb8f7e1e174159/kubernetes/client/api_client.py#L289
I will amend/add this tonight. This is still work in progress and I thank you for the suggestion.
Passing in the kind is acceptable for my use case, I always know what kind of object I am trying to deserialize.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
Alright, I saw that https://github.com/kubernetes-client/python/issues/574#issuecomment-405400414 works... It's awkward, but works...
@davlum try the method at https://github.com/kubernetes-client/python/issues/574#issuecomment-405400414 works... I needed to clone objects and change the names and properties. So I had it transformed into a dict, worked on that lavel, and then converted to yaml... That's how far I got...
https://stackoverflow.com/questions/59977058/clone-kuberenetes-objects-programmatically-python/59977059#59977059
That's not quite what I'm looking for. I would like to go from JSON/YAML to client objects defined within this library. If I'm not mistaken your example returns JSON from the client, modifies it and writes back YAML, never making the conversion from YAML -> client object.
the deserialize() method that exists expects a requests.Response, not JSON
Reading deserialize, it mostly needs the data from a RESTResponse. The method could be reused for JSON by mocking a RESTResponse object, like what @chekolyn did in https://github.com/kubernetes-client/python/pull/989/files#diff-0204f9aa3f2d80c3fe32d62f37882dfaR256-R263
That's how I have to do it now:
class FakeKubeResponse:
def __init__(self, obj):
import json
self.data = json.dumps(obj)
fake_kube_response = FakeKubeResponse(initial_object_json)
v1pod = api_client.deserialize(fake_kube_response, 'V1Pod')
Will be glad to remove this hack after this issue close.
@piroszhog I think that is the right workaround at the moment, and this issue should be closed once #989 is finished.
That's how I have to do it now:
class FakeKubeResponse: def __init__(self, obj): import json self.data = json.dumps(obj) fake_kube_response = FakeKubeResponse(initial_object_json) v1pod = api_client.deserialize(fake_kube_response, 'V1Pod')Will be glad to remove this hack after this issue close.
To make this hack work for me, I had to add explicit datetime deserialization. I'm not sure but I think this may be because I'm also doing funky stuff like deepcopies in my code;
[...]
def _time_dumps(val):
if isinstance(val, datetime.datetime):
return iso8601time(val)
else:
raise TypeError(
"Don't know how to dump {}".format(type(val)))
self.data = json.dumps(body, default=_time_dumps)
[...]
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten
I don't have enough interest to fight the bot ad-vitam. Feel free to close it like you did #340 and pretend it's not a problem. It's not like you own anyone anything, even civility.
Bye!