gapic-generator-python icon indicating copy to clipboard operation
gapic-generator-python copied to clipboard
verified publisher icon googleapis

gapic-generator-python fails to complete with a sufficiently large protobuf tree

Open tmc opened this issue 4 years ago • 5 comments

This generator fails when there is a type reference to a very large protobuf type tree. (For example, the types described at https://github.com/google/fhir/tree/master/proto/google/fhir/proto ).

Initial debugging appears to show that MessageType.with_context is where the majority of the time is spent.

tmc avatar Jul 09 '21 23:07 tmc

MessageType.with_context is also a colossal memory hog, which may be the actual cause. I have definitely generated OOMs when attempting to compile particular APIs. This re-prioritizes an undertaking to do less copying that I'm just about to restart.

software-dov avatar Aug 02 '21 17:08 software-dov

@software-dov Curious if you have anything to share regarding this.

tmc avatar Nov 10 '21 01:11 tmc

@software-dov I experimented and by moving away from FrozenSet and by ensuring the same set() instance is threaded through the service/message/field tree the generation time+memory goes down from non-terminating and consuming 32G to finishing within 2 minutes.

tmc avatar Dec 16 '21 06:12 tmc

Waiting for https://github.com/googleapis/gapic-generator-python/pull/1125 to be merged. @tmc can you please follow up on it? Thanks!

vam-google avatar Jan 19 '22 23:01 vam-google

@vam-google I think that PR needs review.

tmc avatar Aug 26 '22 00:08 tmc