incubator-heron icon indicating copy to clipboard operation
incubator-heron copied to clipboard

terrible resource is needed when submitting to aurora

Open dttlgotv opened this issue 4 years ago • 2 comments

I write a simple example to test resource requirement to submit to aurora. It is very terrible that too more resource is needed. Please check below:

my example: builder.newSource(() -> ThreadLocalRandom.current().nextInt(100)) .setName("random-sentences-source");

      Config config = Config.newBuilder()
                  .setNumContainers(1)
                  .setPerContainerCpu(2)
                  .setPerContainerRamInGigabytes(3)
                  .build();

When I submit this topo to aurora, 4.5 cpu and 6.3G memory are allocated to this topology. It seems that Tmaster used 2.5G cpu and 3.3G memory.

My question is how to limit tmaster resource requirement. It seems that Tmaster task using more resource.

dttlgotv avatar Apr 02 '20 03:04 dttlgotv

In aurora, all containers need to have the same size, and container 0 is mainly for tmaster. Therefore the key is to keep the container size under control. In your example, your cpu and memory usage could be lower if you have 2 containers and 1 core/1.5G per container.

nwangtw avatar Apr 02 '20 05:04 nwangtw

In aurora, all containers need to have the same size, and container 0 is mainly for tmaster. Therefore the key is to keep the container size under control. In your example, your cpu and memory usage could be lower if you have 2 containers and 1 core/1.5G per container.

Let me try, thanks a lot

dttlgotv avatar Apr 02 '20 06:04 dttlgotv