carla
carla copied to clipboard
carla-simulator
When I use new feature about multi-gpu in different servers, carla crash down.
The Primary server run: ./CarlaUE4.sh --nullrhi It can run as usual. But when the Secondary server run: ./CarlaUE4.sh -carla-rpc-port=3000 -carla-primary-host= " the primary server ip" -ini:[/Script/Engine.Renderer Settings]:r.GraphicsAdapter=0 The secondary server start up but can not load the map(all black) , meanwhile the primary server crash down, can not move the viewpoint.
Hoping someone could answer my question, thanks.
Hi, one question. The different servers are running in the same computer? If you are using the same computer for all the servers, please check the GPU memory, because each server loads the map by itself.
About the black screen, by design, the secondary servers will never display anything, as they are only targeted to process sensors. So, they are setup automatically in no rendering mode. That is why you see nothing on those servers.
Regards, Bernat
Hi, one question. The different servers are running in the same computer? If you are using the same computer for all the servers, please check the GPU memory, because each server loads the map by itself.
About the black screen, by design, the secondary servers will never display anything, as they are only targeted to process sensors. So, they are setup automatically in no rendering mode. That is why you see nothing on those servers.
Regards, Bernat
Thanks for your reply. I got different servers running in different computers(local network). Can this work? Another question is: When secondary server connect to the primary server, the primary server stucks with no reply. How can I slove it?
Hi， I am also concerned about this issue.I have two PC: PC1:Linux, run Primary server and Secondary servers PC2:windows, run another Secondary servers I want to configure 8 camera to ego car, every secondary server render 4 camera .But I find PC2 can't connect to Primary server.Should I must use Linux? Another question ,how can I assign camera to which Secondary servers?
Hi， I am also concerned about this issue.I have two PC: PC1:Linux, run Primary server and Secondary servers PC2:windows, run another Secondary servers I want to configure 8 camera to ego car, every secondary server render 4 camera .But I find PC2 can't connect to Primary server.Should I must use Linux? Another question ,how can I assign camera to which Secondary servers?
About Q1, I think you can try to check firewall. Maybe the firewall block tcp messages. You can use telnet command to test the port is open or not. And for Q2, I have no idea either. Maybe it assigns automatically.
I got different servers running in different computers(local network). Can this work?
It should work fine, but the ideal setup is to have all servers in the same computer with several GPU, because you don´t add the delay of the network transmission to each frame. But it should work fine.
Another question is: When secondary server connect to the primary server, the primary server stucks with no reply. How can I slove it?
What happens is that the multi-gpu setup only works on synchronous mode, because primary and secondaries needs to be synchronized each frame. That means that when the primary server detects a secondary connection, it becomes automatically in synchronous mode, and then, unless a client script has a loop doing tick(), all seems to be paused, but that is normal behavior.
I want to configure 8 camera to ego car, every secondary server render 4 camera .But I find PC2 can't connect to Primary server.Should I must use Linux?
No, it should work fine. Check the firewall rules please, as zmz0130 commented.
Another question ,how can I assign camera to which Secondary servers?
Right now the distribution of sensors goes in sequence between all the secondary servers. So, first sensor created goes to first secondary server, and second sensor to the next secondary, and so on. We don't have an API function to manage that, because for the point of view of a client the setup of servers is transparent and unknown. That could change in the future.
I got different servers running in different computers(local network). Can this work?
It should work fine, but the ideal setup is to have all servers in the same computer with several GPU, because you don´t add the delay of the network transmission to each frame. But it should work fine.
Another question is: When secondary server connect to the primary server, the primary server stucks with no reply. How can I slove it?
What happens is that the multi-gpu setup only works on synchronous mode, because primary and secondaries needs to be synchronized each frame. That means that when the primary server detects a secondary connection, it becomes automatically in synchronous mode, and then, unless a client script has a loop doing tick(), all seems to be paused, but that is normal behavior.
It works! Thank you so much. And by the way, another question. Could this multi-gpu feature run on compiled Carla Version? If so, how can I use it?
@bernatx : If different servers(secondary server) running in different computers(local network).How does the sensor data transmit between different servers(secondary server) ? Secondary servers will finished this ,and carla client just run callback function. Or we should write some function to finished sensor data transmit between different servers(secondary server).
How does the sensor data transmit between different servers(secondary server) ? Secondary servers will finished this ,and carla client just run callback function. Or we should write some function to finished sensor data transmit between different servers(secondary server).
From the client side you don't need to do anything specific for multi-gpu. The sensors data is transmitted the same way and directly from the secondary server that is processing each specific sensor, so there are no in the middle servers, the connection is from client to secondary server. The difference is that now the client can connect to different secondary servers for each sensor.
It works! Thank you so much.
Great
And by the way, another question. Could this multi-gpu feature run on compiled Carla Version? If so, how can I use it?
There are some flags you can use when you run Carla, they seems to run fine in Linux but not always in Windows. You can try these commands:
make launch-only ARGS="--editor-flags=\"-game -carla-rpc-port=3000 -carla-primary-host=127.0.0.1\""
It works! Thank you so much.
Great
And by the way, another question. Could this multi-gpu feature run on compiled Carla Version? If so, how can I use it?
There are some flags you can use when you run Carla, they seems to run fine in Linux but not always in Windows. You can try these commands:
make launch-only ARGS="--editor-flags=\"-game -carla-rpc-port=3000 -carla-primary-host=127.0.0.1\""
I will try this and thanks for your reply again.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
@bernatx
primary server: CarlaUE4.exe -carla-primary-port=2002
CarlaUE4.exe -nullrhi this command is not able to open the serever window.
secondary server: CarlaUE4.exe -carla-rpc-port=3000 -carla-primary-host=127.0.0.1 -ini:[/Script/Engine.RendererSettings]:r.GraphicsAdapter=0 --RenderOffscreen
I am running primary and secondary on the same GPU (its not full eben though i run two servers) as soon as i run the scenario runner secondary server crashes and it closes the window (fatal Error). can you help me with this one.
