Hardware recommendations for a large scale (200+ devices) STF

[vovinio]

Hi guys,

I would kindly appreciate if you could share your recommendations and improvement ideas for the following system.

Currently I'm having a 30 square meters lab with 150 devices in it.

• All the devices are being controlled remotely with adb through wifi.
• 60 devices are connected to a stf-provider physical machine (i7 3770 with 32GB RAM) running Ubuntu.
• 90 devices are connected to a stf-provider physical machine (i7 8700K with 32GB RAM) running Ubuntu.
• Each of the stf-provider physical machines has few extra PCIE USB controllers to extend the quantity of the xHCI endpoints for the machine so it could support more USB devices.
• All the devices are connected to the physical machines with Plugable USB 2.0 7 Port Hub with 60W Power Adapter, while, 7 devices are being connected to each Plugable USB hub.
• There is a central server hosting the two stf-provider machines.

- All the devices must have internet access and therefore to be connected by wifi to a router. - It's important to state the the system is working and should work 24/7.

My issues are:

• The devices are regularly losing connection with adb through wifi. Therefore I've connected them by USB to the stf-provider machines so I could run tcpip command by usb to re-establish the wifi connections each time they're lost.

• The reason I'm not controlling the devices simply through USB is because the USB is losing the connectivity much more often than by wifi and is not stable at all (I assume it's due to too many USB connections per machine and the existence of the USB hubs).

In this thread I've got the recommendations to use an ARM like Raspberry PI3 model B, or HypriotOs, OrangePi Zero and Armbian as they already come with docker and docker-compose in it. https://github.com/openstf/stf/issues/966

I'd like to continue this thread here as it's simply regarding the best hardware to use for an stf.

From your experience, I would like to know:

• What would be the most recommended Raspberry PI3 to use? (assuming that they should be on 24/7)
• Would it be better to connect the devices directly to a strong Raspberry PI3 than connecting the devices with a Plugable USB 2.0 7 Port Hub with 60W Power Adapter?
• What would be the best practice for the wifi connections. Meaning, 200 devices in a 30 square meters lab is very dense. Is there a recommended router that can support more than 100 devices? What would be the best practice for setting the wifi network in the lab (how many routers and which ones).
• Is there a recommended switch to connect all the Raspberry PI's to the central server host?
• If there is any other hardware that might fit for setting a lab like that I'd like to hear about that too.

I would really like to hear your ideas and opinions for the best practice to set a lab at this scale.

Thanks.

[gounthar]

I would not say that a Raspberry Pi would be recommended in any way. ;-) It works for me, but I'm not so sure it will be "industrial" enough for your needs. You would need for sure a good USB hub, as the Pi USB ports won't give enough current for your devices. As for the storage, the SDCard is not that good... it shows signs of age after some time. You would be better with a good USB key. Regarding the network part, I would not use Wifi with so many devices in the same room, but reverse tethering like https://github.com/Genymobile/gnirehtet . It has not been compiled for Raspberry Pi yet, but I can give it a try. I'm not surprised with your ADB limitations, as it seems there is a buffer somewhere that can hold up to 49-something Android device IDs. The latest Raspberry Pi 3B+ is supposed to have 1Gb Ethernet, but as it uses the USB bus, it goes only up to 330Mb, so... that's a limitation you have to take into account. Any Gigabit router should do the trick.

[sorccu]

It's very difficult if not borderline impossible to reach perfect connectivity, but there is always room for improvement. I would not use ADB over WiFi as it's way more unreliable than USB in my experience. Since you claim the opposite, there's probably something wrong with your setup, but it's difficult to say what it might be. It could be that the PCIE card you're using isn't great, or perhaps you simply have too many devices per machine. We limited our machines to 28 devices each, so unfortunately I don't have much to say about larger device counts per machine.

You could try decreasing the number of devices and seeing if that helps with stability at all. Perhaps you can find a sweet spot that still allows you to have a relatively high per-machine device count without sacrificing stability.

You can also try industrial USB hubs. Cambrionix is a good one to try out.

[floor23]

Hey guys, I'm here to talk about my expirences. With Raspberry PI3B+, i setup ubuntu arm version,docker,stf. Thus I can connect 6 devices at most. With HP mini pc,industrail hub, ubuntu, docker,I can connect 50 devices at most. Both can run large scale of devices. Hope this will help you all.

[mmirold]

Our experiences with current Raspberry Pis are also mediocre at best. As @floor23 mentioned, hardware-wise 5 oder 6 devices seem to be an upper limit. Maintaining specific ARM images for OpenSTF is no fun either. The main pain point, however, is that due to the low number of devices that may be connected to a Pi, you need quite a lot of them for a complete test lab, which increases the overall susceptibility to hardware failure. And because the usual lab does not have much redundancy, e.g. one device of a kind, you are constantly replacing your Pis (or at least quite often).

After ~9 month of experience, we have decided to switch to Mac minis.

[gounthar]

Interesting. However, I don't agree wholeheartedly regarding the fun of the specific ARM image. What is the main blocking point for you regarding the number of devices? Network, CPU, RAM? I didn't get why you should replace often your Pis (mine have been running for months without a glitch).

[ktoya]

In my experience, insufficient power supply for the Pi when there are more than 6 devices connected, no matter using industrial hub or not.

[yipeng1005]

Hi guy, I am a newbie, have a question I would like to ask, can the phone connect to STF with wifi, when STF and mobile phones are in the same network?

[kontownik]

Depends on USB controllers (you can upgrade it by adding external controllers for PCI and PCI-E) and HUBs quality. I'm running 21 devices on Ubuntu with PC worth about 300$. In my case uploading the .apk it's the most CPU consuming operation.

[vovinio]

Depends on USB controllers (you can upgrade it by adding external controllers for PCI and PCI-E) and HUBs quality. I'm running 21 devices on Ubuntu with PC worth about 300$. In my case uploading the .apk it's the most CPU consuming operation.

It's actually not related to USB controllers. Even if you'll add external PCI-e cards that won't be sufficient for descent scaling. The reason is related to Intel's Chipset architecture that has very limited amount of USB endpoints.

AMD is much more suitable for that particular case. An AMD system could have as much as 254 USB endpoints vs only 96 endpoints with Intel.

[kontownik]

Thanks :) wasn't aware of it. Any particular setup (e.g. chipset + motherboard) that you would recommend? I'm thinking about a new PC for my farm - using Ubuntu.

[avilabss]

@kontownik did you ended up building the hardware? I'm looking for cheapest hardware requirements to connect 200+ usb devices. If over the years you ended up learning more about it then please share. It would really help me a lot.

Thanks :)

[kontownik]

Hey, no... there was no budget to scale it properly... Other thing, worth mentioning, that you need to use multiple adb instances for so many devices. Single instance can work with 10-15 devices in parallel I think.