Horizon: DB Runway: reingest with a HDD disk, and time it against the current SSD

[2opremio]

[bartekn]

Do we need this? I'm 100% sure HDD are just to slow for Horizon.

[kainz]

You would be better served by determining a per-ledger ingest io/operations required estimate and scaling that up. FWIW i've been running a series of stellar horizon nodes with a 7 million ledger retention count.

My numbers are rough, but it looks like an 'idle' (only ingesting, zero api traffic) mainnet horizon instance in my setup (32GB ram, 6GB postgres shared_buffers) consistently uses around 4-7K IOPs (as reported via tps output from linux iostat at 10 second intervals, using XFS as the postgres volume), with periodic spikes into the 10-14k IOPS range. I would guess at needing to burst to 8k IOPS for significant periods of time to be able to keep up with the network. Active API use will increase that depending on your application traffic.

If you can build a HDD array that can meet those TPS numbers you can ingest on HDD with no problem. Getting hard numbers out of HDD performance in the wild generally requires benchmarking, as vendors don't really commit to hard numbers on their datasheets, even for 'enterprise' drives. As an example though we can extrapolate and still get some ballpark figures. If we look at, for example, the 2022 Seagate Exos 4kN SAS drives (https://www.seagate.com/files/www-content/datasheets/pdfs/exos-7e10-DS1957-6M-2104US-en_US.pdf), which are listed across their range of having an 'average' latency of 4.16ms, we can guess that we'd at least be able to achieve 6ms access times/seek speed. Practically this means that to ensure duplication of my above setup, you would need a JBOD of around 100 HDDs to get to the same IOPS rate. (6ms suggest around 166 IOPS per drive).

A modern 'enterprise' SSD or nVME SSD will easily produce orders of magnitude faster operation. For comparison, seagate's 'Nytro' SSD line (https://www.seagate.com/www-content/datasheets/pdfs/nytro-3032-sas-ssd-DS2047-2-2011US-en_US.pdf) claims to have 40k-160k IOPS depending on the drive model.

So yes, HDDs are too slow to run Stellar Horizon, unless you're building a purpose-built array for the task.