What is the recommended hardware to run highly available production setup

[woss]

Hi, currently i have two cheap servers with 8 cores each and 64GB of RAM. I thought this would be enough to process at least 100 images a minute but this is not the case.

So I am asking the community to help me out with finding out good hardware/server setup so I can achieve at least 100 processed images per minute.

I am building the image processing layer on top of IPFS and pre-processing each image in 5 different versions, all images are in avif format, and widths are 80, 500,1024, 2056, and 4000 without any added sharpness or blur, and metadata included.

Any help is appreciated.

the output of lscpu:

[lovell]

with 8 cores each

This looks like a CPU with 4 physical cores.

pre-processing each image in 5 different versions, all images are in avif

The bottleneck will almost certainly be AVIF encoding (via libaom).

At a guess the problem may be too many (hyper-)threads. You'll need to provide a complete, standalone code sample and images that allows someone else to both reproduce and profile if you'd like help optimising this further.

[woss]

Uh, a common language mistake about the cores.

Unfortunately, I cannot extract the example from the source code, it is not public. But the fact that the avif might be the slow part is already a start.

I assume jpg is faster?

At a guess the problem may be too many (hyper-)threads

What do you mean by this?

[lovell]

I assume jpg is faster?

Yes, usually significantly, the degree to which can depend on the type of images you're processing.

At a guess the problem may be too many (hyper-)threads

What do you mean by this?

Without sample code and images to reproduce it's hard to say, but it's possible the overhead of thread management and/or CPU cache thrashing might be slowing things. Sometimes reducing concurrency and/or parallelism can help improve throughput.

[lovell]

I hope this information helped. Please feel free to re-open with more details if further assistance is required.

[woss]

Hey, sorry for not replying i was busy with work, i started on the repo that will test the scenario for this issue, but unfortunately, I didn't finish yet.