Weird issue with Omero

[Hamedhm]

Hi,

We have an Omero server V5.3.1 (I know it is old!) to upload 100K+ images, and have recently encountered some issues with it.

Before I start, the config file looks like below:

./omero config get
Ice.IPv6=0
Ice.MessageSizeMax=500000
omero=True
omero.data.dir=/XXXXX/omero_data_dev
omero.db.host=XXXX
omero.db.name=XXXXX
omero.db.pass=XXXXX
omero.db.user=XXXXX
omero.debug=False
omero.jvmcfg.max_system_memory=31000
omero.jvmcfg.percent.blitz=50
omero.jvmcfg.percent.indexer=20
omero.jvmcfg.percent.pixeldata=20
omero.jvmcfg.system_memory=30000
omero.web.application_host=XXXXX/media/omero:80/
omero.web.application_server=fastcgi-tcp
omero.web.debug=True
omero.web.public.enabled=True
omero.web.public.password=XXXXX
omero.web.public.url_filter=webgateway|webclient/annotation
omero.web.public.user=XXXXX
omero.web.static_url=/mi/media/static/

The issue is that the server quickly becomes unavailable. The log below is produced by the process that may have some signals to specify the issue:

2022-03-08 15:18:59,806 - OmeroService - INFO:dataset=YYYYYYY
!! 03/08/22 15:18:59.946 error: cannot create thread for timer:
   Thread.cpp:752: IceUtil::ThreadSyscallException:
   syscall exception: Resource temporarily unavailable
InternalException: Failed to connect: exception ::Ice::UnknownException
{
    unknown = Thread.cpp:752: IceUtil::ThreadSyscallException:
syscall exception: Resource temporarily unavailable
}
2022-03-08 15:18:59,946 - OmeroService - ERROR:assert failed
2022-03-08 15:18:59,963 - omero.gateway - WARNING:ConnectionLostException on <class 'omero.gateway.OmeroGatewaySafeCallWrapper'> to <3fc15a15-0fda-4ded-afad-485dabcbf5c6omero.api.IQuery> 
!! 03/08/22 15:19:00.125 error: cannot create thread for `Ice.ThreadPool.Client':
   Thread.cpp:752: IceUtil::ThreadSyscallException:
   syscall exception: Resource temporarily unavailable

Please have you seen the same issue before?