Method to upload a large batch at max speed

[marclaporte]

Could / should the Manticore PHP client be adapted?

[sanikolaev]

Hi @marclaporte

1. disable binlog

This has been mitigated here https://github.com/manticoresoftware/manticoresearch/issues/879

2. write to multiple tables at once

This will be resolved once auto-sharding is completed:

• https://github.com/manticoresoftware/manticoresearch-buddy/issues/150 - done
• https://github.com/manticoresoftware/manticoresearch/issues/1087 - yet to do

Also, I believe the client shouldn't manage the schema directly anyway.

3. write with a higher concurrency
4. write with a larger batch

It might make sense to integrate this into the client. The idea would be that when the client receives a large batch of documents, it starts sending them to Manticore at the maximum possible speed, automatically adjusting concurrency and batch size. This could be based on factors like the number of CPU cores and workers available in the Manticore instance, increasing until further gains are no longer feasible.

[marclaporte]

Many things have been addressed.

One remaining blocker: https://github.com/manticoresoftware/manticoresearch/pull/3073

[marclaporte]

AFAICT, the last blocker is https://github.com/manticoresoftware/manticoresearch/issues/1087