C++ thin Client (2.17.0) - Slow write throughput

[hsadia538]

I have my local Apache Ignite setup with 2 nodes that interact with Java and C++ engines. I am experiencing extremely slow write, where I pick jobs in a loop, process data Every time, each job writes ~1320000 records in Ignite. I am using PutAll() in a batch of 3000 records with 2 threads in a C++ based application. 4 such applications write to ignite simultaneously the same number of records.

Specs of machine: memory=25GB
processors=18
swap=12GB

Question: Is it possible to improve the performance of my local setup in terms of the time taken to complete a job? Am I using inefficient structures or inefficiently inserting the records? Initially, when it takes about 309 to do the insertion, I cannot understand why the time is exponentially increasing for subsequent jobs.

The performance measured is

- Job 1: Time taken to write results to Ignite: 309seconds.
  
  - Job 2: Time taken to write results to Ignite: 343 seconds
  - Job 3: 469 seconds
  - Job 4:760 seconds
  - Job 5; 945 seconds  

I am running this whole setup in docker where my ignite config looks like :

        class="org.apache.ignite.configuration.IgniteConfiguration">
        <!--  Set to true to enable distributed class loading for examples, default is false.  -->
        <property name="peerClassLoadingEnabled" value="true" />
        <property name="publicThreadPoolSize" value="16" />
        <property name="systemThreadPoolSize" value="16" />
        <!-- Enable persistence -->
        <property name="dataStorageConfiguration">
            <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
                <property name="defaultDataRegionConfiguration">
                    <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
                    
                        <property name="maxSize" value="#{4L * 1024 * 1024 * 1024}" /> <!-- 4GB -->
                        <property name="initialSize" value="#{1L * 1024 * 1024 * 1024}" /> <!-- 1GB -->

                        <property name="name" value="Default_Region"/>
                        <property name="persistenceEnabled" value="true"/>
                        <property name="checkpointPageBufferSize" value="#{64 * 1024 * 1024}"/>
                    </bean>
                </property>
                <property name="walMode" value="LOG_ONLY"/> <!-- or BACKGROUND for faster but less durable -->
                <property name="walSegmentSize" value="#{64 * 1024 * 1024}"/>
                <property name="walSegments" value="20"/>
            </bean>
        </property>
        <!--  Enable task execution events for examples.  -->
        <property name="includeEventTypes">
            <list>
                <!-- Task execution events -->
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED" />
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED" />
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED" />
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_TIMEDOUT" />
                <util:constant
                    static-field="org.apache.ignite.events.EventType.EVT_TASK_SESSION_ATTR_SET" />
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_REDUCED" />
                <!-- Cache events -->
                <util:constant
                    static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_PUT" />
                <util:constant
                    static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_READ" />
                <util:constant
                    static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_REMOVED" />
            </list>
        </property>
        <!--  Explicitly configure TCP discovery SPI to provide list of initial nodes.  -->
        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                        <property name="addresses">
                            <list>
                                <value>ignite-1:47500..47509</value>
                                <value>ignite-2:47500..47509</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
</beans>```


**Sample code (for reference only, doesn't compile):**

```const size_t BATCH_SIZE = 2500;
const int NUM_THREADS = 2;
const int maxRetries = 3;
const std::chrono::milliseconds retryDelay(1000);
if (analytic_name == "MYANALYTIC") {
    std::map<ResultKey,ResultValue> batch;
    
for (size_t i = startIdx; i < endIdx; ++i) {
        const auto& v = allData[i];
        if (v.size() < 13) continue;

    std::string uniqueId = groupId + "_" + std::to_string(i);
    ResultKey key(uniqueId, v[0]);
    ResultValue val(v[1], v[2], std::stod(v[3]), std::stod(v[4]),
                       v[5], std::stod(v[6]), v[7], std::stod(v[8]),
                       v[9], std::stod(v[10]), v[11], v[12]);

    batch[key] = val;
    if (++rowsWritten % BATCH_SIZE == 0) {
        for (int attempt = 0; attempt < maxRetries; ++attempt) {
            try {
                myCache.PutAll(batch);
                batch.clear();
                break;
            } catch (...) {
                std::this_thread::sleep_for(retryDelay);
            }
        }
    }
}
if (!batch.empty()) {
    for (int attempt = 0; attempt < maxRetries; ++attempt) {
        try {
            myCache.PutAll(batch);
            break;
        } catch (...) {
            std::this_thread::sleep_for(retryDelay);
        }
    }
}

}