Possible schedule performance issues?

[jancoonder]

Hello,

We had setup environment with ES 2.2.0 and JDBC importer 2.2.0.0 . We are trying to fill index with data from MS SQL database, which we were able to do. Problem is that when we schedule it to run once every half an hour to reflect possible changes, it keeps consuming significant amount of RAM (regardless of schedule), actually three times more RAM (765 MB) than the HDD index size itself (238MB). Our configuration (sql select is simplified here without naming each field): { "type":"jdbc", "jdbc": { "driver": "com.microsoft.sqlserver.jdbc.SQLServerDriver", "url":"jdbc:sqlserver://192.168.100.1;databaseName=backend", "user":"elastic", "password":"elasticpass", "elasticsearch": [ { "host": "http://localhost:9100/" } ], "sql":"SELECT * FROM PRODUCT", "index":"products", "type":"normal_product", "schedule": "0 15/30 * 1/1 * ? *" } }

Is there any way to improve performance (like not using so much RAM when not updating at the time)? Thanks in advance.

[jprante]

I do not use com.microsoft.sqlserver.jdbc.SQLServerDriver so I can not comment on performance.

The memory usage of JDBC importer is not related to HDD index size. All I can say is, memory usages under 1 GB for a Java VM is not high, no matter if JDBC importer is running or not.

[poseidonqiu]

We are running 2 scripts with different schedules for import 2 mysql tables into es. The usage of RAM is so much. 10029 root 20 0 24.746g 3.899g 15276 S 0.3 6.2 5:44.30 java 12207 root 20 0 25.572g 4.774g 15276 S 0.3 7.6 11:00.04 java

Is there any better idea to improve performance? Thanks.