create SIARD fails due to memory issues

[Laurira]

One Latvian institution tried to convert their database into SIARD format. They faced a problem during conversion. Below is translated description of the problem. dbptk-app.log.txt

Information about database: SQL type and version MSSQL Server 2012 SP3 DB size (MDF file) ~ 22 GB File storage in DB as BLOB Information about server: CPU: 4 core; 2.8 Ghz RAM: 8 GB

When DBPTK is run with defined parameters, the tool stops processing with error message „Exception in thread “main” java.lang.OutOfMemoryError: Java heap space”, when it tryes to read specific table data in RAM. Table contain largesize files in binary format (BLOB). This error message pops up, when DBPTK processed first 2000 rows in the table.

When trying to run DBPTK with such parameter that it can use 6 GB of RAM, process stops in exactly the same BLOB table, with the same message, but after processing first 30000 rows. So the tool works fine, but the problem is, that DBPTK can’t convert tables piece by piece. If there would be option to choose how many rows read into RAM, before thay are converted, then it would be possible to avoid abovementioned error.

For now, in order to convert DB, in which we have tables with BLOB, the institution needs to improve their server with additional large RAM, which is impossible.

Comand line we used: java -Xmx6g -Ddbptk.jdbc.fetchsize.minimum -jar "-Dfile.encoding=UTF-8" dbptk-app-2.3.0.jar --import microsoft-sql-server --import-server-name=SERVERNAME --import-database="DB_NAME" --import-username=Domain\User --import-password="********" --import-use-integrated-login --import-disable-encryption --export siard-2 --export-file=file.siard