Question: parallel run

[tzipperle]

Is it possible run message parallel with python multiprocess? Or what is the best way to run message parallel?

[gidden]

Normally I run things in parallel with parallel processes on the OS (so have a script that launches everything then run that in parallel). I suspect it would also be possible to use the multiprocess python package to do so in a single script, but have not tried.

On Sat, Oct 13, 2018 at 2:39 PM Thomas Zipperle [email protected] wrote:

Is it possible run message parallel with python multiprocess? Or what is the best way to run message parallel?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/iiasa/ixmp/issues/91, or mute the thread https://github.com/notifications/unsubscribe-auth/ABVAEd7RZLcX38z13h15rGta4Y7QMg9aks5ukd8JgaJpZM4XanFC .

[tzipperle]

I have tried multiprocess with HSQLDB and I got a error. Do I need an oracle data base for parallel run?

[gidden]

Ah, that I do not know... out of curiosity what error did you get?

On Wed, Oct 17, 2018 at 10:30 AM Thomas Zipperle [email protected] wrote:

I have tried multiprocess with HSQLDB and I got a error. Do I need an oracle data base for parallel run?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/iiasa/ixmp/issues/91#issuecomment-430538069, or mute the thread https://github.com/notifications/unsubscribe-auth/ABVAERn62X9Zvb-kX4lMtg1m_XTZ2Yw1ks5uluqJgaJpZM4XanFC .

[danielhuppmann]

If you use HSQLDB in file-based mode, only one process can access it a time. There is a way to launch an HSQLDB instance in server mode, which would allow multiple connections - but I have never tried that...

You will need a configuration file with the following, not sure what the driver should be. The url should probably be the path.

config.name = <some tag>

jdbc.driver = <??>
jdbc.url = <??>
jdbc.user = ixmp
jdbc.pwd = ixmp

[tzipperle]

I have also tried with the server mode. I started the server with:

java -cp ../lib/hsqldb.jar org.hsqldb.server.Server --database.0 file:message --dbname.0 xdb

And with the following config:

config.name = meassage@local

jdbc.driver.1 = org.hsqldb.jdbcDriver
jdbc.url.1 = jdbc:hsqldb:hsql://localhost/xdb
jdbc.user.1 = ixmp
jdbc.pwd.1 = ixmp

jdbc.driver.2 = org.hsqldb.jdbcDriver
jdbc.url.2 = jdbc:hsqldb:hsql://localhost/xdb
jdbc.user.2 = ixmp
jdbc.pwd.2 = ixmp

I could connect to the data base. But only one connection.

[danielhuppmann]

Oh, so you were already on the right track. Sorry that it doesn't work as expected.

@peterkolp, any ideas?

[tzipperle]

Today I did a next try. I have startet a server from the installed ixmp from my conda environment.

java -cp C:\Users\anaconda3\envs\message-ghd\Lib\site-packages\ixmp\lib\hsqldb.jar org.hsqldb.server.Server --database.0 file:C:\Users\.local\ghd_server --dbname.0 ghd_server

But if I try to connect to the server, I get an error:

jpype._jexception.FlywayExceptionPyRaisable: org.flywaydb.core.api.FlywayException: Validate failed: Migration checksum mismatch for migration 1

Do you have any ideas to fix this problem?

[danielhuppmann]

@zikolach, can you take a look?

[zikolach]

@tzipperle to start HSQLDB server you may use following command (same as in your comment):

java -cp ../../ixmp/lib/hsqldb.jar org.hsqldb.server.Server --database.0 file:ixmptest --dbname.0 ixmptest

To check you can connect it use following:

java -cp ../../ixmp/lib/hsqldb.jar org.hsqldb.util.DatabaseManagerSwing \
    --driver org.hsqldb.jdbcDriver \
    --url jdbc:hsqldb:hsql://localhost:9001/ixmptest \
    --user ixmp \
    --password ixmp

If it succeed the JDBC URL is jdbc:hsqldb:hsql://localhost:9001/ixmptest. Since you have migration issue - my assumption is you have done steps from above right!

Now we need to figure out why you have different checksums for migration files. You should see it in the log output as existing and new values. Using HSQLDB client you may try to change checksum in the database manually to fix it (assuming both migrations - the one previously applied and current one - have same changes differs only in formatting/line endings/etc).

Please do a backup copy of directory with database before changing anything manually!

If you attach more detailed log - we can try to figure out what exactly is the problem.

Also you may be interested in using Postgresql database. Recently we added it support (for testing purposes to the moment). But you'll need to care about data migration yourself.

[tzipperle]

java -cp ../../ixmp/lib/hsqldb.jar org.hsqldb.util.DatabaseManagerSwing
--driver org.hsqldb.jdbcDriver
--url jdbc:hsqldb:hsql://localhost:9001/ixmptest
--user ixmp
--password ixmp

I could not check this under windows 10. I get a error unknown driver and the log file is after the run empty.

I start with a new project without a database. It should not be a problem with the data migration

[zikolach]

@tzipperle please ensure the hsqldb.jar file exists in the folder at specified path ../../ixmp/lib/hsqldb.jar - you may need to change path.

If you want to try postgresql option - options for config file may look like:

jdbc.driver = org.postgresql.Driver
jdbc.url = jdbc:postgresql://localhost:5432/ixmptest
jdbc.user = ixmp
jdbc.pwd = ixmp

You'll need to create an empty DB before trying to access it or (if you familiar with Docker) use following compose file to run Postgresql instance in container:

version: "3"
services:
  db:
    image: postgres:10
    environment:
      POSTGRES_USER: ixmp
      POSTGRES_PASSWORD: ixmp
      POSTGRES_DB: ixmptest
    ports:
      - "5432:5432"
    volumes:
      - /path/on/the/host/system:/var/lib/postgresql/data

In the snippets above ixmptest is a name of database schema.

[tzipperle]

I had a problem with the backslash. Now I can connect to the server.

I have tried the following workflow:

1. Deleted the old database
2. Started the server
3. Opened a Platform to the server
4. Migration error

The log-file now contains the following information:

/C1/SET SCHEMA SYSTEM_LOBS INSERT INTO BLOCKS VALUES(0,2147483647,0) COMMIT

[tzipperle]

I could fix the problem with the migration. It was a problem with the conda environment. After I recreated the environment it works. @zikolach Thanks for your help.

[tzipperle]

Hi @zikolach , I have a short question. I use a HSQLDB server and after about 1000 scenario runs the RAM usage of the process "Zulu Platform x64 Architecture" increase highly and the I get an server error.

I think I have to change the table type from memory to cached (hsqldb.default_table_type=cached). But I am not sure how I could do this.

Do you have any ideas what I can do?