Osm - import

[bhjakobsen]

I have been trying to import osm files using the standard osmimport - big and small files.

After some time the imports end with an error (see below).

Does anyone have an idea of what could cause this (i am new to Neo4j) and I really need the osm data at they are the core of my application.

Regards

Bjørn

Error importing OSM file '/Volumes/HD2/osmfiles/denmark-latest.osm': org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:134) at org.neo4j.gis.spatial.osm.OSMImporter.reIndex(OSMImporter.java:322) at org.neo4j.gis.spatial.osm.OSMImporter.reIndex(OSMImporter.java:241) at org.neo4j.gis.spatial.osm.OSMImporter$OSMImportManager.loadTestOsmData(OSMImporter.java:2292) at org.neo4j.gis.spatial.osm.OSMImporter$OSMImportManager.access$1100(OSMImporter.java:2242) at org.neo4j.gis.spatial.osm.OSMImporter.main(OSMImporter.java:2226) Caused by: javax.transaction.RollbackException: Failed to commit, transaction rolled back at org.neo4j.kernel.impl.transaction.TxManager.rollbackCommit(TxManager.java:623) at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:402) at org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:122) at org.neo4j.kernel.TopLevelTransaction.close(TopLevelTransaction.java:124)

[craigtaverner]

I tried the following in the spatial code and it worked: git pull mvn clean install mvn dependency:copy-dependencies rm -Rf osm-db java -cp target/classes:target/dependency/* org.neo4j.gis.spatial.osm.OSMImporter osm-db two-street.osm

I did not (yet) try with a large dataset, like the denmark.osm you tried. Does your problem also occur with the small two-street.osm file?

[bhjakobsen]

Hi

Some of the smaller OSM files like faros-islands works without any problem.

If I try norway or denmark it takes forever and they end with an error. The job takes a lot of resources i set -Xmx25GB and my comp has 24 cores and it uses 75%.

I think the osmimport most be redesigned and able to handle errors in the OSM data which some of the errors could indicate (double nodes).

My assumption is that the database with europe will be around 1TB which is a lot. A planet import to postgresql is less than 500GB.

I really hope that it is possible to make it work as it could be very interesting for my field (logistics - fleet management - task management).

Regards

Bjørn

On 12 Jan 2014, at 17:31, Craig Taverner [email protected] wrote:

I tried the following in the spatial code and it worked: git pull mvn clean install mvn dependency:copy-dependencies rm -Rf osm-db java -cp target/classes:target/dependency/* org.neo4j.gis.spatial.osm.OSMImporter osm-db two-street.osm

I did not (yet) try with a large dataset, like the denmark.osm you tried. Does your problem also occur with the small two-street.osm file?

— Reply to this email directly or view it on GitHub.

[craigtaverner]

It is possible to structure the osm graph much more efficiently. However it would complicate the code somewhat. There are two options I see, either write a different importer that only stores the information needed for your logistics and fleet management needs, or refactor the current code with a number of possible optimizations to improve disk usage and import performance, including memory usage. Neither option is trivial.

[bhjakobsen]

I think option 2 is the best solution for general purposes.

1. News papers want to relate their content to locations,
2. Logistics want to relate their assets to each other, tasks to routes, goods to warehouses etc. - calculate routes - get rid of geocoding/routing services.
3. Police wants to relate …..

They all need substantial amounts of OSM data..

The target should be be able to import and index europe in about 3-4 days on capable hardware.

Regards

Bjørn

On 12 Jan 2014, at 17:56, Craig Taverner [email protected] wrote:

It is possible to structure the osm graph much more efficiently. However it would complicate the code somewhat. There are two options I see, either write a different importer that only stores the information needed for your logistics and fleet management needs, or refactor the current code with a number of possible optimizations to improve disk usage and import performance, including memory usage. Neither option is trivial.

— Reply to this email directly or view it on GitHub.

[jexp]

Björn, we'll try to look into that soon.

[bhjakobsen]

Hi

What does soon mean in real terms?

Regards

Bjørn On 12 Jan 2014, at 20:24, Michael Hunger [email protected] wrote:

Björn, we'll try to look into that soon.

— Reply to this email directly or view it on GitHub.

[Kidikarus]

Craig,

I just did a git clone, mvn clean install and during the unit tests after the build phase I got the following:

TestOSMImport$1.runTest:99 SpatialDatabase java.io.IOException: Cannot overwri... TestOSMImport$1.runTest:99 SpatialDatabase Other node is null. TestOSMImport$1.runTest:99 SpatialDatabase java.io.IOException: Cannot overwri... TestOSMImport$1.runTest:99 SpatialDatabase Unable to commit transaction TestsForDocs.testExportShapefileFromQuery:208->importMapOSM:114 ╗ Runtime java... TestsForDocs.setUp:61->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestsForDocs.setUp:61->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatialQueries>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatialUtils>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatialUtils>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

It looks as though it is having difficulties even with the Sweden osm dataset

[bhjakobsen]

Delete the target database and run it again - might work but the Sweden osm might be to big - try faroe islands or some of the other small ones ( less than 500mb)

Regards

Bjørn

Sendt fra min iPhone

Den 15/01/2014 kl. 19.09 skrev Kidikarus [email protected]:

Craig,

I just did a git clone, mvn clean install and during the unit tests after the build phase I got the following:

TestOSMImport$1.runTest:99 SpatialDatabase java.io.IOException: Cannot overwri... TestOSMImport$1.runTest:99 SpatialDatabase Other node is null. TestOSMImport$1.runTest:99 SpatialDatabase java.io.IOException: Cannot overwri... TestOSMImport$1.runTest:99 SpatialDatabase Unable to commit transaction TestsForDocs.testExportShapefileFromQuery:208->importMapOSM:114 ╗ Runtime java... TestsForDocs.setUp:61->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestsForDocs.setUp:61->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatialQueries>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatialUtils>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime TestSpatialUtils>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

It looks as though it is having difficulties even with the Sweden osm dataset

— Reply to this email directly or view it on GitHub.

[Kidikarus]

Thanks for the advice Bjorn but this is just running the neo4j-spatial mvn install. There is a test Sweden osm data set in there. The XADataManager is shutting down during the regression testing phase of the build. I'm not sure if this is due to the 2.0 update or not. On Jan 15, 2014 2:09 PM, "bhjakobsen" [email protected] wrote:

Delete the target database and run it again - might work but the Sweden osm might be to big - try faroe islands or some of the other small ones ( less than 500mb)

Regards

Bjørn

Sendt fra min iPhone

Den 15/01/2014 kl. 19.09 skrev Kidikarus [email protected]:

Craig,

I just did a git clone, mvn clean install and during the unit tests after the build phase I got the following:

TestOSMImport$1.runTest:99 SpatialDatabase java.io.IOException: Cannot overwri... TestOSMImport$1.runTest:99 SpatialDatabase Other node is null. TestOSMImport$1.runTest:99 SpatialDatabase java.io.IOException: Cannot overwri... TestOSMImport$1.runTest:99 SpatialDatabase Unable to commit transaction TestsForDocs.testExportShapefileFromQuery:208->importMapOSM:114 ╗ Runtime java...

TestsForDocs.setUp:61->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestsForDocs.setUp:61->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatialQueries>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatialUtils>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatialUtils>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

It looks as though it is having difficulties even with the Sweden osm dataset

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/neo4j/spatial/issues/126#issuecomment-32420788 .

[bhjakobsen]

You probably have a database file try and delete it and run mvn clean install.

Regards

Bjørn

Sendt fra min iPad

Den 15/01/2014 kl. 23.58 skrev Kidikarus [email protected]:

Thanks for the advice Bjorn but this is just running the neo4j-spatial mvn install. There is a test Sweden osm data set in there. The XADataManager is shutting down during the regression testing phase of the build. I'm not sure if this is due to the 2.0 update or not. On Jan 15, 2014 2:09 PM, "bhjakobsen" [email protected] wrote:

Delete the target database and run it again - might work but the Sweden osm might be to big - try faroe islands or some of the other small ones ( less than 500mb)

Regards

Bjørn

Sendt fra min iPhone

Den 15/01/2014 kl. 19.09 skrev Kidikarus [email protected]:

Craig,

I just did a git clone, mvn clean install and during the unit tests after the build phase I got the following:

TestOSMImport$1.runTest:99 SpatialDatabase java.io.IOException: Cannot overwri... TestOSMImport$1.runTest:99 SpatialDatabase Other node is null. TestOSMImport$1.runTest:99 SpatialDatabase java.io.IOException: Cannot overwri... TestOSMImport$1.runTest:99 SpatialDatabase Unable to commit transaction TestsForDocs.testExportShapefileFromQuery:208->importMapOSM:114 ╗ Runtime java...

TestsForDocs.setUp:61->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestsForDocs.setUp:61->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSimplePointLayer>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatial.setUp:90->Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatialQueries>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatialUtils>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

TestSpatialUtils>Neo4jTestCase.setUp:88->Neo4jTestCase.setUp:107->Neo4jTestCase.reActivateDatabase:153 ╗ Runtime

It looks as though it is having difficulties even with the Sweden osm dataset

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHubhttps://github.com/neo4j/spatial/issues/126#issuecomment-32420788 .

— Reply to this email directly or view it on GitHub.

[jexp]

The tests work locally for me.

[craigtaverner]

Works for me too. The sweden.osm.administrative dataset is 17MB and took about 10s to import. On my ubuntu laptop.