osm-lib icon indicating copy to clipboard operation
osm-lib copied to clipboard

Array overrun while writing planet.vex

Open abyrd opened this issue 9 years ago • 2 comments

While writing planet.vex:

19:27:55.007 [Thread-0] DEBUG com.conveyal.osmlib.VEXBlock - Contained 5263 entities with type 2.
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1048576
    at com.conveyal.osmlib.DeflatedBlockWriter.write(DeflatedBlockWriter.java:129)
    at java.io.OutputStream.write(OutputStream.java:116)
    at java.io.OutputStream.write(OutputStream.java:75)
    at com.conveyal.osmlib.VarIntOutputStream.writeBytes(VarIntOutputStream.java:52)
    at com.conveyal.osmlib.VarIntOutputStream.writeString(VarIntOutputStream.java:58)
    at com.conveyal.osmlib.VexOutput.writeRelation(VexOutput.java:168)
    at com.conveyal.osmlib.PBFInput.parseRelations(PBFInput.java:190)
    at org.openstreetmap.osmosis.osmbinary.BinaryParser.parse(BinaryParser.java:122)
    at org.openstreetmap.osmosis.osmbinary.BinaryParser.handleBlock(BinaryParser.java:68)
    at org.openstreetmap.osmosis.osmbinary.file.FileBlock.process(FileBlock.java:135)
    at org.openstreetmap.osmosis.osmbinary.file.BlockInputStream.process(BlockInputStream.java:34)
    at com.conveyal.osmlib.PBFInput.copyTo(PBFInput.java:237)
    at com.conveyal.osmlib.main.Converter.main(Converter.java:32)

abyrd avatar Oct 28 '15 18:10 abyrd

This is because there are OSM entities bigger than com.conveyal.osmlib.DeflatedBlockWriter#MAX_MESSAGE_SIZE

abyrd avatar Oct 28 '15 18:10 abyrd

Temporary fix in 26deed74bb2caeedc13a1e1e2854046af1162ccc (just made the buffer bigger). A better solution would be to copy any unfinished message to the beginning of the next block buffer.

abyrd avatar Oct 28 '15 20:10 abyrd