Region fixer crashing when attempting to fix certain worlds

[VL4DST3R]

trafficstars

Describe the bug On my multiverse server I have quite a number of maps (a little over 40 atm) of varying ages and levels of minecraft fuckery/corruption affecting them- that I'm currently trying to deal with. While some of the maps were fixed successfully, others would cause Region fixer to throw out an "Ops! Something went really wrong and regionfixer crashed." error. Bug reports below.

Full copied text from the MS-DOS view All worlds run the same command and arguments, only the path being different: python regionfixer.py --fc --fm --fw -p 24 "E:\Minecraft\Minecraft Bukkit Server\worlds\creative"

DO note that i run this from a batch file to help with the iteration for each world, but i doubt it should cause any issues.

Also, since it seems that the Multiverse way of structuring dimensions in their own wold_[dimension_name] structure is not recognized by Region fixer- eg: "Overworld" worlds path can be left as worlds\creative", and Region fixer also detects the level.dat (and all the other) files present there correctly, Nether/End worlds require full path to the dimensions, Ex: worlds\creative_nether\DIM-1" and worlds\creative_the_end\DIM1 respectively to be read at all, and cannot make use of level.dat or other data present in the world folder.

Expected behavior While i understand it can't literally work miracles with all my wonky worlds, i'm expecting it to at least not completely crash when reading some of them? Again, logs attached below.

Screenshots n/a

Files that would help solving the issue if requested i can provide some of the worlds causing those issues

Desktop (please complete the following information):

• OS: Win10 2004
• Python version: 3.7.3
• Region Fixer Version 0.3.3

Additional context

Bug report 1 (and the most common):

<class 'TypeError'>
Traceback (most recent call last):
  File "E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 374, in fix_problematic_chunks
    chunk = region_file.get_chunk(*local_coords)
  File "E:\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 599, in get_chunk
    return self.get_nbt(x, z)
  File "E:\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 574, in get_nbt
    data = self.get_blockdata(x, z) # This may raise a RegionFileFormatError.
  File "E:\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 566, in get_blockdata
    raise ChunkDataError(err)
nbt.region.ChunkDataError: Error -3 while decompressing data: incorrect data check

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "regionfixer.py", line 588, in <module>
    value = main()
  File "regionfixer.py", line 548, in main
    fix_bad_chunks(args, w)
  File "regionfixer.py", line 71, in fix_bad_chunks
    counter = scanned_obj.fix_problematic_chunks(problem)
  File "E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 1398, in fix_problematic_chunks
    counter += regionset.fix_problematic_chunks(status)
  File "E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 936, in fix_problematic_chunks
    counter += self._set[r].fix_problematic_chunks(status)
  File "E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 391, in fix_problematic_chunks
    out += dc.decompress(i)
TypeError: a bytes-like object is required, not 'int'

Bug report 2 (just one instance):

**********
*** Exception while scanning:
*** 0f31e1bc-88f0-3d0a-95c5-47cd965cb541-6507148097825114692.dat
**********
*** Printing the child's traceback:
*** Exception:<class 'EOFError'>Compressed file ended before the end-of-stream marker was reached
**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\scan.py, line 746, in scan_data
***   _ = nbt.NBTFile(filename=s.path)**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 628, in __init__
***   self.parse_file()**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 655, in parse_file
***   self._parse_buffer(self.file)**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 493, in _parse_buffer
***   tag._parse_buffer(buffer)**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 486, in _parse_buffer
***   name = TAG_String(buffer=buffer).value**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 345, in __init__
***   self._parse_buffer(buffer)**********
*** File E:\Minecraft\Programs\Minecraft Region Fixer\nbt\nbt.py, line 350, in _parse_buffer
***   read = buffer.read(length.value)**********
*** File C:\Users\Vlad\AppData\Local\Programs\Python\Python37-32\lib\gzip.py, line 276, in read
***   return self._buffer.read(size)**********
*** File C:\Users\Vlad\AppData\Local\Programs\Python\Python37-32\lib\_compression.py, line 68, in readinto
***   data = self.read(len(byte_view))**********
*** File C:\Users\Vlad\AppData\Local\Programs\Python\Python37-32\lib\gzip.py, line 482, in read
***   raise EOFError("Compressed file ended before the "
**********

[VincyChannel]

Same for me

[mawoka-myblock]

And for me! Any fix so far?

[VL4DST3R]

This project is very much dead, so don't expect any fixes anytime soon. Also given 1.8 is soon approaching which will probably completely change how worlds are saved yet again, this tool will most likely die with it.

[VincyChannel]

This project is very much dead, so don't expect any fixes anytime soon. Also given 1.8 is soon approaching which will probably completely change how worlds are saved yet again, this tool will most likely die with it.

Yes I know, now this project is dead, rip

[Fenixin]

It's dead for now. I don't have the free time to update this and the NBT library and there are not many people cooperating so, yeah, it's dead for now. I kind of feel bad saying this but right now it feels like a chore (it was fun before).

This project is very much dead, so don't expect any fixes anytime soon. Also given 1.8 is soon approaching which will probably completely change how worlds are saved yet again, this tool will most likely die with it.

I didn't know that 1.8 was changing the save format. Do you have any information/source about this?

[VL4DST3R]

I said "probably"; but it does seem reasonable to expect that, considering world saves now have to deal with a completely new generation algorithm (and probably new parameters to set up for it), increased height limit and introduction of the 3d biomes to achieve these new terrain features.

[Fenixin]

We already had changes in the game like that without changes in the world format. For example: generation algorithm changed a few times, the height limit changed at least once withouth repercuting.

A really hope they don't change the world format.

[mawoka-myblock]

Although the project is dead for now, your tool really helped me and fixed my issue by deleting, not fixing the chunk. Thanks for your great work, @Fenixin ❤️ !

[Fenixin]

Although the project is dead for now, your tool really helped me and fixed my issue by deleting, not fixing the chunk. Thanks for your great work, @Fenixin ❤️ !

Yeah, maybe ut should be called Chunk-remover, because I think that's the most used feature :P

Thaks a lot for the kind comment, it means a lot!

[Fenixin]

@VL4DST3R , ok, so this may be too late but, could you provide onen of the worlds that crash regionfixer? At least, could you provide a region file for type of crash?

You can use PM or post here a link.

[VL4DST3R]

@VL4DST3R , ok, so this may be too late but, could you provide onen of the worlds that crash regionfixer? At least, could you provide a region file for type of crash?

You can use PM or post here a link.

Sure thing, I'm sure i have a couple with both flavors of errors. I'll post a Dropbox link tomorrow when i finish work since I'll need to batch re-run regionfixer on all the maps we have to find them again.

[Fenixin]

@VL4DST3R, thanks! When scanning, please, pull the latest master branch.

[VL4DST3R]

Using the latest build, (no changes in errors btw) here are two levels that have the first issue: overworld and a nether dimension to differentiate a bit, and finally with the latter, which turns out i have only one map affected by it. Do tell me when you get around to downloading them so i can delete them from db.

[Fenixin]

Thanks! I've got the files. You can delete them.

[VL4DST3R]

Great, im curious if you manage to find what the issue is!

[VL4DST3R]

Thanks! I've got the files. You can delete them.

Any updates on this?

[Fenixin]

Not really, sorry. I'm in the end of term marking examns like crazy. I hope soon.

[l1m3r]

I ran into the same fault (exact same error) a few days ago but I found a solution for my specific case. To upgrade from spigot 1.17.1 to 1.18 the spigot team recommended to --forceUpgrade first on 1.17.1 and then again on 1.18 Problem was my 3277-Spigot-3892929-2071671 "failed to read some chunks" leading to Error upgrading world.

Running regionfixer.py after that failed --forceUpgrade lead to the crash documented by OP.

Solution was to update to 3284-Spigot-3892929-0ebef35. Upgrading the map worked flawlessly that way (with regionfixer.py cleaning up some minor stuff before the upgrade).

[Fenixin]

I have pushed a change that adds EOFError to the list of exceptions. Those files that crashed regionfixer should be now correctly flagged as broken data files.

I think this error could be handled better but for now this is enough.

I'll leave this as open for some time, I want to test a few things.

Thanks for the worlds and the reports!

[ab-tools]

Hello Alejandro, first thanks a lot for all your efforts with this great tool! :-)

Not sure if you're still interested in new reports for this issue, but for me the tool still crashes using latest master GIT branch version and this region file while trying to fix the 4 broken chunks: r.-1.-1.zip

Nothing critical in this region, so I can just go ahead and delete them, still wanted to let you know this problem seems to still persist.

Thanks again Andreas

[Fenixin]

Hello @ab-tools!

Thanks very much for your report! I really appreciate it. I have downloaded the file and you can remove it if you want.

I'll look into it when I get the time (who knows when...).

Thanks again!

[ab-tools]

Thanks for your reply, Alejandro, you're welcome and no hurry from my side!

[Fenixin]

Just pushed a new branch with a possible fix for this issue. Could anybody test it?

This is the branch: https://github.com/Fenixin/Minecraft-Region-Fixer/tree/bytes-fix

[VL4DST3R]

Just pushed a new branch with a possible fix for this issue. Could anybody test it?

This is the branch: https://github.com/Fenixin/Minecraft-Region-Fixer/tree/bytes-fix

On it, testing now. BTW tested the one you pushed a few hours ago in the hopes it would handle the corruption better, same issue on that one, but i guess you knew that already.

EDIT: nevermind, it seems its the same with the one i tested, which means that no, the issues i'm experiencing are still here, here is some of the output log (same error shows up for all affected worlds)

Unreadable player files:
No problems found.

Unreadable data files:
No problems found.

Chunk problems:
---------------------------------------------
| Problem | Corrupted  Wrong located  Total |
---------------------------------------------
|  Count  |    16           107       21942 |
---------------------------------------------

Region problems:
No problems found.





######### Repairing chunks with status: Corrupted ##########
Repairing chunks in regionset "Level/Region files for Overworld":
The chunk 25,31 in region file region\r.0.-2.mca couldn't be fixed.
The chunk 27,31 in region file region\r.0.-2.mca couldn't be fixed.


Ops! Something went really wrong and regionfixer crashed.


Bug report:

<class 'TypeError'>
Traceback (most recent call last):
  File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 375, in fix_problematic_chunks
    chunk = region_file.get_chunk(*local_coords)
  File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 622, in get_chunk
    return self.get_nbt(x, z)
  File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 597, in get_nbt
    data = self.get_blockdata(x, z) # This may raise a RegionFileFormatError.
  File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\nbt\region.py", line 589, in get_blockdata
    raise ChunkDataError(err)
nbt.region.ChunkDataError: Error -3 while decompressing data: invalid distance too far back

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "regionfixer.py", line 574, in <module>
    value = main()
  File "regionfixer.py", line 534, in main
    fix_bad_chunks(args, w)
  File "regionfixer.py", line 71, in fix_bad_chunks
    counter = scanned_obj.fix_problematic_chunks(problem)
  File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 1453, in fix_problematic_chunks
    counter += regionset.fix_problematic_chunks(status)
  File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 971, in fix_problematic_chunks
    counter += self._set[r].fix_problematic_chunks(status)
  File "E:\My Games\For Games\Minecraft\Programs\Minecraft Region Fixer\regionfixer_core\world.py", line 392, in fix_problematic_chunks
    out += dc.decompress(i)
TypeError: a bytes-like object is required, not 'int'

[VL4DST3R]

I sent you 2 worlds a while back, and those two are still some of those causing errors on the latest build. Have you by chance tried running the fixer on those maps yourself?

If you need i can provide them again.

[Fenixin]

I sent you 2 worlds a while back, and those two are still some of those causing errors on the latest build. Have you by chance tried running the fixer on those maps yourself?

If you need i can provide them again.

Those are the world I'm testing this with, and they work. The error you posted tells me that you are not in the branch bytes-fix, I really think you downloaded/didn't check out the correct branch. Could you please check that?

[VL4DST3R]

Are you sure-sure the build there is the one thats updated? I double-checked and it acts the same?

[Fenixin]

Ooops... sorry about that, give me a moment...

[VL4DST3R]

LMFAO i was about to go crazy!

[Fenixin]

I'm sorry about that! It should be there now.