coreos-assembler
coreos-assembler copied to clipboard
Published
20 hours ago •
coreos
coreos
OSBuild without compression yields GRUB failures
When we switched OSbuild over to not compressing we started getting failures where images wouldn't boot like:
Booting `Fedora CoreOS 40.20240207.dev.1 (ostree:0)'
error: ../../grub-core/fs/fshelp.c:257:file
`/boot/ostree/fedora-coreos-f25027bf4dfb2213
7b7c8401ed2396924ba112482afc30d62b7a4ab1310b09db/vmlinuz-6.8.0-0.rc0.20240112gi
t70d201a40823.5.fc40.x86_64' not found.
error: ../../grub-core/loader/i386/pc/linux.c:422:you need to load the kernel
first.
Press any key to continue...
some more info.. it appears something with the ext4 filesystem when OSBuild runs the qemu-img convert to get from raw to qcow2 is getting corrupted. I converted the image back into a raw image from the qcow2 and mount up the third partition and when I ls on the ostree/ dir I see:
bash-5.2# ls /mnt/ostree/
[ 7220.329345] EXT4-fs error (device loop0p3): ext4_lookup:1855: inode #65538: comm ls: iget: checksum invalid
ls: cannot access '/mnt/ostree/': Bad message
ok further update.. I became increasingly suspicious of the corruption happening earlier in the OSBuild steps themselves rather than when copying the file out of the supermin VM.
I decided to just add a sanity check into the org.osbuild.qemu OSBuild stage that immediately ran a qemu-img compare after it created the qcow2 from the raw image. Here's the patch:
diff --git a/stages/org.osbuild.qemu b/stages/org.osbuild.qemu
index 642b5146..54e707d4 100755
--- a/stages/org.osbuild.qemu
+++ b/stages/org.osbuild.qemu
@@ -219,22 +219,34 @@ def main(inputs, output, options):
if coroutines:
print(f"qemu-img coroutines: {coroutines}")
extra_args += ["-m", coroutines]
cmd = [
"qemu-img", "convert",
"-O", fmt["type"],
*extra_args,
source, target
]
subprocess.run(
cmd, check=True
)
+ # Sanity check that the image is 100%
+ cmd = [
+ "qemu-img", "compare",
+ "-f", "raw",
+ "-F", fmt["type"],
+ source, target
+ ]
+ subprocess.run(
+ cmd, check=True
+ )
+
+
return 0
if __name__ == '__main__':
args = osbuild.api.arguments()
r = main(args["inputs"], args["tree"], args["options"])
sys.exit(r)
and sure enough one of the first runs I went through complained:
⏱ Duration: 0s
Pipeline qemu: dd43158cea505a60ebd511dc09e0fba0247cef4903ebf9c5b2c70e813b54a2cf
Build
root: <host>
runner: org.osbuild.fedora38 (org.osbuild.fedora38)
org.osbuild.qemu: dd43158cea505a60ebd511dc09e0fba0247cef4903ebf9c5b2c70e813b54a2cf {
"filename": "fedora-coreos-39.20240214.dev.0-qemu.x86_64.qcow2.tmp",
"format": {
"type": "qcow2",
"compression": false,
"compat": "1.1"
}
}
[ 97.999904] audit: type=1400 audit(1707944575.724:72): avc: denied { mounton } for pid=961 comm="mount" path="/srv/cache/osbuild/store/tmp/buildroot-tmp-bzjrq14z/inputs/image/disk.img" dev="vdb1" ino=17809672 scontext=system_u:system_r:ke1
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
Content mismatch at offset 403718656!
Traceback (most recent call last):
File "/run/osbuild/bin/org.osbuild.qemu", line 251, in <module>
r = main(args["inputs"], args["tree"], args["options"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/osbuild/bin/org.osbuild.qemu", line 241, in main
subprocess.run(
File "/usr/lib64/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['qemu-img', 'compare', '-f', 'raw', '-F', 'qcow2', '/run/osbuild/inputs/image/disk.img', '/run/osbuild/tree/fedora-coreos-39.20240214.dev.0-qemu.x86_64.qcow2.tmp']' returned non-zero exit status 1.
⏱ Duration: 4s
Failed
From the Content mismatch at offset 403718656! line we can see that offset 403718656 is ~ 385 MiB which places it squarely in the /boot/ parition 3.
Talked to @jlebon on this. We think (actually more @jlebon, I was just nodding my head to everything he said) that this might be a bug with reflinks on XFS.
The cache qcow2 is an XFS filesystem and when we switch to not using compression for the org.osbuild.qemu stage we think it enables qemu-img convert to use reflinks for some of the data ranges in the image (i.e. to share between the source raw image and newly created qcow2 image).
i.e. this is a theory that could explain why we see issues only after switching off compression. This particular issue reminded @jlebon of https://github.com/coreos/coreos-assembler/pull/935 which we never fully got to the bottom of.
So I don't know if it's this, but the classic failure mode when you create a disk image and then boot from it shortly afterwards is:
- You create the disk image using regular tools, qemu-img etc.
- The data is written to the kernel page cache, but not necessarily synched to disk yet.
- You boot the image using the
cache=nonemode (which usesO_DIRECT) - qemu reads directly from the disk without looking at the page cache
- Result is corruption and woe
If this is the scenario, then you need to modify your image creation pipeline so that it does an fsync or similar on the disk image before qemu runs.
Here's how virt-builder does that:
https://github.com/libguestfs/guestfs-tools/blob/acebc8b9962a78d36afe01cae78c11410b483248/builder/builder.ml#L803
@rwmjones in the failure I detailed in https://github.com/coreos/coreos-assembler/issues/3728#issuecomment-1944607900 all that is happening is:
-
qemu-img convert input.raw output.qcwo2 -
qemu-img compare -f raw -F qcow2 input.raw output.qcow2
and you can see the failure:
Content mismatch at offset 403718656!
Would you anticipate this to be a kernel page cache issue?
No that wouldn't the kernel page cache issue.
It's extremely suspicious though. Usually qcow2 and qemu-img are rock solid tools.
It's extremely suspicious though. Usually qcow2 and qemu-img are rock solid tools.
Yes :). I'd like to take a moment here and thank you and all the other maintainers of those tools over the years. You've truly built something that is load bearing for half of the internet and the world's economy at this point.
I currently don't think it's an issue in qemu-img, but will let you know if that changes.
I do appreciate you weighing in here, though.
OK - I was asked by @sandeen to provide a disk image containing an XFS filesystem with the good and bad generated disk images on it.
I had to make a slight modification to OSBuild to make it save off the bad disk image on failure:
diff --git a/osbuild/pipeline.py b/osbuild/pipeline.py
index af4c3944..93184bf6 100644
--- a/osbuild/pipeline.py
+++ b/osbuild/pipeline.py
@@ -358,6 +358,8 @@ class Pipeline:
results["stages"].append(r)
if not r.success:
+ print(f"XXXX failed tree object at {tree.path} committing")
+ object_store.commit(tree, stage.id)
cleanup(build_tree, tree)
results["success"] = False
return results
Then I ran this in a loop:
while cosa buildextend-qemu --force; do
echo "COUNT is $count";
count=$((count+1));
rm -f cache/cache2.qcow2;
done
It took a couple of iterations, but it did fail with:
⏱ Duration: 0s
Pipeline qemu: e0ab569c34dd263e53d28e56e0b2f9d6f27c4cb7c4fe5d7177f57f8b719fd229
Build
root: <host>
runner: org.osbuild.fedora38 (org.osbuild.fedora38)
org.osbuild.qemu: e0ab569c34dd263e53d28e56e0b2f9d6f27c4cb7c4fe5d7177f57f8b719fd229 {
"filename": "qemu.qcow2",
"format": {
"type": "qcow2",
"compression": false,
"compat": "1.1"
}
}
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
Content mismatch at offset 403718656!
Traceback (most recent call last):
File "/run/osbuild/bin/org.osbuild.qemu", line 251, in <module>
r = main(args["inputs"], args["tree"], args["options"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/run/osbuild/bin/org.osbuild.qemu", line 241, in main
subprocess.run(
File "/usr/lib64/python3.12/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['qemu-img', 'compare', '-f', 'raw', '-F', 'qcow2', '/run/osbuild/inputs/image/disk.img', '/run/osbuild/tree/qemu.qcow2']' returned non-zero exit status 1.
⏱ Duration: 5s
XXXX failed tree object at /srv/cache/osbuild/store/stage/uuid-66d99595d47645869c04b0608ce618f9/data committing
Failed
+ rm -rf /srv/tmp/build.qemu/supermin.out /srv/tmp/build.qemu/supermin.prepare /srv/tmp/build.qemu/supermin.build
+ '[' '!' -f /srv/tmp/build.qemu/rc ']'
++ cat /srv/tmp/build.qemu/rc
+ rc=1
+ '[' -n '' ']'
+ return 1
+ rm -f /srv/builds/39.20240216.dev.0/x86_64/.qemu.building
failed to execute cmd-buildextend-qemu: exit status 1
[coreos-assembler]$
[coreos-assembler]$ rpm -q kernel-core xfsprogs
kernel-core-6.7.3-200.fc39.x86_64
xfsprogs-6.4.0-1.fc39.x86_64
Here's a link to the disk image: cache2.qcow2.zst.
[dustymabe@media cache]$ md5sum cache2.qcow2*
99e49544af7c10ed419afa6040cf3a7e cache2.qcow2
48718e271056210d82e9ecf04fa7f624 cache2.qcow2.zst
Within the disk XFS filesystem the files for input and output of qemu-img convert are located at:
- source
rawimage:-
osbuild/store/objects/0a4be93ab2bb5c17851a57574f3af0cffe135c58ccfc840e4f0ea37dda36f77c/data/tree/disk.img
-
- output
qcow2image:-
osbuild/store/objects/e0ab569c34dd263e53d28e56e0b2f9d6f27c4cb7c4fe5d7177f57f8b719fd229/data/tree/qemu.qcow2
-
I was also asked to run xfs_bmap -vvp against the source raw
image. Here is the output of that:
bash-5.2# xfs_bmap -vvp osbuild/store/objects/0a4be93ab2bb5c17851a57574f3af0cffe135c58ccfc840e4f0ea37dda36f77c/data/tree/disk.img
osbuild/store/objects/0a4be93ab2bb5c17851a57574f3af0cffe135c58ccfc840e4f0ea37dda36f77c/data/tree/disk.img:
EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS
0: [0..39]: 28028432..28028471 3 (2862800..2862839) 40 100000
1: [40..2047]: hole 2008
2: [2048..2063]: 28023792..28023807 3 (2858160..2858175) 16 100000
3: [2064..2071]: 28023608..28023615 3 (2857976..2857983) 8 100000
4: [2072..2079]: 28023480..28023487 3 (2857848..2857855) 8 100000
5: [2080..2095]: 28020720..28020735 3 (2855088..2855103) 16 100000
6: [2096..2103]: 28020536..28020543 3 (2854904..2854911) 8 100000
7: [2104..2111]: 28020216..28020223 3 (2854584..2854591) 8 100000
8: [2112..2119]: 28020088..28020095 3 (2854456..2854463) 8 100000
9: [2120..2127]: 28019768..28019775 3 (2854136..2854143) 8 100000
10: [2128..2143]: 28019440..28019455 3 (2853808..2853823) 16 100000
11: [2144..2151]: 28019256..28019263 3 (2853624..2853631) 8 100000
12: [2152..2167]: 28019120..28019135 3 (2853488..2853503) 16 100000
13: [2168..2175]: 28018616..28018623 3 (2852984..2852991) 8 100000
14: [2176..2183]: 28018296..28018303 3 (2852664..2852671) 8 100000
15: [2184..2191]: 28017528..28017535 3 (2851896..2851903) 8 100000
16: [2192..2199]: 28017144..28017151 3 (2851512..2851519) 8 100000
17: [2200..4095]: hole 1896
18: [4096..4647]: 28028472..28029023 3 (2862840..2863391) 552 100000
19: [4648..19831]: 31393624..31408807 3 (6227992..6243175) 15184 100000
20: [19832..264191]: hole 244360
21: [264192..264207]: 1747104..1747119 0 (1747104..1747119) 16 000000
22: [264208..264719]: 28045576..28046087 3 (2879944..2880455) 512 100000
23: [264720..264743]: 28025056..28025079 3 (2859424..2859447) 24 100000
24: [264744..264751]: 28046112..28046119 3 (2880480..2880487) 8 100000
25: [264752..264775]: hole 24
26: [264776..264783]: 28029152..28029159 3 (2863520..2863527) 8 100000
27: [264784..264791]: 1747952..1747959 0 (1747952..1747959) 8 000000
28: [264792..265799]: hole 1008
29: [265800..265807]: 28027680..28027687 3 (2862048..2862055) 8 100000
30: [265808..280135]: hole 14328
31: [280136..280159]: 28028192..28028215 3 (2862560..2862583) 24 100000
32: [280160..280167]: 28029160..28029167 3 (2863528..2863535) 8 100000
33: [280168..280239]: 28046120..28046191 3 (2880488..2880559) 72 100000
34: [280240..280247]: 1748304..1748311 0 (1748304..1748311) 8 000000
35: [280248..280359]: 28046200..28046311 3 (2880568..2880679) 112 100000
36: [280360..280367]: 28046440..28046447 3 (2880808..2880815) 8 100000
37: [280368..280575]: hole 208
38: [280576..280591]: 28046312..28046327 3 (2880680..2880695) 16 100000
39: [280592..282623]: hole 2032
40: [282624..282631]: 28017016..28017023 3 (2851384..2851391) 8 100000
41: [282632..284671]: hole 2040
42: [284672..313343]: 28178184..28206855 3 (3012552..3041223) 28672 100000
43: [313344..313359]: 28046328..28046343 3 (2880696..2880711) 16 100000
44: [313360..313863]: hole 504
45: [313864..314119]: 28206856..28207111 3 (3041224..3041479) 256 100000
46: [314120..317439]: hole 3320
47: [317440..346111]: 28210432..28239103 3 (3044800..3073471) 28672 100000
48: [346112..346127]: 28046344..28046359 3 (2880712..2880727) 16 100000
49: [346128..350207]: hole 4080
50: [350208..378879]: 28239104..28267775 3 (3073472..3102143) 28672 100000
51: [378880..378895]: 28046360..28046375 3 (2880728..2880743) 16 100000
52: [378896..382975]: hole 4080
53: [382976..411647]: 28267776..28296447 3 (3102144..3130815) 28672 100000
54: [411648..411663]: 28046376..28046391 3 (2880744..2880759) 16 100000
55: [411664..415743]: hole 4080
56: [415744..428039]: 28296448..28308743 3 (3130816..3143111) 12296 100000
57: [428040..502079]: 28404656..28478695 3 (3239024..3313063) 74040 100000
58: [502080..526335]: hole 24256
59: [526336..526343]: 28046392..28046399 3 (2880760..2880767) 8 100000
60: [526344..526351]: 28024936..28024943 3 (2859304..2859311) 8 100000
61: [526352..526367]: hole 16
62: [526368..526375]: 28024344..28024351 3 (2858712..2858719) 8 100000
63: [526376..526399]: hole 24
64: [526400..526567]: 31387216..31387383 3 (6221584..6221751) 168 100000
65: [526568..526575]: 31387408..31387415 3 (6221776..6221783) 8 100000
66: [526576..542783]: hole 16208
67: [542784..542807]: 31387384..31387407 3 (6221752..6221775) 24 100000
68: [542808..543743]: hole 936
69: [543744..543767]: 28024912..28024935 3 (2859280..2859303) 24 100000
70: [543768..543775]: 28014648..28014655 3 (2849016..2849023) 8 100000
71: [543776..544767]: hole 992
72: [544768..544799]: 28024160..28024191 3 (2858528..2858559) 32 100000
73: [544800..544815]: 28024328..28024343 3 (2858696..2858711) 16 100000
74: [544816..544831]: 28023960..28023975 3 (2858328..2858343) 16 100000
75: [544832..544847]: 28023776..28023791 3 (2858144..2858159) 16 100000
76: [544848..544879]: 28023576..28023607 3 (2857944..2857975) 32 100000
77: [544880..544903]: 28023456..28023479 3 (2857824..2857847) 24 100000
78: [544904..544943]: 28023256..28023295 3 (2857624..2857663) 40 100000
79: [544944..544975]: 28023136..28023167 3 (2857504..2857535) 32 100000
80: [544976..545015]: 28022936..28022975 3 (2857304..2857343) 40 100000
81: [545016..545031]: 28022744..28022759 3 (2857112..2857127) 16 100000
82: [545032..545071]: 28022552..28022591 3 (2856920..2856959) 40 100000
83: [545072..545103]: 28022368..28022399 3 (2856736..2856767) 32 100000
84: [545104..545135]: 28022240..28022271 3 (2856608..2856639) 32 100000
85: [545136..545159]: 28022040..28022063 3 (2856408..2856431) 24 100000
86: [545160..545199]: 28021912..28021951 3 (2856280..2856319) 40 100000
87: [545200..545215]: 28022064..28022079 3 (2856432..2856447) 16 100000
88: [545216..545223]: 28021656..28021663 3 (2856024..2856031) 8 100000
89: [545224..545263]: 28021464..28021503 3 (2855832..2855871) 40 100000
90: [545264..545303]: 28021336..28021375 3 (2855704..2855743) 40 100000
91: [545304..545343]: 28021144..28021183 3 (2855512..2855551) 40 100000
92: [545344..545375]: 28021024..28021055 3 (2855392..2855423) 32 100000
93: [545376..545415]: 28020824..28020863 3 (2855192..2855231) 40 100000
94: [545416..545439]: 28020696..28020719 3 (2855064..2855087) 24 100000
95: [545440..545463]: 28020512..28020535 3 (2854880..2854903) 24 100000
96: [545464..545495]: 28020384..28020415 3 (2854752..2854783) 32 100000
97: [545496..545527]: 28020184..28020215 3 (2854552..2854583) 32 100000
98: [545528..545559]: 28020056..28020087 3 (2854424..2854455) 32 100000
99: [545560..545599]: 28019864..28019903 3 (2854232..2854271) 40 100000
100: [545600..545623]: 28019744..28019767 3 (2854112..2854135) 24 100000
101: [545624..545655]: 28019552..28019583 3 (2853920..2853951) 32 100000
102: [545656..545671]: 28019424..28019439 3 (2853792..2853807) 16 100000
103: [545672..545703]: 28019224..28019255 3 (2853592..2853623) 32 100000
104: [545704..545727]: 28019096..28019119 3 (2853464..2853487) 24 100000
105: [545728..545767]: 28018904..28018943 3 (2853272..2853311) 40 100000
106: [545768..545799]: 28018784..28018815 3 (2853152..2853183) 32 100000
107: [545800..545831]: 28018584..28018615 3 (2852952..2852983) 32 100000
108: [545832..545863]: 28018464..28018495 3 (2852832..2852863) 32 100000
109: [545864..545895]: 28018264..28018295 3 (2852632..2852663) 32 100000
110: [545896..545935]: 28018136..28018175 3 (2852504..2852543) 40 100000
111: [545936..545975]: 28017944..28017983 3 (2852312..2852351) 40 100000
112: [545976..546015]: 28017816..28017855 3 (2852184..2852223) 40 100000
113: [546016..546047]: 28017632..28017663 3 (2852000..2852031) 32 100000
114: [546048..546071]: 28017504..28017527 3 (2851872..2851895) 24 100000
115: [546072..546103]: 28017312..28017343 3 (2851680..2851711) 32 100000
116: [546104..546127]: 28017120..28017143 3 (2851488..2851511) 24 100000
117: [546128..546151]: 28016984..28017007 3 (2851352..2851375) 24 100000
118: [546152..546183]: 28016800..28016831 3 (2851168..2851199) 32 100000
119: [546184..546207]: 28016672..28016695 3 (2851040..2851063) 24 100000
120: [546208..546247]: 28016472..28016511 3 (2850840..2850879) 40 100000
121: [546248..546279]: 28016352..28016383 3 (2850720..2850751) 32 100000
122: [546280..546303]: 28016160..28016183 3 (2850528..2850551) 24 100000
123: [546304..546327]: 28015968..28015991 3 (2850336..2850359) 24 100000
124: [546328..546367]: 28015832..28015871 3 (2850200..2850239) 40 100000
125: [546368..546375]: 28015992..28015999 3 (2850360..2850367) 8 100000
126: [546376..546383]: 28016184..28016191 3 (2850552..2850559) 8 100000
127: [546384..546391]: 28016696..28016703 3 (2851064..2851071) 8 100000
128: [546392..546399]: 28017008..28017015 3 (2851376..2851383) 8 100000
129: [546400..546423]: 28015648..28015671 3 (2850016..2850039) 24 100000
130: [546424..546463]: 28015512..28015551 3 (2849880..2849919) 40 100000
131: [546464..546471]: 28015672..28015679 3 (2850040..2850047) 8 100000
132: [546472..546511]: 28015320..28015359 3 (2849688..2849727) 40 100000
133: [546512..546543]: 28015200..28015231 3 (2849568..2849599) 32 100000
134: [546544..546551]: 28014936..28014943 3 (2849304..2849311) 8 100000
135: [546552..546591]: 28014744..28014783 3 (2849112..2849151) 40 100000
136: [546592..546623]: 28014616..28014647 3 (2848984..2849015) 32 100000
137: [546624..546663]: 28013776..28013815 3 (2848144..2848183) 40 100000
138: [546664..546703]: 28013392..28013431 3 (2847760..2847799) 40 100000
139: [546704..546743]: 28013256..28013295 3 (2847624..2847663) 40 100000
140: [546744..546775]: 28013152..28013183 3 (2847520..2847551) 32 100000
141: [546776..546807]: 28013008..28013039 3 (2847376..2847407) 32 100000
142: [546808..546823]: 28012904..28012919 3 (2847272..2847287) 16 100000
143: [546824..546831]: 28013040..28013047 3 (2847408..2847415) 8 100000
144: [546832..546855]: 28012304..28012327 3 (2846672..2846695) 24 100000
145: [546856..546871]: 28012072..28012087 3 (2846440..2846455) 16 100000
146: [546872..546887]: 28014352..28014367 3 (2848720..2848735) 16 100000
147: [546888..546911]: 27322448..27322471 3 (2156816..2156839) 24 100000
148: [546912..546919]: 27322344..27322351 3 (2156712..2156719) 8 100000
149: [546920..546927]: 27321920..27321927 3 (2156288..2156295) 8 100000
150: [546928..546943]: 27324392..27324407 3 (2158760..2158775) 16 100000
151: [546944..546951]: 27324264..27324271 3 (2158632..2158639) 8 100000
152: [546952..546967]: 27334728..27334743 3 (2169096..2169111) 16 100000
153: [546968..546975]: 27334880..27334887 3 (2169248..2169255) 8 100000
154: [546976..546983]: 27334912..27334919 3 (2169280..2169287) 8 100000
155: [546984..546999]: 27342280..27342295 3 (2176648..2176663) 16 100000
156: [547000..547023]: 26091944..26091967 3 (926312..926335) 24 100000
157: [547024..547047]: 26080488..26080511 3 (914856..914879) 24 100000
158: [547048..547063]: 26072072..26072087 3 (906440..906455) 16 100000
159: [547064..547071]: 26070616..26070623 3 (904984..904991) 8 100000
160: [547072..547079]: 26077768..26077775 3 (912136..912143) 8 100000
161: [547080..547095]: 26038728..26038743 3 (873096..873111) 16 100000
162: [547096..547111]: 26038704..26038719 3 (873072..873087) 16 100000
163: [547112..547119]: 26038672..26038679 3 (873040..873047) 8 100000
164: [547120..547127]: 26038816..26038823 3 (873184..873191) 8 100000
165: [547128..547135]: 26038488..26038495 3 (872856..872863) 8 100000
166: [547136..547159]: 25910344..25910367 3 (744712..744735) 24 100000
167: [547160..547175]: 25935920..25935935 3 (770288..770303) 16 100000
168: [547176..547191]: 25941304..25941319 3 (775672..775687) 16 100000
169: [547192..547207]: 25884128..25884143 3 (718496..718511) 16 100000
170: [547208..547223]: 25998880..25998895 3 (833248..833263) 16 100000
171: [547224..547271]: 25169856..25169903 3 (4224..4271) 48 100000
172: [547272..547311]: 25169368..25169407 3 (3736..3775) 40 100000
173: [547312..547319]: 25169904..25169911 3 (4272..4279) 8 100000
174: [547320..547343]: 28769560..28769583 3 (3603928..3603951) 24 100000
175: [547344..547359]: 28769248..28769263 3 (3603616..3603631) 16 100000
176: [547360..547367]: 28785528..28785535 3 (3619896..3619903) 8 100000
177: [547368..547383]: 28813256..28813271 3 (3647624..3647639) 16 100000
178: [547384..547399]: 28832112..28832127 3 (3666480..3666495) 16 100000
179: [547400..547423]: 28874624..28874647 3 (3708992..3709015) 24 100000
180: [547424..547431]: 28835528..28835535 3 (3669896..3669903) 8 100000
181: [547432..547439]: 28832272..28832279 3 (3666640..3666647) 8 100000
182: [547440..547463]: 28932976..28932999 3 (3767344..3767367) 24 100000
183: [547464..547487]: 29268288..29268311 3 (4102656..4102679) 24 100000
184: [547488..547495]: 29234472..29234479 3 (4068840..4068847) 8 100000
185: [547496..547511]: 29308696..29308711 3 (4143064..4143079) 16 100000
186: [547512..547527]: 29430680..29430695 3 (4265048..4265063) 16 100000
187: [547528..547535]: 29459736..29459743 3 (4294104..4294111) 8 100000
188: [547536..547551]: 29520920..29520935 3 (4355288..4355303) 16 100000
189: [547552..547559]: 29497816..29497823 3 (4332184..4332191) 8 100000
190: [547560..547567]: 29459744..29459751 3 (4294112..4294119) 8 100000
191: [547568..559103]: hole 11536
192: [559104..559111]: 1747360..1747367 0 (1747360..1747367) 8 000000
193: [559112..564615]: 28029176..28034679 3 (2863544..2869047) 5504 100000
194: [564616..564639]: 1747752..1747775 0 (1747752..1747775) 24 000000
195: [564640..575487]: hole 10848
196: [575488..575495]: 28045552..28045559 3 (2879920..2879927) 8 100000
197: [575496..575551]: hole 56
198: [575552..575615]: 25166688..25166751 3 (1056..1119) 64 100000
199: [575616..575719]: 25167424..25167527 3 (1792..1895) 104 100000
200: [575720..575743]: hole 24
201: [575744..575951]: 31374264..31374471 3 (6208632..6208839) 208 100000
202: [575952..575999]: hole 48
203: [576000..576319]: 25166368..25166687 3 (736..1055) 320 100000
204: [576320..576383]: 31376408..31376471 3 (6210776..6210839) 64 100000
205: [576384..576463]: 25166752..25166831 3 (1120..1199) 80 100000
206: [576464..576511]: hole 48
207: [576512..576791]: 25166880..25167159 3 (1248..1527) 280 100000
208: [576792..576831]: hole 40
209: [576832..576879]: 25167528..25167575 3 (1896..1943) 48 100000
210: [576880..576895]: hole 16
211: [576896..576991]: 31374744..31374839 3 (6209112..6209207) 96 100000
212: [576992..577023]: hole 32
213: [577024..577287]: 25167160..25167423 3 (1528..1791) 264 100000
214: [577288..577343]: hole 56
215: [577344..577391]: 25166832..25166879 3 (1200..1247) 48 100000
216: [577392..577407]: hole 16
217: [577408..577511]: 31374840..31374943 3 (6209208..6209311) 104 100000
218: [577512..577535]: hole 24
219: [577536..577807]: 31374472..31374743 3 (6208840..6209111) 272 100000
220: [577808..577855]: hole 48
221: [577856..577903]: 31375960..31376007 3 (6210328..6210375) 48 100000
222: [577904..577919]: hole 16
223: [577920..578007]: 31375152..31375239 3 (6209520..6209607) 88 100000
224: [578008..578047]: hole 40
225: [578048..578215]: 31374984..31375151 3 (6209352..6209519) 168 100000
226: [578216..578239]: hole 24
227: [578240..578287]: 31376472..31376519 3 (6210840..6210887) 48 100000
228: [578288..578303]: hole 16
229: [578304..579023]: 31375240..31375959 3 (6209608..6210327) 720 100000
230: [579024..579071]: hole 48
231: [579072..579471]: 31376008..31376407 3 (6210376..6210775) 400 100000
232: [579472..579519]: hole 48
233: [579520..579567]: 31376784..31376831 3 (6211152..6211199) 48 100000
234: [579568..579583]: hole 16
235: [579584..579847]: 31376520..31376783 3 (6210888..6211151) 264 100000
236: [579848..579903]: hole 56
237: [579904..579959]: 31376840..31376895 3 (6211208..6211263) 56 100000
238: [579960..579967]: hole 8
239: [579968..580431]: 31376904..31377367 3 (6211272..6211735) 464 100000
240: [580432..580479]: hole 48
241: [580480..580567]: 31377416..31377503 3 (6211784..6211871) 88 100000
242: [580568..580607]: hole 40
243: [580608..580775]: 31377544..31377711 3 (6211912..6212079) 168 100000
244: [580776..580799]: hole 24
245: [580800..580847]: 31377736..31377783 3 (6212104..6212151) 48 100000
246: [580848..580863]: hole 16
247: [580864..581111]: 31377800..31378047 3 (6212168..6212415) 248 100000
248: [581112..581119]: hole 8
249: [581120..581391]: 31378056..31378327 3 (6212424..6212695) 272 100000
250: [581392..581439]: hole 48
251: [581440..581487]: 31378376..31378423 3 (6212744..6212791) 48 100000
252: [581488..581503]: hole 16
253: [581504..581583]: 31378440..31378519 3 (6212808..6212887) 80 100000
254: [581584..581631]: hole 48
255: [581632..581967]: 31378568..31378903 3 (6212936..6213271) 336 100000
256: [581968..582015]: hole 48
257: [582016..582103]: 31378952..31379039 3 (6213320..6213407) 88 100000
258: [582104..582143]: hole 40
259: [582144..582607]: 31379080..31379543 3 (6213448..6213911) 464 100000
260: [582608..582655]: hole 48
261: [582656..583015]: 31379592..31379951 3 (6213960..6214319) 360 100000
262: [583016..583039]: hole 24
263: [583040..583111]: 31379976..31380047 3 (6214344..6214415) 72 100000
264: [583112..583167]: hole 56
265: [583168..583431]: 31380104..31380367 3 (6214472..6214735) 264 100000
266: [583432..583487]: hole 56
267: [583488..583543]: 31380424..31380479 3 (6214792..6214847) 56 100000
268: [583544..583551]: hole 8
269: [583552..583631]: 31380488..31380567 3 (6214856..6214935) 80 100000
270: [583632..583679]: hole 48
271: [583680..583919]: 31380616..31380855 3 (6214984..6215223) 240 100000
272: [583920..583935]: hole 16
273: [583936..584183]: 31380872..31381119 3 (6215240..6215487) 248 100000
274: [584184..584191]: hole 8
275: [584192..584535]: 31381128..31381471 3 (6215496..6215839) 344 100000
276: [584536..584575]: hole 40
277: [584576..584679]: 31381512..31381615 3 (6215880..6215983) 104 100000
278: [584680..584703]: hole 24
279: [584704..584935]: 31381640..31381871 3 (6216008..6216239) 232 100000
280: [584936..584959]: hole 24
281: [584960..585135]: 31381896..31382071 3 (6216264..6216439) 176 100000
282: [585136..585151]: hole 16
283: [585152..585199]: 31382088..31382135 3 (6216456..6216503) 48 100000
284: [585200..585215]: hole 16
285: [585216..585575]: 31382152..31382511 3 (6216520..6216879) 360 100000
286: [585576..585599]: hole 24
287: [585600..586223]: 31382536..31383159 3 (6216904..6217527) 624 100000
288: [586224..586239]: hole 16
289: [586240..586519]: 31383176..31383455 3 (6217544..6217823) 280 100000
290: [586520..586559]: hole 40
291: [586560..586607]: 31383496..31383543 3 (6217864..6217911) 48 100000
292: [586608..586623]: hole 16
293: [586624..586735]: 31383560..31383671 3 (6217928..6218039) 112 100000
294: [586736..586751]: hole 16
295: [586752..587015]: 31383688..31383951 3 (6218056..6218319) 264 100000
296: [587016..587071]: hole 56
297: [587072..587119]: 31384008..31384055 3 (6218376..6218423) 48 100000
298: [587120..587135]: hole 16
299: [587136..587247]: 31384072..31384183 3 (6218440..6218551) 112 100000
300: [587248..587263]: hole 16
301: [587264..587695]: 31384200..31384631 3 (6218568..6218999) 432 100000
302: [587696..587711]: hole 16
303: [587712..587767]: 31384648..31384703 3 (6219016..6219071) 56 100000
304: [587768..587775]: hole 8
305: [587776..588039]: 31384712..31384975 3 (6219080..6219343) 264 100000
306: [588040..588095]: hole 56
307: [588096..588135]: 31385032..31385071 3 (6219400..6219439) 40 100000
308: [588136..588159]: hole 24
309: [588160..588231]: 31385096..31385167 3 (6219464..6219535) 72 100000
310: [588232..588287]: hole 56
311: [588288..588559]: 31385224..31385495 3 (6219592..6219863) 272 100000
312: [588560..588607]: hole 48
313: [588608..588663]: 31385544..31385599 3 (6219912..6219967) 56 100000
314: [588664..588671]: hole 8
315: [588672..588759]: 31385608..31385695 3 (6219976..6220063) 88 100000
316: [588760..588799]: hole 40
317: [588800..589111]: 31385736..31386047 3 (6220104..6220415) 312 100000
318: [589112..589183]: hole 72
319: [589184..589279]: 31386120..31386215 3 (6220488..6220583) 96 100000
320: [589280..589311]: hole 32
321: [589312..589487]: 31386248..31386423 3 (6220616..6220791) 176 100000
322: [589488..589567]: hole 80
323: [589568..589639]: 31386504..31386575 3 (6220872..6220943) 72 100000
324: [589640..589695]: hole 56
325: [589696..589775]: 31386632..31386711 3 (6221000..6221079) 80 100000
326: [589776..589823]: hole 48
327: [589824..589935]: 31386760..31386871 3 (6221128..6221239) 112 100000
328: [589936..590079]: hole 144
329: [590080..590279]: 31387016..31387215 3 (6221384..6221583) 200 100000
330: [590280..591871]: hole 1592
331: [591872..596551]: 31388808..31393487 3 (6223176..6227855) 4680 100000
332: [596552..596607]: hole 56
333: [596608..596687]: 31393544..31393623 3 (6227912..6227991) 80 100000
334: [596688..673791]: hole 77104
335: [673792..673807]: 28046088..28046103 3 (2880456..2880471) 16 100000
336: [673808..706559]: hole 32752
337: [706560..706575]: 28046400..28046415 3 (2880768..2880783) 16 100000
338: [706576..788479]: hole 81904
339: [788480..788487]: 1748384..1748391 0 (1748384..1748391) 8 000000
340: [788488..788511]: hole 24
341: [788512..788519]: 28046448..28046455 3 (2880816..2880823) 8 100000
342: [788520..788543]: hole 24
343: [788544..788551]: 28046104..28046111 3 (2880472..2880479) 8 100000
344: [788552..804927]: hole 16376
345: [804928..804943]: 28027432..28027447 3 (2861800..2861815) 16 100000
346: [804944..805887]: hole 944
347: [805888..805903]: 28046424..28046439 3 (2880792..2880807) 16 100000
348: [805904..806911]: hole 1008
349: [806912..806919]: 28004856..28004863 3 (2839224..2839231) 8 000000
350: [806920..821247]: hole 14328
351: [821248..821255]: 28027248..28027255 3 (2861616..2861623) 8 100000
352: [821256..1050623]: hole 229368
353: [1050624..1050879]: 28177528..28177783 3 (3011896..3012151) 256 100000
354: [1050880..1050911]: 28210272..28210303 3 (3044640..3044671) 32 100000
355: [1050912..1050943]: 28207112..28207143 3 (3041480..3041511) 32 100000
356: [1050944..1051071]: 28750936..28751063 3 (3585304..3585431) 128 100000
357: [1051072..1051095]: 28750808..28750831 3 (3585176..3585199) 24 100000
358: [1051096..1051135]: 28207296..28207335 3 (3041664..3041703) 40 100000
359: [1051136..1051199]: 28750664..28750727 3 (3585032..3585095) 64 100000
360: [1051200..1075135]: 28308744..28332679 3 (3143112..3167047) 23936 100000
361: [1075136..1075159]: 28210400..28210423 3 (3044768..3044791) 24 100000
362: [1075160..1075191]: 28207648..28207679 3 (3042016..3042047) 32 100000
363: [1075192..1075199]: 28207408..28207415 3 (3041776..3041783) 8 100000
364: [1075200..1075351]: 28791296..28791447 3 (3625664..3625815) 152 100000
365: [1075352..1075391]: 28207416..28207455 3 (3041784..3041823) 40 100000
366: [1075392..1075487]: 28769304..28769399 3 (3603672..3603767) 96 100000
367: [1075488..1075503]: 28207368..28207383 3 (3041736..3041751) 16 100000
368: [1075504..1075519]: 28207456..28207471 3 (3041824..3041839) 16 100000
369: [1075520..1075679]: 29680632..29680791 3 (4515000..4515159) 160 100000
370: [1075680..1075703]: 28207384..28207407 3 (3041752..3041775) 24 100000
371: [1075704..1075775]: 28785456..28785527 3 (3619824..3619895) 72 100000
372: [1075776..1075943]: 28207480..28207647 3 (3041848..3042015) 168 100000
373: [1075944..1078487]: 28207680..28210223 3 (3042048..3044591) 2544 100000
374: [1078488..1080511]: 28485704..28487727 3 (3320072..3322095) 2024 100000
375: [1080512..1080575]: 28762136..28762199 3 (3596504..3596567) 64 100000
376: [1080576..1083015]: 28487792..28490231 3 (3322160..3324599) 2440 100000
377: [1083016..1083031]: 28484008..28484023 3 (3318376..3318391) 16 100000
378: [1083032..1083063]: 28504544..28504575 3 (3338912..3338943) 32 100000
379: [1083064..1083071]: 28506008..28506015 3 (3340376..3340383) 8 100000
380: [1083072..1083135]: 28769624..28769687 3 (3603992..3604055) 64 100000
381: [1083136..1083719]: 28501600..28502183 3 (3335968..3336551) 584 100000
382: [1083720..1085695]: 28502568..28504543 3 (3336936..3338911) 1976 100000
383: [1085696..1087127]: 28504576..28506007 3 (3338944..3340375) 1432 100000
384: [1087128..1089735]: 28506016..28508623 3 (3340384..3342991) 2608 100000
385: [1089736..1089759]: 28479832..28479855 3 (3314200..3314223) 24 100000
386: [1089760..1089791]: 28613592..28613623 3 (3447960..3447991) 32 100000
387: [1089792..1089855]: 28626088..28626151 3 (3460456..3460519) 64 100000
388: [1089856..1094079]: 28621864..28626087 3 (3456232..3460455) 4224 100000
389: [1094080..1094143]: 28791488..28791551 3 (3625856..3625919) 64 100000
390: [1094144..1096639]: 28626152..28628647 3 (3460520..3463015) 2496 100000
391: [1096640..1096703]: 28806648..28806711 3 (3641016..3641079) 64 100000
392: [1096704..1098295]: 28628712..28630303 3 (3463080..3464671) 1592 100000
393: [1098296..1098415]: 28874344..28874463 3 (3708712..3708831) 120 100000
394: [1098416..1098431]: 28628648..28628663 3 (3463016..3463031) 16 100000
395: [1098432..1098559]: 28874496..28874623 3 (3708864..3708991) 128 100000
396: [1098560..1113407]: 28817200..28832047 3 (3651568..3666415) 14848 100000
397: [1113408..1113495]: 29308760..29308847 3 (4143128..4143215) 88 100000
398: [1113496..1113535]: 28832136..28832175 3 (3666504..3666543) 40 100000
399: [1113536..1113695]: 31249120..31249279 3 (6083488..6083647) 160 100000
400: [1113696..1113727]: 28832176..28832207 3 (3666544..3666575) 32 100000
401: [1113728..1113807]: 29459656..29459735 3 (4294024..4294103) 80 100000
402: [1113808..1113855]: 28832288..28832335 3 (3666656..3666703) 48 100000
403: [1113856..1113919]: 28902496..28902559 3 (3736864..3736927) 64 100000
404: [1113920..1116991]: 28832400..28835471 3 (3666768..3669839) 3072 100000
405: [1116992..1117055]: 28905568..28905631 3 (3739936..3739999) 64 100000
406: [1117056..1123319]: 28835536..28841799 3 (3669904..3676167) 6264 100000
407: [1123320..1123415]: 29478144..29478239 3 (4312512..4312607) 96 100000
408: [1123416..1123455]: 28841896..28841935 3 (3676264..3676303) 40 100000
409: [1123456..1123519]: 28932360..28932423 3 (3766728..3766791) 64 100000
410: [1123520..1124791]: 28842000..28843271 3 (3676368..3677639) 1272 100000
411: [1124792..1124799]: 28835512..28835519 3 (3669880..3669887) 8 100000
412: [1124800..1125679]: 28843280..28844159 3 (3677648..3678527) 880 100000
413: [1125680..1125687]: 28835520..28835527 3 (3669888..3669895) 8 100000
414: [1125688..1125695]: 28843272..28843279 3 (3677640..3677647) 8 100000
415: [1125696..1125759]: 28966864..28966927 3 (3801232..3801295) 64 100000
416: [1125760..1140223]: 28952400..28966863 3 (3786768..3801231) 14464 100000
417: [1140224..1140287]: 28978448..28978511 3 (3812816..3812879) 64 100000
418: [1140288..1144799]: 28966928..28971439 3 (3801296..3805807) 4512 100000
419: [1144800..1144919]: 29497696..29497815 3 (4332064..4332183) 120 100000
420: [1144920..1144959]: 28971560..28971599 3 (3805928..3805967) 40 100000
421: [1144960..1145055]: 29497568..29497663 3 (4331936..4332031) 96 100000
422: [1145056..1145087]: 28971696..28971727 3 (3806064..3806095) 32 100000
423: [1145088..1145215]: 29876856..29876983 3 (4711224..4711351) 128 100000
424: [1145216..1151807]: 28971856..28978447 3 (3806224..3812815) 6592 100000
425: [1151808..1151871]: 29004336..29004399 3 (3838704..3838767) 64 100000
426: [1151872..1154463]: 28978512..28981103 3 (3812880..3815471) 2592 100000
427: [1154464..1156095]: 29096888..29098519 3 (3931256..3932887) 1632 100000
428: [1156096..1156159]: 29071544..29071607 3 (3905912..3905975) 64 100000
429: [1156160..1281695]: 29098584..29224119 3 (3932952..4058487) 125536 100000
430: [1281696..1281719]: 29234416..29234439 3 (4068784..4068807) 24 100000
431: [1281720..1281727]: 29225136..29225143 3 (4059504..4059511) 8 100000
432: [1281728..1281759]: 29234440..29234471 3 (4068808..4068839) 32 100000
433: [1281760..1281887]: 30305336..30305463 3 (5139704..5139831) 128 100000
434: [1281888..1281919]: 29225200..29225231 3 (4059568..4059599) 32 100000
435: [1281920..1282007]: 29520648..29520735 3 (4355016..4355103) 88 100000
436: [1282008..1282031]: 29225232..29225255 3 (4059600..4059623) 24 100000
437: [1282032..1282183]: 31249280..31249431 3 (6083648..6083799) 152 100000
438: [1282184..1311615]: 29390296..29419727 3 (4224664..4254095) 29432 100000
439: [1311616..1311679]: 29437264..29437327 3 (4271632..4271695) 64 100000
440: [1311680..1319487]: 29419792..29427599 3 (4254160..4261967) 7808 100000
441: [1319488..1319551]: 29459792..29459855 3 (4294160..4294223) 64 100000
442: [1319552..1321087]: 29427664..29429199 3 (4262032..4263567) 1536 100000
443: [1321088..1321151]: 29470040..29470103 3 (4304408..4304471) 64 100000
444: [1321152..1322471]: 29429264..29430583 3 (4263632..4264951) 1320 100000
445: [1322472..1322583]: 30592744..30592855 3 (5427112..5427223) 112 100000
446: [1322584..1322623]: 29430696..29430735 3 (4265064..4265103) 40 100000
447: [1322624..1322751]: 30597960..30598087 3 (5432328..5432455) 128 100000
448: [1322752..1329151]: 29430864..29437263 3 (4265232..4271631) 6400 100000
449: [1329152..1329215]: 29520832..29520895 3 (4355200..4355263) 64 100000
450: [1329216..1351543]: 29437328..29459655 3 (4271696..4294023) 22328 100000
451: [1351544..1351639]: 29562360..29562455 3 (4396728..4396823) 96 100000
452: [1351640..1351679]: 29459752..29459791 3 (4294120..4294159) 40 100000
453: [1351680..1351743]: 29544208..29544271 3 (4378576..4378639) 64 100000
454: [1351744..1354975]: 29459856..29463087 3 (4294224..4297455) 3232 100000
455: [1354976..1367855]: 29530504..29543383 3 (4364872..4377751) 12880 100000
456: [1367856..1367943]: 29600904..29600991 3 (4435272..4435359) 88 100000
457: [1367944..1371791]: 29558424..29562271 3 (4392792..4396639) 3848 100000
458: [1371792..1371799]: 29562536..29562543 3 (4396904..4396911) 8 100000
459: [1371800..1371815]: 29562304..29562319 3 (4396672..4396687) 16 100000
460: [1371816..1371991]: 31249432..31249607 3 (6083800..6083975) 176 100000
461: [1371992..1372031]: 29562320..29562359 3 (4396688..4396727) 40 100000
462: [1372032..1372119]: 29626888..29626975 3 (4461256..4461343) 88 100000
463: [1372120..1372159]: 29562496..29562535 3 (4396864..4396903) 40 100000
464: [1372160..1372223]: 29577808..29577871 3 (4412176..4412239) 64 100000
465: [1372224..1465447]: 29683952..29777175 3 (4518320..4611543) 93224 100000
466: [1465448..1465599]: 31249608..31249759 3 (6083976..6084127) 152 100000
467: [1465600..1488519]: 29777328..29800247 3 (4611696..4634615) 22920 100000
468: [1488520..1488543]: 29843584..29843607 3 (4677952..4677975) 24 100000
469: [1488544..1488575]: 29808120..29808151 3 (4642488..4642519) 32 100000
470: [1488576..1488735]: 31249760..31249919 3 (6084128..6084287) 160 100000
471: [1488736..1488759]: 29808152..29808175 3 (4642520..4642543) 24 100000
472: [1488760..1488831]: 29843632..29843703 3 (4678000..4678071) 72 100000
473: [1488832..1546751]: 29999984..30057903 3 (4834352..4892271) 57920 100000
474: [1546752..1546815]: 29896544..29896607 3 (4730912..4730975) 64 100000
475: [1546816..1550783]: 30057968..30061935 3 (4892336..4896303) 3968 100000
476: [1550784..1550847]: 29894176..29894239 3 (4728544..4728607) 64 100000
477: [1550848..1554687]: 30062000..30065839 3 (4896368..4900207) 3840 100000
478: [1554688..1554711]: 30057944..30057967 3 (4892312..4892335) 24 100000
479: [1554712..1554751]: 30061936..30061975 3 (4896304..4896343) 40 100000
480: [1554752..1554839]: 29881688..29881775 3 (4716056..4716143) 88 100000
481: [1554840..1554879]: 30057904..30057943 3 (4892272..4892311) 40 100000
482: [1554880..1555007]: 30636536..30636663 3 (5470904..5471031) 128 100000
483: [1555008..1555031]: 30636496..30636519 3 (5470864..5470887) 24 100000
484: [1555032..1555071]: 29949536..29949575 3 (4783904..4783943) 40 100000
485: [1555072..1555175]: 31258192..31258295 3 (6092560..6092663) 104 100000
486: [1555176..1555199]: 29949576..29949599 3 (4783944..4783967) 24 100000
487: [1555200..1555351]: 31258296..31258447 3 (6092664..6092815) 152 100000
488: [1555352..1555391]: 29937232..29937271 3 (4771600..4771639) 40 100000
489: [1555392..1555479]: 31258448..31258535 3 (6092816..6092903) 88 100000
490: [1555480..1555519]: 29937272..29937311 3 (4771640..4771679) 40 100000
491: [1555520..1555679]: 31258536..31258695 3 (6092904..6093063) 160 100000
492: [1555680..1555711]: 29937056..29937087 3 (4771424..4771455) 32 100000
493: [1555712..1555775]: 31258696..31258759 3 (6093064..6093127) 64 100000
494: [1555776..1563975]: 30230928..30239127 3 (5065296..5073495) 8200 100000
495: [1563976..1563999]: 30305040..30305063 3 (5139408..5139431) 24 100000
496: [1564000..1564159]: 31258760..31258919 3 (6093128..6093287) 160 100000
497: [1564160..1569655]: 30327664..30333159 3 (5162032..5167527) 5496 100000
498: [1569656..1574823]: 30382208..30387375 3 (5216576..5221743) 5168 100000
499: [1574824..1574911]: 31258920..31259007 3 (6093288..6093375) 88 100000
500: [1574912..1576895]: 30387464..30389447 3 (5221832..5223815) 1984 100000
501: [1576896..1576959]: 31259008..31259071 3 (6093376..6093439) 64 100000
502: [1576960..1602815]: 30389512..30415367 3 (5223880..5249735) 25856 100000
503: [1602816..1602879]: 31259072..31259135 3 (6093440..6093503) 64 100000
504: [1602880..1609519]: 30415432..30422071 3 (5249800..5256439) 6640 100000
505: [1609520..1609607]: 31259136..31259223 3 (6093504..6093591) 88 100000
506: [1609608..1614783]: 30644032..30649207 3 (5478400..5483575) 5176 100000
507: [1614784..1614943]: 31259224..31259383 3 (6093592..6093751) 160 100000
508: [1614944..1614975]: 30649208..30649239 3 (5483576..5483607) 32 100000
509: [1614976..1615127]: 31259384..31259535 3 (6093752..6093903) 152 100000
510: [1615128..1615135]: 30649240..30649247 3 (5483608..5483615) 8 100000
511: [1615136..1615359]: 31259536..31259759 3 (6093904..6094127) 224 100000
512: [1615360..1617543]: 30649472..30651655 3 (5483840..5486023) 2184 100000
513: [1617544..1618647]: 30680296..30681399 3 (5514664..5515767) 1104 100000
514: [1618648..1628127]: 30681408..30690887 3 (5515776..5525255) 9480 100000
515: [1628128..1628183]: 30699688..30699743 3 (5534056..5534111) 56 100000
516: [1628184..1628207]: 30710824..30710847 3 (5545192..5545215) 24 100000
517: [1628208..1629063]: 30712840..30713695 3 (5547208..5548063) 856 100000
518: [1629064..1629071]: 30710864..30710871 3 (5545232..5545239) 8 100000
519: [1629072..1633703]: 30713704..30718335 3 (5548072..5552703) 4632 100000
520: [1633704..1633719]: 30710872..30710887 3 (5545240..5545255) 16 100000
521: [1633720..1633727]: 30713696..30713703 3 (5548064..5548071) 8 100000
522: [1633728..1633791]: 31259760..31259823 3 (6094128..6094191) 64 100000
523: [1633792..1636375]: 30763832..30766415 3 (5598200..5600783) 2584 100000
524: [1636376..1636407]: 30649424..30649455 3 (5483792..5483823) 32 100000
525: [1636408..1636415]: 30798144..30798151 3 (5632512..5632519) 8 100000
526: [1636416..1636543]: 31259824..31259951 3 (6094192..6094319) 128 100000
527: [1636544..1637719]: 30804752..30805927 3 (5639120..5640295) 1176 100000
528: [1637720..1640663]: 30806808..30809751 3 (5641176..5644119) 2944 100000
529: [1640664..1644943]: 30813400..30817679 3 (5647768..5652047) 4280 100000
530: [1644944..1644967]: 30657304..30657327 3 (5491672..5491695) 24 100000
531: [1644968..1644991]: 30818496..30818519 3 (5652864..5652887) 24 100000
532: [1644992..1645055]: 31259952..31260015 3 (6094320..6094383) 64 100000
533: [1645056..1663615]: 30999832..31018391 3 (5834200..5852759) 18560 100000
534: [1663616..1698367]: 31062368..31097119 3 (5896736..5931487) 34752 100000
535: [1698368..1698431]: 31315560..31315623 3 (6149928..6149991) 64 100000
536: [1698432..1698615]: 31097184..31097367 3 (5931552..5931735) 184 100000
537: [1698616..1698663]: 31315664..31315711 3 (6150032..6150079) 48 100000
538: [1698664..1698687]: 31097416..31097439 3 (5931784..5931807) 24 100000
539: [1698688..1698815]: 31359408..31359535 3 (6193776..6193903) 128 100000
540: [1698816..1702847]: 31097568..31101599 3 (5931936..5935967) 4032 100000
541: [1702848..1702911]: 31315800..31315863 3 (6150168..6150231) 64 100000
542: [1702912..1706311]: 31101664..31105063 3 (5936032..5939431) 3400 100000
543: [1706312..1706335]: 31315712..31315735 3 (6150080..6150103) 24 100000
544: [1706336..1706367]: 31105088..31105119 3 (5939456..5939487) 32 100000
545: [1706368..1706823]: 31359536..31359991 3 (6193904..6194359) 456 100000
546: [1706824..1707959]: 31105576..31106711 3 (5939944..5941079) 1136 100000
547: [1707960..1708039]: 31316064..31316143 3 (6150432..6150511) 80 100000
548: [1708040..1708095]: 31106792..31106847 3 (5941160..5941215) 56 100000
549: [1708096..1708159]: 31315944..31316007 3 (6150312..6150375) 64 100000
550: [1708160..1710423]: 31106912..31109175 3 (5941280..5943543) 2264 100000
551: [1710424..1710431]: 31111200..31111207 3 (5945568..5945575) 8 100000
552: [1710432..1710439]: 31106776..31106783 3 (5941144..5941151) 8 100000
553: [1710440..1712255]: 31173152..31174967 3 (6007520..6009335) 1816 100000
554: [1712256..1712383]: 31359992..31360119 3 (6194360..6194487) 128 100000
555: [1712384..1715527]: 31175096..31178239 3 (6009464..6012607) 3144 100000
556: [1715528..1715535]: 31178560..31178567 3 (6012928..6012935) 8 100000
557: [1715536..1715583]: 31265584..31265631 3 (6099952..6099999) 48 100000
558: [1715584..1716647]: 31360120..31361183 3 (6194488..6195551) 1064 100000
559: [1716648..1716655]: 31265632..31265639 3 (6100000..6100007) 8 100000
560: [1716656..1717519]: 31361184..31362047 3 (6195552..6196415) 864 100000
561: [1717520..1717567]: 31265640..31265687 3 (6100008..6100055) 48 100000
562: [1717568..1717703]: 31362048..31362183 3 (6196416..6196551) 136 100000
563: [1717704..1717751]: 31265824..31265871 3 (6100192..6100239) 48 100000
564: [1717752..1717839]: 31316472..31316559 3 (6150840..6150927) 88 100000
565: [1717840..1717871]: 31265960..31265991 3 (6100328..6100359) 32 100000
566: [1717872..1717967]: 31362184..31362279 3 (6196552..6196647) 96 100000
567: [1717968..1717999]: 31266088..31266119 3 (6100456..6100487) 32 100000
568: [1718000..1718103]: 31362280..31362383 3 (6196648..6196751) 104 100000
569: [1718104..1718135]: 31266224..31266255 3 (6100592..6100623) 32 100000
570: [1718136..1718215]: 31316192..31316271 3 (6150560..6150639) 80 100000
571: [1718216..1718255]: 31266336..31266375 3 (6100704..6100743) 40 100000
572: [1718256..1718503]: 31362384..31362631 3 (6196752..6196999) 248 100000
573: [1718504..1718527]: 31266376..31266399 3 (6100744..6100767) 24 100000
574: [1718528..1718695]: 31362632..31362799 3 (6197000..6197167) 168 100000
575: [1718696..1752799]: 31266568..31300671 3 (6100936..6135039) 34104 100000
576: [1752800..2073087]: hole 320288
577: [2073088..2073135]: 28177784..28177831 3 (3012152..3012199) 48 100000
578: [2073136..2073167]: hole 32
579: [2073168..2073191]: 28207144..28207167 3 (3041512..3041535) 24 100000
580: [2073192..2073375]: 30641224..30641407 3 (5475592..5475775) 184 100000
581: [2073376..2073407]: 28207168..28207199 3 (3041536..3041567) 32 100000
582: [2073408..2073543]: 28769112..28769247 3 (3603480..3603615) 136 100000
583: [2073544..2081879]: 28332680..28341015 3 (3167048..3175383) 8336 100000
584: [2081880..2082063]: 31105120..31105303 3 (5939488..5939671) 184 100000
585: [2082064..2082111]: 28210224..28210271 3 (3044592..3044639) 48 100000
586: [2082112..2082207]: 28841800..28841895 3 (3676168..3676263) 96 100000
587: [2082208..2082239]: 28210368..28210399 3 (3044736..3044767) 32 100000
588: [2082240..2082391]: 29039328..29039479 3 (3873696..3873847) 152 100000
589: [2082392..2082431]: 28027912..28027951 3 (2862280..2862319) 40 100000
590: [2082432..2082527]: 28883968..28884063 3 (3718336..3718431) 96 100000
591: [2082528..2082559]: 28027800..28027831 3 (2862168..2862199) 32 100000
592: [2082560..2082687]: 28971728..28971855 3 (3806096..3806223) 128 100000
593: [2082688..2085103]: 28490232..28492647 3 (3324600..3327015) 2416 100000
594: [2085104..2085127]: 28613640..28613663 3 (3448008..3448031) 24 100000
595: [2085128..2085183]: 28508624..28508679 3 (3342992..3343047) 56 100000
596: [2085184..2085247]: 28808192..28808255 3 (3642560..3642623) 64 100000
597: [2085248..2160167]: 28508744..28583663 3 (3343112..3418031) 74920 100000
598: [2160168..2160175]: 28613664..28613671 3 (3448032..3448039) 8 100000
599: [2160176..2160191]: 28613624..28613639 3 (3447992..3448007) 16 100000
600: [2160192..2160255]: 28832048..28832111 3 (3666416..3666479) 64 100000
601: [2160256..2170559]: 28630304..28640607 3 (3464672..3474975) 10304 100000
602: [2170560..2170623]: 28832208..28832271 3 (3666576..3666639) 64 100000
603: [2170624..2173015]: 28640672..28643063 3 (3475040..3477431) 2392 100000
604: [2173016..2173207]: 31105304..31105495 3 (5939672..5939863) 192 100000
605: [2173208..2173247]: 28640608..28640647 3 (3474976..3475015) 40 100000
606: [2173248..2173375]: 29308568..29308695 3 (4142936..4143063) 128 100000
607: [2173376..2203559]: 28844160..28874343 3 (3678528..3708711) 30184 100000
608: [2203560..2203679]: 31097440..31097559 3 (5931808..5931927) 120 100000
609: [2203680..2203711]: 28874464..28874495 3 (3708832..3708863) 32 100000
610: [2203712..2203863]: 31249920..31250071 3 (6084288..6084439) 152 100000
611: [2203864..2203903]: 28874648..28874687 3 (3709016..3709055) 40 100000
612: [2203904..2203967]: 29021392..29021455 3 (3855760..3855823) 64 100000
613: [2203968..2206847]: 28874752..28877631 3 (3709120..3711999) 2880 100000
614: [2206848..2206911]: 29023504..29023567 3 (3857872..3857935) 64 100000
615: [2206912..2209639]: 28877696..28880423 3 (3712064..3714791) 2728 100000
616: [2209640..2209727]: 29680520..29680607 3 (4514888..4514975) 88 100000
617: [2209728..2213183]: 28880512..28883967 3 (3714880..3718335) 3456 100000
618: [2213184..2213311]: 31174968..31175095 3 (6009336..6009463) 128 100000
619: [2213312..2216871]: 28884096..28887655 3 (3718464..3722023) 3560 100000
620: [2216872..2216959]: 29876728..29876815 3 (4711096..4711183) 88 100000
621: [2216960..2240047]: 28981104..29004191 3 (3815472..3838559) 23088 100000
622: [2240048..2240135]: 30305208..30305295 3 (5139576..5139663) 88 100000
623: [2240136..2240159]: 29004280..29004303 3 (3838648..3838671) 24 100000
624: [2240160..2240415]: 31250072..31250327 3 (6084440..6084695) 256 100000
625: [2240416..2240447]: 29004304..29004335 3 (3838672..3838703) 32 100000
626: [2240448..2240511]: 29039520..29039583 3 (3873888..3873951) 64 100000
627: [2240512..2250055]: 29004400..29013943 3 (3838768..3848311) 9544 100000
628: [2250056..2251071]: 29224120..29225135 3 (4058488..4059503) 1016 100000
629: [2251072..2251079]: 29225256..29225263 3 (4059624..4059631) 8 100000
630: [2251080..2251135]: 29225144..29225199 3 (4059512..4059567) 56 100000
631: [2251136..2251199]: 29258352..29258415 3 (4092720..4092783) 64 100000
632: [2251200..2260351]: 29225264..29234415 3 (4059632..4068783) 9152 100000
633: [2260352..2260415]: 29314904..29314967 3 (4149272..4149335) 64 100000
634: [2260416..2284287]: 29234480..29258351 3 (4068848..4092719) 23872 100000
635: [2284288..2284351]: 29053368..29053431 3 (3887736..3887799) 64 100000
636: [2284352..2290111]: 29258416..29264175 3 (4092784..4098543) 5760 100000
637: [2290112..2290143]: 29520736..29520767 3 (4355104..4355135) 32 100000
638: [2290144..2290175]: 29268104..29268135 3 (4102472..4102503) 32 100000
639: [2290176..2290327]: 31250328..31250479 3 (6084696..6084847) 152 100000
640: [2290328..2290367]: 29268248..29268287 3 (4102616..4102655) 40 100000
641: [2290368..2290463]: 30641080..30641175 3 (5475448..5475543) 96 100000
642: [2290464..2290495]: 29268136..29268167 3 (4102504..4102535) 32 100000
643: [2290496..2290655]: 31250480..31250639 3 (6084848..6085007) 160 100000
644: [2290656..2290687]: 29268168..29268199 3 (4102536..4102567) 32 100000
645: [2290688..2290839]: 31250640..31250791 3 (6085008..6085159) 152 100000
646: [2290840..2297791]: 29463088..29470039 3 (4297456..4304407) 6952 100000
647: [2297792..2297855]: 29596184..29596247 3 (4430552..4430615) 64 100000
648: [2297856..2305895]: 29470104..29478143 3 (4304472..4312511) 8040 100000
649: [2305896..2305991]: 31250792..31250887 3 (6085160..6085255) 96 100000
650: [2305992..2306023]: 29478240..29478271 3 (4312608..4312639) 32 100000
651: [2306024..2306207]: 31250888..31251071 3 (6085256..6085439) 184 100000
652: [2306208..2306239]: 29478272..29478303 3 (4312640..4312671) 32 100000
653: [2306240..2306367]: 31251072..31251199 3 (6085440..6085567) 128 100000
654: [2306368..2325503]: 29478432..29497567 3 (4312800..4331935) 19136 100000
655: [2325504..2325599]: 31251200..31251295 3 (6085568..6085663) 96 100000
656: [2325600..2325631]: 29497664..29497695 3 (4332032..4332063) 32 100000
657: [2325632..2325759]: 31251296..31251423 3 (6085664..6085791) 128 100000
658: [2325760..2348583]: 29497824..29520647 3 (4332192..4355015) 22824 100000
659: [2348584..2348703]: 31251424..31251543 3 (6085792..6085911) 120 100000
660: [2348704..2348735]: 29520768..29520799 3 (4355136..4355167) 32 100000
661: [2348736..2348871]: 31251544..31251679 3 (6085912..6086047) 136 100000
662: [2348872..2349663]: 29520936..29521727 3 (4355304..4356095) 792 100000
663: [2349664..2349687]: 29520896..29520919 3 (4355264..4355287) 24 100000
664: [2349688..2349695]: 29520824..29520831 3 (4355192..4355199) 8 100000
665: [2349696..2349823]: 31251680..31251807 3 (6086048..6086175) 128 100000
666: [2349824..2357695]: 29800248..29808119 3 (4634616..4642487) 7872 100000
667: [2357696..2357759]: 29846904..29846967 3 (4681272..4681335) 64 100000
668: [2357760..2393159]: 29808184..29843583 3 (4642552..4677951) 35400 100000
669: [2393160..2393183]: 29846664..29846687 3 (4681032..4681055) 24 100000
670: [2393184..2393207]: 29843608..29843631 3 (4677976..4677999) 24 100000
671: [2393208..2393279]: 30597672..30597743 3 (5432040..5432111) 72 100000
672: [2393280..2396239]: 29843704..29846663 3 (4678072..4681031) 2960 100000
673: [2396240..2396271]: 29850296..29850327 3 (4684664..4684695) 32 100000
674: [2396272..2396287]: 29846696..29846711 3 (4681064..4681079) 16 100000
675: [2396288..2396431]: 31251808..31251951 3 (6086176..6086319) 144 100000
676: [2396432..2396479]: 29846856..29846903 3 (4681224..4681271) 48 100000
677: [2396480..2396543]: 29881824..29881887 3 (4716192..4716255) 64 100000
678: [2396544..2399871]: 29846968..29850295 3 (4681336..4684663) 3328 100000
679: [2399872..2399935]: 29644952..29645015 3 (4479320..4479383) 64 100000
680: [2399936..2426303]: 29850360..29876727 3 (4684728..4711095) 26368 100000
681: [2426304..2426391]: 30597832..30597919 3 (5432200..5432287) 88 100000
682: [2426392..2426431]: 29876816..29876855 3 (4711184..4711223) 40 100000
683: [2426432..2426559]: 31251952..31252079 3 (6086320..6086447) 128 100000
684: [2426560..2430975]: 29876984..29881399 3 (4711352..4715767) 4416 100000
685: [2430976..2437983]: 30065840..30072847 3 (4900208..4907215) 7008 100000
686: [2437984..2437999]: 29949600..29949615 3 (4783968..4783983) 16 100000
687: [2438000..2438015]: 30061976..30061991 3 (4896344..4896359) 16 100000
688: [2438016..2438111]: 31252080..31252175 3 (6086448..6086543) 96 100000
689: [2438112..2438143]: 29937088..29937119 3 (4771456..4771487) 32 100000
690: [2438144..2438295]: 31252176..31252327 3 (6086544..6086695) 152 100000
691: [2438296..2438335]: 29919328..29919367 3 (4753696..4753735) 40 100000
692: [2438336..2438399]: 29627032..29627095 3 (4461400..4461463) 64 100000
693: [2438400..2438431]: 30641040..30641071 3 (5475408..5475439) 32 100000
694: [2438432..2438463]: 29919368..29919399 3 (4753736..4753767) 32 100000
695: [2438464..2438615]: 31260016..31260167 3 (6094384..6094535) 152 100000
696: [2438616..2438655]: 29919400..29919439 3 (4753768..4753807) 40 100000
697: [2438656..2438751]: 31260168..31260263 3 (6094536..6094631) 96 100000
698: [2438752..2438783]: 29919200..29919231 3 (4753568..4753599) 32 100000
699: [2438784..2438935]: 31260264..31260415 3 (6094632..6094783) 152 100000
700: [2438936..2438975]: 29919232..29919271 3 (4753600..4753639) 40 100000
701: [2438976..2439071]: 31260416..31260511 3 (6094784..6094879) 96 100000
702: [2439072..2439103]: 29937120..29937151 3 (4771488..4771519) 32 100000
703: [2439104..2439231]: 31260512..31260639 3 (6094880..6095007) 128 100000
704: [2439232..2482527]: 30239128..30282423 3 (5073496..5116791) 43296 100000
705: [2482528..2482543]: 30317784..30317799 3 (5152152..5152167) 16 100000
706: [2482544..2482559]: 30305064..30305079 3 (5139432..5139447) 16 100000
707: [2482560..2482687]: 31260640..31260767 3 (6095008..6095135) 128 100000
708: [2482688..2521871]: 30333160..30372343 3 (5167528..5206711) 39184 100000
709: [2521872..2548351]: 30422072..30448551 3 (5256440..5282919) 26480 100000
710: [2548352..2548415]: 31260768..31260831 3 (6095136..6095199) 64 100000
711: [2548416..2577087]: 30448616..30477287 3 (5282984..5311655) 28672 100000
712: [2577088..2577151]: 31260832..31260895 3 (6095200..6095263) 64 100000
713: [2577152..2579527]: 30477352..30479727 3 (5311720..5314095) 2376 100000
714: [2579528..2579567]: 31097368..31097407 3 (5931736..5931775) 40 100000
715: [2579568..2579583]: 30479768..30479783 3 (5314136..5314151) 16 100000
716: [2579584..2579647]: 31260896..31260959 3 (6095264..6095327) 64 100000
717: [2579648..2580479]: 30479848..30480679 3 (5314216..5315047) 832 100000
718: [2580480..2584383]: 30651656..30655559 3 (5486024..5489927) 3904 100000
719: [2584384..2584447]: 31260960..31261023 3 (6095328..6095391) 64 100000
720: [2584448..2586127]: 30655624..30657303 3 (5489992..5491671) 1680 100000
721: [2586128..2586151]: 30641408..30641431 3 (5475776..5475799) 24 100000
722: [2586152..2586175]: 30657328..30657351 3 (5491696..5491719) 24 100000
723: [2586176..2586263]: 31261024..31261111 3 (6095392..6095479) 88 100000
724: [2586264..2586303]: 30657440..30657479 3 (5491808..5491847) 40 100000
725: [2586304..2586463]: 31261112..31261271 3 (6095480..6095639) 160 100000
726: [2586464..2586495]: 30657480..30657511 3 (5491848..5491879) 32 100000
727: [2586496..2586583]: 31261272..31261359 3 (6095640..6095727) 88 100000
728: [2586584..2586623]: 30657600..30657639 3 (5491968..5492007) 40 100000
729: [2586624..2586751]: 31261360..31261487 3 (6095728..6095855) 128 100000
730: [2586752..2588359]: 30657928..30659535 3 (5492296..5493903) 1608 100000
731: [2588360..2589567]: 30665792..30666999 3 (5500160..5501367) 1208 100000
732: [2589568..2595063]: 30690888..30696383 3 (5525256..5530751) 5496 100000
733: [2595064..2600079]: 30718336..30723351 3 (5552704..5557719) 5016 100000
734: [2600080..2600111]: 31056816..31056847 3 (5891184..5891215) 32 100000
735: [2600112..2600127]: 30710848..30710863 3 (5545216..5545231) 16 100000
736: [2600128..2600191]: 31261488..31261551 3 (6095856..6095919) 64 100000
737: [2600192..2611183]: 30766416..30777407 3 (5600784..5611775) 10992 100000
738: [2611184..2611191]: 30809808..30809815 3 (5644176..5644183) 8 100000
739: [2611192..2611199]: 30809752..30809759 3 (5644120..5644127) 8 100000
740: [2611200..2611263]: 31261552..31261615 3 (6095920..6095983) 64 100000
741: [2611264..2614839]: 30809824..30813399 3 (5644192..5647767) 3576 100000
742: [2614840..2614847]: 30809760..30809767 3 (5644128..5644135) 8 100000
743: [2614848..2614863]: 30833816..30833831 3 (5668184..5668199) 16 100000
744: [2614864..2615679]: 30817680..30818495 3 (5652048..5652863) 816 100000
745: [2615680..2615743]: 31261616..31261679 3 (6095984..6096047) 64 100000
746: [2615744..2626783]: 30818560..30829599 3 (5652928..5663967) 11040 100000
747: [2626784..2626807]: 31041760..31041783 3 (5876128..5876151) 24 100000
748: [2626808..2626815]: 30833808..30833815 3 (5668176..5668183) 8 100000
749: [2626816..2626879]: 31261680..31261743 3 (6096048..6096111) 64 100000
750: [2626880..2626903]: 31315736..31315759 3 (6150104..6150127) 24 100000
751: [2626904..2626943]: 30818520..30818559 3 (5652888..5652927) 40 100000
752: [2626944..2627095]: 31362800..31362951 3 (6197168..6197319) 152 100000
753: [2627096..2627135]: 30809768..30809807 3 (5644136..5644175) 40 100000
754: [2627136..2627199]: 31316320..31316383 3 (6150688..6150751) 64 100000
755: [2627200..2629223]: 31109176..31111199 3 (5943544..5945567) 2024 100000
756: [2629224..2629231]: 31106784..31106791 3 (5941152..5941159) 8 100000
757: [2629232..2632831]: 31111208..31114807 3 (5945576..5949175) 3600 100000
758: [2632832..2632895]: 31362952..31363015 3 (6197320..6197383) 64 100000
759: [2632896..2637823]: 31114872..31119799 3 (5949240..5954167) 4928 100000
760: [2637824..2638287]: 31363016..31363479 3 (6197384..6197847) 464 100000
761: [2638288..2648575]: 31120264..31130551 3 (5954632..5964919) 10288 100000
762: [2648576..2648639]: 31363480..31363543 3 (6197848..6197911) 64 100000
763: [2648640..2655167]: 31130616..31137143 3 (5964984..5971511) 6528 100000
764: [2655168..2655207]: 31363544..31363583 3 (6197912..6197951) 40 100000
765: [2655208..2655231]: 31265688..31265711 3 (6100056..6100079) 24 100000
766: [2655232..2656399]: 31363584..31364751 3 (6197952..6199119) 1168 100000
767: [2656400..2656415]: 31265712..31265727 3 (6100080..6100095) 16 100000
768: [2656416..2656583]: 31364752..31364919 3 (6199120..6199287) 168 100000
769: [2656584..2656615]: 31265728..31265759 3 (6100096..6100127) 32 100000
770: [2656616..2656719]: 31364920..31365023 3 (6199288..6199391) 104 100000
771: [2656720..2656735]: 31265760..31265775 3 (6100128..6100143) 16 100000
772: [2656736..2657423]: 31365024..31365711 3 (6199392..6200079) 688 100000
773: [2657424..2657455]: 31265776..31265807 3 (6100144..6100175) 32 100000
774: [2657456..2657831]: 31365712..31366087 3 (6200080..6200455) 376 100000
775: [2657832..2657855]: 31265872..31265895 3 (6100240..6100263) 24 100000
776: [2657856..2658055]: 31366088..31366287 3 (6200456..6200655) 200 100000
777: [2658056..2658103]: 31265896..31265943 3 (6100264..6100311) 48 100000
778: [2658104..2658183]: 31366288..31366367 3 (6200656..6200735) 80 100000
779: [2658184..2658231]: 31266024..31266071 3 (6100392..6100439) 48 100000
780: [2658232..2658319]: 31366368..31366455 3 (6200736..6200823) 88 100000
781: [2658320..2658351]: 31266160..31266191 3 (6100528..6100559) 32 100000
782: [2658352..2658447]: 31366456..31366551 3 (6200824..6200919) 96 100000
783: [2658448..2658487]: 31266288..31266327 3 (6100656..6100695) 40 100000
784: [2658488..2658567]: 31366552..31366631 3 (6200920..6200999) 80 100000
785: [2658568..2658607]: 31266408..31266447 3 (6100776..6100815) 40 100000
786: [2658608..2658735]: 31366632..31366759 3 (6201000..6201127) 128 100000
787: [2658736..2658751]: 31266448..31266463 3 (6100816..6100831) 16 100000
788: [2658752..2658903]: 31366760..31366911 3 (6201128..6201279) 152 100000
789: [2658904..2658935]: 31266464..31266495 3 (6100832..6100863) 32 100000
790: [2658936..2659007]: 31366912..31366983 3 (6201280..6201351) 72 100000
791: [2659008..2673863]: 31300672..31315527 3 (6135040..6149895) 14856 100000
792: [2673864..3095551]: hole 421688
793: [3095552..3095599]: 28177832..28177879 3 (3012200..3012247) 48 100000
794: [3095600..3146439]: 28046456..28097295 3 (2880824..2931663) 50840 100000
795: [3146440..3226703]: hole 80264
796: [3226704..3226727]: 28207200..28207223 3 (3041568..3041591) 24 100000
797: [3226728..3226903]: 30649248..30649423 3 (5483616..5483791) 176 100000
798: [3226904..3226943]: 28207224..28207263 3 (3041592..3041631) 40 100000
799: [3226944..3227071]: 28769432..28769559 3 (3603800..3603927) 128 100000
800: [3227072..3281327]: 28341016..28395271 3 (3175384..3229639) 54256 100000
801: [3281328..3281351]: 28628664..28628687 3 (3463032..3463055) 24 100000
802: [3281352..3281383]: 28478696..28478727 3 (3313064..3313095) 32 100000
803: [3281384..3281407]: 28478864..28478887 3 (3313232..3313255) 24 100000
804: [3281408..3281495]: 28880424..28880511 3 (3714792..3714879) 88 100000
805: [3281496..3281519]: 28478976..28478999 3 (3313344..3313367) 24 100000
806: [3281520..3281535]: 28478888..28478903 3 (3313256..3313271) 16 100000
807: [3281536..3281687]: 29621592..29621743 3 (4455960..4456111) 152 100000
808: [3281688..3281719]: 28479000..28479031 3 (3313368..3313399) 32 100000
809: [3281720..3281727]: 28478904..28478911 3 (3313272..3313279) 8 100000
810: [3281728..3281983]: 31119800..31120055 3 (5954168..5954423) 256 100000
811: [3281984..3282119]: 28478728..28478863 3 (3313096..3313231) 136 100000
812: [3282120..3282879]: 28479032..28479791 3 (3313400..3314159) 760 100000
813: [3282880..3282943]: 28832336..28832399 3 (3666704..3666767) 64 100000
814: [3282944..3285215]: 28479856..28482127 3 (3314224..3316495) 2272 100000
815: [3285216..3285255]: 28484040..28484079 3 (3318408..3318447) 40 100000
816: [3285256..3285311]: 28487728..28487783 3 (3322096..3322151) 56 100000
817: [3285312..3285327]: 28484080..28484095 3 (3318448..3318463) 16 100000
818: [3285328..3291143]: 28492648..28498463 3 (3327016..3332831) 5816 100000
819: [3291144..3291183]: 28835472..28835511 3 (3669840..3669879) 40 100000
820: [3291184..3291199]: 28508680..28508695 3 (3343048..3343063) 16 100000
821: [3291200..3291327]: 29308888..29309015 3 (4143256..4143383) 128 100000
822: [3291328..3321255]: 28583664..28613591 3 (3418032..3447959) 29928 100000
823: [3321256..3321343]: 28971440..28971527 3 (3805808..3805895) 88 100000
824: [3321344..3326935]: 28613680..28619271 3 (3448048..3453639) 5592 100000
825: [3326936..3326943]: 28613672..28613679 3 (3448040..3448047) 8 100000
826: [3326944..3434543]: 28643064..28750663 3 (3477432..3585031) 107600 100000
827: [3434544..3434639]: 28971600..28971695 3 (3805968..3806063) 96 100000
828: [3434640..3434687]: 28750760..28750807 3 (3585128..3585175) 48 100000
829: [3434688..3434775]: 29004192..29004279 3 (3838560..3838647) 88 100000
830: [3434776..3434815]: 28750896..28750935 3 (3585264..3585303) 40 100000
831: [3434816..3434943]: 29430736..29430863 3 (4265104..4265231) 128 100000
832: [3434944..3446015]: 28751064..28762135 3 (3585432..3596503) 11072 100000
833: [3446016..3446079]: 28841936..28841999 3 (3676304..3676367) 64 100000
834: [3446080..3452991]: 28762200..28769111 3 (3596568..3603479) 6912 100000
835: [3452992..3453143]: 31252328..31252479 3 (6086696..6086847) 152 100000
836: [3453144..3453183]: 28769264..28769303 3 (3603632..3603671) 40 100000
837: [3453184..3453279]: 31252480..31252575 3 (6086848..6086943) 96 100000
838: [3453280..3453311]: 28769400..28769431 3 (3603768..3603799) 32 100000
839: [3453312..3453463]: 31252576..31252727 3 (6086944..6087095) 152 100000
840: [3453464..3453503]: 28769584..28769623 3 (3603952..3603991) 40 100000
841: [3453504..3453567]: 29621784..29621847 3 (4456152..4456215) 64 100000
842: [3453568..3465863]: 28769688..28781983 3 (3604056..3616351) 12296 100000
843: [3465864..3471295]: 28887656..28893087 3 (3722024..3727455) 5432 100000
844: [3471296..3471359]: 30309432..30309495 3 (5143800..5143863) 64 100000
845: [3471360..3480703]: 28893152..28902495 3 (3727520..3736863) 9344 100000
846: [3480704..3480767]: 30317824..30317887 3 (5152192..5152255) 64 100000
847: [3480768..3483775]: 28902560..28905567 3 (3736928..3739935) 3008 100000
848: [3483776..3483839]: 30389448..30389511 3 (5223816..5223879) 64 100000
849: [3483840..3486695]: 28905632..28908487 3 (3740000..3742855) 2856 100000
850: [3486696..3494143]: 29013944..29021391 3 (3848312..3855759) 7448 100000
851: [3494144..3494207]: 30415368..30415431 3 (5249736..5249799) 64 100000
852: [3494208..3496255]: 29021456..29023503 3 (3855824..3857871) 2048 100000
853: [3496256..3496319]: 30448552..30448615 3 (5282920..5282983) 64 100000
854: [3496320..3512055]: 29023568..29039303 3 (3857936..3873671) 15736 100000
855: [3512056..3512167]: 31252728..31252839 3 (6087096..6087207) 112 100000
856: [3512168..3512191]: 29039304..29039327 3 (3873672..3873695) 24 100000
857: [3512192..3512343]: 31252840..31252991 3 (6087208..6087359) 152 100000
858: [3512344..3512383]: 29039480..29039519 3 (3873848..3873887) 40 100000
859: [3512384..3512447]: 30477288..30477351 3 (5311656..5311719) 64 100000
860: [3512448..3520727]: 29039584..29047863 3 (3873952..3882231) 8280 100000
861: [3520728..3524655]: 29264176..29268103 3 (4098544..4102471) 3928 100000
862: [3524656..3524751]: 31252992..31253087 3 (6087360..6087455) 96 100000
863: [3524752..3524799]: 29268200..29268247 3 (4102568..4102615) 48 100000
864: [3524800..3524863]: 30479784..30479847 3 (5314152..5314215) 64 100000
865: [3524864..3552607]: 29268312..29296055 3 (4102680..4130423) 27744 100000
866: [3552608..3552767]: 31253088..31253247 3 (6087456..6087615) 160 100000
867: [3552768..3564919]: 29296216..29308367 3 (4130584..4142735) 12152 100000
868: [3564920..3564951]: 28971528..28971559 3 (3805896..3805927) 32 100000
869: [3564952..3564991]: 29308400..29308439 3 (4142768..4142807) 40 100000
870: [3564992..3565079]: 31041592..31041679 3 (5875960..5876047) 88 100000
871: [3565080..3565119]: 29308528..29308567 3 (4142896..4142935) 40 100000
872: [3565120..3565271]: 31253248..31253399 3 (6087616..6087767) 152 100000
873: [3565272..3565311]: 29308720..29308759 3 (4143088..4143127) 40 100000
874: [3565312..3565399]: 31253400..31253487 3 (6087768..6087855) 88 100000
875: [3565400..3565439]: 29308848..29308887 3 (4143216..4143255) 40 100000
876: [3565440..3565567]: 31253488..31253615 3 (6087856..6087983) 128 100000
877: [3565568..3571455]: 29309016..29314903 3 (4143384..4149271) 5888 100000
878: [3571456..3571519]: 30498920..30498983 3 (5333288..5333351) 64 100000
879: [3571520..3629695]: 29314968..29373143 3 (4149336..4207511) 58176 100000
880: [3629696..3629727]: 28947432..28947463 3 (3781800..3781831) 32 100000
881: [3629728..3629759]: 29380744..29380775 3 (4215112..4215143) 32 100000
882: [3629760..3629919]: 31253616..31253775 3 (6087984..6088143) 160 100000
883: [3629920..3629951]: 29380776..29380807 3 (4215144..4215175) 32 100000
884: [3629952..3630039]: 31253776..31253863 3 (6088144..6088231) 88 100000
885: [3630040..3630079]: 29419728..29419767 3 (4254096..4254135) 40 100000
886: [3630080..3630143]: 30600888..30600951 3 (5435256..5435319) 64 100000
887: [3630144..3638919]: 29521728..29530503 3 (4356096..4364871) 8776 100000
888: [3638920..3638943]: 29596112..29596135 3 (4430480..4430503) 24 100000
889: [3638944..3638967]: 29520800..29520823 3 (4355168..4355191) 24 100000
890: [3638968..3639111]: 31253864..31254007 3 (6088232..6088375) 144 100000
891: [3639112..3639935]: 29543384..29544207 3 (4377752..4378575) 824 100000
892: [3639936..3639999]: 30631928..30631991 3 (5466296..5466359) 64 100000
893: [3640000..3654151]: 29544272..29558423 3 (4378640..4392791) 14152 100000
894: [3654152..3654175]: 29621536..29621559 3 (4455904..4455927) 24 100000
895: [3654176..3654207]: 29562272..29562303 3 (4396640..4396671) 32 100000
896: [3654208..3654359]: 31254008..31254159 3 (6088376..6088527) 152 100000
897: [3654360..3654399]: 29562456..29562495 3 (4396824..4396863) 40 100000
898: [3654400..3654463]: 30655560..30655623 3 (5489928..5489991) 64 100000
899: [3654464..3661527]: 29562560..29569623 3 (4396928..4403991) 7064 100000
900: [3661528..3661815]: 29881400..29881687 3 (4715768..4716055) 288 100000
901: [3661816..3661911]: 31254160..31254255 3 (6088528..6088623) 96 100000
902: [3661912..3661951]: 29881784..29881823 3 (4716152..4716191) 40 100000
903: [3661952..3662015]: 31046584..31046647 3 (5880952..5881015) 64 100000
904: [3662016..3674303]: 29881888..29894175 3 (4716256..4728543) 12288 100000
905: [3674304..3674367]: 31097120..31097183 3 (5931488..5931551) 64 100000
906: [3674368..3676671]: 29894240..29896543 3 (4728608..4730911) 2304 100000
907: [3676672..3676735]: 31101600..31101663 3 (5935968..5936031) 64 100000
908: [3676736..3699327]: 29896608..29919199 3 (4730976..4753567) 22592 100000
909: [3699328..3699415]: 31254256..31254343 3 (6088624..6088711) 88 100000
910: [3699416..3699455]: 29919288..29919327 3 (4753656..4753695) 40 100000
911: [3699456..3699583]: 31254344..31254471 3 (6088712..6088839) 128 100000
912: [3699584..3717183]: 29919456..29937055 3 (4753824..4771423) 17600 100000
913: [3717184..3717279]: 31254472..31254567 3 (6088840..6088935) 96 100000
914: [3717280..3717311]: 29937152..29937183 3 (4771520..4771551) 32 100000
915: [3717312..3717439]: 31254568..31254695 3 (6088936..6089063) 128 100000
916: [3717440..3721079]: 29937312..29940951 3 (4771680..4775319) 3640 100000
917: [3721080..3721103]: 29850328..29850351 3 (4684696..4684719) 24 100000
918: [3721104..3721151]: 29937184..29937231 3 (4771552..4771599) 48 100000
919: [3721152..3721215]: 31105496..31105559 3 (5939864..5939927) 64 100000
920: [3721216..3840839]: 30072848..30192471 3 (4907216..5026839) 119624 100000
921: [3840840..3861319]: 30282424..30302903 3 (5116792..5137271) 20480 100000
922: [3861320..3861863]: 30197032..30197575 3 (5031400..5031943) 544 100000
923: [3861864..3863879]: 30302904..30304919 3 (5137272..5139287) 2016 100000
924: [3863880..3897231]: 30197576..30230927 3 (5031944..5065295) 33352 100000
925: [3897232..3897263]: 30479728..30479759 3 (5314096..5314127) 32 100000
926: [3897264..3897279]: 30304920..30304935 3 (5139288..5139303) 16 100000
927: [3897280..3897431]: 31254696..31254847 3 (6089064..6089215) 152 100000
928: [3897432..3897471]: 30305088..30305127 3 (5139456..5139495) 40 100000
929: [3897472..3897535]: 31106712..31106775 3 (5941080..5941143) 64 100000
930: [3897536..3897559]: 31105064..31105087 3 (5939432..5939455) 24 100000
931: [3897560..3897567]: 30305128..30305135 3 (5139496..5139503) 8 100000
932: [3897568..3897815]: 31261744..31261991 3 (6096112..6096359) 248 100000
933: [3897816..3897855]: 30305136..30305175 3 (5139504..5139543) 40 100000
934: [3897856..3898015]: 31261992..31262151 3 (6096360..6096519) 160 100000
935: [3898016..3898047]: 30305176..30305207 3 (5139544..5139575) 32 100000
936: [3898048..3898143]: 31262152..31262247 3 (6096520..6096615) 96 100000
937: [3898144..3898175]: 30305304..30305335 3 (5139672..5139703) 32 100000
938: [3898176..3898303]: 31262248..31262375 3 (6096616..6096743) 128 100000
939: [3898304..3902271]: 30305464..30309431 3 (5139832..5143799) 3968 100000
940: [3902272..3902335]: 31262376..31262439 3 (6096744..6096807) 64 100000
941: [3902336..3907079]: 30309496..30314239 3 (5143864..5148607) 4744 100000
942: [3907080..3907087]: 30316064..30316071 3 (5150432..5150439) 8 100000
943: [3907088..3910215]: 30372344..30375471 3 (5206712..5209839) 3128 100000
944: [3910216..3910231]: 30498880..30498895 3 (5333248..5333263) 16 100000
945: [3910232..3910271]: 30387376..30387415 3 (5221744..5221783) 40 100000
946: [3910272..3910399]: 31262440..31262567 3 (6096808..6096935) 128 100000
947: [3910400..3928599]: 30480680..30498879 3 (5315048..5333247) 18200 100000
948: [3928600..3928623]: 30592680..30592703 3 (5427048..5427071) 24 100000
949: [3928624..3928639]: 30498904..30498919 3 (5333272..5333287) 16 100000
950: [3928640..3928703]: 31262568..31262631 3 (6096936..6096999) 64 100000
951: [3928704..4022399]: 30498984..30592679 3 (5333352..5427047) 93696 100000
952: [4022400..4022423]: 31120240..31120263 3 (5954608..5954631) 24 100000
953: [4022424..4022463]: 30592704..30592743 3 (5427072..5427111) 40 100000
954: [4022464..4022591]: 31262632..31262759 3 (6097000..6097127) 128 100000
955: [4022592..4027391]: 30592872..30597671 3 (5427240..5432039) 4800 100000
956: [4027392..4027479]: 31262760..31262847 3 (6097128..6097215) 88 100000
957: [4027480..4027519]: 30597760..30597799 3 (5432128..5432167) 40 100000
958: [4027520..4027679]: 31262848..31263007 3 (6097216..6097375) 160 100000
959: [4027680..4027711]: 30597800..30597831 3 (5432168..5432199) 32 100000
960: [4027712..4027799]: 31263008..31263095 3 (6097376..6097463) 88 100000
961: [4027800..4027839]: 30597920..30597959 3 (5432288..5432327) 40 100000
962: [4027840..4027975]: 31263096..31263231 3 (6097464..6097599) 136 100000
963: [4027976..4028463]: 30598096..30598583 3 (5432464..5432951) 488 100000
964: [4028464..4031863]: 30659536..30662935 3 (5493904..5497303) 3400 100000
965: [4031864..4036655]: 30667000..30671791 3 (5501368..5506159) 4792 100000
966: [4036656..4036735]: 31263232..31263311 3 (6097600..6097679) 80 100000
967: [4036736..4037015]: 30657640..30657919 3 (5492008..5492287) 280 100000
968: [4037016..4037103]: 30657512..30657599 3 (5491880..5491967) 88 100000
969: [4037104..4037183]: 30657352..30657431 3 (5491720..5491799) 80 100000
970: [4037184..4040439]: 30696384..30699639 3 (5530752..5534007) 3256 100000
971: [4040440..4076903]: 30723352..30759815 3 (5557720..5594183) 36464 100000
972: [4076904..4077055]: 31263312..31263463 3 (6097680..6097831) 152 100000
973: [4077056..4091359]: 30777408..30791711 3 (5611776..5626079) 14304 100000
974: [4091360..4095567]: 30829600..30833807 3 (5663968..5668175) 4208 100000
975: [4095568..4095591]: 31263464..31263487 3 (6097832..6097855) 24 100000
976: [4095592..4095615]: 30833832..30833855 3 (5668200..5668223) 24 100000
977: [4095616..4095679]: 31263488..31263551 3 (6097856..6097919) 64 100000
978: [4095680..4099575]: 30833920..30837815 3 (5668288..5672183) 3896 100000
979: [4099576..4099591]: 30833872..30833887 3 (5668240..5668255) 16 100000
980: [4099592..4099607]: 30833856..30833871 3 (5668224..5668239) 16 100000
981: [4099608..4099623]: 30699664..30699679 3 (5534032..5534047) 16 100000
982: [4099624..4099647]: 30833888..30833911 3 (5668256..5668279) 24 100000
983: [4099648..4099711]: 31263552..31263615 3 (6097920..6097983) 64 100000
984: [4099712..4108743]: 31018392..31027423 3 (5852760..5861791) 9032 100000
985: [4108744..4108775]: 31366984..31367015 3 (6201352..6201383) 32 100000
986: [4108776..4108799]: 31041536..31041559 3 (5875904..5875927) 24 100000
987: [4108800..4108951]: 31367016..31367167 3 (6201384..6201535) 152 100000
988: [4108952..4108991]: 31041720..31041759 3 (5876088..5876127) 40 100000
989: [4108992..4109055]: 31367168..31367231 3 (6201536..6201599) 64 100000
990: [4109056..4111783]: 31137144..31139871 3 (5971512..5974239) 2728 100000
991: [4111784..4118015]: hole 6232
992: [4118016..4118063]: 28177880..28177927 3 (3012248..3012295) 48 100000
993: [4118064..4118095]: hole 32
994: [4118096..4118111]: 28207264..28207279 3 (3041632..3041647) 16 100000
995: [4118112..4118271]: 29296056..29296215 3 (4130424..4130583) 160 100000
996: [4118272..4118303]: 29308368..29308399 3 (4142736..4142767) 32 100000
997: [4118304..4118335]: 28207336..28207367 3 (3041704..3041735) 32 100000
998: [4118336..4118399]: 28210304..28210367 3 (3044672..3044735) 64 100000
999: [4118400..4118463]: 28750832..28750895 3 (3585200..3585263) 64 100000
1000: [4118464..4127847]: 28395272..28404655 3 (3229640..3239023) 9384 100000
1001: [4127848..4128031]: 31120056..31120239 3 (5954424..5954607) 184 100000
1002: [4128032..4128039]: 31105560..31105567 3 (5939928..5939935) 8 100000
1003: [4128040..4128063]: 28478912..28478935 3 (3313280..3313303) 24 100000
1004: [4128064..4128159]: 29430584..29430679 3 (4264952..4265047) 96 100000
1005: [4128160..4128191]: 28478936..28478967 3 (3313304..3313335) 32 100000
1006: [4128192..4128343]: 29777176..29777327 3 (4611544..4611695) 152 100000
1007: [4128344..4128383]: 28479792..28479831 3 (3314160..3314199) 40 100000
1008: [4128384..4128519]: 29846712..29846847 3 (4681080..4681215) 136 100000
1009: [4128520..4130399]: 28482128..28484007 3 (3316496..3318375) 1880 100000
1010: [4130400..4130415]: 28628688..28628703 3 (3463056..3463071) 16 100000
1011: [4130416..4130431]: 28484024..28484039 3 (3318392..3318407) 16 100000
1012: [4130432..4130495]: 28874688..28874751 3 (3709056..3709119) 64 100000
1013: [4130496..4132095]: 28484104..28485703 3 (3318472..3320071) 1600 100000
1014: [4132096..4135231]: 28498464..28501599 3 (3332832..3335967) 3136 100000
1015: [4135232..4135615]: 28502184..28502567 3 (3336552..3336935) 384 100000
1016: [4135616..4135639]: 28640648..28640671 3 (3475016..3475039) 24 100000
1017: [4135640..4135679]: 28508696..28508735 3 (3343064..3343103) 40 100000
1018: [4135680..4135807]: 29478304..29478431 3 (4312672..4312799) 128 100000
1019: [4135808..4138399]: 28619272..28621863 3 (3453640..3456231) 2592 100000
1020: [4138400..4141871]: 28781984..28785455 3 (3616352..3619823) 3472 100000
1021: [4141872..4141951]: 29308440..29308519 3 (4142808..4142887) 80 100000
1022: [4141952..4147711]: 28785536..28791295 3 (3619904..3625663) 5760 100000
1023: [4147712..4147863]: 31051512..31051663 3 (5885880..5886031) 152 100000
1024: [4147864..4147903]: 28791448..28791487 3 (3625816..3625855) 40 100000
1025: [4147904..4147967]: 28877632..28877695 3 (3712000..3712063) 64 100000
1026: [4147968..4163063]: 28791552..28806647 3 (3625920..3641015) 15096 100000
1027: [4163064..4163135]: 28915776..28915847 3 (3750144..3750215) 72 100000
1028: [4163136..4164607]: 28806720..28808191 3 (3641088..3642559) 1472 100000
1029: [4164608..4164671]: 28893088..28893151 3 (3727456..3727519) 64 100000
1030: [4164672..4169655]: 28808256..28813239 3 (3642624..3647607) 4984 100000
1031: [4169656..4169687]: 28750728..28750759 3 (3585096..3585127) 32 100000
1032: [4169688..4169711]: 28813272..28813295 3 (3647640..3647663) 24 100000
1033: [4169712..4169727]: 28813240..28813255 3 (3647608..3647623) 16 100000
1034: [4169728..4169823]: 31254848..31254943 3 (6089216..6089311) 96 100000
1035: [4169824..4169855]: 28027536..28027567 3 (2861904..2861935) 32 100000
1036: [4169856..4170015]: 25169472..25169631 3 (3840..3999) 160 100000
1037: [4170016..4170047]: 28027400..28027431 3 (2861768..2861799) 32 100000
1038: [4170048..4170207]: 25169632..25169791 3 (4000..4159) 160 100000
1039: [4170208..4170239]: 28027216..28027247 3 (2861584..2861615) 32 100000
1040: [4170240..4170303]: 25169792..25169855 3 (4160..4223) 64 100000
1041: [4170304..4174207]: 28813296..28817199 3 (3647664..3651567) 3904 100000
1042: [4174208..4181495]: 28908488..28915775 3 (3742856..3750143) 7288 100000
1043: [4181496..4181567]: 31254944..31255015 3 (6089312..6089383) 72 100000
1044: [4181568..4198079]: 28915848..28932359 3 (3750216..3766727) 16512 100000
1045: [4198080..4198143]: 31106848..31106911 3 (5941216..5941279) 64 100000
1046: [4198144..4198591]: 28932424..28932871 3 (3766792..3767239) 448 100000
1047: [4198592..4198719]: 31255016..31255143 3 (6089384..6089511) 128 100000
1048: [4198720..4213119]: 28933000..28947399 3 (3767368..3781767) 14400 100000
1049: [4213120..4213183]: 31114808..31114871 3 (5949176..5949239) 64 100000
1050: [4213184..4218119]: 28947464..28952399 3 (3781832..3786767) 4936 100000
1051: [4218120..4218143]: 28932952..28932975 3 (3767320..3767343) 24 100000
1052: [4218144..4218175]: 28947400..28947431 3 (3781768..3781799) 32 100000
1053: [4218176..4218327]: 31255144..31255295 3 (6089512..6089663) 152 100000
1054: [4218328..4218367]: 28932872..28932911 3 (3767240..3767279) 40 100000
1055: [4218368..4218455]: 31255296..31255383 3 (6089664..6089751) 88 100000
1056: [4218456..4218495]: 28932912..28932951 3 (3767280..3767319) 40 100000
1057: [4218496..4218559]: 31130552..31130615 3 (5964920..5964983) 64 100000
1058: [4218560..4224063]: 29047864..29053367 3 (3882232..3887735) 5504 100000
1059: [4224064..4224127]: 31178616..31178679 3 (6012984..6013047) 64 100000
1060: [4224128..4242239]: 29053432..29071543 3 (3887800..3905911) 18112 100000
1061: [4242240..4242303]: 31255384..31255447 3 (6089752..6089815) 64 100000
1062: [4242304..4243263]: 29071608..29072567 3 (3905976..3906935) 960 100000
1063: [4243264..4243327]: 31255448..31255511 3 (6089816..6089879) 64 100000
1064: [4243328..4267583]: 29072632..29096887 3 (3907000..3931255) 24256 100000
1065: [4267584..4267607]: 29098560..29098583 3 (3932928..3932951) 24 100000
1066: [4267608..4267647]: 29098520..29098559 3 (3932888..3932927) 40 100000
1067: [4267648..4267743]: 31255512..31255607 3 (6089880..6089975) 96 100000
1068: [4267744..4267775]: 29072568..29072599 3 (3906936..3906967) 32 100000
1069: [4267776..4267935]: 31255608..31255767 3 (6089976..6090135) 160 100000
1070: [4267936..4267967]: 29072600..29072631 3 (3906968..3906999) 32 100000
1071: [4267968..4268351]: 31255768..31256151 3 (6090136..6090519) 384 100000
1072: [4268352..4275951]: 29373144..29380743 3 (4207512..4215111) 7600 100000
1073: [4275952..4276039]: 31256152..31256239 3 (6090520..6090607) 88 100000
1074: [4276040..4285503]: 29380832..29390295 3 (4215200..4224663) 9464 100000
1075: [4285504..4285535]: 28884064..28884095 3 (3718432..3718463) 32 100000
1076: [4285536..4285567]: 29427600..29427631 3 (4261968..4261999) 32 100000
1077: [4285568..4285727]: 31256240..31256399 3 (6090608..6090767) 160 100000
1078: [4285728..4285759]: 29427632..29427663 3 (4262000..4262031) 32 100000
1079: [4285760..4285911]: 31256400..31256551 3 (6090768..6090919) 152 100000
1080: [4285912..4285951]: 29429200..29429239 3 (4263568..4263607) 40 100000
1081: [4285952..4286015]: 31256552..31256615 3 (6090920..6090983) 64 100000
1082: [4286016..4294199]: 29569624..29577807 3 (4403992..4412175) 8184 100000
1083: [4294200..4294271]: 31256616..31256687 3 (6090984..6091055) 72 100000
1084: [4294272..4312503]: 29577880..29596111 3 (4412248..4430479) 18232 100000
1085: [4312504..4312527]: 29429240..29429263 3 (4263608..4263631) 24 100000
1086: [4312528..4312575]: 29596136..29596183 3 (4430504..4430551) 48 100000
1087: [4312576..4312639]: 31256688..31256751 3 (6091056..6091119) 64 100000
1088: [4312640..4317295]: 29596248..29600903 3 (4430616..4435271) 4656 100000
1089: [4317296..4317383]: 31256752..31256839 3 (6091120..6091207) 88 100000
1090: [4317384..4337927]: 29600992..29621535 3 (4435360..4455903) 20544 100000
1091: [4337928..4337951]: 29419768..29419791 3 (4254136..4254159) 24 100000
1092: [4337952..4337983]: 29621560..29621591 3 (4455928..4455959) 32 100000
1093: [4337984..4338135]: 31256840..31256991 3 (6091208..6091359) 152 100000
1094: [4338136..4338175]: 29621744..29621783 3 (4456112..4456151) 40 100000
1095: [4338176..4338239]: 31256992..31257055 3 (6091360..6091423) 64 100000
1096: [4338240..4343279]: 29621848..29626887 3 (4456216..4461255) 5040 100000
1097: [4343280..4343375]: 31257056..31257151 3 (6091424..6091519) 96 100000
1098: [4343376..4343423]: 29626984..29627031 3 (4461352..4461399) 48 100000
1099: [4343424..4343487]: 31257152..31257215 3 (6091520..6091583) 64 100000
1100: [4343488..4361343]: 29627096..29644951 3 (4461464..4479319) 17856 100000
1101: [4361344..4361407]: 31257216..31257279 3 (6091584..6091647) 64 100000
1102: [4361408..4396911]: 29645016..29680519 3 (4479384..4514887) 35504 100000
1103: [4396912..4396999]: 31257280..31257367 3 (6091648..6091735) 88 100000
1104: [4397000..4397023]: 29680608..29680631 3 (4514976..4514999) 24 100000
1105: [4397024..4397183]: 31257368..31257527 3 (6091736..6091895) 160 100000
1106: [4397184..4400343]: 29680792..29683951 3 (4515160..4518319) 3160 100000
1107: [4400344..4408871]: 29940952..29949479 3 (4775320..4783847) 8528 100000
1108: [4408872..4409071]: 31257528..31257727 3 (6091896..6092095) 200 100000
1109: [4409072..4409087]: 29949480..29949495 3 (4783848..4783863) 16 100000
1110: [4409088..4409215]: 31257728..31257855 3 (6092096..6092223) 128 100000
1111: [4409216..4459575]: 29949624..29999983 3 (4783992..4834351) 50360 100000
1112: [4459576..4459671]: 31257856..31257951 3 (6092224..6092319) 96 100000
1113: [4459672..4459711]: 29949496..29949535 3 (4783864..4783903) 40 100000
1114: [4459712..4459775]: 31257952..31258015 3 (6092320..6092383) 64 100000
1115: [4459776..4464335]: 30192472..30197031 3 (5026840..5031399) 4560 100000
1116: [4464336..4464511]: 31258016..31258191 3 (6092384..6092559) 176 100000
1117: [4464512..4464543]: 31263616..31263647 3 (6097984..6098015) 32 100000
1118: [4464544..4464575]: 30304936..30304967 3 (5139304..5139335) 32 100000
1119: [4464576..4464735]: 31263648..31263807 3 (6098016..6098175) 160 100000
1120: [4464736..4464767]: 30304968..30304999 3 (5139336..5139367) 32 100000
1121: [4464768..4465175]: 31263808..31264215 3 (6098176..6098583) 408 100000
1122: [4465176..4465215]: 30305000..30305039 3 (5139368..5139407) 40 100000
1123: [4465216..4465279]: 31264216..31264279 3 (6098584..6098647) 64 100000
1124: [4465280..4467103]: 30314240..30316063 3 (5148608..5150431) 1824 100000
1125: [4467104..4467111]: 30317800..30317807 3 (5152168..5152175) 8 100000
1126: [4467112..4468823]: 30316072..30317783 3 (5150440..5152151) 1712 100000
1127: [4468824..4468847]: 29380808..29380831 3 (4215176..4215199) 24 100000
1128: [4468848..4468863]: 30317808..30317823 3 (5152176..5152191) 16 100000
1129: [4468864..4468927]: 31264280..31264343 3 (6098648..6098711) 64 100000
1130: [4468928..4478703]: 30317888..30327663 3 (5152256..5162031) 9776 100000
1131: [4478704..4485439]: 30375472..30382207 3 (5209840..5216575) 6736 100000
1132: [4485440..4485463]: 31264344..31264367 3 (6098712..6098735) 24 100000
1133: [4485464..4485503]: 30387416..30387455 3 (5221784..5221823) 40 100000
1134: [4485504..4485567]: 31264368..31264431 3 (6098736..6098799) 64 100000
1135: [4485568..4487871]: 30598584..30600887 3 (5432952..5435255) 2304 100000
1136: [4487872..4487935]: 31264432..31264495 3 (6098800..6098863) 64 100000
1137: [4487936..4518911]: 30600952..30631927 3 (5435320..5466295) 30976 100000
1138: [4518912..4518975]: 31264496..31264559 3 (6098864..6098927) 64 100000
1139: [4518976..4523479]: 30631992..30636495 3 (5466360..5470863) 4504 100000
1140: [4523480..4523511]: 31264560..31264591 3 (6098928..6098959) 32 100000
1141: [4523512..4523519]: 30636528..30636535 3 (5470896..5470903) 8 100000
1142: [4523520..4523647]: 31264592..31264719 3 (6098960..6099087) 128 100000
1143: [4523648..4528023]: 30636664..30641039 3 (5471032..5475407) 4376 100000
1144: [4528024..4528055]: 31264720..31264751 3 (6099088..6099119) 32 100000
1145: [4528056..4528063]: 30641072..30641079 3 (5475440..5475447) 8 100000
1146: [4528064..4528159]: 31264752..31264847 3 (6099120..6099215) 96 100000
1147: [4528160..4528191]: 30641176..30641207 3 (5475544..5475575) 32 100000
1148: [4528192..4528343]: 31264848..31264999 3 (6099216..6099367) 152 100000
1149: [4528344..4528359]: 30641208..30641223 3 (5475576..5475591) 16 100000
1150: [4528360..4528575]: 31265000..31265215 3 (6099368..6099583) 216 100000
1151: [4528576..4531167]: 30641440..30644031 3 (5475808..5478399) 2592 100000
1152: [4531168..4534023]: 30662936..30665791 3 (5497304..5500159) 2856 100000
1153: [4534024..4542527]: 30671792..30680295 3 (5506160..5514663) 8504 100000
1154: [4542528..4542535]: 30681400..30681407 3 (5515768..5515775) 8 100000
1155: [4542536..4542543]: 30699744..30699751 3 (5534112..5534119) 8 100000
1156: [4542544..4542567]: 30699640..30699663 3 (5534008..5534031) 24 100000
1157: [4542568..4542583]: 30649456..30649471 3 (5483824..5483839) 16 100000
1158: [4542584..4542591]: 30699680..30699687 3 (5534048..5534055) 8 100000
1159: [4542592..4542655]: 31265216..31265279 3 (6099584..6099647) 64 100000
1160: [4542656..4553727]: 30699752..30710823 3 (5534120..5545191) 11072 100000
1161: [4553728..4553791]: 31265280..31265343 3 (6099648..6099711) 64 100000
1162: [4553792..4555743]: 30710888..30712839 3 (5545256..5547207) 1952 100000
1163: [4555744..4559759]: 30759816..30763831 3 (5594184..5598199) 4016 100000
1164: [4559760..4559783]: 31265344..31265367 3 (6099712..6099735) 24 100000
1165: [4559784..4559791]: 30797312..30797319 3 (5631680..5631687) 8 100000
1166: [4559792..4559807]: 30797608..30797623 3 (5631976..5631991) 16 100000
1167: [4559808..4559871]: 31265368..31265431 3 (6099736..6099799) 64 100000
1168: [4559872..4565471]: 30791712..30797311 3 (5626080..5631679) 5600 100000
1169: [4565472..4565759]: 30797320..30797607 3 (5631688..5631975) 288 100000
1170: [4565760..4566279]: 30797624..30798143 3 (5631992..5632511) 520 100000
1171: [4566280..4566287]: 30809816..30809823 3 (5644184..5644191) 8 100000
1172: [4566288..4572887]: 30798152..30804751 3 (5632520..5639119) 6600 100000
1173: [4572888..4573767]: 30805928..30806807 3 (5640296..5641175) 880 100000
1174: [4573768..4735783]: 30837816..30999831 3 (5672184..5834199) 162016 100000
1175: [4735784..4749895]: 31027424..31041535 3 (5861792..5875903) 14112 100000
1176: [4749896..4749919]: 31265432..31265455 3 (6099800..6099823) 24 100000
1177: [4749920..4749951]: 31041560..31041591 3 (5875928..5875959) 32 100000
1178: [4749952..4750015]: 31265456..31265519 3 (6099824..6099887) 64 100000
1179: [4750016..4750039]: 31315864..31315887 3 (6150232..6150255) 24 100000
1180: [4750040..4750079]: 31041680..31041719 3 (5876048..5876087) 40 100000
1181: [4750080..4750143]: 31265520..31265583 3 (6099888..6099951) 64 100000
1182: [4750144..4754943]: 31041784..31046583 3 (5876152..5880951) 4800 100000
1183: [4754944..4755007]: 31367232..31367295 3 (6201600..6201663) 64 100000
1184: [4755008..4759863]: 31046648..31051503 3 (5881016..5885871) 4856 100000
1185: [4759864..4759903]: 31367296..31367335 3 (6201664..6201703) 40 100000
1186: [4759904..4759911]: 31051504..31051511 3 (5885872..5885879) 8 100000
1187: [4759912..4760063]: 31367336..31367487 3 (6201704..6201855) 152 100000
1188: [4760064..4761791]: 31051664..31053391 3 (5886032..5887759) 1728 100000
1189: [4761792..4761855]: 31367488..31367551 3 (6201856..6201919) 64 100000
1190: [4761856..4765175]: 31053456..31056775 3 (5887824..5891143) 3320 100000
1191: [4765176..4765247]: 31367552..31367623 3 (6201920..6201991) 72 100000
1192: [4765248..4770767]: 31056848..31062367 3 (5891216..5896735) 5520 100000
1193: [4770768..4770775]: 31053440..31053447 3 (5887808..5887815) 8 100000
1194: [4770776..4770815]: 31056776..31056815 3 (5891144..5891183) 40 100000
1195: [4770816..4771727]: 31367624..31368535 3 (6201992..6202903) 912 100000
1196: [4771728..4771775]: 31053392..31053439 3 (5887760..5887807) 48 100000
1197: [4771776..4771903]: 31368536..31368663 3 (6202904..6203031) 128 100000
1198: [4771904..4805183]: 31139872..31173151 3 (5974240..6007519) 33280 100000
1199: [4805184..4805191]: 31105568..31105575 3 (5939936..5939943) 8 100000
1200: [4805192..4805511]: 31178240..31178559 3 (6012608..6012927) 320 100000
1201: [4805512..4805519]: 31097560..31097567 3 (5931928..5931935) 8 100000
1202: [4805520..4805567]: 31178568..31178615 3 (6012936..6012983) 48 100000
1203: [4805568..4805631]: 31368664..31368727 3 (6203032..6203095) 64 100000
1204: [4805632..4876071]: 31178680..31249119 3 (6013048..6083487) 70440 100000
1205: [4876072..4877583]: 31368728..31370239 3 (6203096..6204607) 1512 100000
1206: [4877584..4877623]: 31266120..31266159 3 (6100488..6100527) 40 100000
1207: [4877624..4877903]: 31370240..31370519 3 (6204608..6204887) 280 100000
1208: [4877904..4877911]: 31266192..31266199 3 (6100560..6100567) 8 100000
1209: [4877912..4878407]: 31370520..31371015 3 (6204888..6205383) 496 100000
1210: [4878408..4878423]: 31266200..31266215 3 (6100568..6100583) 16 100000
1211: [4878424..4878567]: 31371016..31371159 3 (6205384..6205527) 144 100000
1212: [4878568..4878583]: 31266256..31266271 3 (6100624..6100639) 16 100000
1213: [4878584..4879983]: 31371160..31372559 3 (6205528..6206927) 1400 100000
1214: [4879984..4879999]: 31266272..31266287 3 (6100640..6100655) 16 100000
1215: [4880000..4880071]: 31372560..31372631 3 (6206928..6206999) 72 100000
1216: [4880072..4880127]: 31266496..31266551 3 (6100864..6100919) 56 100000
1217: [4880128..4880271]: 31372632..31372775 3 (6207000..6207143) 144 100000
1218: [4880272..4880303]: 31265992..31266023 3 (6100360..6100391) 32 100000
1219: [4880304..4880415]: 31372776..31372887 3 (6207144..6207255) 112 100000
1220: [4880416..4880431]: 31266072..31266087 3 (6100440..6100455) 16 100000
1221: [4880432..4880607]: 31372888..31373063 3 (6207256..6207431) 176 100000
1222: [4880608..4880639]: 31315528..31315559 3 (6149896..6149927) 32 100000
1223: [4880640..4880719]: 31373064..31373143 3 (6207432..6207511) 80 100000
1224: [4880720..4880743]: 31315640..31315663 3 (6150008..6150031) 24 100000
1225: [4880744..4880839]: 31373144..31373239 3 (6207512..6207607) 96 100000
1226: [4880840..4880879]: 31315760..31315799 3 (6150128..6150167) 40 100000
1227: [4880880..4880967]: 31373240..31373327 3 (6207608..6207695) 88 100000
1228: [4880968..4881023]: 31315888..31315943 3 (6150256..6150311) 56 100000
1229: [4881024..4881095]: 31373328..31373399 3 (6207696..6207767) 72 100000
1230: [4881096..4881143]: 31316016..31316063 3 (6150384..6150431) 48 100000
1231: [4881144..4881223]: 31373400..31373479 3 (6207768..6207847) 80 100000
1232: [4881224..4881271]: 31316144..31316191 3 (6150512..6150559) 48 100000
1233: [4881272..4881351]: 31373480..31373559 3 (6207848..6207927) 80 100000
1234: [4881352..4881399]: 31316272..31316319 3 (6150640..6150687) 48 100000
1235: [4881400..4881479]: 31373560..31373639 3 (6207928..6208007) 80 100000
1236: [4881480..4881511]: 31316400..31316431 3 (6150768..6150799) 32 100000
1237: [4881512..4881631]: 31373640..31373759 3 (6208008..6208127) 120 100000
1238: [4881632..4881639]: 31316432..31316439 3 (6150800..6150807) 8 100000
1239: [4881640..4881751]: 31373760..31373871 3 (6208128..6208239) 112 100000
1240: [4881752..4881775]: 31316440..31316463 3 (6150808..6150831) 24 100000
1241: [4881776..4882071]: 31373872..31374167 3 (6208240..6208535) 296 100000
1242: [4882072..4882079]: 31316464..31316471 3 (6150832..6150839) 8 100000
1243: [4882080..4882175]: 31374168..31374263 3 (6208536..6208631) 96 100000
1244: [4882176..4925015]: 31316568..31359407 3 (6150936..6193775) 42840 100000
1245: [4925016..5140223]: hole 215208
1246: [5140224..5140479]: 28177928..28178183 3 (3012296..3012551) 256 100000
1247: [5140480..5142487]: hole 2008
1248: [5142488..5142527]: 28028392..28028431 3 (2862760..2862799) 40 100000
1249: [5142528..20971519]: hole 15828992
FLAG Values:
0100000 Shared extent
0010000 Unwritten preallocated extent
0001000 Doesn't begin on stripe unit
0000100 Doesn't end on stripe unit
0000010 Doesn't begin on stripe width
0000001 Doesn't end on stripe width
As a lazy way to test this in other environments I opened a test PR against OSBuild that would then run in their CI
- https://github.com/osbuild/osbuild/pull/1594
Sure enough CI failed on two environments:
Thanks. So the disk.img does indeed have many reflinked extents, for starters (extents with flags 100000) When I compare the original disk image to the converted-back-to-raw qcow2 image, the qcow2 image seems to have many ranges of zeros rather than correct data:
# qemu-img convert osbuild/store/objects/e0ab569c34dd263e53d28e56e0b2f9d6f27c4cb7c4fe5d7177f57f8b719fd229/data/tree/qemu.qcow2 ../disk.img
# cmp -l ./osbuild/store/objects/0a4be93ab2bb5c17851a57574f3af0cffe135c58ccfc840e4f0ea37dda36f77c/data/tree/disk.img ../disk.img | awk '{print $3}' | sort | uniq
0
Not sure what to make of this yet... my first thought is that it's possibly related to preallocated/unwritten extents in the original disk.img but that's just a guess.
Hi,
Can you upload the pre-convert raw image somewhere, or is there an easy way for me to get it from some pipeline (like https://github.com/osbuild/osbuild/pull/1594)? (I’ve tried downloading the cache2.qcow2.zst that you linked, but the download is very slow for me (~30 kB/s), so would take more than a day to download.)
Some more context here about what OSBuild is doing that might be unique here.
OSbuild builds things in stages. We have stages that do one or two things, but the entire output from that stage gets saved and copied into the next stages for the next work item.
A simplified version of what is going on here is:
- pipeline:
tree- create FS tree with things like /etc/passwd, /usr/bin/bash, etc
- pipeline:
raw-image- create a raw disk image, partition, mkfs on partitions, etc
- copy the FS tree into a raw disk image
- run bootloader install
- pipeline:
raw-qemu-image- update kernel arguments in bls config entry file of /boot filesystem on raw disk image
- pipeline:
qemu- convert the raw disk image from the previous stage into
qcow2
- convert the raw disk image from the previous stage into
So what's happening here is there is a disk.img from the raw-image stage that gets copied into the raw-qemu-image stage and then modified slightly which then gets qemu-img converted in the qemu stage.
So there is an original disk.img from the raw-image pipeline that propably shares reflinks with the disk.img from the raw-qemu-image pipeline that then shares reflinks with the qemu.qcow2 from the qemu pipeline.
This series of steps may be what is contributing to the problem here.
Also note this isn't 100% reproducible. Sometimes I can reproduce it every time and other times only 10% or so.
More data:
I patched OSBuild to not cp using --refink=auto when it copies the first disk.img to the second disk.img in the workflow described above:
diff --git a/stages/org.osbuild.copy b/stages/org.osbuild.copy
index 5bc9067f..a7570ee7 100755
--- a/stages/org.osbuild.copy
+++ b/stages/org.osbuild.copy
@@ -157,7 +157,7 @@ def main(args, options):
print(f"copying '{src}' -> '{dst}'")
- cmd = ["cp", "-a", "--reflink=auto"]
+ cmd = ["cp", "-a", "--reflink=never"]
if remove_destination:
cmd.append("--remove-destination")
subprocess.run(cmd + [src, dst], check=True)
I ran the loop 30 times and didn't see any failures.
So far, I haven’t found any reproducer on my system (and I’d like to have some simple local reproducer :)).
What I’ve tried that didn’t work (50 iterations for most items, 1000 for the last):
- Copy /osbuild/store/objects/0a4be93ab2bb5c17851a57574f3af0cffe135c58ccfc840e4f0ea37dda36f77c/data/tree/disk.img from cache2.qcow2 onto a local disk (btrfs on luks), and run qemu-img convert and qemu-img compare there
- Mount cache2.qcow2 on the host (via qemu-storage-daemon’s FUSE export), and convert disk.img from there to a qcow2 file on the host filesystem
- Mount cache2.qcow2 on the host, and convert disk.img from there to a qcow2 file on the mounted filesystem (i.e. just work inside of cache2.qcow2)
- Attach cache2.qcow2 to a VM (via virtio-blk), and convert disk.img inside the VM to a qcow2 file on the same filesystem
However, I haven’t scrubbed the image between runs (i.e. basically rm cache2.qcow2 && zstd -d cache2.qcow2.zst), that’s one thing I still ought to try (but will of course take much longer).
However, I haven’t scrubbed the image between runs (i.e. basically rm cache2.qcow2 && zstd -d cache2.qcow2.zst), that’s one thing I still ought to try (but will of course take much longer).
I’ve done that now for the VM case (doesn’t take that long with hot-plugging and -unplugging), and still couldn’t reproduce it (50 runs).
OK I might have a more minimal reproducer that doesn't include OSBuild.
I took a look at exactly what the OSbuild stages were doing when copying things around and wrote a script to try to simulate it more closely. The summary is that we are now:
- copying the first
disk.imginto a newdisk.imgwith--reflink=always - truncating it to a new size to make it a larger disk
- mounting one of the partitions and editing a file
- unmounting the partition
- then running
qemu-img convertto generate theqcow2
Using the filesystem I uploaded before, mount it and cd into the root of that filesystem and then run this script:
[builder@2f7c40c67cf8 srv]$ cat repro.sh
#/usr/bin/bash
set -eux -o pipefail
[ ! -e /dev/loop-control ] && mknod /dev/loop-control c 10 237
mkdir -p mnt
count=1
while true; do
echo "COUNT is $count"; count=$((count+1))
cp --remove-destination --reflink=always \
osbuild/store/objects/06ead42642e12a8f1ab109bf29935661ed4060a0bc59accbae7549a2f0cd900f/data/tree/disk.img ./
truncate --size=$((10 * 1024 * 1024 * 1024)) disk.img
loopdev=$(losetup -f --show --partscan disk.img)
sleep 1
mount ${loopdev}p3 ./mnt/
sed -i -E 's/^options (.*)$/options \1 ignition.platform.id=qemu console=ttyS0 console=ttyS0,115200n8 ignition.platform.id=qemu/' \
./mnt/loader/entries/*.conf
umount ./mnt
qemu-img convert -O qcow2 -o compat=1.1 disk.img qemu.qcow2
qemu-img compare -f raw -F qcow2 disk.img qemu.qcow2
done
bash-5.2# bash /srv/repro.sh
+ '[' '!' -e /dev/loop-control ']'
+ mkdir -p mnt
+ count=1
+ true
+ echo 'COUNT is 1'
COUNT is 1
+ count=2
+ cp --remove-destination --reflink=always osbuild/store/objects/06ead42642e12a8f1ab109bf29935661ed4060a0bc59accbae7549a2f0cd900f/data/tree/disk.img ./
+ truncate --size=10737418240 disk.img
++ losetup -f --show --partscan disk.img
[ 234.459387] loop0: detected capacity change from 0 to 20971520
[ 234.461171] GPT:Primary header thinks Alt. header is not at the end of the disk.
[ 234.462675] GPT:5142527 != 20971519
[ 234.463235] GPT:Alternate GPT header not at the end of the disk.
[ 234.463979] GPT:5142527 != 20971519
[ 234.464393] GPT: Use GNU Parted to correct GPT errors.
[ 234.465016] loop0: p1 p2 p3 p4
+ loopdev=/dev/loop0
+ sleep 1
+ mount /dev/loop0p3 ./mnt/
[ 235.488936] EXT4-fs (loop0p3): mounted filesystem 96d15588-3596-4b3c-adca-a2ff7279ea63 r/w with ordered data mode. Quota mode: none.
+ sed -i -E 's/^options (.*)$/options \1 ignition.platform.id=qemu console=ttyS0 console=ttyS0,115200n8 ignition.platform.id=qemu/' ./mnt/loader/entries/ostree-1-fedora-coreos.conf
+ umount ./mnt
[ 235.515810] EXT4-fs (loop0p3): unmounting filesystem 96d15588-3596-4b3c-adca-a2ff7279ea63.
+ qemu-img convert -O qcow2 -o compat=1.1 disk.img qemu.qcow2
+ qemu-img compare -f raw -F qcow2 disk.img qemu.qcow2
Content mismatch at offset 403718656!
Nice job on the reproducer! If I omit the step where it modifies the copied disk.img via sed, it passes. If I md5sum the copied disk.img before mounting it, it passes.
Not sure what to make of this yet.
Thanks, that works indeed!
With debugging information put into qemu-img, I can see that it believes the offset is zero, whereas it is not zero when you actually inspect it. qemu-img gets this information via SEEK_HOLE/SEEK_DATA.
There seems to be some inconsistency in this hole information. Fully reading disk.img seems to update this information, so putting a cat disk.img > /dev/null before the qemu-img convert has the test pass.
Consequently, neither qcow2 nor qemu-img nor what target filesystem you use seem to be of importance, but beware that if you use qemu-img compare, it will again check hole information, so the following passes for me:
qemu-img convert -f raw -O raw disk.img /tmp/qemu.img
qemu-img compare -f raw -F raw disk.img /tmp/qemu.img
(Side note, I actually have no idea why it does report a mismatch when using qcow2. The hole information in disk.img should still be wrong, making qemu-img compare skip the same areas as convert did. Perhaps the qcow2 clustering changes something about the compare granularity.)
So this says both images are identical, but if you use diff -q instead of qemu-img compare (or compare a hexdump), you will indeed see that they differ, because diff actually reads the whole image. Same if you put a cat disk.img > /dev/null in between qemu-img convert and qemu-img compare (because this seems to update the hole information).
So replacing qemu-img convert+qemu-img compare by the following reproduces the problem completely without qemu or qcow2:
cp --sparse=always disk.img /tmp/qemu.img
diff -q disk.img /tmp/qemu.img
(Again, note that using qemu-img compare instead of diff would report for both images to be identical because it checks hole information and skips everything that’s reportedly zero. But if you run it after something has read the whole image (e.g. after diff, which does that), it reports a difference.)
A lot to unpack there. Thanks @XanClic! I'm not sure quite what to make of it all, but there is one thing you mention in there that seems interesting:
Consequently, neither qcow2 nor qemu-img nor what target filesystem you use seem to be of importance
I'd argue that the target filesystem is of importance here. We switched from xfs to ext4 to get unblocked on this and that is working fine in our pipeline. I've also been running builds on Fedora Cloud VMs (BTRFS, which also supports reflinks) and haven't seen any issues there.
I'm not saying it is 100% filesystem related, but there is at least some data to point towards it being related.
Oh, absolutely, the (source) filesystem does matter. I just meant the target filesystem of the copy/convert operation (for qemu.qcow2), which I changed to tmpfs, and still saw the problem.
I do believe that the (seemingly) incorrect hole information for disk.img is XFS-related.
Patch from dchinner:
https://lore.kernel.org/linux-xfs/[email protected]/T/#u
Still trying to craft a simple reproducer though.
Landed upstream in https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4b2f459d86252619448455013f581836c8b1b7da
Hasn't been backported to any stable branches yet.
This was backported to Fedora 6.7 series in https://gitlab.com/cki-project/kernel-ark/-/commit/c0412c5250f7bdb2ea61b61f27eada0d8a135692 and should land in F39 with kernel-6.7.11-200 (https://bodhi.fedoraproject.org/updates/FEDORA-2024-2fcce4ffb7)