bees

bees copied to clipboard

Please check #97 It's from the same run

19:49:38 gentoo beesd[29713]: crawl_258[29748]: Opening src bfr 4K [0xb002000..0xb003000] fid = 4480:685 fd = 94 '/run/bees/mnt/6a5e0f31-4857-4352-811a-c55dcdd44433/subvol_snapshots/1775/snapshot/data1'
19:49:38 gentoo beesd[29713]: crawl_258[29748]: chase_extent_ref ino BtrfsInodeOffsetRoot { .m_inum = 685, .m_offset = 0xb000000, .m_root = 4480 } bbd BeesBlockData { 4K 0x28801000 fd = 7 '/run/bees/mnt/6a5e0f31-4857-4352-811a-c55dcdd44433/data2', data[4096] }
19:49:38 gentoo beesd[29713]: crawl_258[29748]: ino_off_root BtrfsInodeOffsetRoot { .m_inum = 685, .m_offset = 0xb000000, .m_root = 4480 }
19:49:38 gentoo beesd[29713]: crawl_258[29748]: for_each_extent_ref BeesBlockData { 4K 0x28801000 fd = 7 '/run/bees/mnt/6a5e0f31-4857-4352-811a-c55dcdd44433/data2', data[4096] } at 0xe7579000z2000: 41 found
19:49:38 gentoo beesd[29713]: crawl_258[29748]: overlap_bfr 0 (..0x0]
19:49:38 gentoo beesd[29713]: crawl_258[29748]: Opening dst bfr 4K [0x28801000..0x28802000] fd = 7 '/run/bees/mnt/6a5e0f31-4857-4352-811a-c55dcdd44433/data2'
19:49:38 gentoo beesd[29713]: crawl_258[29748]: replace_dst dst_bfr 4K [0x28801000..0x28802000] fd = 7 '/run/bees/mnt/6a5e0f31-4857-4352-811a-c55dcdd44433/data2'
19:49:38 gentoo beesd[29713]: crawl_258[29748]: finding one match (out of 41) at 0xe7579000z2000 for BeesBlockData { 4K 0x28801000 fd = 7 '/run/bees/mnt/6a5e0f31-4857-4352-811a-c55dcdd44433/data2', address = 0x46a401000, hash = 0x6b035ccf87d5e020, data[4096] }
19:49:38 gentoo beesd[29713]: crawl_258[29748]: resolved_addrs.size() = 220
19:49:38 gentoo beesd[29713]: crawl_258[29748]: Pushing hash 0x6b035ccf87d5e020 addr 0x46a401000 bbd BeesBlockData { 4K 0x28801000 fd = 7 '/run/bees/mnt/6a5e0f31-4857-4352-811a-c55dcdd44433/data2', address = 0x46a401000, hash = 0x6b035ccf87d5e020, data[4096] }
19:49:38 gentoo beesd[29713]: crawl_258[29748]: scan bbd BeesBlockData { 4K 0x28801000 fd = 7 '/run/bees/mnt/6a5e0f31-4857-4352-811a-c55dcdd44433/data2', address = 0x46a401000, hash = 0x6b035ccf87d5e020, data[4096] }
19:49:38 gentoo beesd[29713]: crawl_258[29748]: Extent { begin = 0x28800000, end = 0x288111b5, physical = 0x46a400000, flags = FIEMAP_EXTENT_LAST, physical_len = 0x12000, logical_len = 0x12000 } block_count 18
19:49:38 gentoo beesd[29713]: crawl_258[29748]: scan extent Extent { begin = 0x28800000, end = 0x288111b5, physical = 0x46a400000, flags = FIEMAP_EXTENT_LAST, physical_len = 0x12000, logical_len = 0x12000 }
19:49:38 gentoo beesd[29713]: crawl_258[29748]:
19:49:38 gentoo beesd[29713]: crawl_258[29748]: *** EXCEPTION ***
19:49:38 gentoo beesd[29713]: crawl_258[29748]:         exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack
19:49:38 gentoo beesd[29713]: crawl_258[29748]: ***

I think this is another incarnation of #89 and can be closed in favor of the other issue.

Yeah this one is breaking out of a finite loop with billions of possible iterations. Like #99, we have to cross a dozen stack frames to kill a loop, so we use an exception. Long-term fix is a redesign of scan_one_extent.

Maybe in short-term, you could create a separate exception class, i.e. "ExitLoopException", catch that, and prevent the stack trace from being printed and just print an informational message received via the exception?

I agree with @kakra, when I see exceptions, I get nervous that my data would be lost.

So we should follow-up on this in #89 because this raises concerns of users.

I'm seeing the same error a lot too on a drive with many ro snapshots. Using bees from master branch from today.

2019-06-27 11:37:46 32228.32231<7> crawl_11471: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack
2019-06-27 11:37:46 32228.32232<6> crawl_11471: dedup: src 32K [0x32000..0x3a000] {0x3fe12023000z12000} /run/bees/mnt/f021a039-87d6-4498-a0f5-6bbba3dfb1f1/volumes/var_db/var_db.20190627T0000/smokeping/bahnhof/cloudflare.rrd
2019-06-27 11:37:46 32228.32232<6> crawl_11471:        dst 32K [0x1c0000..0x1c8000] {0x3fe1209c000z0} /run/bees/mnt/f021a039-87d6-4498-a0f5-6bbba3dfb1f1/volumes/var_db/var_db.20190519T0000/smokeping/bahnhof/cloudflare.rrd
2019-06-27 11:37:46 32228.32232<7> crawl_11471: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack
2019-06-27 11:37:46 32228.32231<6> crawl_11471: dedup: src 32K [0x32000..0x3a000] {0x3fe12023000z12000} /run/bees/mnt/f021a039-87d6-4498-a0f5-6bbba3dfb1f1/volumes/var_db/var_db.20190627T0000/smokeping/bahnhof/cloudflare.rrd
2019-06-27 11:37:46 32228.32231<6> crawl_11471:        dst 32K [0x1e0000..0x1e8000] {0x3fe1209d000z0} /run/bees/mnt/f021a039-87d6-4498-a0f5-6bbba3dfb1f1/volumes/var_db/var_db.20190519T0000/smokeping/bahnhof/cloudflare.rrd
2019-06-27 11:37:47 32228.32231<7> crawl_11471: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack
2019-06-27 11:37:47 32228.32232<6> crawl_11471: dedup: src 32K [0x32000..0x3a000] {0x3fe12023000z12000} /run/bees/mnt/f021a039-87d6-4498-a0f5-6bbba3dfb1f1/volumes/var_db/var_db.20190627T0000/smokeping/bahnhof/cloudflare.rrd
2019-06-27 11:37:47 32228.32232<6> crawl_11471:        dst 32K [0x200000..0x208000] {0x3fe1209e000z0} /run/bees/mnt/f021a039-87d6-4498-a0f5-6bbba3dfb1f1/volumes/var_db/var_db.20190519T0000/smokeping/bahnhof/cloudflare.rrd
2019-06-27 11:37:47 32228.32231<6> crawl_11471: dedup: src 12K [0x32000..0x35000] {0x3fe12023000z12000} /run/bees/mnt/f021a039-87d6-4498-a0f5-6bbba3dfb1f1/volumes/var_db/var_db.20190627T0000/smokeping/bahnhof/cloudflare.rrd
2019-06-27 11:37:47 32228.32231<6> crawl_11471:        dst 12K [0x27d000..0x280000] {0x3fe120bb000z1d000} /run/bees/mnt/f021a039-87d6-4498-a0f5-6bbba3dfb1f1/volumes/var_db/var_db.20190519T0000/smokeping/bahnhof/cloudflare.rrd
2019-06-27 11:37:47 32228.32231<7> crawl_11471: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack
2019-06-27 11:37:47 32228.32232<7> crawl_11471: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack
2019-06-27 11:37:47 32228.32231<6> crawl_11471: dedup: src 32K [0x32000..0x3a000] {0x3fe12023000z12000} /run/bees/mnt/f021a039-87d6-4498-a0f5-6bbba3dfb1f1/volumes/var_db/var_db.20190627T0000/smokeping/bahnhof/cloudflare.rrd
2019-06-27 11:37:47 32228.32231<6> crawl_11471:        dst 32K [0x280000..0x288000] {0x3fe120c8000z0} /run/bees/mnt/f021a039-87d6-4498-a0f5-6bbba3dfb1f1/volumes/var_db/var_db.20190519T0000/smokeping/bahnhof/cloudflare.rrd
2019-06-27 11:37:47 32228.32231<7> crawl_11471: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack

This will get triggered a lot when the data consists of adjacent repeating blocks. These are not good candidates for dedupe with the existing algorithms, and they're also hard to fix with the existing code.

I plan to replace the code with a better algorithm (i.e. one that doesn't have this corner case). When that code is ready, this FIXME (and all the code surrounding it) will be removed.

I too receive a lot of "bailing out here" exceptions:

2020-08-29 17:17:52 20424.20438<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20441<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20440<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20441<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20438<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20439<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20441<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20438<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20441<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20438<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20441<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20438<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20441<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20438<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20441<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20438<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20441<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20439<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20438<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20441<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20441<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20440<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20438<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack 2020-08-29 17:17:52 20424.20439<7> crawl_1455: exception (ignored): exception type std::runtime_error: FIXME: bailing out here, need to fix this further up the call stack

As dim-geo commented, I too get very nervous when exceptions arise in a program that in managing my data...

Maybe also look here: https://github.com/Zygo/bees/issues/149 - it's harmless.

Maybe also look here: #149 - it's harmless.

I've read it's harmless, nevertheless I am scared by exceptions :-D

I think output of Bees is too much complicated/technical and scaring for the average user.