sheepdog

sheepdog copied to clipboard

after an OS crash some of the files in the cache were truncated. sheep was crashing when these were being accessed and the log messages were less than informative.

I'd like to know

• how serious is this?
• what was likely to be lost (i.e. are these files referring to blocks that were being written and not yet confirmed to the qemu/guest OS as written or were some blocks that were supposed to be on disk lost?
• does the cache code use O_DIRECT and could there be a problem related to the meta-data not being written while the data was?
• can you add code to check for this and repair if possible
• can you add a utility to check for consistency (something like fsck)
• would using a journal prevent this?

In my specific case I doubt I would have lost any actual data, the last few hundred megabytes of modified data was copied and all seems ok, but I worry that this may happen again with more damage.

Versions: Sheepdog daemon version 0.7.5 Linux compucom 3.16.0-53-generic #72~14.04.1-Ubuntu SMP Fri Nov 6 18:17:23 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.19), Copyright (c) 2003-2008 Fabrice Bellard

sheep started as: /usr/sbin/sheep -c local /mnt/sheep/0/ -z 0 -p 7000 -r 127.0.0.1:4000 -w size=4096 dir=/mnt/sheep/cache/

the last part of the log:

Nov 12 12:32:26 INFO [main] md_add_disk(141) /mnt/sheep/0/obj, nr 1 Nov 12 12:32:27 INFO [main] send_join_request(770) IPv4 ip:127.0.0.1 port:7000 Nov 12 12:32:27 ERROR [main] for_each_object_in_stale(383) /mnt/sheep/0/obj/.stale Nov 12 12:32:28 NOTICE [main] http_init(451) http service is not complied Nov 12 12:32:28 ERROR [main] check_host_env(461) WARN: Allowed open files 1024 too small, suggested 1024000 Nov 12 12:32:28 INFO [main] main(853) sheepdog daemon (version 0.7.5) started Nov 12 12:32:28 INFO [main] recover_object_main(624) object c1968a000007d3 is recovered (1/9182) Nov 12 12:32:28 INFO [main] recover_object_main(624) object c1968c00000520 is recovered (2/9182) Nov 12 12:32:28 INFO [main] recover_object_main(624) object 2de2f7000001ee is recovered (3/9182) ...... more of the same ..... Nov 12 12:32:29 INFO [main] recover_object_main(624) object 2de2f7000001bc is recovered (9180/9182) Nov 12 12:32:29 INFO [main] recover_object_main(624) object c1968c000009ac is recovered (9181/9182) Nov 12 12:32:29 INFO [main] recover_object_main(624) object 128a750000011b is recovered (9182/9182) Nov 12 12:33:02 ERROR [oc_push 2661] read_cache_object_noupdate(362) size 0, count:4194304, offset 0 Success Nov 12 12:33:02 EMERG [oc_push 2661] do_push_object(901) PANIC: push failed but should never fail Nov 12 12:33:02 EMERG [oc_push 2661] crash_handler(250) sheep exits unexpectedly (Aborted). Nov 12 12:33:02 EMERG [oc_push 2661] sd_backtrace(857) /usr/sbin/sheep() [0x405bd7] Nov 12 12:33:02 EMERG [oc_push 2661] sd_backtrace(857) /lib/x86_64-linux-gnu/libpthread.so.0(+0x1033f) [0x7f5df413633f] Nov 12 12:33:02 EMERG [oc_push 2661] sd_backtrace(857) /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7f5df346ecc8] Nov 12 12:33:02 EMERG [oc_push 2661] sd_backtrace(857) /lib/x86_64-linux-gnu/libc.so.6(abort+0x147) [0x7f5df34720d7] Nov 12 12:33:02 EMERG [oc_push 2661] sd_backtrace(857) /usr/sbin/sheep() [0x415c5e] Nov 12 12:33:02 EMERG [oc_push 2661] sd_backtrace(857) /usr/sbin/sheep() [0x428f49] Nov 12 12:33:02 EMERG [oc_push 2661] sd_backtrace(857) /lib/x86_64-linux-gnu/libpthread.so.0(+0x8181) [0x7f5df412e181] Nov 12 12:33:02 EMERG [oc_push 2661] sd_backtrace(857) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6c) [0x7f5df353247c] Nov 12 12:33:04 INFO [oc_push 2661] dump_stack_frames(796) Cannot get info from GDB Nov 12 12:33:04 INFO [oc_push 2661] dump_stack_frames(798) Set /proc/sys/kernel/yama/ptrace_scope to zero if you are using Ubuntu. Nov 12 12:33:04 EMERG [oc_push 2661] __sd_dump_variable(731) dump __sys Nov 12 12:33:07 ERROR [main] crash_handler(490) sheep pid 2648 exited unexpectedly.

Hi @hmex , sorry for my late reply.

Please don't use object cache and journaling. They are broken and don't improve performance.

For consistency checking, dog vdi check can be used.

Could you use v0.9.3? v0.7.5 is quite old.

I compiled 0.9.3_7_g796c88d, but .... (see next bug report)

However relating to this bug report, I am seriously concerned that you tell me that both cache and journal are unsafe "features", yet, both are enabled and the man page doesn't say "IF YOU USE THIS YOUR DATA WILL END UP IN THE NEAREST BLACK HOLE". .... two concerns here, one, I was under the impression that sheepdog is production quality, meaning all features are stable and have a known reliability, even if it's not 100% (and if the cache ends up with zero length files it might warn? recover? explain?), second, if there are two features that seem OK for use, how many more features will end up losing me data, how many times do I need to get burnt before I figure out the right combination of options and features for a stable system.

Anyway, I hope you don't mind the constructive criticism. .... For myself I've stopped using cache and forgot about journalling, but for other potential users, I'd recommend indicating the problems clearly in the manual, or better yet, either making sure it works or deleting it. A cache that drops data is worse than no cache at all. And if you're right about it not improving performance, then the option should be deleted or made into a no-op (be ignored).

@mitake may be disable danger options for sheep binary? In this case we don't need delete code, but users can't break their data 25 нояб. 2015 г. 21:24 пользователь "hmex" [email protected] написал:

I compiled 0.9.3_7_g796c88d, but .... (see next bug report)

However relating to this bug report, I am seriously concerned that you tell me that both cache and journal are unsafe "features", yet, both are enabled and the man page doesn't say "IF YOU USE THIS YOUR DATA WILL END UP IN THE NEAREST BLACK HOLE". .... two concerns here, one, I was under the impression that sheepdog is production quality, meaning all features are stable and have a known reliability, even if it's not 100% (and if the cache ends up with zero length files it might warn? recover? explain?), second, if there are two features that seem OK for use, how many more features will end up losing me data, how many times do I need to get burnt before I figure out the right combination of options and features for a stable system.

Anyway, I hope you don't mind the constructive criticism. .... For myself I've stopped using cache and forgot about journalling, but for other potential users, I'd recommend indicating the problems clearly in the manual, or better yet, either making sure it works or deleting it. A cache that drops data is worse than no cache at all. And if you're right about it not improving performance, then the option should be deleted or made into a no-op (be ignored).

— Reply to this email directly or view it on GitHub https://github.com/sheepdog/sheepdog/issues/195#issuecomment-159693819.

@hmex sorry for inconvenience. I agree with deleting the features. Especially object cache is harmful because of not only its buggy implementation but also its broken design. The reason why the design is broken is described in this post: http://lists.wpkg.org/pipermail/sheepdog-users/2015-May/015023.html

I'm just keeping the object cache because some people at NIMBOXX (e.g. @imrook ) is working on improvements. However, recent workqueue parallerization by @zhanghongzhou and me improved write performance significantly (thanks for testing, @pttpzp !). You can find the changes in this PR: https://github.com/sheepdog/sheepdog/pull/197 If the improvements achieved by #197 is enough, there's no reason to refine object cache. How do you think, @imrook ? If you need it be kept, I'll just keep. But at least it should be locked as @vtolstov says.

@tmenjo could you work on deleting object cache?

@mitake ok, i'll try it.