reallymine
reallymine copied to clipboard
andlabs
Data Sectors for Symwave
Per contribution guidelines, submitting reference data files for Symwave-based drives. These are binary images of the key sector from my personal drive, along with the first two leading sectors. Two variants of each are included in the zip file attached:
keyblk-straight.img is the raw key sector, as extracted directly from the drive. keyblk-reversed.img is a "corrected" version of the key sector. See below. header-encrypted.img contains the drive's first two sectors, as read directly from the drive. header-decrypted.img contains the decrypted form of the same two sectors.
I was fortunate enough to be able to get a decrypted version of the MBR and several of the following sectors while the bridge was still at least somewhat functional. This should, I hope, prove useful for comparison. I also noted that the raw key sector, extracted directly from my drive, did NOT quite match the form of the version recently published. Specifically, whereas the published sector begins with the letters, 'SYMW', I found them in reverse order in mine: 'WMYS'. Suspecting a big/little-endian issue, I played a hunch, and came up with a version that flips around each set of four characters. Rather than force you to do the same (simple as that might be), I went ahead and saved a "corrected" version of the key sector, along with the original.
I believe this reversal is a correct operation, based on results obtained in my own tests. In the paper recently published, the Symwave analysis discusses the presence of two data encryption keys (DEKs), the second of which 'unwraps' to a string of zero bytes. I believe this is by design, perhaps a form of error check. Testing against my own corrected key sector obtained the same result: the second DEK unwrapped to a string of nulls. That this should come by chance seems to me highly unlikely, enough to suggest that the process I've employed is almost certainly correct.
That said, I believe there must be at least one more stage in the decryption process. While I have not yet attempted to decrypt or encrypt either leading sectors in their entirety, I HAVE tested the first DEK against a 16-byte block of nulls. You may note the presence of several such blocks in both forms of my leading sectors, in fact, this is why I included the sector immediately following the MBR. Its raw form on disk is completely zeroed, which the bridge "decrypts" into a block of apparent gibberish, albeit patterned gibberish. Likewise, several blocks of zeroes are found in the decrypted MBR, which may be readily mapped to similar patterns in the encrypted form. What I've found, however, is that neither of these patterns correspond to either an encryption or decryption of a block of zeroes using my unwrapped DEK. I'm certain there has to be another step involved to obtain the correct key.
With that in mind, I would like to call your attention to another part of the key sector. Following the wrapped keys, the sector is almost entirely zeroes, however, there is one more strip of data near the end. It starts around byte 448 (0x1c0), if you go with the reversed sector. It's not very long, but I suspect the data there indicates some sort of additional processing required to obtain the final key. Maybe you'll see something in the firmware. Here's hoping, at least.
Thanks a lot! I'll go through this later and finish up Symwave.go, and the final version will also do the endianness reversal for you. (And yes, I'm not too surprised that the bytes are flipped as the 68000 is a big-endian CPU, though it does make me wonder if the encryption is done in software...)
Excellent, and my pleasure! Best of luck, and let me know if you have any questions. Not sure I'll have an answer, but I'll do my best!
On Sun, Jan 10, 2016 at 9:18 PM, Pietro Gagliardi [email protected] wrote:
Thanks a lot! I'll go through this later and finish up Symwave.go, and the final version will also do the endianness reversal for you. (And yes, I'm not too surprised that the bytes are flipped as the 68000 is a big-endian CPU, though it does make me wonder if the encryption is done in software...)
— Reply to this email directly or view it on GitHub https://github.com/andlabs/reallymine/issues/1#issuecomment-170422589.
Sorry for the delay. Does the current master decrypt your drive successfully? (You'll need to build it manually; if you used go get to get reallymine you'll need to get it again since there's now an external dependency for the key-wrapping algorithm.)
I might need a few days to check it out (things are kind of busy, here, right now), but I'll let you know when I get some results.
Thank you!
Sent from TypeApp
On Mar 30, 2016, 12:51, at 12:51, Pietro Gagliardi [email protected] wrote:
Sorry for the delay. Does the current master decrypt your drive successfully? (You'll need to build it manually; if you used go get to get reallymine you'll need to get it again since there's now an external dependency.)
You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/andlabs/reallymine/issues/1#issuecomment-203550197
All right, here's the latest: I have not yet attempted to build the master, but rather updated my own little tool with the new information from your latest update. Namely, after extracting DEK1 from the hidden sector, reverse every 4-byte sequence to obtain the final key. Once that was obtained, I was finally able to decrypt the entire drive successfully.
The reason I haven't attempted to build the master yet is that I can't really use it as intended at this point. As I understand it, this tool is designed primarily as a filter, decrypting data from the input stream after extracting its key, and dumping the 'plaintext' to output. To do that, I would need a second 2TB drive to receive the data, which isn't the most viable option for me, right now.
What I was able to do instead, once I had the key, was to mount the encrypted filesystem read-only, using cryptmount. I'll supply details in a follow-up. This permitted me to copy and verify the contents to a smaller drive, without having to decrypt an entire 1.8TB partition to get it first. Again, details will be forthcoming.
If I might make a suggestion, I think a good idea would be to break the key extractor(s) out into a separate tool, one the filter could call to prepare for normal decryption, but also available to the user if all he/she needs is the key. I also see a possibility here to generate ready-made cmtab entries for cryptmount, which could simplify things immensely in cases like mine.
Now I've successfully recovered my data, I'll go ahead and try building the tool, and let you know how that goes. I might also put in some work on my ideas above, time permitting.
You guys are pure genius, keep up the great work On Apr 4, 2016 11:51 AM, "athomic1" [email protected] wrote:
All right, here's the latest: I have not yet attempted to build the master, but rather updated my own little tool with the new information from your latest update. Namely, after extracting DEK1 from the hidden sector, reverse every 4-byte sequence to obtain the final key. Once that was obtained, I was finally able to decrypt the entire drive successfully.
The reason I haven't attempted to build the master yet is that I can't really use it as intended at this point. As I understand it, this tool is designed primarily as a filter, decrypting data from the input stream after extracting its key, and dumping the 'plaintext' to output. To do that, I would need a second 2TB drive to receive the data, which isn't the most viable option for me, right now.
What I was able to do instead, once I had the key, was to mount the encrypted filesystem read-only, using cryptmount. I'll supply details in a follow-up. This permitted me to copy and verify the contents to a smaller drive, without having to decrypt an entire 1.8TB partition to get it first. Again, details will be forthcoming.
If I might make a suggestion, I think a good idea would be to break the key extractor(s) out into a separate tool, one the filter could call to prepare for normal decryption, but also available to the user if all he/she needs is the key. I also see a possibility here to generate ready-made cmtab entries for cryptmount, which could simplify things immensely in cases like mine.
Now I've successfully recovered my data, I'll go ahead and try building the tool, and let you know how that goes. I might also put in some work on my ideas above, time permitting.
— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/andlabs/reallymine/issues/1#issuecomment-205389767
Is cryptmount something that live-decrypts the drive as the kernel reads it? I originally wanted to try something like that via NBD, but couldn't wrap my head around the NBD spec :/ I'd love to hear the details of what you did, though! And yeah, I can totally make it spit out the decryption key as well, though it would also have to tell you what you would need to do to use that key successfully (swap each group of 4 bytes before and after decrypting, for instance).
For what it's worth I did indeed decrypt my own drives straight from the drive, but into partition images on another drive. My main concern there is that I'm afraid that I'd be wearing the drive out if I encouraged doing it. I could perhaps say that doing this does work, but hasn't been tested enough? Implementing something like ddrescue (or even implementing this whole thing inside ddrescue itself, though it would require logfile changes) would also work.
Whatever the case, glad to see that you got your data out! That only leaves the PLX/Oxford Semiconductor drives (assuming something that doesn't use AES-256-ECB doesn't pop up, like the paper implies might...).
Finally, Real Results!
Having finally built and tested the reallymine application, I can now confirm that it successfully located and extracted my encrypted drive's Symwave sector and key, and decrypted about 350MB of data before I killed the process. I'm certain, had I permitted it to continue, it would have successfully decrypted the entire drive, except for that nasty out-of-space error looming in its all-too-near future. I did verify correctness of data actually recovered.
Looking good!
I would like to note that against the doc's recommendation, I took input directly from the device, having no place to store a 1.8TB image. Thankfully, restriction against this is not yet enforced. I would suggest a switch to override it at the user's discretion, with appropriate warnings, etc, etc. For some people (like me), there just might not be any other option.
Details on cryptmount are still forthcoming. I will try to have that up later today. I also want to put up some information about known locations of key sectors, which I found on a forum elsewhere. With this information, the key search might be sped up in at least some cases. The application could look first in known locations before resorting to an exhaustive "brute force" scan. Other than that, I think we've got this issue nailed. You can probably go ahead and close it, if you want.
Thanks again for all your work on this!
I'd like to get rid of that restriction now that I know with my own drives it does work. This is why I was wondering if I should just give a warning that this program does not take precautions to prevent wear like ddrescue does. Yes, requiring an image is a really dumb idea, especially since 4TB is still the largest drive size that's commonly available =P
For what it's worth, I was planning on tidying up the UI on this thing and doing a quick test of the keys before continuing by trying to decrypt various bootsectors and seeing if the partition tables made any sense, then asking the user if that's what they know their drive is. Of course, thinking about it now I'm not sure what point that would serve either...
Let's keep this open; this discussion is very interesting and helpful :)
Narrowing the Search
I debated opening up another issue for this and the cryptmount information, but for now, I think I'll just include it here. I'm sure we can open it later, if appropriate.
The following is a list of known or suspected "magic sector" locations, arranged by bridge type and capacity. Much of this information comes from this forum post: http://www.hddoracle.com/viewtopic.php?f=3&t=1166#p6163
Initio 500GB 976769032 0x03A385008 750GB 1465143304 0x057545008 1000GB 1953519624 0x074705808
Symwave 500GB 976770435 0x03A385583 750GB 1465144707 0x057545583 1000GB 1953521027 0x074705D83 2000GB 3907026307 0x0E8E07D83
Jmicron 500GB 976769056 0x03A385020 750GB 1465143328 0x057545020 1000GB 1953519648 0x074705820 2000GB* 3907024928 0x0E8E07820 3000GB 5860528160 0x15D509020
Here's a text file that might be slightly easier to use: key-sectors-v2.txt
Information I added includes the column of hex values,and sector locations for 2TB drives. The 2GB Symwave is my own, discovered through a little trial-and-error, with a lot of help from the original list. The 2GB Jmicron is thoroughly speculative, hence the asterisk. Assuming such a unit actually exists "in the wild," I'd love to hear from anyone who can verify or disprove it.
A clear pattern seems to emerge when the addresses are shown in hexadecimal, however, it starts to break apart at higher values. The lowest byte of the address looks rock solid for each bridge type. The next one up is also consistent through the lower addresses, but its high bit seems to be set for those at 1TB and above. That is, until one hits 3TB, if that last one on the Jmicron list is to be believed. I'm pretty sure there's some sort of logic, here, but I have yet to puzzle it out completely.
In any case, I believe there's enough information here to narrow the search for keys to just a few areas of the drive or image. Maybe even just one, if we assume the likely address to be the highest valid one, or maybe it'll give you enough to find the logic in the firmware?
A Report on Mounting Pre-encrypted Volumes
All right, here's the one you've been waiting for!
First, a little bit of backstory: Once I'd confirmed the key against the master boot record, I was able to extract the first of two partitions through a little filter I built with it in Python. At only about 100GB, it wasn't too hard to find a place to put it.
The challenge came with the second partition. I knew I wouldn't be able to extract a full 1.8TB, but with a spare 500GB drive, I thought I could still get the data, since I hadn't stored much more than about 300GB. Turns out XFS isn't so forgiving about this, having superblocks spaced out through the whole address space. Without the whole thing, it didn't look like I was going to get anywhere.
Time to look at mounting encrypted.
Turns out we have a few options, some better developed than others. Unfortunately, most of the instructions online relate to CREATING them, with little to no help on mounting one already made. At least, not without a known password. I looked at loop-aes, losetup, and a couple of others, but they all seemed to assume a key either generated through a password (good luck finding that!), or encrypted with GPG. Neither seemed to clearly address my situation. What I really needed was something that would let me supply the raw key in a clear and concise manner. I finally found my answer in cryptmount.
According to the documentation, "cryptmount allows a user to mount an encrypted filing system without requiring superuser privileges, and assists the superuser in creating new encrypted filesystems." Aside from being perhaps more user-friendly, this is essentially no different from what other packages offer, however, if offers one feature no others clearly do: a raw key option.
To use cryptmount, you'll need to set up /etc/cryptmount/cmtab to give the target drive a name and make it accessible. This normally has to be done as root, for security reasons. See the man page links below for more detailed information, but here is the configuration (with some obfuscations) that worked for me:
mybook {
dev=/dev/disk/by-id/orwhatever
startsector=#number-goes-here#
dir=/mnt
fstype=xfs
mountoptions=ro
cipher=aes-ecb
keyformat=raw
keyfile=/key/goes/here
}
The key settings here (no pun intended) are keyformat=raw, and cipher=aes-ecb. Unfortunately, there seems to be precious little information about cipher options, or how to find them. /proc/crypto is mentioned in a one or two places, but proved only barely informative. I went through several rounds of trial and error before finally hitting on a setting that worked. Hopefully, this will save others the same trouble.
Note that mountoptions replaces fsoptions, at least in my cryptmount package. Absolutely make sure ro (read-only) is set, unless you really want to keep using the encrypted filesystem as is.
The user is on his own determining startsector, sorry to say, but a partial extraction, like the first few hundred megabytes, should get enough information to locate at least the first one.
Everything else here should be self-explanatory. If not, just have a look at the man pages linked here:
http://linux.die.net/man/8/cryptmount http://linux.die.net/man/5/cmtab
Just note that some points may differ. Yes, fsoptions, I'm looking at you.
startsector shouldn't be too much of an issue since we can just read that from the partition table, whatever format it may be =P reallymine could possibly generate that whole configuration file. Thanks for all the information; I'll be sure to take it into consideration.
My pleasure, once again, and thanks for all your efforts in developing this. You've saved me a LOT of trouble and expense. Thanks again!
I had started a thread at hddoracle on reallymine a while back. I posted my experience With it On Apr 6, 2016 6:13 PM, "athomic1" [email protected] wrote:
My pleasure, once again, and thanks for all your efforts in developing this. You've saved me a LOT of trouble and expense. Thanks again!
— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/andlabs/reallymine/issues/1#issuecomment-206614909
I'd be interested in reading that thread, if it's still up. MrDecay, can you give us a link?
http://www.hddoracle.com/viewtopic.php?f=22&t=1488 On Apr 6, 2016 7:09 PM, "athomic1" [email protected] wrote:
I'd be interested in reading that thread, if it's still up. MrDecay, can you give us a link?
— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/andlabs/reallymine/issues/1#issuecomment-206627742
Hello guys. I am grateful to have found this project and appreciate all of the hard work you guys have been putting into it. I have a situation where I have 2 External WD driver with failing USB boards (they both failed at around the same time). The first one to fail is a WD MyBook 3TB drive. This one has some important memories. I have no backup because at the time it didn't matter to me much, however it has the last photos of a friend of mine who died suddenly at age 20 and now it would be a great thing to recover. She suffered from seizures and had one in her sleep and suffocated. I am sure her family would be grateful to see her last memories, at the time I thought nothing of the photos or backing them up since all way fine with her and it was kind of a freak situation. On this 3TB drive I threw away the USB board since after some googling it seemed like it would be no problem to open the enclosure and insert it into my desktop, and after I opened it and saw what it looked like it was obvious I could do that. Of course I was unaware of the hardware encryption, and didn't think of the possibility especially since I never had a password. The second drive is a WD Elements 1.5TB which also appears to be encrypted. For that one I kept the board and should be able to buy that specific revision although I don't believe that particular drive has much important data if any (these drives failed 4 years ago now). For the 3TB drive I even purchased another one with the same model number and hooked the old one up which I read others were successful with but I had no luck. I am guessing it has a different revision of the USB connector especially since they were purchased a good 5 years apart. I have tried data recovery software on both drives to no avail, I only find random files of varying sizes that obviously aren't the actual files I had on the drive. I found the paper about the security flaws in these drives especially with the default KEK keys and that gave me hope that I may still be able to recover the data without spending hundreds that I can't afford for data recovery specialists. I am a technology major but have no experience in drive partitions or data recovery and although I understood the concepts I had no idea how to use the default KEK and attempt to de-crypt the files although it did lead me here. I have tried the reallymine tool in Linux Mint (since I already have it ready in a dual-boot setup on my old laptop). I understand how to run it. On the 3TB Drive it does not work, and I'm guessing I have the wrong type of board (one of the ones not implemented yet) based on the error. I did see that the source code was updated recently with one more kind although I was unable to compile the source code mainly because I have no idea how or even what language .go files are :p On the 1.5TB drive reallymine hangs at "Finding key sector..." so far I have left it for 14 hours. The computer is fully responsive but it just hangs there. I have tried unplugging it and plugging it in again a few times but it always seems to hang. I was wondering if you guys could help me out in understanding what possible issues could be happening on the 1.5TB drive and/or give me some tips to compile the newer code. I know you need it tested and if it works for me that would be a good results to have I assume. If you are feeling really kind you could even make a new release but I am not against trying to learn if you don't feel like it is ready for a release yet! I appreciate any help you can give me, and am also willing to help in any way I can to benefit this great project.
@athomic1 out of curiosity, how will cryptmount specify the method needed to flip the bytes before and after decryption?
I don't recall needing to specify a method to do that. Actually, the byte-flipping was only needed to decrypt the key properly. Once you have that, just drop it raw into a file, and reference it per cmtab, as noted above. Note the option "keyformat=raw" indicates to use the key as is. You basically have to store it in native binary form (not a hex dump) in your key file. Check "man 5 cmtab" for more info.
Again, you only need to rearrange bytes to recover the decryption key. You DON'T need it to recover the data itself.
To emulate the JS chip, I wrote a crypt module that reverses each block of 16 bytes. So cryptmount needs to be used three times, to sandwich the AES decryption between two layers of reversal.
So, the JS chip reverses the byte order of the data blocks, as well as the encryption key? Yikes! I don't think the Symwave does that. Once I extracted the key, I was able just to mount it as described above, no layering needed. With that kind of overhead, no wonder these things fail!
binxycraft, can you dump the last 10 megabytes of the drive into a file? Then you could look for the key block in a hexeditor. The JMicro, it starts with WDv1. For Symwave, with SYMW or WMYS.
If you look at the Decrypt() method of each bridge chip source file in the source, you will see that each chip needs it, including Symwave, and each chip uses a different approach. I don't know about the PLX one, as I don't have data to test it with.
athomic1, for my WD 4TB with Jmicron chip, the key block is at 0x1D1C0A820, counting in blocks of 512 bytes.
Very interesting. Based on my data, that's a difference of 0x74701800 from the 3TB location. Between 1TB and 2TB on the Symwave, the difference is 0x74702000.
There seems to be SOME consistency of differences, indicating some logic determining the key sector location. For example, all Initio addresses end with 08, while Symwave's end with 83, and JMicron's with 20. The next higher byte value seems to be USUALLY the same, though bit 2 (hex value 08) or 5 (0x20) seems to vary: on for the bigger drives, off for smaller.
It just seems there should be some way to narrow down the search for the key sector, and reduce the search time immensely, but unless someone can poke at the firmware, I'm not sure how easily this can be done, if it can.
Thanks for the info, maddoc!
athomic1, BTW, I have been using my WD 4TB as an internal disk for about 12 months now. It has the Jmicro chip, and I use cryptsetup, although not with a cmtab like yours. More like just a command like this: echo 2850018b2aae7afbb7eb8b6d755f5cc60a241f03dda312e600eb76cc5560af3b | xxd -p -r | cryptsetup -c aes-ecb --key-size=256 -d - --hash=plain create wd /dev/sdc The xxd program comes with vim and converts between binary and hex. [Note: the jmicro chip also reverses each 16-byte block, so I have to use a filter for that, too]
Sounds good! Probably just as good your way as mine, especially for a one-shot deal, which is what I wanted to do. My plan, if I can ever get back to it, is to rewrite the DECRYPTED data back to the drive, and use it like any normal one, or do something like that. If I were going to continue working with the encrypted drive as is, I'd probably rely on a cmtab as before, but that's me. I'll definitely keep your method in mind, though! Might come in handy yet.
I have just now seen my first 2TB JMS MyBook. The keyblock was at 3907024928, so that number is now confirmed-ish.