ltopers
ltopers copied to clipboard
amiaopensource
Wishlist: `migratelto` script
Not entirely sure what this would entail yet. Would possibly be new script for reading ltos, recording checksums and creating log, process log that records when lto fails to read/write. Could have component that checks packet contents, records what is missing.
In 2014 we migrated our internal 5.7 PB archive from LTO-4 to LTO-6. En passant, we also transcoded all the Y′CBCR 4:2:2 files into Matroska/FFV1. Therefore this has been a more complex process as the one you propose. But at the beginning of next year, we’ll migrate LTO-5 to LTO-7 for a client. That could give the possibility to develop a more generic solution here. Thoughts?
I'm interested and curious about what the process would look like. What information would a generic use case want to record after reading materials off a tape? Would we be operating under the assumption that users would move files from tape to a staging external hard drive/volume before writing to LTO 7? I think this is possible but I'm curious what a "generic" use case would look like.
Is there an extant script that will compare checksums on an existing tape with the original checksums when the tape was created?
As said on Twitter (in discussions around the IASA conference), my company uses a script that reads and pipes directly to md5/sha1/sha256 checking, without local recording on SSD or HDD. That could also be an idea for the next AMIA hack day. I'm currently delving deeper into ltopers and mmfunctions, in order to know how much work the integration here would need.
At CUNY we use collectionchecksum to make a manifest of checksums for files in a directory to be written to tape, but that process depends on a local packaging structure. Then the -v option in writelto grabs a checksum list from the tape by reading it back. So we diff the output of collectionchecksum with the output of writelto -v. But this requires some local expectations and needs to be more generic.
There is a LTFS copy utility, the ltfscopy command, that provides binary copying from LTO to disk or from LTO to LTO with hash values verification. This may be useful for a migration tool.
@retokromer I haven't checked ltfscopy yet. I'll run some tests using it. Thanks!
We use it, and are happy with. (Perhaps I can contribute some of our ideas to your script.)
Sorry, @CSchloss385, what is the exact purpose of your migratelto script? I suspect our goals are a little different. Personally, I was thinking about migration from one LTO generation to another, let’s say from LTO-5 to LTO-7. This should allow to work both ways:
- manually: the computer tells to the operator when which cartridge is to put in or out on which tape desk
- automated: at least all the needed LTO-5 cartridges and one LTO-7 cartridge are in a tape library [therefore I was working in generalising the barcodes used]
My intend is to avoid to store on HDD or SSD all the data between the LTO-5 and the LTO-7. To pipe the reading from LTO-5 [to checking,] to writing onto LTO-7 [and to checking]. This means at least two desks have to be managed in parallel. Perhaps more, different scripts would be an appropriate way to go. Thoughts, @dericed and @dinahhandel?
I’m also thinking about a solution working on macOS, Linux and Linux on Windows.
@retokromer we are using the migratelto script to transfer one LTO-5 tape to a 6TB hard drive. This is part of our migration from LTO-5 to LTO-7 but the process is LTO 5 to external hard drive and then the content from the drive to LTO-7.
@CSchloss385 Thank you very much for this clarification! Then I suggest that we prepare (at least) two different migration scripts, based on different workflows. I guess, (a variant of) your script could also be used as a readlto, the reverse operation of writelto?
Changed the title, because we – it’s actually mainly @CSchloss385’s œuvre! – are working on that script.
@CSchloss385 Caveat: I guess the ltfscopy function I mentioned is specific to the LTFS implementation by HP we use, and does not exist in the LTFS implementation by Quantum used for LTOpers.
@CSchloss385 and let’s have a new migratelto which combines readlto and writelto for migration!
If there is no opposition, tomorrow I’ll:
- rename the current
migrateltotoreadlto - update the pull request https://github.com/amiaopensource/homebrew-amiaos/pulls by @NMAAHC
- create a release
cc @dericed @CSchloss385