Add redundancy to storage

[dswd]

If one or more bundles get lost, it is important to have some redundancy to reconstruct them. Redundancy can be added by taking N bundles, calculating their parity (XORing all their contents including headers) and storing this parity together with the bundles. If one of the bundles gets lost or corrupted, the parity can be used together with the remaining bundles to reconstruct the lost bundle.

Ok the plan is as follows:

• [ ] Add a format for parity files containing
  • A magic header to identify the file type
  • A header containing the bundle ids and sizes of the contained bundles
  • The raw parity data
• [ ] Add a configuration option that defines how many bundles are combined into a parity file
• [ ] Add a parity writer that creates parity files while writing bundles.
• [ ] Upload parity files to the same directory hierarchy as bundle files.
• [ ] Extend the bundle map to also include parity file information.
  • Add new parity files automatically
  • Fetch missing parity information automatically
• [ ] Try to repair broken bundles with the help of parity files.
• [ ] Add information to info commands to display parity size
• [ ] Modify/delete parity files on vacuum