snpr icon indicating copy to clipboard operation
snpr copied to clipboard
verified publisher icon openSNP

Properly extract files (FamilyTree especially)

Open philippbayer opened this issue 7 years ago • 4 comments

Right now the preparsing job uses the zip gem to unzip uploaded files, but uploaded files can come in a variety of flavours. FTDNA just updated their file format to be gz.

To do:

  • [ ] In the preparsing job, test more which file format the user has uploaded (zip, tar.gz, gz, flat text, rar?)
  • [ ] then use the right gem to unzip instead of going for default zip (probably need to add a few gems as it looks like there's no 'generic' ruby gem for all kinds of compressions)

philippbayer avatar Mar 27 '18 02:03 philippbayer

Parsing some files in the data dump was not a smooth experience. There are file with names such as user7170_file5562_yearofbirth_unknown_sex_unknown.23andme.txt but are zipped in reality. I feel like the code checks for that so I'm not sure why this is happened.

raivivek avatar Dec 03 '18 06:12 raivivek

Yep, this is a fix we really(tm) should get to. @philippbayer has already started working on this in #492 I think. @raivivek any chance you wanna pick up that PR and give it more testing etc? 😄

gedankenstuecke avatar Dec 03 '18 18:12 gedankenstuecke

Sure! I'll be happy to help iron it out.

raivivek avatar Dec 03 '18 23:12 raivivek

That would be great, thanks!

gedankenstuecke avatar Dec 03 '18 23:12 gedankenstuecke