rubydora icon indicating copy to clipboard operation
rubydora copied to clipboard

Assigning large files as datastream content fails

Open cbeer opened this issue 13 years ago • 1 comments

Large files (for some definition of large??) can't be read into Ruby, so they can't be added as object datastreams:

/Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/rubydora-0.5.7/lib/rubydora/datastream.rb:121:in `read': Invalid argument - HDS/barcode334930_1.mov (Errno::EINVAL)
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/rubydora-0.5.7/lib/rubydora/datastream.rb:121:in `content'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activemodel-3.2.2/lib/active_model/dirty.rb:143:in `attribute_change'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activemodel-3.2.2/lib/active_model/dirty.rb:117:in `block in changes'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activemodel-3.2.2/lib/active_model/dirty.rb:117:in `map'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activemodel-3.2.2/lib/active_model/dirty.rb:117:in `changes'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/rubydora-0.5.7/lib/rubydora/datastream.rb:245:in `to_api_params'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/rubydora-0.5.7/lib/rubydora/datastream.rb:210:in `block in create'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:403:in `_run__3645590890494944274__create__594327490570200131__callbacks'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:405:in `__run_callback'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:385:in `_run_create_callbacks'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/rubydora-0.5.7/lib/rubydora/datastream.rb:209:in `create'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/rubydora-0.5.7/lib/rubydora/datastream.rb:221:in `block in save'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:403:in `_run__3645590890494944274__save__594327490570200131__callbacks'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:405:in `__run_callback'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/rubydora-0.5.7/lib/rubydora/datastream.rb:220:in `save'

See also:

1.9.3p0 :009 > f = open('HDS/barcode334930_1.mov')
 => #<File:HDS/barcode334930_1.mov> 
1.9.3p0 :010 > f.size
 => 3799020967 
1.9.3p0 :011 > f.read; 1
Errno::EINVAL: Invalid argument - HDS/barcode334930_1.mov
    from (irb):11:in `read'
    from (irb):11
    from /Users/chris_beer/.rvm/rubies/ruby-1.9.3-p0/bin/irb:16:in `<main>'

=== vs ====

1.9.3p0 :013 > f = open('/Users/chris_beer/Desktop/tmp')
 => #<File:/Users/chris_beer/Desktop/tmp> 
1.9.3p0 :014 > f.size
 => 3261 
1.9.3p0 :015 > f.read
 => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<xsl:stylesheet xmlns=\"http://www.pbcore.org/PBCore/PBCoreNamespace.html\" xmlns:xsl=\"http://www.w3.org/1999/XSL/Transform\" version=\"1.0\">\n    <xsl:output indent=\"yes\"/>\n    <xsl:strip-space elements=\"*\"/>\n    <xsl:template match=\"//ROW\">\n        <pbcoreDescriptionDocument xmlns=\"http://www.pbcore.org/PBCore/PBCoreNamespace.html\" schemaVersion=\"2.0\">\n            <pbcoreIdentifier>\n                <xsl:attribute name=\"source\">MARS Program Record ID</xsl:attribute>\n                <xsl:value-of select=\"ID_PROGRAM/text()\" />              \n            </pbcoreIdentifier>\n            <xsl:apply-templates select=\"*[normalize-space()]\"></xsl:apply-templates>\n        </pbcoreDescriptionDocument>\n    </xsl:template>\n                  \n    <xsl:template match=\"TITLE_SERIES\">\n        <pbcoreTitle>\n            <xsl:attribute name=\"titleType\">Series</xsl:attribute>\n            <xsl:value-of select=\"text()\" />\n        </pbcoreTitle>\n    </xsl:template>    \n\n    <xsl:template match=\"TITLE_PROGRAM\">\n      <pbcoreTitle>\n        <xsl:attribute name=\"titleType\">Program</xsl:attribute>\n        <xsl:value-of select=\"text()\" />\n      </pbcoreTitle>\n    </xsl:template>\n\n    <xsl:template match=\"DATE_RELEASE\">\n        <pbcoreAssetDate>\n            <xsl:attribute name=\"dateType\">broadcast</xsl:attribute>\n            <xsl:value-of select=\"text()\" />\n        </pbcoreAssetDate>\n    </xsl:template>\n\n    <xsl:template match=\"COVERAGE_DATE_PORTRAYED\">\n        <pbcoreCoverage>\n            <coverage>\n            <xsl:value-of select=\"text()\" />\n            </coverage>\n            <coverageType>temporal</coverageType>\n        </pbcoreCoverage>\n    </xsl:template>\n\n    \n    <xsl:template match=\"RIGHTS_SUMMARY\">\n        <pbcoreRightsSummary>\n            <rightsSummary>\n                <xsl:attribute name=\"annotation\">rights summary</xsl:attribute>\n                <xsl:attribute name=\"source\">WGBH MARS</xsl:attribute>\n                <xsl:value-of select=\"text()\"/>\n            </rightsSummary>\n        </pbcoreRightsSummary>\n    </xsl:template>\n\n    <xsl:template match=\"ID_NOLA\">\n        <pbcoreIdentifier>\n            <xsl:attribute name=\"source\">NOLA Code</xsl:attribute>\n            <xsl:value-of select=\"text()\" />\n        </pbcoreIdentifier>\n    </xsl:template>\n    \n    <xsl:template match=\"SOURCE_PROGRAM_NUMBER\">\n        <pbcoreTitle> \n            <xsl:attribute name=\"titleType\">Episode</xsl:attribute>\n            <xsl:value-of select=\"text()\"/>\n        </pbcoreTitle>\n    </xsl:template>\n                   \n    <xsl:template match=\"DESCRIPTION_PROGRAM\">\n        <pbcoreDescription>\n            <xsl:attribute name=\"descriptionType\">Program</xsl:attribute>\n            <xsl:attribute name=\"descriptionTypeRef\">http://metadataregistry.org/concept/show/id/1702.html</xsl:attribute>\n            <xsl:value-of select=\"text()\" />\n        </pbcoreDescription>\n    </xsl:template>\n    \n    <xsl:template match=\"Producer\">\n        <pbcoreCreator>\n            <creator><xsl:value-of select=\"text()\"/></creator>\n            <creatorRole ref=\"http://metadataregistry.org/concept/show/id/1425.html\">Producer</creatorRole>\n        </pbcoreCreator>\n    </xsl:template>\n    \n    <xsl:template match=\"text()\" />\n</xsl:stylesheet>" 

cbeer avatar Mar 21 '12 12:03 cbeer

See https://gist.github.com/2147407 for large file ingest experiments

cbeer avatar Mar 21 '12 16:03 cbeer