migrate all DrupalNodeMainImage to native Image records

[ebarry]

Some nodes use a legacy main image type: DrupalNodeMainImage -- not sure how many are left.

Example: http://publiclab.org/wiki/somerville-massachusetts (though this may have changed?)

This could be done by:

• [ ] writing a query which returns all nodes with a DrupalMainImage (by using logic like this: https://github.com/publiclab/plots2/blob/master/app/models/drupal_node.rb#L296-L308)
• [ ] by using this code in a Rails Migration to create a new native Image from the path of the node's DrupalNodeMainImage: https://github.com/publiclab/plots2/blob/master/app/controllers/images_controller.rb#L10-L13
• [ ] deleting all legacy DrupalMainImage code, mostly in this method and elsewhere in that file: https://github.com/publiclab/plots2/blob/master/app/models/drupal_node.rb#L296-L308

These steps could be fleshed out and broken up more into separate issues.

[btbonval]

Last time this happened, the images were referencing old id's on the HD which aliased to new images on the new system's HD.

We had to recover the original image from the old system (I think the images have been backed up), load it onto the new system with a new filename, and then fix the reference in the database. Although Jeff did all that work, not me, so I might have the steps wrong.

[jywarren]

This had also happened on the air-quality-monitoring and providence pages... All with old-style Drupal lead images and not the newer wiki lead image system which is rails.

I wonder how many old tool and place pages now have the wrong image and if we could simply manually update them all and ditch the Drupal specific code.

[btbonval]

Oh I didn't realize there was different code for the two systems.

Yeah, one less drupal dripping would probably be a good thing! :)

[btbonval]

• DrupalFile
  • https://github.com/publiclab/plots2/blob/master/app/models/drupal_file.rb
• DrupalMainImage
  • https://github.com/publiclab/plots2/blob/master/app/models/drupal_main_image.rb

vs

• Image
  • https://github.com/publiclab/plots2/blob/master/app/models/image.rb

did I miss anything?

[jywarren]

http://publiclab.org/tools and http://publiclab.org/places list those types which had a drupal lead image type.

[jywarren]

that's right! https://github.com/publiclab/plots2/blob/master/app/models/drupal_node.rb#L193

[jywarren]

Yeah, i don't see any others that look wrong. We could indeed just download/upload all the images and then rip out that drupal code, it seems.

[jywarren]

Liz, do you have any time to download and re-upload all the full-res tool and place page lead images (that are old enough to have been made on the old site) and we can tackle the code changes? Or do you see this as not high priority, and we can take our time on it?

[jywarren]

There are also a lot of older research notes with Drupal-style lead images, like http://publiclab.org/sites/default/files/imagecache/default/IMG_0907.JPG on http://publiclab.org/notes/warren/10-10-2012/comparisons-oil-samples-and-residues-uv-light

[grvsachdeva]

@jywarren could you sum-up what goal we want to achieve here and maybe repost the issue if still required? Thanks!

[jywarren]

I've looked pretty hard at this and I wanted to be able to suggest something like:

DrupalMainImage.all.each do |dimage|
    image = Image.new(uid: dimage.uid,
                        remote_url: dimage.file.path)
    image.save
end

But I think remote URL loading is off by default in Paperclip now, for security reasons: https://github.com/thoughtbot/paperclip#usage

So, I think we have to figure out how to convert image files using drupal_main_image.file.path local path into Rails-native Image records.

We can then run this on all records, either in a migration, or manually. Then we can delete all the DrupalMainImage code!

https://github.com/publiclab/plots2/blob/master/app/models/drupal_main_image.rb

https://github.com/publiclab/plots2/blob/master/app/models/node.rb#L309-L327

I think this needs a bit of research, some testing out, then maybe we can tidy it all up in a new issue?

[jywarren]

OK, I've collected nids for all nodes which have a DrupalMainImage record: https://gist.github.com/jywarren/c9ca204d69a833c4d423eee4170ab423

There are over 1000, unfortunately. And for each, we need to check that there is not already a node.main_image_id record (13 of them do have this) because that means a new main image was added in the new system and we can ignore those.

[stale[bot]]

Hi :smile:, this issue has been automatically marked as stale because it has not had recent activity. Don't worry you can continue to work on this and ask @publiclab/reviewers to add "work in progress" label :tada: . Otherwise, it will be closed if no further activity occurs in 5 days -- but you can always re-open it if you like! :100: Thank you for your contributions :raised_hands: :balloon:.

[ebarry]

Commenting to remove the 'stale' label -- we don't want to lose the old images.

[jywarren]

Just looking over this again... I wonder if we can download the images locally, then use this guidance on uploading from the rails console to achieve this:

https://github.com/thoughtbot/paperclip/issues/1447

We could either do this all in a migration (though we'd have to see if we can fetch remote files in Ruby, or use something like system('wget https://path.to/image.png') in some local folder... so we'd need wget or curl to be installed... maybe better we could use Ruby's Tempfile command? https://www.rubyguides.com/2019/05/ruby-tempfile/ or https://duckduckgo.com/?q=ruby+tempfile+download&ia=web or maybe https://gist.github.com/cheeyeo/10011283#file-temp-rb-L40-L42 ?

Some more resources:

• https://github.com/thoughtbot/paperclip/issues/1447

[anirudhprabhakaran3]

Hmm yeah this also looks good. I'll look into the tempfiles as well - if not we can always use wget or curl. Thanks for the resources!