GSOC 2020: Mapknitter Image Export and Spectral Workbench upgrade

[alaxalves]

Here, we can discuss about weekly goals and the plans for the work to be accomplished during the whole project. Original proposal is available here: https://publiclab.org/notes/alaxallves/03-06-2020/gsoc-proposal-2020-spectral-workbench-rails-and-devops-upgrades?_=1583504846

Spectral Workbench

Tasks

Pre Steps

• [x] Get test coverage percentage tool set up
• [x] Define a desired stable test coverage with mentor
• [x] Increase test coverage percentage until the desired one
• [x] Improve CI builds
• [x] Containerize the development environment, App and Database
• [x] Include and configure SimpleCov to monitor test coverage
• [x] Set a ruby coding stylesheet as in Mapknitter and Plots2 projects
• [x] Create intermediate "development" branch

Upgrade

• [x] Increase test coverage percentage until the desired one
• [x] Switch javascript dependencies manager, Bower ~> Yarn
• [x] Create stable docker environments
• [x] Create stable CI/CD pipelines
• [x] Update Ruby version
• [x] Update gems versions and lock them on the Gemfile
• [x] Update Yarn version
• Update Rails
  • [x] Update to Rails 4.2
  • [x] Update to Rails 5.0
  • [x] Update to Rails 5.2
  • [ ] Update to Rails 6.0

Mapknitter Exporter

Tasks

Pre Steps

• [x] Identify which parts of the export module need to be upgraded
• [x] Define how the upgrades should happen and where(several MK exporter related repos)
• [x] Define which new features or flows should be changed or included on the upgrade

Upgrade

• [x] Get Mapknitter running into the cloud
• Code Performance
  • [ ] Break def self.generate_perspectival_dist method into multiple smaller methods
  • [ ] Break def self.run_export method into multiple smaller methods
  • [ ] Break def self.generate_composite_tiff method into multiple smaller methods
  • [ ] Break def self.distort_warpables method into multiple smaller methods
  • [ ] Reduce the amount of params passing
• [x] Container-wrap any Exporter related projects

Discussion

• [ ] Collecting a set of image URLs and their corner coordinates
• [ ] Determining the image dimensions in pixel and converting corner coordinates to pixel positions
• [ ] Producing an SVG artifact containing images at relative positions for less memory usage
• [ ] Exporter module as a microservice

[alaxalves]

Test coverage tool set up: https://github.com/publiclab/spectral-workbench/pull/496

[alaxalves]

Improve CI builds: https://github.com/publiclab/spectral-workbench/pull/480 https://github.com/publiclab/spectral-workbench/pull/487

[alaxalves]

Intermediate "development" branch: https://github.com/publiclab/spectral-workbench/tree/development

[alaxalves]

Containerize the development environment: https://github.com/publiclab/spectral-workbench/pull/470

[alaxalves]

Update Ruby, Node and Rails: https://github.com/publiclab/spectral-workbench/pull/495

[alaxalves]

Get Mapknitter running into the cloud: https://github.com/publiclab/mapknitter/pull/1273

[alaxalves]

Container-wrap any Exporter related projects: https://github.com/publiclab/mapknitter-exporter/pull/27 https://github.com/publiclab/mapknitter/pull/363

[alaxalves]

Currently we're running on ~68% of code coverage. Via:

@gr455 has kindly offered his help to get the percentage up, but I still @jywarren to define with me an acceptable percentage, in order to get https://github.com/publiclab/spectral-workbench/pull/499 merged and consequently the upcoming upgrades.

[jywarren]

Just linking in my suggestion from the check-in -- Do you think that system tests which walk through the steps in the tutorials in https://publiclab.org/wiki/spectral-workbench will be possible? -- this will increase the number of required tests but since those interactions (like calibration, or adding operations, for example) are really critical, it'll help to have system tests for them!

For the live capture interface, we could... make a special testing page which includes a short looping video of a live spectral capture. This would be pretty intense though. Perhaps we can coordinate with the team we've begun working with on spectral-workbench.js who may be developing such tests on the client end! https://github.com/publiclab/spectral-workbench.js/issues/159

[jywarren]

To the question of acceptable coverage, I think system tests of the critical functions documented in the tutorials would be make me very confident in merging #499. What do you think of that, Alax, does that makes sense to you too?

[jywarren]

• https://publiclab.org/wiki/spectral-workbench-snapshots
• https://publiclab.org/wiki/spectral-workbench-calibration
• https://publiclab.org/wiki/spectral-workbench-tools
• https://publiclab.org/wiki/spectral-workbench-operations

(working on fixing these videos today, sorry!)

[alaxalves]

Hello Jeff, thx for the feedback. Right now, me and Ruturaj are working with the models and controllers tests to assure #499 is more trustful to be merged. I felt like after those testing upgrades we can be confident regarding that. Now when it comes to the system tests, I think it's a great idea and I believe that it could fit our schedules, but I don't think that it should push us back from merging #499 once we have nice coverage through the models and controllers tests. Because if we wait on system tests to get it merged I think it'll delay a lot the work to come.

[jywarren]

Hi @alaxalves I think this is OK because the status quo right now is that these are manually tested. What we can do is if you can go through the videos and guides on the above links to confirm that they work (i.e. calibration, using operations, using snapshots, and basic intro video on https://publiclab.org/wiki/spectral-workbench-tools, and confirm each of these locally, we can merge #499, as that is the current state of our testing. And then yes, let's aim to improve on this as it's quite a slow process we have now, and system tests will speed that up substantially -- how does that sound? Thank you!

Also - fixed the videos finally today. Thank you for your patience!

[alaxalves]

Great, I'll manually confirm that those features actually work and I'll give the input here and fix whatever has to be fixed.

I have already an effort on setting system tests here --> https://github.com/publiclab/spectral-workbench/tree/feat/system-tests but it'll be waaaay less painful when we have Rails >5 here.