oam-browser
oam-browser copied to clipboard
hotosm
WIP: Update dependencies to work on recent node versions
TODO
- [ ] Figure out StatusImage breakage
- [ ] Remove my facebook test credentials and make the oam staging ones work
Background
I was trying to get started hacking on oam-browser and ran into some setup issues with some of the dependencies, mostly around webdriverio. It seems like the versions that oam-browser is using are not working anymore with recent node version (I'm testing on 13).
So I figured out I need to update some packages and the webdriverio ones seems to be the most problematic ones (they depend on an old fibers version that doesn't build anymore).
Closes hotosm/oam-browser#295
Status
I'm almost there with the integration tests running locally against oam-api (I opened https://github.com/hotosm/oam-api/pull/153 for a fix for oam-api).
The only tests that seem to fail are the one simulating a click on 'View image' after upload. But is this functionality supposed to work ? Because it seems the metadata returned by oam-api doesn't contain a uuid and oam-browser is looking for a uuid, in StatusImage.js:imageClick. I am testing against the develop branch of oam-api. Guidance on this would be appreciated.
StatusImage.js issue
If I print the metadata that the imageClick function is receiving, I'm seeing this:
{"acquisition_end":"2020-04-10T10:08:05.077Z","acquisition_start":"2020-04-09T22:00:00.000Z","contact":"Bob Mary,[email protected]","platform":"satellite","provider":"42","properties":{"license":"CC-BY 4.0","sensor":"42"},"title":"test"}
This corresponds that what's stored in the uploads table of the DB, and the frontend is indeed getting that from the /uploads/{id} route of oam-api.
Now, what this function seems to expect is the actual image metadata as stored on S3 (in the XXX_meta.json file or in the metas DB table.
So I feel there is an issue there that the frontend is not calling the right oam-api route to get this metadata.
Question
- Is modernizing the oam-browser packages something that is welcome ? If yes, I'll try to finish this
- Are the automated regression tests supposed to pass on
develop? Looking at the commits, seems the test are broken since a while. I'd be willing to help getting them green again - If someone can help me understand better the
StatusImage.js:imageClickissue, I'd be happy
Thanks so much for contributing @julienr ! As you can see there are some issues with the integration tests, which for me have been with the facebook and google logins. To address your points:
- A PR to modernize the packages would be welcome for sure.
- If you continue to run into issues with the integration tests, feel free to DM me on the HOTOSM Slack slack.hotosm.org. I think some changes in the FB and Google auth apis have changed. And if you think we can merge safely without the tests passing in the interim, I think that's ok for now.
- I'll look into the StatusImage.js issue and get back to you.
Looking a bit more into this StatusImage issue, right after i upload an image, here is what I have in the oam-api mongodb:
> db.uploads.find().sort({"createdAt":-1})[0]
{
"_id" : ObjectId("5ea5754cbaf14b002737be98"),
"scenes" : [
{
"contact" : null,
"title" : "everest",
"provider" : "42",
"platform" : "satellite",
"sensor" : "42",
"acquisition_start" : "2020-04-25T22:00:00.000Z",
"acquisition_end" : "2020-04-26T11:14:11.212Z",
"tms" : "https://github.com/openimagerynetwork/oin-meta-generator/blob/master/test/fixtures/everest-utm.gtiff?raw=true",
"license" : "CC-BY 4.0",
"tags" : "",
"urls" : [
"https://github.com/openimagerynetwork/oin-meta-generator/blob/master/test/fixtures/everest-utm.gtiff?raw=true"
],
"images" : [
ObjectId("5ea5754cbaf14b002737be99")
]
}
],
"user" : ObjectId("5e7f2001af7e5a00119e4fd6"),
"createdAt" : ISODate("2020-04-26T11:49:32.518Z")
}
Now, finding the image associated with this upload (which is what's returned by the uploads/{id} endpoint, I get this
> db.images.findOne({ _id: ObjectId("5ea5754cbaf14b002737be99") })
{
"_id" : ObjectId("5ea5754cbaf14b002737be99"),
"url" : "https://github.com/openimagerynetwork/oin-meta-generator/blob/master/test/fixtures/everest-utm.gtiff?raw=true",
"status" : "finished",
"user_id" : ObjectId("5e7f2001af7e5a00119e4fd6"),
"messages" : [
"",
"Downloading https://github.com/openimagerynetwork/oin-meta-generator/blob/master/test/fixtures/everest-utm.gtiff?raw=true...",
"Transcoding 3 band(s)...",
"Adding overviews...",
"Creating cloud-optimized GeoTIFF...",
"Generating thumbnail...",
"Generating footprint...",
"Uploading..."
],
"metadata" : {
"acquisition_end" : "2020-04-26T11:14:11.212Z",
"acquisition_start" : "2020-04-25T22:00:00.000Z",
"contact" : "OAM test user,[email protected]",
"platform" : "satellite",
"provider" : "42",
"properties" : {
"license" : "CC-BY 4.0",
"sensor" : "42"
},
"title" : "everest"
},
"startedAt" : ISODate("2020-04-26T11:49:34.993Z"),
"stoppedAt" : ISODate("2020-04-26T11:49:41.112Z")
}
Looking in the meta table, I think this is the corresponding metadata (although I'm unclear as to how one should find the metadata corresponding to an image. Here, the timestamps seem to match):
> db.metas.find().sort({"uploaded_at":-1})[0]
{
"_id" : ObjectId("5ea57555baf14b002737be9b"),
"acquisition_end" : ISODate("2020-04-26T11:14:11.212Z"), "acquisition_start" : ISODate("2020-04-25T22:00:00Z"), "contact" : "OAM test user,[email protected]",
"platform" : "satellite",
"provider" : "42",
"properties" : {
"wmts" : "http://tiles.staging.openaerialmap.org/5ea5754cbaf14b002737be98/0/5ea5754cbaf14b002737be99/wmts",
"tms" : "http://tiles.staging.openaerialmap.org/5ea5754cbaf14b002737be98/0/5ea5754cbaf14b002737be99/{z}/{x}/{y}",
"thumbnail" : "https://oam-julien-oin-bucket.s3.amazonaws.com/development/5ea5754cbaf14b002737be98/0/5ea5754cbaf14b002737be99.png",
"resolution" : [
350,
350
],
"colorinterp" : [
"red",
"green",
"blue"
],
"dtype" : "uint8",
"filename" : "5ea5754cbaf14b002737be99.tif",
"resolution_in_meters" : 350,
"bands" : 3,
"url" : "s3://oam-julien-oin-bucket/development/5ea5754cbaf14b002737be98/0/5ea5754cbaf14b002737be99.tif",
"dimensions" : [
66,
66
],
"crs" : "EPSG:32645",
"sensor" : "42",
"license" : "CC-BY 4.0"
},
"title" : "everest",
"user" : ObjectId("5e7f2001af7e5a00119e4fd6"),
"uuid" : "https://oam-julien-oin-bucket.s3.amazonaws.com/development/5ea5754cbaf14b002737be98/0/5ea5754cbaf14b002737be99.tif",
"geojson" : ...,
"footprint" : "POLYGON ((86.808984 27.884612, 87.044128 27.884612, 87.044128 28.093268, 86.808984 28.093268, 86.808984 27.884612))",
"gsd" : 350,
"file_size" : 21100,
"projection" : "PROJCS[\"WGS 84 / UTM zone 45N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTH
ORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",87],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHO
RITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32645\"]]",
"meta_uri" : "https://oam-julien-oin-bucket.s3.amazonaws.com/development/5ea5754cbaf14b002737be98/0/5ea5754cbaf14b002737be99_meta.json",
"uploaded_at" : ISODate("2020-04-26T11:49:41.115Z"),
"bbox" : [
86.808984,
27.884612,
87.044128,
28.093268
],
"__v" : 0
}
I feel what's in the meta table is what's the frontend is expecting the /uploads/{id} endpoint to return.
Looking at the history of oam-api, it seems like this commit has changed the 'update image metadata' logic. Before this commit, IIUC, it was duplicating the metadata in image.metadata, but after this it's only copying some of the metadata:
https://github.com/hotosm/oam-api/commit/84d3492d4fd513a9ef767b7dcd5f35c6ddcb5b5b#diff-647a467e336f16a2a502ee92eac324f2L286
I don't understand enough of the overall architecture to fully grasp the intent behind these changes, so it's very possible that I am missing something here. I could try assigning the whole meta object to image.metadata and see if it fixes the problem.
Indeed, copying the metadata in the image in oam-api seems to help with the StatusImage issue. I have a WIP fix on this branch: https://github.com/julienr/oam-api/commit/818d43dc1d4200d2ed5355c4c22a716a5a35b6a0 (needs some oam-browser fixes as well: https://github.com/julienr/oam-browser/commit/0eca44e2bd62012b893fdf417861092764cb70ec)
I'm not quite sure this is the right fix though, because I guess this metadata will not be present for images in the current prod db, so this would break backward compatibility.
I feel getting these tests to pass again is going to require a number of changes. Do you know the last commits of oam-browser/oam-api for which these tests were passing ? Do you prefer that I try to fix everything at once in a big PR or we do it step-by-step in multiple small increments ?