web-monitoring-processing icon indicating copy to clipboard operation
web-monitoring-processing copied to clipboard

Keep track of canonical URLs from page markup in `source_metadata`

Open Mr0grog opened this issue 3 years ago • 1 comments

Some pages have a <link rel="canonical" href="{url}"> element in their markup, indicating a correct, “canonical” URL for the page (some more info here: https://en.wikipedia.org/wiki/Canonical_link_element). When importing data from the Wayback Machine, it would be great to include the canonical URL in the source_metadata field.

We already parse HTML pages that we’re importing to get their titles, and we could get the canonical link (if present) at a similar point in the process. Ideally we should create a way to only parse the page once to get title, canonical link, and anything else we might want to extract from the page content in the future.

Where we already parse markup for titles:

https://github.com/edgi-govdata-archiving/web-monitoring-processing/blob/21512eb804d3212cb4d4458fbbfb8e3e308628c0/web_monitoring/cli/cli.py#L408-L413

https://github.com/edgi-govdata-archiving/web-monitoring-processing/blob/21512eb804d3212cb4d4458fbbfb8e3e308628c0/web_monitoring/utils.py#L98-L112

Mr0grog avatar Jun 29 '21 06:06 Mr0grog

It might be useful to do this for <link rel="shortlink">, too. I had totally forgotten about that one!

Mr0grog avatar Jun 29 '21 06:06 Mr0grog