Add all "optional" attributes getters to OpinionSiteLinear, and correct failing sources
This surfaced on #1020, where the _get_docket_document_numbers getter was needed for the new OpinionSiteLinear scraper to keep the old behavior. However, when adding this and other getters to the base OpinionSiteLinear class, it breaks the tests because some fields that were supossed to be scraped, were actually not passed into the scrape caller. They were present on site.cases[0], but not on site[0]. With the getter change, theyare now added to the example tests. For example, on fla and "disposition": disposition.text_content().strip(),
It may be worth to check freelawproject/courtlistener#4042, maybe we can get rid of some of the unused attributes in this PR
I found another type of silent failure for OpinionSiteLinear
For example, in haw, we collect a "lower_courts" key. However, this won't appear on the final dictionary used in CL, nor in the example file because the OpinionSiteLinear getter expects a "lower_court" key. So, it fails silently https://github.com/freelawproject/juriscraper/blob/92d27210adebfe7efa3b5ff2777667d3cd0de78f/juriscraper/opinions/united_states/state/haw.py#L44
A similar instance is in mo.py, where "judge" and "judges" keys are both collected, and only the first one is used
https://github.com/freelawproject/juriscraper/blob/92d27210adebfe7efa3b5ff2777667d3cd0de78f/juriscraper/opinions/united_states/state/mo.py#L42-L43
I will write a test to catch this problem of unused keys in OpinionSite Linear, and will also correct the straightforward ones
I have extended AbstractSite._check_sanity for OpinionSiteLinear, to check for "wrong" case key strings, which were not being used by the OpinionSiteLinear cases. Found and corrected these:
KeyError: "Invalid key 'author' for case dictionary juriscraper.opinions.united_states.federal_appellate.ca7"
KeyError: "Invalid key 'cite' for case dictionary juriscraper.opinions.united_states.federal_special.armfor"
KeyError: "Invalid key 'lower_courts' for case dictionary juriscraper.opinions.united_states.state.haw"
KeyError: "Invalid key 'judges' for case dictionary juriscraper.opinions.united_states.territories.nmariana"
KeyError: "Invalid key 'dispostion' for case dictionary juriscraper.opinions.united_states.state.mo" (this one is a typo, should be 'disposition')
KeyError: "Invalid key 'judges' for case dictionary juriscraper.opinions.united_states.state.mo"
KeyError: "Invalid key 'judges' for case dictionary juriscraper.opinions.united_states.state.tenn"