phoenix_pipeline

phoenix_pipeline copied to clipboard

Think about adding a pre-pipeline coding step that geocodes complete articles (rather than sentences) to the country. This would be useful for two things:

1. Associating actors that don't have country codes with the correct country. E.g., "Egyptian police fired on protesters" -> EGYCOP, ~CVL. If the document is geocoded to one country, we could say with enough confidence that ~CVL = EGYCVL. This process would happen in the pipeline rather than in Petrarch.
2. Mordecai's country coding works much better at the document level than the sentence level, and its place function can take in a country limit as an argument. Doing a first-pass whole article country-level coding could help the sentence level geocoding.

Because the pipeline operates at the sentence level, actually geocoding the articles would have to happen outside the pipeline. Changes to the pipeline would just be in order to use the new info.

Downside is the overhead when you have a large corpus that contains mostly junk (e.g. recently worked with one generated by the data-provider-who-shall-not-be-named with 2.5M stories, only about 2% of which generated events). We could do this just as easily by doing the substitutions in a post-processing phase, right?

I had forgotten that the geocoding step in postprocess.py hits the db again. Since that's also where it does the actor splitting, it would be easy to put a step in there where it geocodes the full article and updates the db, but only if an event has already been coded from that article.