backend

backend copied to clipboard

CBC doesn't seem to ingesting new stories since March 2020. I checked the feeds in https://sources.mediacloud.org/#/sources/7333/feeds and the ones that I checked had recent content. There were a few that...

dsjen

Picks descriptions provided [here](https://github.com/mediacloud/backend/issues/734) and adds them to the docker-compose file as descriptions for each app. Finishes https://github.com/mediacloud/backend/issues/723

esirK

add short description of what each "app" does

4

It is a little hard to figure out exactly what each "app" container is for, even thought the names are descriptive. I think it'd be helpful to have a high-level,...

rahulbot

In attach_story_data_to_stories(), don't overwrite existing fields

1

Hey Hal! I know you're superbusy with other stuff, but could you have a quick look? In https://github.com/mediacloud/backend/issues/729 (and possibly https://github.com/mediacloud/backend/issues/725), it looks like `MediaWords::DBI::Stories::attach_story_data_to_stories()` used to reset `$list_field` key...

pypt

stories_public/list empty `story_tags` when more than 100 rows requested

2

(Moved from #725.) More confusingly - asking to page with more rows than 100 seems to make the story_tags disaster in results. This code returns a story 105831 with story_tags...

pypt

recent stories missing tags (not getting processed fully?)

2

I noticed that recent stories don't have any tags on them. Perhaps some services aren't running as we transition? ```python q = '*' fq = mc.dates_as_query_clause(dt.date(2020,8,20), dt.date(2020,8,24)) tag_sets_id = mediacloud.tags.TAG_SET_NYT_THEMES_VERSION...

rahulbot

Source metadata (e.g. end date and num_stories_X) incorrect

3

We've found a number of inconsistencies relating to end dates in source manager: * https://github.com/mitmedialab/MediaCloud-Web-Tools/issues/1953 * https://github.com/mitmedialab/MediaCloud-Web-Tools/issues/1991 I think I've zeroed in on where the problem may lie. The `mediaHealth`...

dsjen

Hey Jason! The AP crawler that you wrote (and Hal adapted) crashes pretty often with the following: ``` 2020-01-29T20:23:40.670845000Z INFO crawler_ap.ap: Found new story (guid: 3816eb886cea605916a1f879a5d516d8, version: 0), 2020-01-29T20:23:40.868011000Z INFO...

pypt

Account Activation Failed

1

I tried to register for an account, but the activation mail did not arrive at '.ac.in' domain. I tried using my organization's Gsuite email as well as my personal Gmail...

dipanjanm2

add topic ingestion function for PushShift verified twitter

2

Same as the reddit one (https://github.com/berkmancenter/mediacloud/issues/598). I think this has already happened, but we aren't tracking it on an issue. We want to be able to search PushShift.io verified twitter...

rahulbot