Error 503 when (probably) hitting maximum result limit edge case (50000) in /api/v2/search/spending_by_transaction/

[agnessa]

I am fetching data from /api/v2/search/spending_by_transaction/ in pages of 100 results and consistently hit http error 503 when requesting page 500 of the result set. I think the error is due to the fact that there is an edge case where the result set size == 50 000, which is not handled well. Here are 3 curl requests to reproduce the condition:

• requesting page 499 - works (200)

curl -i -X POST -H "Content-Type: application/json" -d '{"filters":{"time_period": [{"start_date":"2010-01-01", "end_date": "2011-01-01"}], "keywords":["food waste", "222 million tons", "412 food rescue", "aerobic digest", "ample harvest", "ampleharvest", "anaerobic digest", "baltimore free farm", "berkeley student food collective", "borderlands", "boston area gleaners", "boulder food rescue", "ca farm to family", "california farm to family", "campus kitchens project", "city harvest", "city soil", "cold chain", "cold storage", "cold-chain", "commonwealth kitchen", "community food b", "community harvest", "community kitchen", "compost", "compost cats", "controlled atmosphere", "daily table", "date label", "dc central kitchen", "digester", "donate don''t dump", "drexel food lab", "equal heart", "expiration date", "extra food", "fair food", "falling food", "feed more", "feeding america", "food ba", "food bank", "food basket", "food bk", "food bridge", "food closet", "food connect", "food cupboard", "food deposit", "food depot", "food desposit", "food finders", "food for free", "food forward", "food hall", "food keeper", "food kitchen", "food lifelife", "food lifeline", "food link", "food loss", "food ministr", "food mission", "food pantr", "food recover", "food recovery network", "food rescue", "food rescue partnership", "food rescue us", "food reserv", "food runner", "food share", "food shelf", "food shift", "food shuttle", "foodbank", "foodbasket", "foodbridge", "foodhall", "foodkeeper", "foodlink", "foodrunner", "foodshelf", "foodshift", "forgotten harvest", "freezer", "friendships donation network", "fruit cycle", "fruitcycle", "galloping grace", "gitm foundation", "gleaner", "gleaning", "good food recovery", "green mountain compost", "greywater", "grow a row", "grow nyc", "grow-a-row", "growing soul", "growingsould", "grownyc", "healthy food for all", "hidden harvest", "how2compost", "ilsr", "insinkerator", "institute for local self-reliance", "island grown initiative", "k-12 food rescue", "k12 food rescue", "keep austin fed", "kitchens for good", "la kitchen", "la soupe", "leanpath", "little herds", "lovin spoonfuls", "lovin'' spoonfuls", "manna food center", "matching excess and need for stability", "means database", "mobile produce pantry", "move for hunger", "new venture fund", "no tray", "no-tray", "nourish now", "off spec", "off-spec", "peels & wheels", "peels and wheels", "peninsula food runners", "philabuncance", "philabundance", "plate size", "post harvest", "post-harvest", "postharvest", "postharvest education foundation", "produce good", "produce matchmaker", "producegood", "rebel produce", "recover food", "recovered food", "recovering food", "refed", "refrig", "replate", "rescue food", "rescued food", "rescuing food", "rescuing leftover cuisine", "resell", "rolling harvest", "rooster soup", "rotary first harvest", "salvage supperclub", "salvation farms", "san francisco wholesale produce", "seattle food rescue", "seattle''s table", "seattles table", "second helpings", "share meals", "small plate", "smaller plate", "smaller portion", "soup kitchen", "spoil", "swipe out hunger", "three square", "transfernation", "trayless", "trayless dining", "ugly fruit and veg", "urban gleaner", "urban harvester", "vermiculture", "vermont commodity program", "vermont foodbank", "walk-in cooler", "waste", "waste analy", "waste audit", "waste no food", "waste not", "waste tracking", "we don''t waste", "we dont waste", "what a waste", "wonky", "zero waste", "organic & food"], "award_type_codes": ["02", "03"]},"fields":["Action Date","generated_internal_id"], "page": 499, "sort": "Action Date", "limit": 100}' https://api.usaspending.gov/api/v2/search/spending_by_transaction/

• requesting page 500 - doesn't work (503 every time)

curl -i -X POST -H "Content-Type: application/json" -d '{"filters":{"time_period": [{"start_date":"2010-01-01", "end_date": "2011-01-01"}], "keywords":["food waste", "222 million tons", "412 food rescue", "aerobic digest", "ample harvest", "ampleharvest", "anaerobic digest", "baltimore free farm", "berkeley student food collective", "borderlands", "boston area gleaners", "boulder food rescue", "ca farm to family", "california farm to family", "campus kitchens project", "city harvest", "city soil", "cold chain", "cold storage", "cold-chain", "commonwealth kitchen", "community food b", "community harvest", "community kitchen", "compost", "compost cats", "controlled atmosphere", "daily table", "date label", "dc central kitchen", "digester", "donate don''t dump", "drexel food lab", "equal heart", "expiration date", "extra food", "fair food", "falling food", "feed more", "feeding america", "food ba", "food bank", "food basket", "food bk", "food bridge", "food closet", "food connect", "food cupboard", "food deposit", "food depot", "food desposit", "food finders", "food for free", "food forward", "food hall", "food keeper", "food kitchen", "food lifelife", "food lifeline", "food link", "food loss", "food ministr", "food mission", "food pantr", "food recover", "food recovery network", "food rescue", "food rescue partnership", "food rescue us", "food reserv", "food runner", "food share", "food shelf", "food shift", "food shuttle", "foodbank", "foodbasket", "foodbridge", "foodhall", "foodkeeper", "foodlink", "foodrunner", "foodshelf", "foodshift", "forgotten harvest", "freezer", "friendships donation network", "fruit cycle", "fruitcycle", "galloping grace", "gitm foundation", "gleaner", "gleaning", "good food recovery", "green mountain compost", "greywater", "grow a row", "grow nyc", "grow-a-row", "growing soul", "growingsould", "grownyc", "healthy food for all", "hidden harvest", "how2compost", "ilsr", "insinkerator", "institute for local self-reliance", "island grown initiative", "k-12 food rescue", "k12 food rescue", "keep austin fed", "kitchens for good", "la kitchen", "la soupe", "leanpath", "little herds", "lovin spoonfuls", "lovin'' spoonfuls", "manna food center", "matching excess and need for stability", "means database", "mobile produce pantry", "move for hunger", "new venture fund", "no tray", "no-tray", "nourish now", "off spec", "off-spec", "peels & wheels", "peels and wheels", "peninsula food runners", "philabuncance", "philabundance", "plate size", "post harvest", "post-harvest", "postharvest", "postharvest education foundation", "produce good", "produce matchmaker", "producegood", "rebel produce", "recover food", "recovered food", "recovering food", "refed", "refrig", "replate", "rescue food", "rescued food", "rescuing food", "rescuing leftover cuisine", "resell", "rolling harvest", "rooster soup", "rotary first harvest", "salvage supperclub", "salvation farms", "san francisco wholesale produce", "seattle food rescue", "seattle''s table", "seattles table", "second helpings", "share meals", "small plate", "smaller plate", "smaller portion", "soup kitchen", "spoil", "swipe out hunger", "three square", "transfernation", "trayless", "trayless dining", "ugly fruit and veg", "urban gleaner", "urban harvester", "vermiculture", "vermont commodity program", "vermont foodbank", "walk-in cooler", "waste", "waste analy", "waste audit", "waste no food", "waste not", "waste tracking", "we don''t waste", "we dont waste", "what a waste", "wonky", "zero waste", "organic & food"], "award_type_codes": ["02", "03"]},"fields":["Action Date","generated_internal_id"], "page": 500, "sort": "Action Date", "limit": 100}' https://api.usaspending.gov/api/v2/search/spending_by_transaction/

• requesting page 501 - works (422 + gives you the error message about exceeding max result limit)

curl -i -X POST -H "Content-Type: application/json" -d '{"filters":{"time_period": [{"start_date":"2010-01-01", "end_date": "2011-01-01"}], "keywords":["food waste", "222 million tons", "412 food rescue", "aerobic digest", "ample harvest", "ampleharvest", "anaerobic digest", "baltimore free farm", "berkeley student food collective", "borderlands", "boston area gleaners", "boulder food rescue", "ca farm to family", "california farm to family", "campus kitchens project", "city harvest", "city soil", "cold chain", "cold storage", "cold-chain", "commonwealth kitchen", "community food b", "community harvest", "community kitchen", "compost", "compost cats", "controlled atmosphere", "daily table", "date label", "dc central kitchen", "digester", "donate don''t dump", "drexel food lab", "equal heart", "expiration date", "extra food", "fair food", "falling food", "feed more", "feeding america", "food ba", "food bank", "food basket", "food bk", "food bridge", "food closet", "food connect", "food cupboard", "food deposit", "food depot", "food desposit", "food finders", "food for free", "food forward", "food hall", "food keeper", "food kitchen", "food lifelife", "food lifeline", "food link", "food loss", "food ministr", "food mission", "food pantr", "food recover", "food recovery network", "food rescue", "food rescue partnership", "food rescue us", "food reserv", "food runner", "food share", "food shelf", "food shift", "food shuttle", "foodbank", "foodbasket", "foodbridge", "foodhall", "foodkeeper", "foodlink", "foodrunner", "foodshelf", "foodshift", "forgotten harvest", "freezer", "friendships donation network", "fruit cycle", "fruitcycle", "galloping grace", "gitm foundation", "gleaner", "gleaning", "good food recovery", "green mountain compost", "greywater", "grow a row", "grow nyc", "grow-a-row", "growing soul", "growingsould", "grownyc", "healthy food for all", "hidden harvest", "how2compost", "ilsr", "insinkerator", "institute for local self-reliance", "island grown initiative", "k-12 food rescue", "k12 food rescue", "keep austin fed", "kitchens for good", "la kitchen", "la soupe", "leanpath", "little herds", "lovin spoonfuls", "lovin'' spoonfuls", "manna food center", "matching excess and need for stability", "means database", "mobile produce pantry", "move for hunger", "new venture fund", "no tray", "no-tray", "nourish now", "off spec", "off-spec", "peels & wheels", "peels and wheels", "peninsula food runners", "philabuncance", "philabundance", "plate size", "post harvest", "post-harvest", "postharvest", "postharvest education foundation", "produce good", "produce matchmaker", "producegood", "rebel produce", "recover food", "recovered food", "recovering food", "refed", "refrig", "replate", "rescue food", "rescued food", "rescuing food", "rescuing leftover cuisine", "resell", "rolling harvest", "rooster soup", "rotary first harvest", "salvage supperclub", "salvation farms", "san francisco wholesale produce", "seattle food rescue", "seattle''s table", "seattles table", "second helpings", "share meals", "small plate", "smaller plate", "smaller portion", "soup kitchen", "spoil", "swipe out hunger", "three square", "transfernation", "trayless", "trayless dining", "ugly fruit and veg", "urban gleaner", "urban harvester", "vermiculture", "vermont commodity program", "vermont foodbank", "walk-in cooler", "waste", "waste analy", "waste audit", "waste no food", "waste not", "waste tracking", "we don''t waste", "we dont waste", "what a waste", "wonky", "zero waste", "organic & food"], "award_type_codes": ["02", "03"]},"fields":["Action Date","generated_internal_id"], "page": 501, "sort": "Action Date", "limit": 100}' https://api.usaspending.gov/api/v2/search/spending_by_transaction/