503 Server Capacity Error

[csmcallister]

Description of problem

When calling the api/v2/bulk_download/awards/ endpoint in batches overnight, I intermittently receive a 503 response stating Service Unavailable: Back-end server is at capacity.

Steps to recreate

Although this happens seemingly at random, here's how I was making the requests:

import json

import requests

def get_awards_by_agency(agency: tuple, date_range: dict, out_dir: str) -> None:
    """Use the bulk download API to get awards from 1-month time-span per agency

    Arguments:
        agency {tuple} -- The agency's name and their id, as returned by api/v2/bulk_download/list_agencies/
        date_range {dict} -- e.g. {"start_date":"2017-09-01","end_date":"2017-09-30"}
    """    
    agency_name, agency_id = agency
    url = 'https://api.usaspending.gov/api/v2/bulk_download/awards/'
    headers = {'Content-Type': 'application/json'}
    payload = json.dumps({
        "award_levels": ["prime_awards"],
        "filters": {
            "award_types": ["contracts"],
            "prime_award_types": ["02", "03", "04", "05", "A", "B", "C", "D"],
            "sub_award_types": ["procurement"],
            "agency": str(agency_id),
            "date_type": "action_date",
            "date_range": date_range
            },
        "file_format": "csv"})

    try:
        r = requests.post(url, data=payload, headers=headers)
        r.raise_for_status()
    except Exception as e:
        logger.critical(f"{e} unable to post bulk download request for {agency_id}", exc_info=True)
        return

As you can see, I was making requests for one agency at a time for 1 month of contract award data. Any idea why this would occasionally fail? Or perhaps any tips for how I should restructure these requests?

[lisaam]

Hi Chris, Your code is good.

We're trying to understand a little more about what's happening. How often is it occurring? Every night or just one night? When did it start?

We do a lot of data loading at night. Does the same thing occur when you run request during the day?

[csmcallister]

Hi @lisaam. Thanks for the speedy follow up!

I was making the requests overnight starting around 10pm EST. The errors responses started very late (~3am) and persisted for nearly every request thereafter. Unfortunately I've deleted those logs as my laptop was running out of space during a large download and I had to make some hasty decisions 🤷‍♂️

Interestingly, I tried the script again last night using the same 1 month date range with individual agency requests, but only going back to 10/1/2019 instead of 10/1/2017 like originally. That worked just fine, but the process might have finished before any of your data uploading started late in the night

To give y'all more insight, I can re-run the original process overnight and share the full log report so we can pinpoint the times at which the errors are raised. I can also run the same process during the day to see if the issue persists. Just let me know and thanks again for your awesome work with making so much of this data publicly available!

[lisaam]

No problem about deleting the data due to space.

I think you could help us most by running it now during the day.

[csmcallister]

I think you could help us most by running it now during the day.

I'll give it a go tomorrow and report back.