What is the BEST way to authorize credentials? In a loop that runs every so often, or before every function that uses gspread_asyncio?

[Swimmerhair]

Right now, I have an authorization loop that runs every 60 seconds that automatically re-authorizes the credentials for the sheet I am using.

async def refresh_google_creds(agcm):
    await bot.wait_until_ready() #waits until discord.client has initialized and is ready
    while not bot.is_closed:  #returns true if the discord.client is running
        if bot.creds.access_token_expired:
            print('Refreshed google creds at ' + time.ctime())
        bot.gclient = await agcm.authorize()
        bot.sheet = await bot.gclient.open_by_key('sheetkey')
        bot.database_sheet = await bot.gclient.open_by_key('sheet key')
        await asyncio.sleep(60) # task runs every 60 seconds

All my other code is calling on the regular functions and methods to read/write to the spreadsheet that is being authorized in in the refresh_google_creds function.

@bot.command(pass_context=True)
async def function(ctx):
    ws = await bot.sheet.get_worksheet(0)
    rcount = ws.row_count
    msg = await ws.cell(random.randint(1,rcount),1)
    await bot.send_message(ctx.message.channel, msg.value)

The reason I ask is because I still am getting some blocking code for some reason. Last night, my bot crashed because of the error Task was destroyed but it is pending! citing the loop refresh_google_creds as the offending line. Any ideas?

[dgilman]

Not sure if I can help you with the "task was destroyed but it was pending" error without a traceback. Also not sure what you mean by "blocking code" as the sentence after that is sort of a non sequitur, or at least I'm not understanding how improperly blocking code leads to the "task was destroyed" error.

The only thing that jumps out at me here is that you need to go through the whole agcm.authorize, open_by_key, get_worksheet dance every time your bot.command is called. If not, in that first line, where you call get_worksheet(0), that could be using an AGCM under the hood whose credentials are an hour old and have expired. The only way to make sure you have up-to-date credentials is to do the whole chain, starting with agcm.authorize, every time your bot command function is called. It is ok to call all those functions many times, even concurrently. And of course if your bot.command doesn't take a whole hour to run you only need to call them once at the beginning of the function but you do need to call them every time.

[Swimmerhair]

The error happened again last night. See below for the traceback including some debug info from the logger:

INFO:oauth2client.client:Refreshing access_token
INFO:oauth2client.client:Refreshing access_token
ERROR:root:Error while calling getattr (<Spreadsheet 'Clash Data' id:1Hkwo9gSpk3NjgPLPkG8kh0zBNw2nxsYWRw0cVdn0JA0>, 'title') {}. Sleeping for 1.1 seconds.
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<refresh_google_creds() running at Main.py:315> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x74e880f0>()]>>

The only way to make sure you have up-to-date credentials is to do the whole chain, starting with agcm.authorize, every time your bot command function is called. It is ok to call all those functions many times, even concurrently.

That was my next route of how to program this properly. If I call a function that includes the agcm.authorize, open_by_key, etc... would this function need to be awaited as well? Or would a general non-awaited function suffice?

Thanks for the help!

[dgilman]

You need to enable debug mode for your event loop to get the traceback. Honestly I just leave it on all the time, it's useful and you can even tweak the output a little bit https://docs.python.org/3.4/library/asyncio-dev.html#pending-task-destroyed

If you were worried about accidentally putting blocking code into the async code (which is very easy to do, I've done it several times) the debug mode helps warn you when that happens and points to the offender.

I think if you try to use await in a function that is not declared as async def fn() you get a syntax error. The gspread_asyncio methods all have to be awaited so whatever functions call them have to be awaited, whatever functions call those functions have to be awaited and so on until almost all of your code is async def fn().

[Swimmerhair]

I turned on the debugging, but of course now the program is working without any bugs. I see Warnings, but I'm waiting for it to crash to see where it went wrong. I will get back to you when it crashes. Thanks again for the help so far! : )

[Swimmerhair]

So I noticed something when running some tests. The code below is not using anything from the discord.py library.

import asyncio
import random
import gspread_asyncio
from oauth2client.service_account import ServiceAccountCredentials

scope = ['https://spreadsheets.google.com/feeds',
      'https://www.googleapis.com/auth/drive']

async def run(agcm):
    while True:
        gclient = await agcm.authorize()
        sheet = await gclient.open_by_key('sheet_key')
        ws = await sheet.get_worksheet(0)
        rcount = ws.row_count
        msg = await ws.cell(random.randint(1,rcount),1)
        msg = msg.value
        print(msg)
        await asyncio.sleep(5)
        #loop.stop()

def get_creds():
   return ServiceAccountCredentials.from_json_keyfile_name('client_secret.json', scope)

agcm = gspread_asyncio.AsyncioGspreadClientManager(get_creds)
loop = asyncio.get_event_loop()
loop.set_debug(True)
loop.create_task(run(agcm))
loop.run_forever()

Even though I am not using the discord library, I am still getting a warning at the very beginning saying that there is a piece of blocking code:

WARNING:asyncio:Executing <Handle <TaskWakeupMethWrapper object at 0x000001F02B504B28>(<Future finis...events.py:275>) created at C:\Users\wmiller\AppData\Local\Programs\Python\Python36\lib\asyncio\futures.py:347> took 0.781 seconds

Any ideas?

[dgilman]

The purpose of those warnings is to alert you to parts of the code that are potentially blocking. Ideally you have lots of little calls to await which gives the asyncio code lots of opportunities to pause your execution and run another coroutine. If you accidentally blocked while running your code a coroutine would run for a while without hitting an await and that throws up this warning. You getting this warning could indicate that there is an accidentally blocking call down inside of gspread_asyncio but I don't have enough information here to diagnose that. I'm also wondering if the lack of competing coroutines in your example is causing all this time to be spent in the big one but I also don't know enough about asyncio to know that for sure.

Your call to random.randint is blocking, although I am skeptical that it is the culprit. It is still good async hygiene to run that call in an executor. You'd farm that out to an executor like this: rand_val = await agcm.loop.run_in_executor(None, random.randint, 1, rcount)