Using pipe_to and pipe_from

[Shmookoff]

I can't really find anything in docs about the use of pipe_to. The only clarification I could find is this, where bytes get just printed out.

What buffer does it need? How do I use it? Can someone please clarify this for me.

[Shmookoff]

As I understand, pipe_to should work with an async buffer, couldn't find one though.

[omarryhan]

Hey @Shmookoff ,

An example has been added here: https://github.com/omarryhan/aiogoogle/commit/0a275894f4e140c508083fc9c74bbc65c3ab2be5

[omarryhan]

Oh, sorry, just realized that all you can see is print

[omarryhan]

Does this help: https://github.com/omarryhan/aiogoogle/commit/5008af099387778451ed3d97419b61e487b380d5 ?

[Shmookoff]

Does this help: 5008af0 ?

Not much. Ideally, I would want some std Python buffer to work with this. My main goal is to download a file from drive to a buffer and upload it to Telegram using this, that uses IOBase. Is it possible?

[omarryhan]

Maybe you can wrap the IOBase with a generator? pipe_to expects a class with a generator method. You think that would work?

[Shmookoff]

I think it is fair to say that my programming skills are not on this level yet, so.. I would really love some more explanation 🙂

[Shmookoff]

I think I found something. AsyncBufferedIOBase and AsyncBufferedReader from aiofiles.tempfile Is this what I need? I still don't really understand how to use it though..

[Shmookoff]

I worked it out!

This is a compiled example from the pieces of my project.

from io import BytesIO
from aiofiles.tempfile import TemporaryFile

async def main():
    async with Aiogoogle(user_creds=user_creds, client_creds=client_creds) as aiogoogle:
        drive_v3 = await aiogoogle.discover("drive", "v3")
        async with TemporaryFile("wb+") as async_buffer:
            await aiogoogle.as_user(
                drive_v3.files.get(fileId=file_id, pipe_to=async_buffer, alt="media")
            )
            await async_buffer.seek(0)
            contents = await async_buffer.read()
    buffer = BytesIO(contents)

I do not guarrante the example working, as I haven't tested this. For all I can say, in my project it works. I think the idea is clear though. I strongly believe there should be examples like this, as one (including me) may not know what to do with async buffered IO.

@omarryhan, what do you think?

[omarryhan]

I think it is fair to say that my programming skills are not on this level yet, so.. I would really love some more explanation 🙂

Oh nevermind, I thought the current API expected a generator. Sorry, was on my phone yesterday all day, so couldn't look in depth into this. Even if it was a generator, I'm not really sure if that would work :)

Great solution! I'm open to adding a better API if you think it would be more efficient and simpler than the current approach. Also, for sure adding an example with the code you shared above would be a good idea.

Thanks!! @Shmookoff

[Shmookoff]

Sorry for long time to answer, had to focus on my own project for a little while.

As I said, I don't think I have the knowledge to think of a new API. However, I really think the current API is complicated to understand, and not very much decoupled.

Developing my project, I also ran into pipe_from parameter. I would think it would work with the same TemporaryFile from aiofiles.tempfile. This doesn't seem to be the case though.

async with Aiogoogle(
    service_account_creds=ServiceAccountCreds(**config.google, scopes=scopes)
) as aiogoogle:
    drive = await aiogoogle.discover("drive", "v3")

    with open("README.md", "rb") as file:
        contents = file.read()
    async with TemporaryFile("wb+") as async_buffer:
        await async_buffer.write(contents)
        await async_buffer.seek(0)

        req = drive.files.create(pipe_from=async_buffer)
        await aiogoogle.as_service_account(req)

Raises: TypeError: object of type 'generator' has no len() Traceback: GitHub Gists

I tried to bypass pipe_from altogether by assigning media_upload after the Request object was instansiated

async with Aiogoogle(
    service_account_creds=ServiceAccountCreds(**config.google, scopes=scopes)
) as aiogoogle:
    drive = await aiogoogle.discover("drive", "v3")

    with open("README.md", "rb") as file:
        contents = file.read()
    req = drive.files.create()
    req.media_upload = MediaUpload(contents)
    await aiogoogle.as_service_account(req)

But, as I quickly realized, it pretty much is imposiible. MediaUpload needs additional data such as upload_path, which can only be recieved from Method.__call__. I think Method.__call__ should be decoupled into small methods with particular function and result such as construct_upload_path. Thus, the API would be more flexible, and allow for (for example) use-case described above.

I'll be adding a PR with examples for pipe_to and pipe_from. I need your help in understanding the pipe_from though.

[Shmookoff]

For now, I found a workaround to my problem.

req = drive.files.create(pipe_from=True)
req.media_upload.file_body = contents
req.media_upload.pipe_from = None

This is, of course, fairly ugly and not intended.

[omarryhan]

For now, I found a workaround to my problem.

req = drive.files.create(pipe_from=True)
req.media_upload.file_body = contents
req.media_upload.pipe_from = None

This is, of course, fairly ugly and not intended.

Nice workaround

And yeah, I agree having Method.__call__ take everything wasn't the best design decision. I'll see if I can think of a better implementation for pipe_to/from and will try to add more examples. Thanks for sharing your efforts!