Uploading files >100 MB to the Firebase Storage fails (Firebase Emulator Suite)

[JustinTrvz]

Step 2: Describe your environment

• Operating System version: Linux Ubuntu 22.04.1
• Firebase SDK version: 12.5.4
• Firebase Product: Storage
• Python version: 3.10.12
• Pip version: 22.0.2

Step 3: Describe the problem

I am using the Firebase Emulator Suite - Storage and firebase-admin in Python.

I wand to upload files to the Firebase Emulator Suite - Storage and I already set the system's environment variable STORAGE_EMULATOR_HOST to http://localhost:9199.

When I use the function blob.upload_from_filename(filename=file_path) and I upload a file that is smaller than the _DEFAULT_CHUNKSIZE (100 MB), which is defined in blob.py, everything is fine. But if my file exceeds the chunksize (> 100 MB), the file will not be uploaded correctly.

I tried to upload a 200 MB large file via the Firebase Emulator Suite UI in my browser (http://127.0.0.1:4000/storage/<project-id>) and it worked. So the problem is not the emulator itsself.

These are the files I want to upload, which are larger than 700 MB per file.

This is the UI after uploading two of the files. The maximum file size I can reach is 104.86 MB...

The Firebase Emulator Suite does not throw any error, neither my code does.

Relevant Code:

In the following code I am creating the Firebase app object in the __init__ function and use the storage bucket to upload files using the function upload_from_filename() in my custom function upload_zip_from_path():

def __init__(self, app_name: str = "[DEFAULT]", config_yml_path: str = "config.yml"):
        configs = FileUtils.load_config_yml(config_yml_path)
        self.DEBUG = bool(configs["environment"]["debug"])
        self.DATABASE_URL = configs["database"]["url"]
        self.BUCKET_URL = configs["storage"]["url"]
        token_file_path = configs["database"]["token_file_path"]

        # Set host to local Firebase Emulator Suite Storage
        if self.DEBUG:
            os.environ["STORAGE_EMULATOR_HOST"] = "http://localhost:9199"

        try:
            self.APP_OBJ = firebase_admin.get_app(app_name)
        except ValueError:
            # If app does not exist, a value error is thrown
            if len(token_file_path) > 0:
                # Production: Authentication token needed
                token = credentials.Certificate(token_file_path)
                self.APP_OBJ = firebase_admin.initialize_app(
                    credential=token,
                    options={
                        "databaseURL": self.DATABASE_URL,
                        "storageBucket": self.BUCKET_URL,
                    },
                    name=app_name,
                )
            else:
                # Development: No token needed using Firebase Emulator Suite
                self.APP_OBJ = firebase_admin.initialize_app(
                    options={
                        "databaseURL": self.DATABASE_URL,
                        "storageBucket": self.BUCKET_URL,
                    },
                    name=app_name,
                )
        # References
        self.SID_ROOT = db.reference("sid")
        self.USER_ROOT = db.reference("user")
        # Firebase app specific
        self.APP_NAME = app_name
        self.BUCKET = storage.bucket()


def upload_zip_from_path(self, zip_path:str):
        # Set file name
        if "/" in zip_path:
            zip_name = zip_path.split("/")[-1]  # "test/abc.zip" -> ["test", "abc.zip"] -> "abc.zip"
        else:
            zip_name = zip_path
        # Get a reference to the Firebase Storage bucket
        zip_folder = self.BUCKET.blob(f"zip/{zip_name}")
        zip_folder.upload_from_filename(zip_path)
        return zip_folder.public_url

On the other hand I tried to chunk the file by myself, but this did not went well. With the following code the current chunk will overwrite the last chunk - this is not wanted... I wanted to append it to the file.

def resumable_upload(self, file_path, chunk_size=25 * 1024 * 1024):
        # Set file name
        if "/" in file_path:
            destination_blob_name = file_path.split("/")[-1]  # "test/abc.zip" -> ["test", "abc.zip"] -> "abc.zip"
        else:
            destination_blob_name = file_path

        # Get a reference to the Firebase Storage bucket
        bucket = storage.bucket()

        # Create a resumable upload session
        blob = bucket.blob(destination_blob_name)
        blob.chunk_size = chunk_size
        chunk_count = 0
        chunk_start = 0
        total_size = os.path.getsize(file_path)

        with open(file_path, 'rb') as file:
            while chunk_start < total_size:
                chunk_end = min(chunk_start + chunk_size, total_size)
                chunk_data = file.read(chunk_size)
                blob.upload_from_string(chunk_data, content_type="application/zip")
                
                chunk_start = chunk_end
                chunk_count += 1
                print(f'Uploaded chunk {chunk_count} / {total_size // chunk_size} ({chunk_start}/{total_size} bytes)')

        print('Resumable upload completed.')
        return blob.public_url

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[mrkrosenberg]

Did anyone find a solution to this?

[mcelroyengineering]

Any updates on this? I'm running into something similar Update: Fortunately I was able to bring my file sizes under the limit by compressing them, but I'd still be interested hearing solutions for cases with larger files

[Judimax]

Greetings figured it out it set by an internal variable

what you have to do

    blob = self.bucket.blob(blob_name)
///
    blob.chunk_size =int(Mib(256*8).bytes) (increase the size)
///
    blob.upload_from_string(file_string, content_type=content_type)

on digging in firebase admin emulator does not do resumable uploads mabye there is a variable somehwere to enable uploads but your best bet is to make a dev enviromnent and never use the emulator :(

I redacted the payload the response from the emulator should have been 308