Python: URI error when downloading a file that was just uploaded

[sarg3nt]

We have one function that uploads a file and then another that loops through the vault to get all files and concatenate them. When it gets to the file that was just uploaded it fails with the following error:

requests.exceptions.MissingSchema: Invalid URL 'None': No scheme supplied. Perhaps you meant https://None?

Adding a sleep(3) after the file upload fixes it but this has a bad code smell and we don't know if it will always be stable.

Is this a bug? Is there a better way to make sure the file is fully uploaded? Seems to me that secrets_manager.upload_file(secret, file) should not return until the file is fully available in the vault.

I've found a fix that will get us by for the moment in case someone else comes across this but I'd still like to know if this is expected behavior or not. The fix is to add a for loop with a try, except in it to keep trying to open the file and download the bytes, see code below. Note this for loop must be around the call to get_secret_by_title. Putting it around the get_file_data does not fix it.

total_loops: int = 30
for i in range(total_loops):
    secret = self.secrets_manager.get_secret_by_title(file_title)
    file_data: bytes = None
    if secret is None:
        logger.error(
            f"Could not find the '{file_title}' file in the secrets store.",
        )
        return None

    # Get the last file in the file list, this should be the most recent if there are more than one.
    if secret.files:
        file = secret.files[-1]
        if not file:
            logger.warning("File not found.")
            return None

        logger.debug(f"File: {file}")

        try:
            file_data = file.get_file_data()
            logger.trace("File Data:")
            logger.trace(file_data)
            break
        except MissingSchema:
            logger.warning(
                f"Failed getting file data, trying again. {i + 1} of {total_loops}",
            )
        except Exception as ex:
            logger.warning(
                f"Exception {type(ex).__name__} ocurred connecting getting file data.",
            )

        sleep(2)
        continue

    logger.warning("No files found.")
    return None

if file_data:
    logger.debug("File data found, returning its bytes.")
    return file_data

logger.warning("No file data found.")
return None

Some of the code to upload the file

# get the first secret matching the record title
secret = self.secrets_manager.get_secret_by_title(file_title)

if secret is None:
    logger.debug("Creating a new record.")
    record = RecordCreate("file", file_title)
    record_uid = self.secrets_manager.create_secret(folder_uid, record)
    logger.debug(record_uid)

    # Get an individual secret by UID to attach the file to
    secret = self.secrets_manager.get_secrets([record_uid])[0]
elif len(secret.files) > 0:
    logger.warning(f"Found existing record that contains files: '{secret}'")
    for file in secret.files:
        logger.warning(f"{file}")

# Prepare file data for upload
file = KeeperFileUpload.from_file(file_path, file_name)

# Upload file attached to the owner record and get the file UID
file_uid = self.secrets_manager.upload_file(secret, file)

[idimov-keeper]

We are working on a fix which will add new timeout parameter to secrets_manager.upload_file(secret, file, timeout) to wait until file upload completes or timeout expires. The fix depends on some changes to the KSM back-end to return the file upload status (even with a valid URL file may still be incomplete even with non-zero byte count) and then we can start adding new upload file functionality to all KSM SDKs.

[maksimu]

This is expected behavior due to backend eventual consistency. When upload_file() completes, there's a brief processing delay (typically 1-3 seconds) before the file's download URL becomes available. Your retry loop solution is the correct approach to handle this at this time. We'll consider adding a wait_until_ready parameter to upload_file() in a future release to handle this automatically. Closing this as working as designed.