nsxt_upgrade_upload_mub with "file" does not work because of size limit

[lvlie]

Describe the bug

When trying to upload the mub file from the ansible controller, using the following task:

- name: Upload upgrade file to NSX-T manager coordinator node from file
  vmware.ansible_for_nsxt.nsxt_upgrade_upload_mub:
    hostname: "{{ coordinator }}"
    username: "{{ username }}"
    password: "{{ password }}"
    validate_certs: False
    file: "{{ nsx_upgrade_mub_file }}"

The task will fail with the following error message:

fatal: [nsxt]: FAILED! => {
    "changed": true,
    "invocation": {
        "module_args": {
            "file": "***/VMware-NSX-upgrade-bundle-3.1.3.5.0.19068434.mub",
            "hostname": "***",
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "port": 443,
            "timeout": null,
            "url": null,
            "username": "***",
            "validate_certs": false
        }
    },
    "msg": "Error: string longer than 2147483647 bytes"

Maybe the upgrade bundle was smaller than 2Gb at some point, but now its not anymore. Manually uploading the mub works fine, but getting it from a webserver requires it to run on http only (no TLS) because it seems to fail on our internal CA certs (even when they are added to the normal truststore using the certificates task).

Reproduction steps

1. Install python (latest)
2. Install ansible (latest)
3. Run the nsxt_upgrade_upload_mub with the file parameter

Expected behavior

mub file is uploaded and ready to be used

Additional context

No response

[marcowork]

+1

[B-Lourens]

The 2GB+ filesize is the issue. You can find multiple references to the error, usualy refrring to the httplib, urllib or ssl.. One solution is to use streaming upload.

This is what we did to make the upload work.

Install request-toolbelt package

Edit nsxt_upgrade_upload_mub.py Add:

import requests
from requests_toolbelt.multipart import encoder
from requests.auth import HTTPBasicAuth 

Replace line 140 - 174 with:

        session = requests.Session()
        with open(file_path, 'rb') as src_file:
             body = encoder.MultipartEncoder({
                 "file": (src_file.name, src_file, "application/octet-stream")
             })
             headers = {"Prefer": "respond-async", "Content-Type": body.content_type}
             resp = session.post(mgr_url + endpoint, auth=HTTPBasicAuth(mgr_username, mgr_password), timeout=None, verify=False, data=body, headers=headers)
             bundle_id = 'latest'#resp['bundle_id']
             headers = dict(Accept="application/json")
             headers['Content-Type'] = 'application/json'
             try:
                 wait_for_operation_to_execute(mgr_url,
                     '/upgrade/bundles/%s/upload-status'% bundle_id,
                     mgr_username, mgr_password, validate_certs,
                     ['status'], ['SUCCESS'], ['FAILED'])
             except Exception as err:
                 module.fail_json(msg='Error while uploading upgrade bundle. Error [%s]' % to_native(err))
             module.exit_json(changed=True, ip_address=ip_address,
             message='The upgrade bundle %s got uploaded successfully.' % module.params[mub_type])
        session.close()

NOTE: This will break the URL upload!

The response will show:

changed: [127.0.0.1] => {
    "changed": true,
    "invocation": {
        "module_args": {
            "file": "/upgrade_bundle/VMware-NSX-upgrade-bundle-3.2.1.1.0.20115686.mub",
            "hostname": "<ip>",
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "port": 443,
            "timeout": 9000,
            "url": null,
            "username": "admin",
            "validate_certs": false
        }
    },
    "ip_address": "<ip>",
    "message": "The upgrade bundle /upgrade_bundle/VMware-NSX-upgrade-bundle-3.2.1.1.0.20115686.mub got uploaded successfully."
}