Boto3 update and create job api calls when adding additional default arguments, the new job parameters do not get reflected in the UI

[portswigger-katie]

Describe the bug

I have a python script that creates and updates glue jobs. If it detects additional arguments ( job parameters) it appends these to the default arguments of the job.

In the AWS console the 'job run input arguments' part of the UI reflects the new job parameters but the 'job detail' is failing to show them in the UI in 'job parameters' under 'Advanced properties'

Expected Behavior

I would expect In the AWS console the 'job run' part of the UI to reflect the new job parameters.

Current Behavior

The UI does not reflect the new job parameters. There are no errors as the api calls are succeeding.

Reproduction Steps

    job_name: str,
    role: str,
    script_location: str,
    temp_dir: str,
    extra_py_files: str,
    config: dict,
    glue_client,
) -> dict[str, Any]:
    """Create or update an AWS Glue job."""
    print(config.get("library-set"))

    # Initialize the common job_params dictionary
    job_params = {
        "Description": config.get("job_description", "Job created by GitHub Actions"),
        "Role": role,
        "ExecutionProperty": {
            "MaxConcurrentRuns": config.get("max_concurrent_runs", 1)
        },
        "Command": {
            "Name": config.get("command_name", "glueetl"),
            "ScriptLocation": script_location,
            "PythonVersion": config.get("python_version", "3"),
        },
        "DefaultArguments": {
            "--extra-py-files": extra_py_files,
            "--enable-job-insights": str(
                config.get("enable_job_insights", True)
            ).lower(),
            "--job-language": "python",
            "--TempDir": temp_dir,
            "--enable-auto-scaling": str(
                config.get("enable_auto_scaling", True)
            ).lower(),
            "--enable-glue-datacatalog": str(
                config.get("enable-glue-datacatalog", False)
            ).lower(),
        },
        "MaxRetries": config.get("max_retries", 0),
        "Timeout": config.get("timeout_minutes", 120),
        "NumberOfWorkers": int(config.get("number_of_workers", 10)),
        "GlueVersion": str(config.get("glue_version", "4.0")),
        "WorkerType": config.get("worker_type", "G.1X"),
        "ExecutionClass": config.get("execution_class", "STANDARD"),
    }

    # Add library-set if it exists in the config
    if config.get("library-set"):
        job_params["DefaultArguments"]["library-set"] = str(
            config["library-set"]
        ).lower()

    connections = config.get("connections")
    if connections:
        job_params["Connections"] = {"Connections": connections}

    try:
        glue_client.get_job(JobName=job_name)
        logger.info(f"Glue job '{job_name}' exists. Updating it.")
        glue_client.update_job(JobName=job_name, JobUpdate=job_params)
        logger.info(f"Glue job '{job_name}' updated successfully.")

    except glue_client.exceptions.EntityNotFoundException:
        logger.info(f"Glue job '{job_name}' does not exist. Creating it.")
        glue_client.create_job(Name=job_name, **job_params)
        logger.info(f"Glue job '{job_name}' created successfully.")
    except Exception as e:
        logger.error(f"Error creating or updating Glue job '{job_name}': {e}")
        sys.exit(1)

Possible Solution

When theses api calls are made in addition to updating the 'Job run input arguments' also update the 'Job parameters' in the job detail section of the UI.

Additional Information/Context

boto3 aws-glue-sessions aws-glue-libs @ git+https://github.com/awslabs/[email protected] botocore

SDK version used

boto3 1.35.16

Environment details (OS name and version, etc.)

Python 3.11.4 on MACOS 14.6.1