boto3 icon indicating copy to clipboard operation
boto3 copied to clipboard
verified publisher icon boto

execute_statement will result in an error if you execute an SQL statement that returns an array that contains null.

Open s-isobe opened this issue 9 months ago • 4 comments

Describe the bug

I am running execute_statement against RDS (postgresql).

When retrieving array type data, an error occurs if the array contains null.

Regression Issue

  • [ ] Select this option if this issue appears to be a regression.

Expected Behavior

The expected response is:

{'ResponseMetadata': {'RequestId': '29fe6071-8b56-4717-8db6-8bbe2ea79de2', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '29fe6071-8b56-4717-8db6-8bbe2ea79de2', 'date': 'Fri, 18 Apr 2025 01:09:48 GMT', 'content-type': 'application/json', 'content-length': '155', 'connection': 'keep-alive'}, 'RetryAttempts': 0}, 'records': [[{'arrayValue': {'longValues': [123, null, 456]}}]], 'numberOfRecordsUpdated': 0}

Current Behavior

I am getting the below error.

boto3 version: 1.35.76
Traceback (most recent call last):
  File "/home/s.isobe/workspace/python_test/rds_test.py", line 13, in <module>
    recs = rds_client.execute_statement(
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/client.py", line 1005, in _make_api_call
    http, parsed_response = self._make_request(
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/client.py", line 1029, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/endpoint.py", line 119, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/endpoint.py", line 197, in _send_request
    success_response, exception = self._get_response(
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/endpoint.py", line 239, in _get_response
    success_response, exception = self._do_get_response(
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/endpoint.py", line 313, in _do_get_response
    parsed_response = parser.parse(
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 253, in parse
    parsed = self._do_parse(response, shape)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 894, in _do_parse
    self._add_modeled_parse(response, shape, final_parsed)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 904, in _add_modeled_parse
    self._parse_payload(response, shape, member_shapes, final_parsed)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 946, in _parse_payload
    body_parsed = self._parse_shape(shape, original_parsed)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 333, in _parse_shape
    return handler(shape, node)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 665, in _handle_structure
    final_parsed[member_name] = self._parse_shape(
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 333, in _parse_shape
    return handler(shape, node)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 1005, in _handle_list
    return super()._handle_list(shape, node)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 341, in _handle_list
    parsed.append(self._parse_shape(member_shape, item))
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 333, in _parse_shape
    return handler(shape, node)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 1005, in _handle_list
    return super()._handle_list(shape, node)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 341, in _handle_list
    parsed.append(self._parse_shape(member_shape, item))
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 333, in _parse_shape
    return handler(shape, node)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 665, in _handle_structure
    final_parsed[member_name] = self._parse_shape(
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 333, in _parse_shape
    return handler(shape, node)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 665, in _handle_structure
    final_parsed[member_name] = self._parse_shape(
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 333, in _parse_shape
    return handler(shape, node)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 1005, in _handle_list
    return super()._handle_list(shape, node)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 341, in _handle_list
    parsed.append(self._parse_shape(member_shape, item))
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 333, in _parse_shape
    return handler(shape, node)
  File "/home/s.isobe/workspace/python_test/venv/lib64/python3.9/site-packages/botocore/parsers.py", line 1033, in _handle_integer
    return int(value)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Reproduction Steps

If you run a script like the following, you will get an error.

import boto3
import botocore
import json

print(f'boto3 version: {boto3.__version__}')

rds_client = boto3.client('rds-data')

sql = '''
select array[123::int4, null::int4, 456::int4];
'''

recs = rds_client.execute_statement(
    secretArn = 'secretsmanager_arn',
    resourceArn = 'resource_arn',
    database = 'mydb',
    sql = sql
)

print(f'{recs}')

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.35.76

Environment details (OS name and version, etc.)

AlmaLinux release 9.5 (Teal Serval) 5.14.0-503.35.1.el9_5.x86_64

s-isobe avatar Apr 18 '25 01:04 s-isobe

Based on the contents of the stack trace, we changed the problem area and ran the program, which produced the expected results.

Is this change a difficult conversation?

    def _handle_integer(self, shape, value):
        if value is None:
            return None
        else:
            return int(value)

s-isobe avatar Apr 21 '25 01:04 s-isobe

The following worked without any problems:

select array[123::text, null::text]

response:

['123', None]

For now, when extracting the array, cast it with ::text.

s-isobe avatar Apr 21 '25 02:04 s-isobe

Hello @s-isobe, thanks for reaching out. I have replicated the issue and got the same error. I will discuss to the team for the change provided and I will let you know if we have any updates. Thank you.

adev-code avatar May 15 '25 18:05 adev-code

Hello @s-isobe, thanks for the patience. I have reached out to the RDSDataService in relation to their service model that the response returns a null as Boto is not expecting a null and I also reached out for a fix. I will update as soon as I get any response from the team. If you have any questions, please let me know. Thanks.

For Internal Tracking: P258881794

adev-code avatar Jun 23 '25 17:06 adev-code