ion.load throws The encoded data was not valid for encoding utf-8

[ori-maci]

I am currently working with QLDB -> Kinesis stream -> AWS lambda (node.js 18+) and I am encountering the following when loading an ion record in the lambda:

"errorType": "Runtime.UnhandledPromiseRejection",
    "errorMessage": "TypeError: The encoded data was not valid for encoding utf-8",
    "reason": {
        "errorType": "TypeError",
        "errorMessage": "The encoded data was not valid for encoding utf-8",
        "code": "ERR_ENCODING_INVALID_ENCODED_DATA",

Package versions are:

    "ion-js": "^4.3.0",
    "jsbi": "^4.3.0"

Here is an example code to reproduce this issue:

import { load } from 'ion-js';

const data = "84mawgpANDk2ZTVmYzM4NmUwZjU3ZTQ1MmFjM2U0OWI0NWE1NDUwYjU5MGQ1OWI1ZjI2Mzg3MDQyYzBhMTEzNzVhYTQ4MQpANDljNmM0MzE2Y2Q2YzlkNjM0Zjk0N2FkNzA1NjVkYjYxZmU0MmE4ODI1ZmU1MmYxMWM2YzVjYTYwNmE0ZDc0ZhrhCAgAGtwI4AEA6u4Cu4GD3gK2h74Cso1xbGRiU3RyZWFtQXJuinJlY29yZFR5cGWHcGF5bG9hZIxibG9ja0FkZHJlc3OIc3RyYW5kSWSKc2VxdWVuY2VOb410cmFuc2FjdGlvbklkjo5ibG9ja1RpbWVzdGFtcIlibG9ja0hhc2iLZW50cmllc0hhc2iOkXByZXZpb3VzQmxvY2tIYXNojo9lbnRyaWVzSGFzaExpc3SOj3RyYW5zYWN0aW9uSW5mb4pzdGF0ZW1lbnRziXN0YXRlbWVudIlzdGFydFRpbWWOj3N0YXRlbWVudERpZ2VzdIlkb2N1bWVudHOOlkFkM0FXYlpQNnp1OUhFVGROZzZwcEGJdGFibGVOYW1lh3RhYmxlSWSOkXJldmlzaW9uU3VtbWFyaWVzhGhhc2iKZG9jdW1lbnRJZN4Gl4qOzGFybjphd3M6cWxkYjp1cy1lYXN0LTE6MDEzMjQyMTkwMTg0OnN0cmVhbS9zdGctT3JkZXJzL0tyVTVjVXpkVzM1S0hQWkVjM043STSLjUJMT0NLX1NVTU1BUlmM3gW1jd6djo6WMHJFTjFheHM3UWpKQ25KWXVQQzVSYo8iHguQjpYydXl4WjBZOHk3NDRuT09ydURodWJqkWuAD+eEj5O4osMAhJKuoEluX8OG4PV+RSrD5JtFpUULWQ1ZtfJjhwQsChE3WqSBk66gqfuETMGDyOUE0M6bw5Xy5LZrR6g/Pg3yw7rLHAgTNvmUrqDEt5D5RfCLqxqoN9aYlp53vM43fzQ11k8jdmeUO3fbuJW+AYiuoCIpiZTEUHJR54AxsdM/V6dZXNbY/3behh9NlifFnzxVrqBJxsQxbNbJ1jT5R61wVl22H+QqiCX+UvEcbFymBqTXT66gNM1SpPLZqpH3L2W+RiST6lCt/1CXHpwnu09d7WnTWzSuoEncf6mN7E2qxjhSlH0vwaEgcI3+osU2iA0O5mJ8tDjvlt4CtZe+Aobe2JiOplNFTEVDVCAqIGZyb20gT3JkZXJzIFdIRVJFIG9yZGVySWQgPSA/mWqAD+eEj5O4osMtmq6gJCAdJXhOXYA4eNcQSo7MnuvZHel3OYTzj2e9tSmqptreAamYjvdVUERBVEUgT3JkZXJzIGFzIHQgQlkgaWQgU0VUIHQub3JkZXJTdGF0dXMgPSA/LCB0LnVwZGF0ZWRBdCA9ID8sIHQudXBkYXRlZEJ5ID0gPywgdC5sbXNEYWlseUV2ZW50ID0gPyBXSEVSRSBvcmRlcklkID0gP5lqgA/nhI+TuKLDXJquoL2d2K96cDxjFa3q4DjahDo322blvdjH8Y4p+zzSgFqEm96onN6lnYZPcmRlcnOejpYwbHVMVzBqOXVlR0Vtemp3OTZKWVBsl7IhAZ++vt68oK6gScbEMWzWydY0+UetcFZdth/kKogl/lLxHGxcpgak10+hjpZBZDNBV2JaUDZ6dTlIRVRkTmc2cHBBGsgJCAEawwngAQDq7gPngYPeA+KHvgPejXFsZGJTdHJlYW1Bcm6KcmVjb3JkVHlwZYdwYXlsb2FkiXRhYmxlSW5mb4l0YWJsZU5hbWWHdGFibGVJZIhyZXZpc2lvboxibG9ja0FkZHJlc3OIc3RyYW5kSWSKc2VxdWVuY2VOb4RoYXNohGRhdGGHb3JkZXJJZIlwcm9ncmFtSWSOjm9yZ2FuaXphdGlvbklkjpZsb2FuWWVhcmx5RHVyYXRpb25EYXlzjo5leHRlcm5hbExvYW5JZItvcmRlckFjdGlvbo6TZnVuZGluZ1N1YnNpZGlhcnlJZIxjdXN0b21lck5hbWWOkGludGVyZXN0UmF0ZVR5cGWOkGZsb2F0aW5nUmF0ZVR5cGWHZmVlVHlwZYp0ZXJtRGF0ZUF0hW5vdGVzhmFtb3VudIxpbnRlcmVzdFJhdGWJZmVlQW1vdW50jo53cml0ZU9mZkFtb3VudI6Tb3duZXJPcmdhbml6YXRpb25JZIljcmVhdGVkQnmJY3JlYXRlZEF0gmlki29yZGVyU3RhdHVziXVwZGF0ZWRBdIl1cGRhdGVkQnmNbG1zRGFpbHlFdmVudIlldmVudFR5cGWOj2FjY3J1ZWRJbnRlcmVzdIhtZXRhZGF0YYZ0eFRpbWWEdHhJZN4F0oqOzGFybjphd3M6cWxkYjp1cy1lYXN0LTE6MDEzMjQyMTkwMTg0OnN0cmVhbS9zdGctT3JkZXJzL0tyVTVjVXpkVzM1S0hQWkVjM043STSLjpBSRVZJU0lPTl9ERVRBSUxTjN4E7I3eoY6GT3JkZXJzj46WMGx1TFcwajl1ZUdFbXpqdzk2SllQbJDeBMSR3p2SjpYwckVOMWF4czdRakpDbkpZdVBDNVJikyIeC5SuoEnGxDFs1snWNPlHrXBWXbYf5CqIJf5S8RxsXKYGpNdPld4DuJaOpGEwNTlmN2M3LWMyMGYtNDk3OS05MWIwLTc5NmE0MjBlMThjNpeOpGRkNzQzYjFkLTAyYmEtNGM1Ny1iMmRkLTE0ZTM4ZmU1YTg1YpiOpDY5MGZhNDlmLWU5ZDUtNDhjNC1iNDliLWQ3MDFlN2E0YmU2YpkiAWiaijg1NzY1NDYyNjibIQGcjqRhNDRiZjliYy1jYzU0LTRiZTUtYjlkZi1kOTAxYzQ5MDM4MDGdikJyYXZvIFRpbGWeIJ8goIChgKKAoyMBhqCkSEAVAAAAAAAApSCmIKeOpDYwOGFjZTI0LWY1MGItNDdmYy04MzdkLWE4NDlmNWI4MzFhZKiOlmFkbWluQGFyY2FkaWFmdW5kcy5jb22pjpgyMDIzLTA0LTEzVDIwOjAxOjM0LjcwOFqqjpZBZDNBV2JaUDZ6dTlIRVRkTmc2cHBBqyECrI6YMjAyMy0wNC0xNVQxOTo1NjozNS4zNjFarY6Wb21hY2lAYXJjYWRpYWZ1bmRzLmNvba7esq8hA6mOmDIwMjMtMDQtMTVUMDQ6MDA6MDAuMDAwWrBIQC0qqqqqqqukSEAVAAAAAAAAsd7Cqo6WQWQzQVdiWlA2enU5SEVUZE5nNnBwQYUhLbJrgA/nhI+TuKLDAIKzjpYydXl4WjBZOHk3NDRuT09ydURodWJq5Zv539x9CYjfG+BB7dduxQ==";

const res = Buffer.from(data, 'base64')

const ionRecord = load(res); // Error is throw here

This issue happens intermittently as well. Any help or insight would be greatly appreciated.

[popematt]

Your base64 data does not start with the Ion Version Marker (IVM), so your data is not Ion binary format. Therefore, the load function assumes that you are trying to provide UTF-8 encoded Ion text, which is why you get a UTF-8 encoding error.

That being said, there is an IVM that occurs after the first 144 bytes. After dropping the first 144 bytes, you can see your Ion data by running:

echo "4AEA ..." | base64 -d | ion dump

I do not know how it is that you ended up with non-Ion data around the Ion data, but hopefully this information helps you find out where the problem is introduced.

[popematt]

The non-Ion data is probably being added as a wrapper around your Ion data in one of the steps between QLDB and your application, not by QLDB itself. I would start by looking at whether Kinesis and/or Lambda adds any metadata or wrappers to event data.