states:GetExecutionHistory fails in ruby3.4 Lambda environment

[amancevice]

Describe the bug

In the Lambda ruby3.4 runtime using the latest version of the aws-sdk-states gem, the states:GetExecutionHistory encounters a JSON parsing error if the execution history contains unicode characters.

Regression Issue

• [ ] Select this option if this issue appears to be a regression.

Expected Behavior

Aws::States::Client.new.get_execution_history(execution_arn: 'arn:aws...') should succeed

Current Behavior

When a Step Function state machine's execution history contains an emdash (Unicode U+2014) the Ruby SDK throws an error parsing the JSON response.

/var/lang/lib/ruby/gems/3.4.0/gems/json-2.10.2/lib/json/common.rb:248:in 'String#encode': "\xE2" on US-ASCII (Encoding::InvalidByteSequenceError)

Reproduction Steps

1. Create a state machine that contains an emdash (—) character in its output or variable assignment
2. Create an execution for the state machine and take note of the execution ARN
3. Open Lambda ruby3.4 runtime shell: docker run --rm -it --entrypoint=bash public.ecr.aws/lambda/ruby
4. Install the states gem: gem install aws-sdk-states
5. Open an IRB session: irb
6. Try to get the execution history: Aws::States::Client.new.get_execution_history(execution_arn: 'arn:aws:...')
7. See error

Possible Solution

No response

Additional Information/Context

No response

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-states

Environment details (Version of Ruby, OS environment)

Lambda ruby3.4

[mullermp]

Thanks for opening an issue. The error appears to be in the JSON gem. Do you have a full SDK stack trace? You may want to consider opening an issue with the json maintainers. You can also try adding gem oj to your gemfile and use it as your preferred parser and see if that works.

[amancevice]

@mullermp thanks that makes sense. I wonder if this is instead a bug to file with the Lambda maintainers — maybe the ruby3.4 runtime needs to be updated to include the json and bigdecimal libraries (aws-sdk-dynamodb depends on bigdecimal). Is there a good place for me to file a ticket for them?

[mullermp]

I'm not sure if it's a lambda issue either. What part of the code is calling .encode()? Can we try to get a root cause of why that can't be parsed with the json gem?

[amancevice]

Here's a sample SAM template that can replicate the issue (forgive the overly-permissive IAM policy):

# template.yml
---
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31
Description: AWS Ruby SDK Test
Resources:
  Function:
    Type: AWS::Serverless::Function
    Properties:
      Architectures:
        - arm64
      CodeUri: .
      Description: Test aws-sdk-states
      Environment:
        Variables:
          STATE_MACHINE_ARN: !Ref StateMachine
      Handler: index.handler
      Policies:
        - Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Action: states:*
              Resource: "*"
      Runtime: ruby3.4
  StateMachine:
    Type: AWS::Serverless::StateMachine
    Properties:
      Definition:
        QueryLanguage: JSONata
        StartAt: Pass
        States:
          Pass:
            Type: Pass
            End: true
            Output:
              utf8: — # emdash character

And here's my Gemfile + index.rb:

# Gemfile
source 'https://rubygems.org'

gem 'aws-sdk-states'

# index.rb
require 'aws-sdk-states'

STATES = Aws::States::Client.new
STATE_MACHINE_ARN = ENV['STATE_MACHINE_ARN']

def handler(event:, context:)
  execution_arn = STATES.list_executions(state_machine_arn: STATE_MACHINE_ARN).executions.first.execution_arn
  STATES.get_execution_history(execution_arn:)
end

And here is the result of invoking the Lambda:

{
  "errorMessage": "\"\\xE2\" on US-ASCII",
  "errorType": "Function<Encoding::InvalidByteSequenceError>",
  "stackTrace": [
    "/var/lang/lib/ruby/3.4.0/json/common.rb:221:in 'String#encode'",
    "/var/lang/lib/ruby/3.4.0/json/common.rb:221:in 'JSON::Ext::Parser.parse'",
    "/var/lang/lib/ruby/3.4.0/json/common.rb:221:in 'JSON.parse'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/json/json_engine.rb:10:in 'Aws::Json::JsonEngine.load'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/json.rb:43:in 'Aws::Json.load'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/json/parser.rb:22:in 'Aws::Json::Parser#parse'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/json/handler.rb:65:in 'Aws::Json::Handler#parse_body'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/json/handler.rb:37:in 'Aws::Json::Handler#parse_response'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/json/handler.rb:14:in 'block in Aws::Json::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/response.rb:52:in 'block in Seahorse::Client::Response#on'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/http/response.rb:146:in 'block in Seahorse::Client::Http::Response#on_success'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/http/response.rb:173:in 'block in Seahorse::Client::Http::Response#listener'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/http/response.rb:137:in 'Seahorse::Client::Http::Response#on_done'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/http/response.rb:144:in 'Seahorse::Client::Http::Response#on_success'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/response.rb:51:in 'Seahorse::Client::Response#on'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/json/handler.rb:14:in 'Aws::Json::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/recursion_detection.rb:18:in 'Aws::Plugins::RecursionDetection::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-states-1.87.0/lib/aws-sdk-states/plugins/endpoints.rb:43:in 'block in Aws::States::Plugins::Endpoints::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/user_agent.rb:69:in 'Aws::Plugins::UserAgent.metric'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-states-1.87.0/lib/aws-sdk-states/plugins/endpoints.rb:57:in 'Aws::States::Plugins::Endpoints::Handler#with_metrics'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-states-1.87.0/lib/aws-sdk-states/plugins/endpoints.rb:43:in 'Aws::States::Plugins::Endpoints::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/endpoint_discovery.rb:84:in 'Aws::Plugins::EndpointDiscovery::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/endpoint.rb:46:in 'Seahorse::Client::Plugins::Endpoint::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/param_validator.rb:26:in 'Aws::Plugins::ParamValidator::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/raise_response_errors.rb:16:in 'Seahorse::Client::Plugins::RaiseResponseErrors::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:169:in 'Aws::Plugins::ChecksumAlgorithm::OptionHandler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in 'Aws::Plugins::JsonvalueConverter::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/invocation_id.rb:16:in 'Aws::Plugins::InvocationId::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in 'Aws::Plugins::IdempotencyToken::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in 'Aws::Plugins::ParamConverter::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/request_callback.rb:89:in 'Seahorse::Client::Plugins::RequestCallback::OptionHandler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in 'Aws::Plugins::ResponsePaging::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/plugins/response_target.rb:24:in 'Seahorse::Client::Plugins::ResponseTarget::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in 'block in Aws::Plugins::Telemetry::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/telemetry/no_op.rb:29:in 'Aws::Telemetry::NoOpTracer#in_span'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:53:in 'Aws::Plugins::Telemetry::Handler#span_wrapper'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in 'Aws::Plugins::Telemetry::Handler#call'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.219.0/lib/seahorse/client/request.rb:72:in 'Seahorse::Client::Request#send_request'",
    "/var/runtime/ruby/3.4.0/gems/aws-sdk-states-1.87.0/lib/aws-sdk-states/client.rb:1721:in 'Aws::States::Client#get_execution_history'",
    "/var/task/index.rb:8:in 'Object#handler'",
    "/var/runtime/gems/aws_lambda_ric-3.0.0/lib/aws_lambda_ric/lambda_handler.rb:28:in 'LambdaHandler#call_handler'",
    "/var/runtime/gems/aws_lambda_ric-3.0.0/lib/aws_lambda_ric.rb:86:in 'AwsLambdaRIC::LambdaRunner#run_user_code'",
    "/var/runtime/gems/aws_lambda_ric-3.0.0/lib/aws_lambda_ric.rb:64:in 'AwsLambdaRIC::LambdaRunner#start_runtime_loop'",
    "/var/runtime/gems/aws_lambda_ric-3.0.0/lib/aws_lambda_ric.rb:47:in 'AwsLambdaRIC::LambdaRunner#run'",
    "/var/runtime/gems/aws_lambda_ric-3.0.0/lib/aws_lambda_ric.rb:219:in 'AwsLambdaRIC::Bootstrap#bootstrap_handler'",
    "/var/runtime/gems/aws_lambda_ric-3.0.0/lib/aws_lambda_ric.rb:201:in 'AwsLambdaRIC::Bootstrap#start'",
    "/var/runtime/index.rb:4:in '<main>'"
  ]
}

[jterapin]

This issue looks very familiar to what I encountered last year with the json gem. See: https://github.com/ruby/json/issues/697

I will test with your reproduction to see if I can replicate the error. In the mean time, have you tried using a different json engine like oj as mullermp mentioned? To see if the similar error surfaces?

[amancevice]

~~I've tried but I'm doing something wrong with SAM — Lambda returns cannot load such file -- oj~~

After adding the oj gem to the Gemfile & building with sam build --use-container I was able to get it to work. Still, I think it's suboptimal for the default Ruby v3.4 runtime for Lambda to have trouble executing AWS SDK ops out of the box

[jterapin]

Hi @amancevice - I have relayed your feedback to the team that works on the Ruby Lambda. They are now made aware of this and have it on their backlog to investigate. I will circle back when I hear back from them.

[joshuay03]

We (@buildkite) ran into this when calling Aws::SSM::Client#describe_parameters in a ruby3.4 lambda:

{
    "errorMessage": "\"\\xE2\" on US-ASCII",
    "errorType": "Function<Encoding::InvalidByteSequenceError>",
    "stackTrace": [
        "/var/lang/lib/ruby/3.4.0/json/common.rb:221:in 'String#encode'",
        "/var/lang/lib/ruby/3.4.0/json/common.rb:221:in 'JSON::Ext::Parser.parse'",
        "/var/lang/lib/ruby/3.4.0/json/common.rb:221:in 'JSON.parse'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/json/json_engine.rb:10:in 'Aws::Json::JsonEngine.load'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/json.rb:43:in 'Aws::Json.load'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/json/parser.rb:22:in 'Aws::Json::Parser#parse'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/json/handler.rb:65:in 'Aws::Json::Handler#parse_body'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/json/handler.rb:37:in 'Aws::Json::Handler#parse_response'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/json/handler.rb:14:in 'block in Aws::Json::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/seahorse/client/response.rb:52:in 'block in Seahorse::Client::Response#on'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/seahorse/client/http/response.rb:146:in 'block in Seahorse::Client::Http::Response#on_success'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/seahorse/client/http/response.rb:173:in 'block in Seahorse::Client::Http::Response#listener'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/seahorse/client/http/response.rb:137:in 'Seahorse::Client::Http::Response#on_done'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/seahorse/client/http/response.rb:144:in 'Seahorse::Client::Http::Response#on_success'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/seahorse/client/response.rb:51:in 'Seahorse::Client::Response#on'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/json/handler.rb:14:in 'Aws::Json::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/plugins/recursion_detection.rb:18:in 'Aws::Plugins::RecursionDetection::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-ssm-1.202.0/lib/aws-sdk-ssm/plugins/endpoints.rb:43:in 'block in Aws::SSM::Plugins::Endpoints::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/plugins/user_agent.rb:90:in 'Aws::Plugins::UserAgent.metric'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-ssm-1.202.0/lib/aws-sdk-ssm/plugins/endpoints.rb:57:in 'Aws::SSM::Plugins::Endpoints::Handler#with_metrics'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-ssm-1.202.0/lib/aws-sdk-ssm/plugins/endpoints.rb:43:in 'Aws::SSM::Plugins::Endpoints::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/plugins/endpoint_discovery.rb:84:in 'Aws::Plugins::EndpointDiscovery::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/seahorse/client/plugins/endpoint.rb:46:in 'Seahorse::Client::Plugins::Endpoint::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/plugins/param_validator.rb:26:in 'Aws::Plugins::ParamValidator::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/seahorse/client/plugins/raise_response_errors.rb:16:in 'Seahorse::Client::Plugins::RaiseResponseErrors::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:169:in 'Aws::Plugins::ChecksumAlgorithm::OptionHandler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in 'Aws::Plugins::JsonvalueConverter::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/plugins/invocation_id.rb:16:in 'Aws::Plugins::InvocationId::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in 'Aws::Plugins::IdempotencyToken::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in 'Aws::Plugins::ParamConverter::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/seahorse/client/plugins/request_callback.rb:89:in 'Seahorse::Client::Plugins::RequestCallback::OptionHandler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in 'Aws::Plugins::ResponsePaging::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/seahorse/client/plugins/response_target.rb:24:in 'Seahorse::Client::Plugins::ResponseTarget::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in 'block in Aws::Plugins::Telemetry::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/telemetry/no_op.rb:29:in 'Aws::Telemetry::NoOpTracer#in_span'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/plugins/telemetry.rb:53:in 'Aws::Plugins::Telemetry::Handler#span_wrapper'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/aws-sdk-core/plugins/telemetry.rb:39:in 'Aws::Plugins::Telemetry::Handler#call'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-core-3.232.0/lib/seahorse/client/request.rb:72:in 'Seahorse::Client::Request#send_request'",
        "/var/runtime/ruby/3.4.0/gems/aws-sdk-ssm-1.202.0/lib/aws-sdk-ssm/client.rb:5368:in 'Aws::SSM::Client#describe_parameters'",
        ...
    ]
}

Looks like it's incorrectly attempting to parse an ASCII response.

[jterapin]

Hi! Have you tried updating the json gem to the latest (in your Gemfile) to see if the same issue occurs? I'm thinking that the Ruby runtime 3.4 shipped with an older version of that gem. Alternative, you can use oj gem to the Gemfile to use a different JSON parser.

[joshuay03]

Hi! Have you tried updating the json gem to the latest (in your Gemfile) to see if the same issue occurs? I'm thinking that the Ruby runtime 3.4 shipped with an older version of that gem.

I checked in a new ruby3.4 lambda and the json version used is 2.9.1, which is what is bundled with Ruby 3.4.0 to 3.4.6 (latest): https://stdgems.org/json. All our lambdas are dependency-free (with the exception of the aws gems of course) and don't currently have a Gemfile, and we'd prefer to keep it that way. My understanding is that this is a valid encoding error, meaning the aws-sdk-core gem should be forcing UTF-8 before attempting to parse the JSON response. I also don't see anything in the changelogs for all the json versions that follow that would indicate that this is a bug: https://github.com/ruby/json/releases

Alternative, you can use oj gem to the Gemfile to use a different JSON parser.

Unfortunately, introducing another dependency isn't an option for us at this stage, we're pretty much blocked from upgrading the runtime until this has been resolved.

[mullermp]

Can you check the output of Encoding.default_external on lambda? Is it ASCII?

[amancevice]

@mullermp it's "US-ASCII"

[mullermp]

Can you set it to UTF-8 before your function runs? That should make it work. Changing the parsing behavior by forcing everything to utf8 may be a breaking change.

[amancevice]

Will test this but for the record a ruby3.3 Lambda function also reports "US-ASCII". My 2¢ is that the AWS Ruby SDK should "just work" in the bare Lambda ruby3.4 runtime without having to install any additional gems or changing the environment.

[mullermp]

I understand your concern that it should just work. It effectively does that. The issue is that your response data from the service is not ASCII supported. We cannot coerce the response data to a different format because it can change the output data from a service call. I think the issue is ultimately with lambda containers not having utf8 as the default. I would recommend changing the encoding before your function, and we will pressure the lambda team to fix this upstream.

[amancevice]

@mullermp I'm totally open to the idea that this is a bug for AWS Lambda specifically and not the ruby SDK (in which case I'm happy to file a bug there but would need a pointer where to do that). But, just to restate, this is a problem that doesn't exist in the ruby3.3 runtime.

Here's an example function that works perfectly fine in ruby3.3 but fails in ruby3.4 when the response contains a UTF-8 character

require 'aws-sdk-ssm'

def handler(event:, context:)
  Aws::SSM::Client.new.get_parameter(name: 'example_with_utf8').parameter.value
end

[mullermp]

Ok. I see your earlier post about the encoding being the same for 3.3 and 3.4. To be clear about the issue, the json gem bundled with newer ruby has stricter parsing. I think lambda having ascii only encoding was a latent "bug" if you will. I still think that the best solution would be for lambda to support utf8 by default. As far as mitigation options go, I again think forcing strings to utf8 is a breaking change for our parser because we can't assume input formats. For now, there's a few mitigation options: 1) use older JSON gem, 2) use oj gem for parsing, or 3) add one line at the top of your function to change the default external encoding. I would otherwise still encourage you to push lambda team to fix that. I will support you internally and I've tried to start that discussion with them yesterday.

[mullermp]

@amancevice @joshuay03 I've got a point of contact with the lambda team and we're working on a solution.