Zappa
Zappa copied to clipboard
Miserlou
Cannot invoke task: TypeError: Object of type 'LambdaContext' is not JSON serializable
Context
When invoking tasks a TypeError is raised as zappa tries to JSON encode the LambdaContext object.
Expected Behavior
The task should be invoked
Actual Behavior
A TypeError is raised as context.identity (CognitoIdenity instance) cannot be serialized as it has no '__dict__' attribute and hence cannot be serialized.
Object of type 'LambdaContext' is not JSON serializable: TypeError
Traceback (most recent call last):
File "/var/task/handler.py", line 505, in lambda_handler
return LambdaHandler.lambda_handler(event, context)
File "/var/task/handler.py", line 242, in lambda_handler
return handler.handler(event, context)
File "/var/task/handler.py", line 341, in handler
result = self.run_function(app_function, event, context)
File "/var/task/handler.py", line 272, in run_function
result = app_function(event, context) if varargs else app_function()
File "/var/task/zappa/async.py", line 344, in _run_async
aws_region=aws_region).send(task_path, args, kwargs)
File "/var/task/zappa/async.py", line 143, in send
self._send(message)
File "/var/task/zappa/async.py", line 151, in _send
payload = json.dumps(message).encode('utf-8')
File "/var/lang/lib/python3.6/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/var/lang/lib/python3.6/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/var/lang/lib/python3.6/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/var/lang/lib/python3.6/json/encoder.py", line 180, in default
o.__class__.__name__)
TypeError: Object of type 'LambdaContext' is not JSON serializable
Possible Fix
Remove unserializable types before serialization or define custom encoder. It's a horrible solution, I admit.
# create dict from context with serializable types (this list almost certainly missing many valid types!)
context_serializable = {k:v for k, v in context.__dict__.items() if type(v) in [int, float, bool, str, list, dict]}
json.dumps(context_serializable)
Steps to Reproduce
- Create test task
from zappa.async import task
@task
def my_func():
print("All good!")
- Deploy it
- Invoke it
zappa invoke bugtest 'test.my_func'
Your Environment
- Zappa version used: 0.43.2
- Operating System and Python version: Ubuntu, Python 3.6
- The output of
pip freeze:
argcomplete==1.8.2
base58==0.2.4
boto3==1.4.5
botocore==1.5.40
certifi==2017.7.27.1
chardet==3.0.4
click==6.7
docutils==0.14
durationpy==0.5
future==0.16.0
futures==3.1.1
hjson==3.0.0
idna==2.6
jmespath==0.9.3
kappa==0.6.0
lambda-packages==0.16.1
placebo==0.8.1
python-dateutil==2.6.1
python-slugify==1.2.4
PyYAML==3.12
requests==2.18.4
s3transfer==0.1.11
six==1.10.0
toml==0.9.2
tqdm==4.15.0
troposphere==1.9.5
Unidecode==0.4.21
urllib3==1.22
Werkzeug==0.12
wsgi-request-logger==0.4.6
zappa==0.43.2
- Your
zappa_settings.py:
{
"bugtest": {
"app_function": "test.my_func",
"aws_region": "eu-west-2",
"profile_name": "turbogram",
"s3_bucket": "zappa-bug-test"
}
}
I'm not opposed to this solution. I was attempting to return another raised error type and its lack of JSON-afinity made it super ugly.
So I have been playing around with this today to try and come to a solution, however, I believe I may have stumbled on another issue.
First, my solution:
import json
from zappa.async import task, ASYNC_CLASSES, LambdaAsyncResponse
# Define a custom encoder which casts non base types to string
class LambdaMessageEncoder(json.JSONEncoder):
def default(self, obj):
if not type(obj) in [int, float, complex, dict, tuple, list, bool] and obj is not None:
return str(obj)
return json.JSONEncoder.default(self, obj)
# Extend LambdaAsyncResponse to rework the _send method
# The send method is only included due to a bug where response_id is not set
class TestLambdaAsyncResponse(LambdaAsyncResponse):
def send(self, task_path, args, kwargs):
"""
Create the message object and pass it to the actual sender.
"""
message = {
'task_path': task_path,
'capture_response': getattr(self, 'capture_response', False),
'response_id': getattr(self, 'response_id', None),
'args': args,
'kwargs': kwargs
}
self._send(message)
return self
def _send(self, message):
"""
Given a message, directly invoke the lamdba function for this task.
"""
message['command'] = 'zappa.async.route_lambda_task'
# Use custom encoder class
payload = json.dumps(message, cls=LambdaMessageEncoder).encode('utf-8')
if len(payload) > 128000: # pragma: no cover
raise AsyncException("Payload too large for async Lambda call")
self.response = self.client.invoke(
FunctionName=self.lambda_function_name,
InvocationType='Event', #makes the call async
Payload=payload
)
self.sent = (self.response.get('StatusCode', 0) == 202)
# Add service to ASYNC_CLASSES
ASYNC_CLASSES['test'] = TestLambdaAsyncResponse
@task()
def my_func_default_encoder(*args, **kwargs):
print("All good, default encoder!")
@task(service='test')
def my_func_with_encoder(*args, **kwargs):
print("All good, new encoder!")
This fixes our issues with LambdaContext being unserializable but raises a new issue as Zappa is trying to call the method asynchronously when we invoke it through the command line. The response object, TestLambdaAsyncResponse, is in turn unserializable. Output below:
Calling invoke for stage bugtest..
[DEBUG] 2017-09-12T11:52:17.97Z d327d303-97b0-11e7-86c5-114d325c08e9 StringToSign:
AWS4-HMAC-SHA256
20170912T115217Z
20170912/eu-west-2/lambda/aws4_request..............
[DEBUG] 2017-09-12T11:52:17.97Z d327d303-97b0-11e7-86c5-114d325c08e9 Signature:
..................
[DEBUG] 2017-09-12T11:52:17.98Z d327d303-97b0-11e7-86c5-114d325c08e9 Sending http request: <PreparedRequest [POST]>
[INFO] 2017-09-12T11:52:17.98Z d327d303-97b0-11e7-86c5-114d325c08e9 Starting new HTTPS connection (2): lambda.eu-west-2.amazonaws.com
[DEBUG] 2017-09-12T11:52:17.211Z d327d303-97b0-11e7-86c5-114d325c08e9 "POST /2015-03-31/functions/zappabug-bugtest/invocations HTTP/1.1" 2020
[DEBUG] 2017-09-12T11:52:17.211Z d327d303-97b0-11e7-86c5-114d325c08e9 Response headers: {'content-type': '', 'date': 'Tue, 12 Sep 2017 11:52}
[DEBUG] 2017-09-12T11:52:17.211Z d327d303-97b0-11e7-86c5-114d325c08e9 Response body:
<botocore.response.StreamingBody object at 0x7f72a708df98>
[DEBUG] 2017-09-12T11:52:17.211Z d327d303-97b0-11e7-86c5-114d325c08e9 Event needs-retry.lambda.Invoke: calling handler <botocore.retryhandle>
[DEBUG] 2017-09-12T11:52:17.212Z d327d303-97b0-11e7-86c5-114d325c08e9 No retry needed.
Result of test.my_func_with_encoder:
<test.TestLambdaAsyncResponse object at 0x7f72a7cf1d30>
An error occurred during JSON serialization of response: <test.TestLambdaAsyncResponse object at 0x7f72a7cf1d30> is not JSON serializable
Traceback (most recent call last):
File "/var/lang/lib/python3.6/json/__init__.py", line 238, in dumps
**kw).encode(obj)
File "/var/lang/lib/python3.6/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/var/lang/lib/python3.6/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/var/runtime/awslambda/bootstrap.py", line 110, in decimal_serializer
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <test.TestLambdaAsyncResponse object at 0x7f72a7cf1d30> is not JSON serializable
[END] RequestId: d327d303-97b0-11e7-86c5-114d325c08e9
[REPORT] RequestId: d327d303-97b0-11e7-86c5-114d325c08e9
Duration: 131.44 ms
Billed Duration: 200 ms
Memory Size: 512 MB
Max Memory Used: 34 MB
I'm not sure my understanding of the internals of Zappa is quite strong enough to tackle this one, would appreciate some guidance/thoughts on how to tackle this.
Further to the above, I believe an error in the TestLambdaAsyncResponse method was causing Zappa to retry calling our method.
I need to find the time to work through this issue as right now I have very little confidence in my solution.
I am getting same error.
I added task decorator to function and I am calling the function from zappa_settings.json shown as below:
"events": [
{
"function": "tasks.print_for_test_lambda",
"expression": "rate(1 minute)"
}
],
Project directory structure:
rootDir/
tasks.py
zappa_settings.json
...
@mesuutt to have a method execute from an event it can't have the task decorator (based on my last tests ~2 months ago)
I believe that when the event calls a function with a task decoy or the event launches a separate lambda function to do the task.
Sent from my iPhone
On Nov 27, 2017, at 11:52 AM, Rob Watkiss [email protected] wrote:
@mesuutt to have a method execute from an event it can't have the task decorator (based on my last tests ~2 months ago)
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.
@mcrowson Yes, I do agree with you but I am still confused that If the task function has some dependencies on imported modules then how would it going to run.
Please look at below snippet.
import os
from flask import Flask
from flask_restful import Api as FlaskRestfulAPI, Resource,reqparse
from elasticsearch import Elasticsearch
from zappa.async import task
from config import config
from analyzer import Paradigmatic
app = Flask(__name__)
app.config.from_object(config['dev'])
class ParadigmaticResource(Resource):
post_parser = reqparse.RequestParser()
post_parser.add_argument('word1', required=True, location='json')
post_parser.add_argument('word2', required=True, location='json')
@task
def calculate_paradigmatic(self, *args, **kwargs):
paradigmatic = Paradigmatic()
score = paradigmatic.similarity(kwargs['word1'], kwargs['word2'])
return {'score': score }
def post(self):
args = self.post_parser.parse_args()
data = self.calculate_paradigmatic(word1=args['word1'], word2=args['word2'])
return {'score': data}
with app.app_context():
api = FlaskRestfulAPI(app)
api.add_resource(ParadigmaticResource, '/paradigmatic')
if __name__ == '__main__':
app.run(debug=True)
Now When I execute this endpoint then I am getting the below-mentioned error.
[1521814433242] [DEBUG] 2018-03-23T14:13:53.226Z 6a8d7647-2ea4-11e8-8f8b-3f7ca6a6d141 Zappa Event: {'resource': '/{proxy+}', 'path': '/paradigmatic', 'httpMethod': 'POST', 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8', 'cache-control': 'no-cache', 'CloudFront-Forwarded-Proto': 'https', 'CloudFront-Is-Desktop-Viewer': 'true', 'CloudFront-Is-Mobile-Viewer': 'false', 'CloudFront-Is-SmartTV-Viewer': 'false', 'CloudFront-Is-Tablet-Viewer': 'false', 'CloudFront-Viewer-Country': 'IN', 'content-type': 'application/json', 'Host': 'voxvx70ba3.execute-api.ap-south-1.amazonaws.com', 'origin': 'chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop', 'postman-token': '0c8ebf3f-22b9-c825-e54a-0325e486171d', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36', 'Via': '2.0 affc54995ded64e4c5647f6363ed884e.cloudfront.net (CloudFront)', 'X-Amz-Cf-Id': 'zt9m6CYbOqvOWZOPE9BQftFnEqD4qDK2WTkE5k3mIJ5gnr19MogfYA==', 'X-Amzn-Trace-Id': 'Root=1-5ab50ba1-a06594aae7a25b8e84eefedc', 'X-Forwarded-For': '103.19.39.2, 54.182.245.48', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https'}, 'queryStringParameters': None, 'pathParameters': {'proxy': 'paradigmatic'}, 'stageVariables': None, 'requestContext': {'requestTime': '23/Mar/2018:14:13:53 +0000', 'path': '/dev/paradigmatic', 'accountId': '445420586144', 'protocol': 'HTTP/1.1', 'resourceId': 'ne01c4', 'stage': 'dev', 'requestTimeEpoch': 1521814433148, 'requestId': '6a85fbf4-2ea4-11e8-9c0b-3f97e930b286', 'identity': {'cognitoIdentityPoolId': None, 'accountId': None, 'cognitoIdentityId': None, 'caller': None, 'sourceIp': '103.19.39.2', 'accessKey': None, 'cognitoAuthenticationType': None, 'cognitoAuthenticationProvider': None, 'userArn': None, 'userAgent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36', 'user': None}, 'resourcePath': '/{proxy+}', 'httpMethod': 'POST', 'apiId': 'voxvx70ba3'}, 'body': 'ewoJIndvcmQxIjogIm15c3FsIiwKCSJ3b3JkMiI6ICJqcXVlcnkiCn0=', 'isBase64Encoded': True}
[1521814433243] Object of type 'ParadigmaticResource' is not JSON serializable
I am not sure what is missing, I followed as per the documentation.
I want to execute this function asynchronously.
Please do let me know if someone has any idea about it.
Thanks
Have you tried creating a wrapper function that's not in the class? The async function call stuff needs a function name, not a method of a class.
I'm getting this error when trying to call a task from inside a task.
@task(...)
def test():
inside_task()
@task(...)
def inside_task():
pass
With the following traceback:
Traceback (most recent call last):
File "/var/task/handler.py", line 567, in lambda_handler
return LambdaHandler.lambda_handler(event, context)
File "/var/task/handler.py", line 240, in lambda_handler
return handler.handler(event, context)
File "/var/task/handler.py", line 372, in handler
result = self.run_function(app_function, event, context)
File "/var/task/handler.py", line 275, in run_function
result = app_function(event, context) if varargs else app_function()
File "/var/task/zappa/async.py", line 424, in _run_async
capture_response=capture_response).send(task_path, args, kwargs)
File "/var/task/zappa/async.py", line 170, in send
self._send(message)
File "/var/task/zappa/async.py", line 178, in _send
payload = json.dumps(message).encode('utf-8')
File "/var/lang/lib/python3.6/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/var/lang/lib/python3.6/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/var/lang/lib/python3.6/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/var/lang/lib/python3.6/json/encoder.py", line 180, in default
o.__class__.__name__)
TypeError: Object of type 'LambdaContext' is not JSON serializable
I'm getting this error when trying to call a task from inside a task.
@task(...) def test(): inside_task() @task(...) def inside_task(): passWith the following traceback:
Traceback (most recent call last): File "/var/task/handler.py", line 567, in lambda_handler return LambdaHandler.lambda_handler(event, context) File "/var/task/handler.py", line 240, in lambda_handler return handler.handler(event, context) File "/var/task/handler.py", line 372, in handler result = self.run_function(app_function, event, context) File "/var/task/handler.py", line 275, in run_function result = app_function(event, context) if varargs else app_function() File "/var/task/zappa/async.py", line 424, in _run_async capture_response=capture_response).send(task_path, args, kwargs) File "/var/task/zappa/async.py", line 170, in send self._send(message) File "/var/task/zappa/async.py", line 178, in _send payload = json.dumps(message).encode('utf-8') File "/var/lang/lib/python3.6/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/var/lang/lib/python3.6/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/var/lang/lib/python3.6/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/var/lang/lib/python3.6/json/encoder.py", line 180, in default o.__class__.__name__) TypeError: Object of type 'LambdaContext' is not JSON serializable
you should not use @taskdecorator to the function that used for events in zappa_settings.json
So I have been playing around with this today to try and come to a solution, however, I believe I may have stumbled on another issue.
First, my solution:
import json from zappa.async import task, ASYNC_CLASSES, LambdaAsyncResponse # Define a custom encoder which casts non base types to string class LambdaMessageEncoder(json.JSONEncoder): def default(self, obj): if not type(obj) in [int, float, complex, dict, tuple, list, bool] and obj is not None: return str(obj) return json.JSONEncoder.default(self, obj) # Extend LambdaAsyncResponse to rework the _send method # The send method is only included due to a bug where response_id is not set class TestLambdaAsyncResponse(LambdaAsyncResponse): def send(self, task_path, args, kwargs): """ Create the message object and pass it to the actual sender. """ message = { 'task_path': task_path, 'capture_response': getattr(self, 'capture_response', False), 'response_id': getattr(self, 'response_id', None), 'args': args, 'kwargs': kwargs } self._send(message) return self def _send(self, message): """ Given a message, directly invoke the lamdba function for this task. """ message['command'] = 'zappa.async.route_lambda_task' # Use custom encoder class payload = json.dumps(message, cls=LambdaMessageEncoder).encode('utf-8') if len(payload) > 128000: # pragma: no cover raise AsyncException("Payload too large for async Lambda call") self.response = self.client.invoke( FunctionName=self.lambda_function_name, InvocationType='Event', #makes the call async Payload=payload ) self.sent = (self.response.get('StatusCode', 0) == 202) # Add service to ASYNC_CLASSES ASYNC_CLASSES['test'] = TestLambdaAsyncResponse @task() def my_func_default_encoder(*args, **kwargs): print("All good, default encoder!") @task(service='test') def my_func_with_encoder(*args, **kwargs): print("All good, new encoder!")This fixes our issues with
LambdaContextbeing unserializable but raises a new issue as Zappa is trying to call the method asynchronously when we invoke it through the command line. The response object,TestLambdaAsyncResponse, is in turn unserializable. Output below:Calling invoke for stage bugtest.. [DEBUG] 2017-09-12T11:52:17.97Z d327d303-97b0-11e7-86c5-114d325c08e9 StringToSign: AWS4-HMAC-SHA256 20170912T115217Z 20170912/eu-west-2/lambda/aws4_request.............. [DEBUG] 2017-09-12T11:52:17.97Z d327d303-97b0-11e7-86c5-114d325c08e9 Signature: .................. [DEBUG] 2017-09-12T11:52:17.98Z d327d303-97b0-11e7-86c5-114d325c08e9 Sending http request: <PreparedRequest [POST]> [INFO] 2017-09-12T11:52:17.98Z d327d303-97b0-11e7-86c5-114d325c08e9 Starting new HTTPS connection (2): lambda.eu-west-2.amazonaws.com [DEBUG] 2017-09-12T11:52:17.211Z d327d303-97b0-11e7-86c5-114d325c08e9 "POST /2015-03-31/functions/zappabug-bugtest/invocations HTTP/1.1" 2020 [DEBUG] 2017-09-12T11:52:17.211Z d327d303-97b0-11e7-86c5-114d325c08e9 Response headers: {'content-type': '', 'date': 'Tue, 12 Sep 2017 11:52} [DEBUG] 2017-09-12T11:52:17.211Z d327d303-97b0-11e7-86c5-114d325c08e9 Response body: <botocore.response.StreamingBody object at 0x7f72a708df98> [DEBUG] 2017-09-12T11:52:17.211Z d327d303-97b0-11e7-86c5-114d325c08e9 Event needs-retry.lambda.Invoke: calling handler <botocore.retryhandle> [DEBUG] 2017-09-12T11:52:17.212Z d327d303-97b0-11e7-86c5-114d325c08e9 No retry needed. Result of test.my_func_with_encoder: <test.TestLambdaAsyncResponse object at 0x7f72a7cf1d30> An error occurred during JSON serialization of response: <test.TestLambdaAsyncResponse object at 0x7f72a7cf1d30> is not JSON serializable Traceback (most recent call last): File "/var/lang/lib/python3.6/json/__init__.py", line 238, in dumps **kw).encode(obj) File "/var/lang/lib/python3.6/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/var/lang/lib/python3.6/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/var/runtime/awslambda/bootstrap.py", line 110, in decimal_serializer raise TypeError(repr(o) + " is not JSON serializable") TypeError: <test.TestLambdaAsyncResponse object at 0x7f72a7cf1d30> is not JSON serializable [END] RequestId: d327d303-97b0-11e7-86c5-114d325c08e9 [REPORT] RequestId: d327d303-97b0-11e7-86c5-114d325c08e9 Duration: 131.44 ms Billed Duration: 200 ms Memory Size: 512 MB Max Memory Used: 34 MBI'm not sure my understanding of the internals of Zappa is quite strong enough to tackle this one, would appreciate some guidance/thoughts on how to tackle this.
I'm facing same issue, any update on this?
As mentioned in the comment, a scheduled event cannot use a @task decorator.
If the function is called in code other than the scheduled event, and you still want it to run as a scheduled event, I believe if you create a simple handler (my_async_task_handler() below) to manage the scheduled event you can work around this issue. (same when calling a task in a task).
...
"events": [
{
"function": "tasks.my_async_task_handler",
"expression": "rate(1 minute)"
}
],
...
tasks.py:
@task(...)
def my_async_task():
print("code here")
def my_async_task_handler():
my_async_task()
