Zappa
Zappa copied to clipboard
Miserlou
Parameter validation failed when using @task_sns.
Context
I want to invoke a task which use @task_sns.
E.g. code:
@task_sns
def my_async_task(data):
a = data.get('a')
b = data.get('b')
return a + b
def my_endpoint():
my_async_task({
'a': 4,
'b': 10
})
return jsonify({
'success': True
})
This code produces this error:
Parameter validation failed:
Invalid type for parameter Message, value: b'{"task_path": "my_async_task", "capture_response": false, "response_id": null, "args": [{"a": 4, "b": 10}], "kwargs": {}, "command": "zappa.async.route_sns_task"}', type: <class 'bytes'>, valid types: <class 'str'>: ParamValidationError
Traceback (most recent call last):
File "/var/task/handler.py", line 509, in lambda_handler
return LambdaHandler.lambda_handler(event, context)
File "/var/task/handler.py", line 240, in lambda_handler
return handler.handler(event, context)
File "/var/task/handler.py", line 345, in handler
result = self.run_function(app_function, event, context)
File "/var/task/handler.py", line 270, in run_function
result = app_function(event, context) if varargs else app_function()
File "/var/task/main.py", line 81, in update_results
my_async_task(search_serialized)
File "/var/task/zappa/async.py", line 421, in _run_async
capture_response=capture_response).send(task_path, args, kwargs)
File "/var/task/zappa/async.py", line 170, in send
self._send(message)
File "/var/task/zappa/async.py", line 249, in _send
Message=payload
File "/var/runtime/botocore/client.py", line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 586, in _make_api_call
api_params, operation_model, context=request_context)
File "/var/runtime/botocore/client.py", line 641, in _convert_to_request_dict
api_params, operation_model)
File "/var/runtime/botocore/validate.py", line 291, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
Expected Behavior
Invoking the task should not fail. Message argument sent to botocore must be a string, not bytes.
Actual Behavior
We can't invoke the task using a SNS channel, because botocore expect a string, not bytes.
Possible Fix
Steps to Reproduce
- use the code pasted above
-
zappa invoke dev main.my_endpoint - See the result.
Your Environment
- Zappa version used: 0.45.1
- Operating System and Python version: macOS, Python 3.6.3
- The output of
pip freeze:
argcomplete==1.9.2
arrow==0.12.0
base58==0.2.4
boto3==1.5.7
botocore==1.8.21
certifi==2017.11.5
cfn-flip==1.0.0
chardet==3.0.4
click==6.7
dnspython==1.15.0
docutils==0.14
durationpy==0.5
Flask==0.12.2
flask-mongoengine==0.9.3
Flask-WTF==0.14.2
future==0.16.0
hjson==3.0.1
idna==2.6
itsdangerous==0.24
Jinja2==2.10
jmespath==0.9.3
kappa==0.6.0
lambda-packages==0.19.0
MarkupSafe==1.0
marshmallow==2.15.0
marshmallow-mongoengine==0.9.1
mongoengine==0.15.0
placebo==0.8.1
pymongo==3.6.0
python-dateutil==2.6.1
python-slugify==1.2.4
PyYAML==3.12
raven==6.4.0
requests==2.18.4
s3transfer==0.1.12
six==1.11.0
toml==0.9.4
tqdm==4.19.1
troposphere==2.1.2
Unidecode==0.4.21
urllib3==1.22
Werkzeug==0.12
wsgi-request-logger==0.4.6
WTForms==2.1
zappa==0.45.1
- Link to your project (optional): none
- Your
zappa_settings.py:
{
"dev": {
"app_function": "main.app",
"aws_region": "eu-west-3",
"profile_name": "default",
"project_name": "serverless",
"runtime": "python3.6",
"s3_bucket": "******",
"timeout": 500,
"async_source": "sns",
"async_resources": true,
"async_response_table": "******-async-response",
"async_response_table_read_capacity": 1,
"async_response_table_write_capacity": 1
}
}
It's been over six months, and this issue is still open. As a user who would dearly love to see this get fixed - can anyone please look into this? Any idea when we can expect to see a fix be pushed in an update?
It's been eight months since I dropped my last comment into this issue, and I've had to essentially monkey-patch the async file for all this time.
This seems to me like an issue related to supporting both Python2 and Python3's understanding of strings simultaneously. Python2 is reaching the end of its life this year, and so it's unreasonable for there to be broken functionality in Python3 in the interest of supporting old code. Python2 users should expect to be pinning versions left and right.
When can we expect this problem to be solved?
I'm having the same issue, any advice for how to get this working @relwell ?
@RoryOfByrne here's a gist of my workaround: https://gist.github.com/relwell/3eb9d68526ba8eadb41b3b1399e747a9
you would import that file instead of the normal zappa_async module.
Brilliant, thank you @relwell . Am I right in thinking that the only change is removing .encode('utf-8') - here? I'm working on my own patch right now and that's the change I'm making.
I think so, @RoryOfByrne. It's been a while since I inspected the solution.
Also bumped into the same problem here. @relwell approach wasn't a fit for me, as it is not so easy to apply his monkey-patch on the env I'm working with, although it worked just fine. I declared the boto3 session manually, used a fixed ARN for the publish request and hard-coded the dict that zappa builds before sending to SNS.
import boto3
message = {
"task_path": "module.my_async_method",
"capture_response": False,
"response_id": None,
"args": ["myarg1", "myarg2"],
"kwargs": {},
"command": "zappa.asynchronous.route_sns_task"
}
message = json.dumps(message)
aws_session = boto3.Session()
client = aws_session.client('sns')
client.publish(
TargetArn="arn:aws:sns:zone:123123123:sns-topic-name",
Message=message
)
It is basically the same as _send function from zappa but without the payload size exception and SNS message Id response. I hope the problem is solved soon so I can properly make SNS calls 😄
