Add support for AWS Rekognition

[Leonidas-from-XIV]

I don't know if this is the right way to do it but I need to access Rekognition and thought that since the definitions here are auto-generated I can just use the definitions from Botocore to get bindings directly.

The "test suite" is taken directly from the RDS tests for now…

[Leonidas-from-XIV]

I don't quite understand what all the modules are. Currently I need to use the make functions from Types_internal because they don't seem to be exposed anywhere else, which is a bit odd. I saw that the ec2 bindings has a proper (non-internal) types module, but don't know how that came to be.

[tmcgilchrist]

Types vs types_internal is my fault, try pulling in the latest master and rerunning the generation. It should now generate just Types.ml for each service.

See https://github.com/inhabitedtype/ocaml-aws/blob/master/src/aws_gen.ml#L194

[Leonidas-from-XIV]

I've regenerated the files now, Types.ml exists now.

What does not work is the generated endpoint rekognition.amazonaws.com since there is no region-free hostname for Rekognition. In #55 it is solved by reading the endpoints from the botocore declarations and deriving the right hostnames. I think eventually this is the right thing to do, but for now I just hacked in .us-east-1 into the hostname in lib/aws.ml and the Rekognition endpoints.

Unfortunately when I try it it does not work (and it also looks like the error handling also failed):

HttpError(500 - BadResponse): Could not find <Error> nodes for error response code.. Body: <InternalFailure/>

HTTP 500 is not much to go about. I'm wondering whether this is due to the URL encoding of slashes in the query params, Action=DetectModerationLabels&Image.S3Object.Bucket=hest&Image.S3Object.Name=hest%2Fjpg%2Fpage_1.jpg&Version=2016-06-27 but I don't know.

When I try the same with aws from awscli I can see that instead of encoding everything in a query string for POST it constructs the data as JSON. Not sure if this is because Rekognition does not support query params.

[Leonidas-from-XIV]

I implemented that bit by hand using Core.Time (instead of CalendarLib) and Digestif (instead of nocrypto) and decided to send the data as a JSON POST body, so I had to sign the payload as well. That failed with the exact same InternalFailure error, so I added an (unsigned) content-type application/x-amz-json-1.1 header at which point it told me it doesn't know which target and adding x-amz-target as a header (RekognitionService.DetectModerationLabels was what I was going for) fixed the issue. So maybe setting the content-type in this PR, even if the payload is empty, might be the way to go.

[tmcgilchrist]

@Leonidas-from-XIV the endpoint generation has been merged into master, if you want to rebase off that for future work.

[Leonidas-from-XIV]

@tmcgilchrist Thanks! I'll take a look when I am back from Berlin. I seem to recall having seen you around here, in such case maybe we can meet for a short chat about ocaml-aws.

[tmcgilchrist]

Of course I’m in the OCaml workshop for the afternoon. I’ve got a red JaneStreet T-shirt.

On Fri, 23 Aug 2019 at 10:33 am, Marek Kubica [email protected] wrote:

@tmcgilchrist https://github.com/tmcgilchrist Thanks! I'll take a look when I am back from Berlin. I seem to recall having seen you around here, in such case maybe we can meet for a short chat about ocaml-aws.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/inhabitedtype/ocaml-aws/pull/60?email_source=notifications&email_token=AABJXOPUADVLMG2FGHQENR3QF6OEBA5CNFSM4IHRH3YKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD47Q4SI#issuecomment-524226121, or mute the thread https://github.com/notifications/unsubscribe-auth/AABJXONJOUZTCK4ADTUEZEDQF6OEBANCNFSM4IHRH3YA .

[Leonidas-from-XIV]

I regenerated the source with the current master and I am still seeing the InternalFailure. Setting application/x-amz-json-1.1 as content-type makes it worse since then it really tries to decode the body which is empty since everything is encoded as query param. Setting x-amz-target does not help.

I fear to get Rekognition going we are going to need to encode the request in the body instead of the query params which is quite a big change.

[Leonidas-from-XIV]

Okay so after reaching out to Amazon support, they told me that the Query Parameter encoding is not the way to go and apparently does not work for Rekognition.

Therefore we would need to submit the body as JSON at least for Rekognition, but I assume it is the recommended way to do anyway for all services so changing all existing services would probably be the right thing to do.

I would suggest making that change (which would match what the Haskell bindings do), what do you think @tmcgilchrist?

[tmcgilchrist]

I'm in favour. I wonder what else needs to be modified to submit as a JSON body? Do you have an idea on what else needs changing?

[Leonidas-from-XIV]

What needs to be changed:

• [ ] Set content-type to application/x-amz-json-1.1
• [ ] x-amz-target needs to be set (which is basically equivalent to the Action query param)
• [ ] We need to encode the params as JSON and sign this new body payload (unlike our current empty body)

Overall it is not too complex, but I'll only have time to work on this in a few weeks.

[pm-mck]

@Leonidas-from-XIV I might have some time to help with this. Are there any AWS docs that you've referred to on this before?

I would imagine UTF8 support for JSON payloads will be extremely important, will we need to bring in any UTF8 support via https://github.com/dbuenzli/uutf or some other lib?