Amazonka 2.0

[endgame]

Per the release announcement, Amazonka 2.0.0-rc1 is available for testing. The release announcement also explains how to change your cabal.project or stack.yaml file to test Amazonka in your own projects.

Because of the magnitude of the accumulated changes, we are now in a stabilisation period before we release to Hackage. During this period, we'd greatly appreciate people testing amazonka against their projects, and reporting issues against the 2.0 milestone. We are aiming for a four-week stabilisation period (which would end on 2021-12-16), but may shorten or lengthen this depending on the volume of issues raised and any impact from the holiday season.

[endgame]

Status update: We won't hit the the planned release date, and with Christmas nearly upon us that's going to slow everything down more. Mid-January seems more realistic. There are things I want to test by porting my work codebase to amazonka-2.0, and that work stalled a bit due to the massive outage in us-east-1 last week.

[ocharles]

Hi folks, how are things looking for a 2.0 release? There's no rush from my side, I'm just looking to plan when we can move our work codebase to aeson-2, and amazonka is one of the blockers there at the moment (though we are using 2.0 with source-repository-package)

[endgame]

About time for a status update, thanks for a poke:

• January is basically over
• The auth refactor was a huge feature that lots of people want but needs resting.
• Work codebase is now running on amazonka-2, so I have some confidence that things work properly
• There are a few other things I want to get done on the 2.0 milestone.
• There's probably enough major changes (e.g., giant auth refactor) which probably warrant another, shorter, RC round
• I'm currently assigned to project work at work for another five weeks; after that, I'll get two weeks where I can work solidly on amazonka on work time. Until then, it's only nights and weekends for me.

I would love to say April for final 2.0 release, but we might drag out longer than that. Any chance you can pitch in a PR for any of the milestone items?

P.S.: Glad to hear you're using 2.0 with source-repository-package. I assume that it's been reasonably stable for you, since I haven't seen that many issues get filed?

[ocharles]

P.S.: Glad to hear you're using 2.0 with source-repository-package. I assume that it's been reasonably stable for you, since I haven't seen that many issues get filed?

Unfortunately, I'm not sure we actually use amazonka for anything - it's just a dependency of something else. As such, I'm not sure I can volunteer any PRs as I've never used the library :sweat_smile:

[mbj]

I would love to say April for final 2.0 release, but we might drag out longer than that. Any chance you can pitch in a PR for any of the milestone items?

I recommend to move some of the open milestones into post 2.0 releases. Some, to me: Appear not in the category of "needs to be done in a breaking change so should not delay 2.0".

These would IMO not cause a breaking change post 2.0 once they are addressed, to me they look like pending additions which could go into amazonka-2.0.x, or amazonka-2.1:

• https://github.com/brendanhay/amazonka/issues/584
• https://github.com/brendanhay/amazonka/issues/747
• https://github.com/brendanhay/amazonka/issues/748

These milestone items, to me look more like bugfixes which have a minor breaking change character, but would from my (downstream perspective) not be counted as a breaking change, despite the fact I may have to adjust some imports:

• https://github.com/brendanhay/amazonka/issues/728

Consequence of not considering these as blocking may be a bit earlier release, which allows downstream libraries to adapt much earlier, increasing the total value provided by amazonka-2.0.

This is my 2¢, not in any case authoritative. Just another amazonka downstream consumers perspective.

[endgame]

Yeah, perhaps. It's something I'll look at once I put a chunk of time into things and see what sort of momentum I can achieve.

[magthe]

I just thought I'd report that I've spent some time porting one of the services at work to use 2.0.0rc1. So far it works well. I'm particularly happy about the support for AssumeRoleWithWebIdentity (and so is our platform team as we're now down to a single service using access keys in envvars).

A few observations:

• It was surprising to find that the newX functions aren't in the same package as the lenses to work with them. It led to these import statements to work with an SQS queue:

    import Amazonka.SQS
      ( newDeleteMessageBatch,
        newDeleteMessageBatchRequestEntry,
        newReceiveMessage,
      )
    import Amazonka.SQS.DeleteMessageBatch (deleteMessageBatch_entries)
    import Amazonka.SQS.ReceiveMessage
      ( receiveMessageResponse_httpStatus,
        receiveMessageResponse_messages,
        receiveMessage_maxNumberOfMessages,
      )
    import Amazonka.SQS.Types (message_body, message_receiptHandle)
  

  I'm neither disgusted, nor enthusiastic, just surprised.

• The move away from a custom monad, AWS', is nice. Using resourceT directly is a win, even if the Env has to be passed around a bit.

All in all it was rather straight forward to move from 1.6 to 2.0. Very nice work!

Hopefully you'll find the time to make the official release soon.

[endgame]

Thanks for the feedback. The lenses are in their own module because generic-lens and/or generic-optics are now the recommended ways to access fields in request/response records. See the use of #buckets (with -XOverloadedLabels) in the S3 example:

https://github.com/brendanhay/amazonka/blob/f73a957d05f64863e867cf39d0db260718f0fadd/examples/src/S3.hs#L53

If you don't like that, you can also use field with an explicit type application, and if you don't like that you can get all lenses for a a single service from Amazonka.<ServiceName>.Lens (e.g., Amazonka.SQS.Lens).

[ysangkok]

I have also patched a codebase from the now-deleted release/1.7.0 tag to the current Amazonka master, but I am not testing it since we don't have infrastructure to gradually deploy things, so I think it is too risky. The patch is 93 insertions(+), 95 deletions(-).

I would love to say April for final 2.0 release, but we might drag out longer than that.

Any chance a new estimate could be made? Because it seems like it is certain to drag on longer now. I ask because Amazonka is keeping us on Aeson 1.5, and I want to know how much effort I should put into maybe getting a new Aeson 1.5 release.

There is no rush of course, I thank you for you work. Just thought it would be useful with a new estimate if it is easy to do.

[brendanhay]

@magthe If you're using the generated (legacy) lenses, the usage for your example is intended to be:

import Amazonka.SQS
    ( newDeleteMessageBatch,
      newDeleteMessageBatchRequestEntry,
      newReceiveMessage,
    )
import Amazonka.SQS.Lens
    ( deleteMessageBatch_entries,
      receiveMessageResponse_httpStatus,
      receiveMessageResponse_messages,
      receiveMessage_maxNumberOfMessages,
      message_body,
      message_receiptHandle
    )

[endgame]

April is no longer realistic - you are right. The work on tidying up service definitions and adding the new AWS services is stalled because we currently cannot regenerate services, and I'd like @brendanhay to review #761 before I charge off and merge it. Once that's unblocked, I'll see if I can get a little momentum up. However, my life has also become significantly busier of late, which makes evening/weekend contributions more difficult. Work does let me do some Amazonka work on company time, but it's usually in the two-week blocks between six-week projects, and the next one of those isn't for another ~four weeks.

[magthe]

@brendanhay: aha, I used the links in the docs to guide me to find the functions/lenses I needed. I guess that's not the way, even for legacy lenses then.

I didn't find any mention of -XOverloadedLabels or anything like #buckets (like @endgame mentioned above) in the documentation. If there is a documentation of "intended use" somewhere then I missed it. I also found no mention of there even being "legacy lenses".

Your comments make me think that maybe this is something that is difficult for users to just discover on their own. Especially when having used 1.6 somewhat casually, like I have. Well, at least it was for me :grin:

Not even the first example anyone is likely to bump into, the example in Amazonka, seem to be written using the "recommended ways".

I hope I'm not coming across as confrontational. It's not my intention. Just trying to express how I approached porting to 2.0 as someone who has a complete lack of context beyond what I found in the documentation.

[brendanhay]

This is helpful feedback, @magthe, thanks! The documentation for much of the work that has come along with 2.0 still needs improvement, and feedback like this lets us know what we should focus on.

[istathar]

We've ported a fair bit of 1.6.1 code to 2.0.0 and for the modules we are using amazonka is running fine. Your work has been great, and while I know you're wanting everything to be perfect before you upload 2.0.0 to Hackage we'd encourage you to iterate fixes and documentation as you go after the milestone release. Nothing wrong with lots of micro releases afterwards. You've done a really amazing job.

[NorfairKing]

I'm looking forward to this release as well!

[endgame]

So am I. This week's objective is to regenerate bindings to existing services using latest botocore, and from there start generating bindings for the services we don't have yet.

Unfortunately, four services generate code that doesn't compile and the reasons all appear to be different. So I'll dig into that tomorrow.

[prikhi]

We've been using the following libraries in production w/ no issue for the last month or so:

  - git: https://github.com/brendanhay/amazonka/
    commit: 0ccede621e56fb6f240e4850e205cde82d0e4a4b
    subdirs:
      - lib/amazonka
      - lib/amazonka-core
      - lib/services/amazonka-sso
      - lib/services/amazonka-sts
      - lib/services/amazonka-cloudwatch
      - lib/services/amazonka-ecs
      - lib/services/amazonka-s3
      - lib/services/amazonka-ses
      - lib/services/amazonka-sns
      - lib/services/amazonka-sqs
      - lib/services/amazonka-ssm

[endgame]

Thanks for the updates everyone. #806 is the PR where all current services are regenerated against latest botocore - I would appreciate it if people can test against it and report back.

#807 Adds the configuration for a bunch of new services. After this goes through, I'll generate them. That will get us up to botocore minus five or six services that don't compile (usually because of circular module imports).

[pwm]

Had some down time so bumped our codebase from 51408b16b4 to 0ed63c6cc0.

We only use:

amazonka
amazonka-cognito-idp
amazonka-core
amazonka-ses
amazonka-sso
amazonka-sts
amazonka-test

It went smoothly, I only had to fix one thing which was removing dealing with a Maybe:

old: https://github.com/brendanhay/amazonka/blob/51408b16b4/lib/services/amazonka-sts/gen/Amazonka/STS/AssumeRole.hs#L925 new: https://github.com/brendanhay/amazonka/blob/0ed63c6cc0/lib/services/amazonka-sts/gen/Amazonka/STS/AssumeRole.hs#L988

then it all compiled and all our tests passed.

[dustin]

Does anyone have a recipe for how I might build nix-based projects until this is in hackage? This fails with recent symbols not being present in nix builds and is making it a bit difficult to use my projects across multiple machines.

(alternatively, just releasing it or even releasing it as amazonka2 would be awesome)

[endgame]

We've been using this at work with haskell.nix without such issues - can you file a bug with a minimal reproducer?

[dustin]

Can you give an example of how you do this? I don't think there's much of a point in filing a bug if I'm just doing it entirely wrong.

I'm using stack (with nix) and have the following in my stack.yaml:

extra-deps:
- github: brendanhay/amazonka
  commit: e64638ae16dc6d549d6e7c21262245e29761ede7
  subdirs:
  - lib/amazonka
  - lib/amazonka-core
  - lib/services/amazonka-s3
  - lib/services/amazonka-sts
  - lib/services/amazonka-sqs
  - lib/services/amazonka-lambda

nix complains that it can't find that revision (same with current main). This works fine with regular stack, but nix-build fails. amazonka is the only dependency I have to treat differently currently since it's not in the normal package repo, so it's quite likely there's a better way to do it.

[endgame]

It would help keep threads of discussion from becoming confused with each other (this issue is a bit of a mess already), and there have been similar requests for help in issues before: #750

I need to know how you are using nix (haskell.nix, nixpkgs, some other mechanism?) and a minimal reproduction repo would help a lot, as that snippet looks correct. I haven't used stack for years, as I've found it caused me more problems than it solved, so I cannot infer what exactly you are doing. Please help me so I can try and help you.

[dustin]

Understood. I filed #815, but had mentioned it here specifically because closing this bug would be the least-effort way to fix this (from my side, anyway -- obviously if it were easy on your side, the bug wouldn't be open).

[endgame]

I'd love for it to be on Hackage too, but there are still quite a few blockers.

[endgame]

@brendanhay: aha, I used the links in the docs to guide me to find the functions/lenses I needed. I guess that's not the way, even for legacy lenses then.

I didn't find any mention of -XOverloadedLabels or anything like #buckets (like @endgame mentioned above) in the documentation. If there is a documentation of "intended use" somewhere then I missed it. I also found no mention of there even being "legacy lenses".

The specifics of how you use a generic lens library are up to you (which is why only some of the examples use them), but there's actually a recommendation in each package description:

https://github.com/brendanhay/amazonka/blob/a4356591e151df86519b3bd2ccca1518c22e8b17/lib/services/amazonka-s3/amazonka-s3.cabal#L26

As well as a recommendation in each request's haddock:

https://github.com/brendanhay/amazonka/blob/a4356591e151df86519b3bd2ccca1518c22e8b17/lib/services/amazonka-s3/gen/Amazonka/S3/AbortMultipartUpload.hs#L118

I think this is a funny edge case where it's harder for experienced users to notice things. Once we're back on Hackage I think it will be a lot more obvious, as these recommendations will be rendered in a much more visible way.

(FWIW, I didn't actually notice these recommendations until I was hacking on the generator and actually had to change the templates.)

[endgame]

About time I gave another update. Amazonka now has bindings to every service in botocore except for two (s3control and lexv2-runtime).

In a couple of weeks I should get another couple of blocks of work-sponsored time to work on things; the goal will be to tick off a bunch of the smaller tasks that have been hanging around in the milestone for some time.

[pbrisbin]

Thanks for the update @endgame! Sorry if you've mentioned this elsewhere but is there a list of what specifically is blocking a Hackage release? Do you think those "couple of blocks" you'll get in a couple of weeks could get the project to a release, or is there still more?

I'm interested in pitching in if I can, and would love some kind of "blocks 2.0 on Hackage" label to work against. Or is it safe to say any Issue that doesn't have the "post 2.0" label is that list?

[endgame]

The Milestone is pretty up-to-date, so if you have time I would recommend trying to pick things off from that. #689, #638, #754, and #810 are all pretty routine. #745 would be nice, and probably isn't too much work either.

I would really like to release with bindings for all services in botocore, but that's going to mean working out how to support SigV4A.

[pbrisbin]

Ah, the Milestone! That makes sense, I'll try to take a look at those issues next week.

I would really like to release with bindings for all services in botocore

I might be missing something, but it seems to me that missing bindings would be the perfect thing to not hold the release for, since adding them later is an additive change, and so would only require a minor version bump. Unless, maybe you expect the SigV4A work to require compatibility breaks elsewhere?