aws-serverless-data-lake-framework icon indicating copy to clipboard operation
aws-serverless-data-lake-framework copied to clipboard

Compatibility install option for SDLFv2

Open cnfait opened this issue 1 year ago • 5 comments

Having SDLFv1 and SDLFv2 side-by-side in the same accounts. [more details to follow]

  • [ ] SSM parameters prefixed with /SDLF2/ instead of /SDLF/
  • [ ] KMS aliases starting with sdlf2- instead of sdlf-
  • [ ] CodeCommit repositories starting with sdlf2- instead of sdlf-
  • [ ] DynamoDB table names using a prefix distinct from octagon- (such as sdlf2-)
  • [ ] import team keys when deploying CloudFormation stacks

cnfait avatar Feb 26 '24 14:02 cnfait

#290 helps handling the CodeCommit part of this issue.

cnfait avatar Mar 25 '24 18:03 cnfait

#291 helps handling the DynamoDB part - but there is more work required.

cnfait avatar Mar 25 '24 18:03 cnfait

#295 and #296 are also linked to this task.

cnfait avatar Mar 26 '24 17:03 cnfait

There is a branch with the changes necessary to install SDLFv2 side-by-side with SDLFv1: https://github.com/awslabs/aws-serverless-data-lake-framework/tree/sdlf1-backcompatibility

Some of the changes have been incorporated into main (see previous comments) but not all are relevant/desirable for all installations of SDLFv2. This is an upgrade path more than a good solution long-term.

Important caveats:

  • when deploying foundations, do not use datalake as the domain name (we suggest lake).
  • when deploying stageA and stageB in SDLFv2, do not use the names a and b (lower-case) for pStageName.

cnfait avatar Mar 26 '24 19:03 cnfait

Other changes that may become part of main:

removing FunctionName for the replicate lambda in sdlf-foundations https://github.com/awslabs/aws-serverless-data-lake-framework/commit/c968be52b3b3be3c4fedc02951ab5c8b719c2704

retrieving the Glue crawler name from SSM in sdlf-stageB fetch-metadata Lambda function: https://github.com/awslabs/aws-serverless-data-lake-framework/commit/612e15454100680edeff42db786d7febb5b03f1e

cnfait avatar Mar 26 '24 19:03 cnfait