[Bug] Unable to update Implementation Guide: Lambda too big to deploy with IG

[ericfuxealth]

Describe the bug Trying to follow the steps here to add implementation guide but ran into issues with the Lambda package too big to deploy. https://github.com/awslabs/fhir-works-on-aws-deployment/blob/mainline/USING_IMPLEMENTATION_GUIDES.md

To Reproduce Steps to reproduce the behavior:

1. Follow the steps here https://github.com/awslabs/fhir-works-on-aws-deployment/blob/mainline/USING_IMPLEMENTATION_GUIDES.md
2. Download the US core package from https://www.hl7.org/fhir/us/core/package.tgz
3. Unzip the package, rename the folder to "hl7.fhir.us.core"
4. cd hl7.fhir.us.core
5. Run "npm --registry https://packages.simplifier.net i"
6. mv node_modules/* .. I now have these packages:

• hl7.fhir.r4.core
• hl7.fhir.us.core
• hl7.fhir.uv.bulkdata
• us.nlm.vsac

7. cd .. (now in fhir-works-on-aws-deployment)
8. run "yarn run compile-igs"
9. cd javaHapiValidatorLambda (now in fhir-works-on-aws-deployment/javaHapiValidatorLambda)
10. mvn clean install
11. serverless deploy
12. Error " An error occurred: ValidatorLambdaFunction - Resource handler returned message: "Unzipped size must be smaller than 262144000 bytes (Service: Lambda, Status Code: 400, Request ID: 8a035ca9-1d30-4993-baa0-8b837d71c684, Extended Request ID: null)" (RequestToken: 3704c919-1ac0-ca0b-5571-7cb26bda4b19, HandlerErrorCode: InvalidRequest)."

I checked the unzipped jar file size, the issue the implementation guides are too big.

Expected behavior No error installing the Lambda

Versions (please complete the following information):

• Release Version installed 3.0.0

Additional context Are the above steps correct?

[SanketD92]

Hello @ericfuxealth, thank you for reporting this error. We've added it to our backlog and we'll work it into our sprints based on priority.

[Bingjiling]

Hi @ericfuxealth, thanks for reaching out! I checked my dev environment and integ test environment used for GitHub workflow. Both environments have US Core IG deployed, the size of Hapi validator lambda is within the limit and the size of the IG package is small. Could you please try following these steps in deploying US Core and see if it resolves the issue?

https://github.com/awslabs/fhir-works-on-aws-deployment/blob/9aba394bc27f204e77cc2ca255746fc95d9cbb38/.github/workflows/deploy.yaml#L75-L92

If the issue persists, could you please share a screenshot of the file sizes in your unzipped jar file?

[ericfuxealth]

Thanks. No issue with just US-core. The concern is when we add more profiles in the future we'll hit the Lambda limit size. It's not a blocking issue right now.

[Bingjiling]

Got it, thanks for the clarification! I'll add this information to our ticket to help prioritization. The limit of Lambda package size is a hard limit, so we need to look into alternative ways to attach the IG packages to allow arbitrary number of profiles.

At the mean time, we've seen pruning and cleaning npm packages to be useful in trimming down the package size. Hope it can help ease up the situation.

[Abdulkhan2]

Hi, is there any update on this issue/enhancement?

[floofydoug]

Hi @ericfuxealth, thanks for reaching out! I checked my dev environment and integ test environment used for GitHub workflow. Both environments have US Core IG deployed, the size of Hapi validator lambda is within the limit and the size of the IG package is small. Could you please try following these steps in deploying US Core and see if it resolves the issue?

https://github.com/awslabs/fhir-works-on-aws-deployment/blob/9aba394bc27f204e77cc2ca255746fc95d9cbb38/.github/workflows/deploy.yaml#L75-L92

If the issue persists, could you please share a screenshot of the file sizes in your unzipped jar file?

I tried running the hapivalidator independently and I still cannot deploy the lambda. I tried it on US Core 5.0.1, 4.0.0, and the same version the tests are on: 3.1.1. Even when using the same version the tests are on, I get a jar file MUCH bigger than the 30mb file that comes with FWOA originally. my file is ~ 86mb zipped, and unzipped it is 275mb. with US CORE 5.0.1, it was close to 300mb zipped. Screenshot of unzipped:

here is a screenshot of the error on cloudformation:

I followed the instructions @ericfuxealth gave above and have tried your suggestion, @Bingjiling to mimic the jobs as close as possible :

 mvn --batch-mode --update-snapshots --no-transfer-progress clean install 
 serverless deploy --stage dev --region us-west-2 --conceal 

I've also tried updating the memory in the lambda:

package:
  artifact: target/fwoa-hapi-validator-dev.jar

functions:
  validator:
    handler: software.amazon.fwoa.Handler
    timeout: 50 <<<< (doubled)
    memorySize: 10240 <<<<<<<
    provisionedConcurrency: 5

but that does not work either. 
           
    

[floofydoug]

Requesting that this enhancement task be upgraded to bug.

[floofydoug]

when installing us.core, it lists hl7.fhir.r4.core as a dependency.

Is it expected to have hl7.fhir.r4.core and other dependencies in the unzipped jar? (because r4 is already 40mb) However inspecting each file and folder in this unzipped jar does not add up to the total size... i am unsure where the bloat is coming from.

[nguyen102]

You can run this command to download the IG

~/Downloads/test: $ curl http://hl7.org/fhir/us/core/STU3.1.1/package.tgz | tar xz -C implementationGuides
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  598k  100  598k    0     0  2698k      0 --:--:-- --:--:-- --:--:-- 2784k
~/Downloads/test: $ ls
implementationGuides/
~/Downloads/test: $ du -sh implementationGuides/
5.3M	implementationGuides/

The IG should be 5.3 mb in size. Then you can run the commands here to deploy the HAPI Validator. There should be a lambda name fhir-service-validator-dev-validator in your account. It should be about 75mb in size.

[floofydoug]

Thanks for the response,@nguyen102. if i do not npm i -- registry https://packages.simplifier.net/ 3.1.1 dependencies, then my uncompressed jar file comes in at 247mb. just barely enough to pass the lambda and continue running. Can you confirm that the validator lambda method is only sufficient for 3.1.1 core without its dependencies and all other US CORE versions and packages would be too large?

[floofydoug]

when trying the subsequent releases of core (4.0.0, and 5.0.1) , yarn run compile-igs required dependencies to be installed and errored when they were not included. I'm surprised that 3.1.1 does not expect the dependency of hl7.fhir.r4.core to be included in the ig.

[floofydoug]

You can run this command to download the IG

~/Downloads/test: $ curl http://hl7.org/fhir/us/core/STU3.1.1/package.tgz | tar xz -C implementationGuides
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  598k  100  598k    0     0  2698k      0 --:--:-- --:--:-- --:--:-- 2784k
~/Downloads/test: $ ls
implementationGuides/
~/Downloads/test: $ du -sh implementationGuides/
5.3M	implementationGuides/

The IG should be 5.3 mb in size. Then you can run the commands here to deploy the HAPI Validator. There should be a lambda name fhir-service-validator-dev-validator in your account. It should be about 75mb in size.

I can confirm that STU3.1.1 is only 5.3mb in size. But the validator package failing lambda has to do with the size of uncompressed fhir-service-validator-dev-validator that i showed in cloudformation screenshot above. attempting to include any IG dependencies like VSAC or thesaurus definitions causes validator to become too big because the test scripts currently running are already at 247mb. uncompressed limit is around 250mb.

[nguyen102]

The current IG publisher only allows US Core 3.1.1 to work with FWoA, as other IG will require the us.nlm.vsac dependency which drives up the size of the validator.jar. As far as r4 core not being required, that's because FWoA loads that by default, and so it isn't required in the implementationGuides folder.

[ssvegaraju]

Hi, we've introduced a fix for this issue here: https://github.com/awslabs/fhir-works-on-aws-deployment/pull/789 Unfortunately we're unable to merge the code into mainline branch at this time due to conflicts on our end, but this code has been tested and will work to fix the issue of being unable to deploy implementation guides that are too large. We've done this by uploading implementation guides to an S3 Bucket and then drawing from there into the validator, which reduces the size of the Lambda Function being deployed.

Thanks, Sukeerth

[nisankep]

FHIR Works on AWS has been moved to maintenance mode. While in maintenance, we will not add any new features to this solution. All security issues should be reported directly to AWS Security at [[email protected]] (mailto:[email protected]). If you are new to this solution, we advise you to explore using [HealthLake] (https://aws.amazon.com/healthlake), which is our managed service for building FHIR based transactional and analytics applications. You can get started by contacting your AWS Account team. If you are an existing customer of FHIR Works on AWS, and have additional questions or need immediate help, please reach out to [email protected] or contact your AWS Account team.