openaq-fetch icon indicating copy to clipboard operation
openaq-fetch copied to clipboard

Published 20 hours ago • verified publisher icon openaq

Determine the best deployment strategy for the lambda fetchers

Open caparker opened this issue 2 years ago • 7 comments

Created to discuss the best way to deploy the updated fetchers. A quick overview of how its set up to work

  1. event triggers the scheduler lambda
  2. scheduler lambda then determines which sources should be run and submits one SQS message that includes all the selected sources passed in the body of the message
  3. Fetcher lambda is responding to SQS and looks in the event to find the sources (if available)
  4. The fetcher then combines the source information with the env for the deployment and builds the file

So with this setup we will need a different instance of the lambda stack for each deployment that we want to run. That means deploying a different stack for each, with a different .env file for each, which doesnt seem sustainable in the long term but good for now in my opinion.

Long term I am thinking we need to just have one deployment of the stack and that the scheduler pulls from the database to see what it should do. @russbiggs @majesticio

caparker avatar Oct 28 '22 14:10 caparker

And right now I think we have at least two deployments to manage

  1. All sources, no offset, assuming that all can be finished in the 15min limit
  2. AirNow, 24 hour offset

caparker avatar Oct 28 '22 14:10 caparker

The two deployment option seems reasonable

russbiggs avatar Oct 28 '22 15:10 russbiggs

Are you proposing to just keep the two deployment option long term? The downside of that is the need to manually deploy and not use github actions.

caparker avatar Oct 28 '22 15:10 caparker

not long term, "good for now" as you said in the original comment

russbiggs avatar Oct 28 '22 19:10 russbiggs

I assume we have 3 lambda deployments now with Mexico/Sinaica? And we are considering using several in the future based off of geography and how much time an adapter needs?

majesticio avatar Jan 19 '23 17:01 majesticio

Mexico/SINAICA is in the main deployment I believe. perhaps @caparker can confirm

russbiggs avatar Jan 19 '23 18:01 russbiggs

Correct. We have 3 deployments of the lambda; the main, London and the air now offset deployment. The Mexico data comes form the main one.

On Thu, Jan 19, 2023 at 10:15 AM Russ Biggs @.***> wrote:

Mexico/SINAICA is in the main deployment I believe. perhaps @caparker https://github.com/caparker can confirm

— Reply to this email directly, view it on GitHub https://github.com/openaq/openaq-fetch/issues/920#issuecomment-1397411189, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOE5OXATBV5JUU7XEBWBRDWTGAENANCNFSM6AAAAAARRDKKO4 . You are receiving this because you were mentioned.Message ID: @.***>

caparker avatar Jan 19 '23 18:01 caparker