mergestat-lite icon indicating copy to clipboard operation
mergestat-lite copied to clipboard

feat(actions): First iteration of GITHUB_ACTIONS sync

Open ramiroJCB opened this issue 2 years ago • 0 comments

With this pull request we open our first iteration over the new sync type GITHUB_ACTIONS this one is a complex one because itself divides in three main categories related in order workflows-> workflow runs -> workflow jobs -> workflow logs Also we don't use the graphql client neither the mergestat-light dependency for this sync .Mainly because we are moving away from mergestat-light dependency and the graphql api from Github does not provided the information we require for this sync .

*NOTE * There are a lot of transient errors from GitHub API that we are not handling in this PR , those errors will be handled in #408 .

For this specific problem we choosed this algorithm to manage the iteration of getting all github actions

`

  • WorkflowIterator(owner,repo,repoID)

    • Pull a pages of Workflows from the API
      • For each workflows_page
        • HandleWorkflowsUpsert([]workflows ,repoID)
      • WorkflowrunsIterator(owner,repo,[]workflows)
  • WorkflowrunsIterator(owner,repo,[]workflows,repoID)

    • Pull a page of Workflow runs from the API
      • For each workflowRuns_Page
        • HandleWorkflowRunsUpsert([]workflowRuns,repoID)
      • WorkflowjobsIterator(owner,repo,[]workflowRuns)
  • WorkflowjobsIterator([]runs)

    • Pull a pages of Workflow jobs from the API
      • For each []jobs_page
        • Per job we pull a log getWorkflorJobLog(owner,repo,jobID)
        • HandleUpsertWorkflowJob (job, log,repoID) `

    and a picture :) resolves #366
    #367 #368

Screen Shot 2022-10-12 at 10 19 26 AM

ramiroJCB avatar Oct 12 '22 15:10 ramiroJCB