glow icon indicating copy to clipboard operation
glow copied to clipboard

Published 20 hours ago • verified publisher icon projectglow

Test recalibration + DBR 9.1 Docker images

Open a0x8o opened this issue 2 years ago • 4 comments

What changes are proposed in this pull request?

  1. Temporarily stop Hail tests (until these are upgraded for where breaking Hail changes are presently failing)
  2. Fix broken links in Getting Started document source for Databricks Repos Sync/Git Clone
  3. Provide Docker images for DBR 9.1

How is this patch tested?

  • [X] Unit tests
  • [X] Integration tests
  • [X] Manual tests

I am aiming to address three things here:

  1. Hail-IS made some changes that are breaking the Hail tests in projectglow. I'm switching off these tests until someone can fix the Hail tests.
  2. Getting Started had another broken link which I've fixed. I also have found that these tests should have a higher timeout value as frequently the tests fail due to short load timeouts.
  3. I believe we need to provide the DBR9.1 Docker images in addition to DBR10.4 images in master. I've reintroduced these to master.

a0x8o avatar Jun 30 '22 21:06 a0x8o

@williambrandler @karenfeng I am trying to address three things here:

  1. Hail-IS made some changes that are breaking the Hail tests in Projectglow. I'm switching off these tests until someone (I or another engineer) can fix the tests.
  2. Getting Started had another broken link which I've fixed. I also have found that these tests should have a higher timeout value as frequently the tests fail due to short load timeouts.
  3. I believe we need to provide the DBR9.1 Docker images in addition to DBR10.4 images in master.

a0x8o avatar Jun 30 '22 21:06 a0x8o

ah ok, thanks @a0x8o

I removed 9.1 when bumping to 10.4, but it seems reasonable to maintain both versions until 9.1 is obsolete

Fine to remove hail tests to get the ci passing again, can add back in once https://github.com/hail-is/hail/issues/11707 is resolved. The Hail team are waiting for EMR and dataproc to update to spark 3.2.0, after which I expect the tests to pass (once the Hail version is updated)

I see EMR is on 3.2.0 with emr-6.6.0, however dataproc is still on spark 3.1.3 (dataproc versioning)

williambrandler avatar Jul 01 '22 01:07 williambrandler

@williambrandler any idea why we are seeing the following: https://app.circleci.com/pipelines/github/projectglow/glow/3131/workflows/a1e5ccf6-ada6-480b-91d8-9983139c4cd9/jobs/9682

a0x8o avatar Jul 07 '22 16:07 a0x8o

@williambrandler any idea why we are seeing the following: https://app.circleci.com/pipelines/github/projectglow/glow/3131/workflows/a1e5ccf6-ada6-480b-91d8-9983139c4cd9/jobs/9682

either a transient issue with circleci or permissions related @a0x8o

Try this link to join the project? https://app.circleci.com/pipelines/github/projectglow/glow?invite=true

williambrandler avatar Jul 07 '22 21:07 williambrandler