Dan King
Dan King
``` starting test is.hail.io.fs.GoogleStorageFSSuite.largeDirectoryOperations... Exception: org.scalatest.exceptions.TestFailedException: files not deleted: gs://hail-test-ezlis/fs-suite/delete-many-files/72e64985-c4b9-46ff-9191-93b8c4589083/1890.suffix at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528) at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527) at org.scalatest.testng.TestNGSuite.newAssertionFailedException(TestNGSuite.scala:67) at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501) at is.hail.io.fs.FSSuite.largeDirectoryOperations(FSSuite.scala:445) at is.hail.io.fs.FSSuite.largeDirectoryOperations$(FSSuite.scala:430) at is.hail.io.fs.GoogleStorageFSSuite.largeDirectoryOperations(GoogleStorageFSSuite.scala:10) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at...
Dan to connect Hana with Ed. Reassign once that's done.
Hmm. OK, this is an interesting one. You're basically trying to do: 1. For each gene-interval, get all the variants in that gene. 2. Filter to variants having the primary...
@hanars , for my own curiosity, what was the timing on this query before 0.2.126?
One possible point of confusion, there's a Hail Query "LocalBackend" which is a mostly internal thing and that's different than spark-in-local-mode (aka `hl.init(..., master='local[*]')`); I'm pretty sure we're using spark-in-local-mode...
Next steps: 1. upload the profile, the `mt.describe()`, metadata.json.gz from the MT/HT to the team chat and get feedback (Chris, Patrick take a look). Decode appears quite slow.
Hey @shengqh ! Yeah, this is a bug in Kryo, a serialization library used by Spark, which cannot handle the size of data you're producing. This is partly a deficiency...
> We have a 35K cohort. The VCF format of chr1 is 2.4T. Heh. So, yes, "project" VCFs grow super-linearly in the number of samples. I (and others) are currently...
Hail team suggests this is a good opportunity to work together with gnomAD team to transfer knowledge on how to build these kinds of aggregators. I will talk to gnomAD...