cromwell icon indicating copy to clipboard operation
cromwell copied to clipboard
verified publisher icon broadinstitute

Format of docker-image-cache-manifest-file?

Open monkollek opened this issue 3 years ago • 4 comments

According to https://cromwell.readthedocs.io/en/stable/backends/Google/ The format for the json file should be

{
  "biocontainers/samtools:1.3.1": "projects/broad-dsde-cromwell-dev/global/images/v1-docker-biocontainers-samtools-1-3-1",
  "gcr.io/gcp-runtimes/ubuntu_16_0_4:latest": "projects/broad-dsde-cromwell-dev/global/images/v1-docker-gcr-io-gcp-runtimes-ubuntu-16-0-4-latest",
  ...
}

I followed this format but got this error

[2022-11-20 18:17:16,88] [warn] Failed to build PipelinesApiConfigurationAttributes on attempt 1 of 3, retrying.
cromwell.backend.google.pipelines.common.PipelinesApiConfigurationAttributes$$anon$1: Google Pipelines API configuration is not valid: Errors:
Attempt to decode value on failed cursor: DownField(manifestFormatVersion)
        at cromwell.backend.google.pipelines.common.PipelinesApiConfigurationAttributes$.apply(PipelinesApiConfigurationAttributes.scala:307)
        at cromwell.backend.google.pipelines.common.PipelinesApiBackendLifecycleActorFactory.defaultBuildAttributes$1(PipelinesApiBackendLifecycleActorFactory.scala:32)
        at cromwell.backend.google.pipelines.common.PipelinesApiBackendLifecycleActorFactory.$anonfun$papiAttributes$1(PipelinesApiBackendLifecycleActorFactory.scala:34)
        at scala.util.Try$.apply(Try.scala:210)
        at cromwell.backend.google.pipelines.common.PipelinesApiBackendLifecycleActorFactory$.cromwell$backend$google$pipelines$common$PipelinesApiBackendLifecycleActorFactory$$build$1(PipelinesApiBackendLifecycleActorFactory.scala:109)
        at cromwell.backend.google.pipelines.common.PipelinesApiBackendLifecycleActorFactory$.robustBuildAttributes(PipelinesApiBackendLifecycleActorFactory.scala:120)
        at cromwell.backend.google.pipelines.common.PipelinesApiBackendLifecycleActorFactory.<init>(PipelinesApiBackendLifecycleActorFactory.scala:34)
        at cromwell.backend.google.pipelines.v2beta.PipelinesApiLifecycleActorFactory.<init>(PipelinesApiLifecycleActorFactory.scala:10)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at cromwell.engine.backend.BackendConfigurationEntry.$anonfun$asBackendLifecycleActorFactory$1(BackendConfiguration.scala:13)
        at scala.util.Try$.apply(Try.scala:210)
        at cromwell.engine.backend.BackendConfigurationEntry.asBackendLifecycleActorFactory(BackendConfiguration.scala:14)
        at cromwell.engine.backend.CromwellBackends.$anonfun$backendLifecycleActorFactories$1(CromwellBackends.scala:14)
        at scala.collection.immutable.List.map(List.scala:246)
        at cromwell.engine.backend.CromwellBackends.<init>(CromwellBackends.scala:14)
        at cromwell.engine.backend.CromwellBackends$.initBackends(CromwellBackends.scala:42)
        at cromwell.server.CromwellSystem.$init$(CromwellSystem.scala:68)
        at cromwell.CromwellEntryPoint$$anon$1.<init>(CromwellEntryPoint.scala:123)
        at cromwell.CromwellEntryPoint$.$anonfun$buildCromwellSystem$1(CromwellEntryPoint.scala:123)
        at scala.util.Try$.apply(Try.scala:210)
        at cromwell.CromwellEntryPoint$.buildCromwellSystem(CromwellEntryPoint.scala:123)
        at cromwell.CromwellEntryPoint$.runSingle(CromwellEntryPoint.scala:66)
        at cromwell.CromwellApp$.runCromwell(CromwellApp.scala:14)
        at cromwell.CromwellApp$.delayedEndpoint$cromwell$CromwellApp$1(CromwellApp.scala:25)
        at cromwell.CromwellApp$delayedInit$body.apply(CromwellApp.scala:3)
        at scala.Function0.apply$mcV$sp(Function0.scala:39)
        at scala.Function0.apply$mcV$sp$(Function0.scala:39)
        at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
        at scala.App.$anonfun$main$1(App.scala:76)
        at scala.App.$anonfun$main$1$adapted(App.scala:76)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:926)
        at scala.App.main(App.scala:76)
        at scala.App.main$(App.scala:74)
        at cromwell.CromwellApp$.main(CromwellApp.scala:3)
        at cromwell.CromwellApp.main(CromwellApp.scala)

which to me implies I got the format incorrect. The format is a lot different to reference-disk-localization-manifests so just checking.

Are there any examples that can be shared as gs://gcp-public-data--broad-references/refdisk_manifest.json and the script at https://github.com/broadinstitute/cromwell/blob/develop/scripts/reference_disks/create_images.sh was very helpful!

Thanks!

monkollek avatar Nov 20 '22 18:11 monkollek

I found the code relating to the format. Please update the documentation :o)

https://github.com/broadinstitute/cromwell/blob/32d5d0cbf07e46f56d3d070f457eaff0138478d5/supportedBackends/google/pipelines/common/src/main/scala/cromwell/backend/google/pipelines/common/PipelinesApiDockerCacheMappingOperations.scala

monkollek avatar Nov 20 '22 18:11 monkollek

Here is an example that worked for me for those that are interested.

{
  "manifestFormatVersion": 2,
  "dockerImageCacheMap": {"us.gcr.io/broad-gatk/gatk:4.3.0.0":
                            {"dockerImageDigest": "sha256:e7996ba655225c1cde0a1faec6a113e217758310af2cf99b00d61dae8ec6e9f2",
                            "diskImageName":"projects/${PROJECT}/global/images/${IMAGE}"}
                         }

}

There is some details on how the disk image should look on the Google API website

"The Compute Engine Disk Images to use as a Docker cache. The disks will be mounted into the Docker folder in a way that the images present in the cache will not need to be pulled. The digests of the cached images must match those of the tags used or the latest version will still be pulled. The root directory of the ext4 image must contain image and overlay2 directories copied from the Docker directory of a VM where the desired Docker images have already been pulled. Any images pulled that are not cached will be stored on the first cache disk instead of the boot disk. Only a single image is supported."

After all that it actually took longer using the disk cache :(

monkollek avatar Nov 20 '22 19:11 monkollek

How do you get "path" and "crc32c" ?

nicholema avatar Aug 23 '23 19:08 nicholema

@monkollek was this also part of creating your reference disk? I ask this as you mention that you were able to make this also with the create_images.sh

jbakerpmc avatar Aug 12 '24 09:08 jbakerpmc