Cromwell fails to find the docker image when the digest is none (<none>)

[nh13]

I am trying to run Cromwell with docker images that were loaded with docker load. This means that the digests are unavailable (i.e. <none>). Unforutnately, this means that when looking up the image locally (i.e. when the config docker.hash-lookup.method="local" is used), the image is not found. The offending lines of code are: https://github.com/broadinstitute/cromwell/blob/1898d8103a06d160dc721d464862313e78ee7a2c/dockerHashing/src/main/scala/cromwell/docker/local/DockerCliClient.scala#L26 https://github.com/broadinstitute/cromwell/blob/1898d8103a06d160dc721d464862313e78ee7a2c/dockerHashing/src/main/scala/cromwell/docker/local/DockerCliClient.scala#L78-L92

Can we instead use the image ID instead of the digest when using local images?

log output

[INFO] [09/16/2019 11:07:14.821] [cromwell-system-akka.dispatchers.engine-dispatcher-40] [akka://cromwell-system/user/SingleWorkflowRunnerActor/JobExecutionTokenDispenser] Not triggering log of token queue status. Effective log interval = None
[INFO] [09/16/2019 11:07:14.830] [cromwell-system-akka.dispatchers.engine-dispatcher-76] [akka://cromwell-system/user/SingleWorkflowRunnerActor/JobExecutionTokenDispenser] Assigned new job execution tokens to the following groups: 2b766fe6: 1
[2019-09-16 11:07:16,20] [error] Docker pull failed
java.lang.RuntimeException: Error running: docker pull <image>
Exit code: 1
Error response from daemon: pull access denied for <image> repository does not exist or may require 'docker login': denied: requested access to the resource is denied

	at cromwell.docker.local.DockerCliClient.$anonfun$forRun$1(DockerCliClient.scala:58)
	at scala.util.Try$.apply(Try.scala:213)
	at cromwell.docker.local.DockerCliClient.forRun(DockerCliClient.scala:50)
	at cromwell.docker.local.DockerCliClient.pull(DockerCliClient.scala:37)
	at cromwell.docker.local.DockerCliClient.pull$(DockerCliClient.scala:36)
	at cromwell.docker.local.DockerCliClient$.pull(DockerCliClient.scala:94)
	at cromwell.docker.local.DockerCliFlow$.pull(DockerCliFlow.scala:101)
	at cromwell.docker.local.DockerCliFlow.$anonfun$run$1(DockerCliFlow.scala:35)
	at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:139)
	at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:351)
	at cats.effect.internals.IORunLoop$RestartCallback.run(IORunLoop.scala:362)
	at cats.effect.internals.Trampoline.cats$effect$internals$Trampoline$$immediateLoop(Trampoline.scala:70)
	at cats.effect.internals.Trampoline.startLoop(Trampoline.scala:36)
	at cats.effect.internals.TrampolineEC$JVMTrampoline.super$startLoop(TrampolineEC.scala:93)
	at cats.effect.internals.TrampolineEC$JVMTrampoline.$anonfun$startLoop$1(TrampolineEC.scala:93)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
	at cats.effect.internals.TrampolineEC$JVMTrampoline.startLoop(TrampolineEC.scala:93)
	at cats.effect.internals.Trampoline.execute(Trampoline.scala:43)
	at cats.effect.internals.TrampolineEC.execute(TrampolineEC.scala:44)
	at cats.effect.internals.ForwardCancelable.loop$1(ForwardCancelable.scala:46)
	at cats.effect.internals.ForwardCancelable.$anonfun$cancel$1(ForwardCancelable.scala:52)
	at cats.effect.internals.ForwardCancelable.$anonfun$cancel$1$adapted(ForwardCancelable.scala:52)
	at cats.effect.internals.IORunLoop$RestartCallback.start(IORunLoop.scala:337)
	at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:119)
	at cats.effect.internals.IORunLoop$.start(IORunLoop.scala:34)
	at cats.effect.IO.unsafeRunAsync(IO.scala:258)
	at cats.effect.internals.IORace$.onSuccess$1(IORace.scala:40)
	at cats.effect.internals.IORace$.$anonfun$simple$4(IORace.scala:79)
	at cats.effect.internals.IORace$.$anonfun$simple$4$adapted(IORace.scala:77)
	at cats.effect.internals.IORunLoop$.cats$effect$internals$IORunLoop$$loop(IORunLoop.scala:136)
	at cats.effect.internals.IORunLoop$RestartCallback.signal(IORunLoop.scala:351)
	at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:372)
	at cats.effect.internals.IORunLoop$RestartCallback.apply(IORunLoop.scala:312)
	at cats.effect.internals.IOShift$Tick.run(IOShift.scala:36)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:49)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

A workaround is setting up a registry to host the images (so we can have digests), but I ask anyway.

[illusional]

Hi @nh13, not from Broad but have you tried turning the docker-digest lookup off with the following in your config:

docker.hash-lookup.enabled = false

[asalimih]

I'm having the same issue, is there any solution other than docker.hash-lookup.enabled = false because it may cause problems for call caching

[gr8ape]

I am also encountering this issue and have yet to find resolution.

[aednichols]

I think call caching can't work on images without a hash/digest anyway, since the hash is taken into account to evaluable caching eligibility.