I installed Miniconda3-latest-MacOSX-x86_64.sh, Bigdata script (bds_Darwin.tar). Also delete anaconda in my Mac and removed Anaconda path from .bash_profile. But when I run "bash install_dependencies.sh", it still use anaconda. the detail as below:

Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

trim-galore==0.4.1
bowtie2==2.2.6

Current channels:

https://repo.anaconda.com/pkgs/main/osx-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/osx-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/osx-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/osx-64
https://repo.anaconda.com/pkgs/pro/noarch
https://conda.anaconda.org/bioconda/osx-64
https://conda.anaconda.org/bioconda/noarch
https://conda.anaconda.org/r/osx-64
https://conda.anaconda.org/r/noarch
https://conda.anaconda.org/bcbio/osx-64
https://conda.anaconda.org/bcbio/noarch
https://conda.anaconda.org/daler/osx-64
https://conda.anaconda.org/daler/noarch
https://conda.anaconda.org/asmeurer/osx-64
https://conda.anaconda.org/asmeurer/noarch

To search for alternate channels that may provide the conda package you're looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

how to fix it? It seems that I can't delete anaconda completely. Ps: I use the method showed in this link (https://docs.anaconda.com/anaconda/install/uninstall) to delete anaconda.

Jul 11 '18 15:07 llz-hiv

You can try removing trim-galore and change the version of bowtie2 in requirements.txt. Check available versions for MacOS here https://anaconda.org/bioconda/bowtie2/files.

However, we strongly recommend to install docker and use our new pipeline which supports docker https://github.com/ENCODE-DCC/atac-seq-pipeline/.

Jul 11 '18 21:07 leepc12

Hi Jin,

Thank you for your suggestion! I was trying to install the new ENCODE-ATAC-seq-pipeline on my local computer with docker yesterday. But when I ran the pipeline, it seems the pipeline was stuck in "bowtie2" step. The computer keep running till now and seems will run forever and become very hot. Below is content of the input.json and hg38.tsv I used in pipeline. The details of running progress is also shown below as well:

Here is the input.json: "atac.pipeline_type" : "atac", "atac.genome_tsv" : "/Users/liangzhu/data/encode-pipeline-genome-data/hg38.tsv", "atac.fastqs" : [[ [ "/Users/liangzhu/git/atacomate/SRR891275/SRR891275_1.fastq.gz", "/Users/liangzhu/git/atacomate/SRR891275/SRR891275_2.fastq.gz" ]]],

"atac.paired_end" : true,

"atac.trim_adapter.auto_detect_adapter" : true,

"atac.smooth_win" : 73,
"atac.enable_idr" : true,
"atac.idr_thresh" : 0.05,

"atac.qc_report.name" : "SRR891275",
"atac.qc_report.desc" : "hESC_ATAC"

}

Here is the hg38.tsv

ref_fa /Users/liangzhu/data/encode-pipeline-genome-data/hg38/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz bowtie2_idx_tar /Users/liangzhu/data/encode-pipeline-genome-data/hg38/bowtie2_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar bwa_idx_tar /Users/liangzhu/data/encode-pipeline-genome-data/hg38/bwa_index/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar chrsz /Users/liangzhu/data/encode-pipeline-genome-data/hg38/hg38.chrom.sizes blacklist /Users/liangzhu/data/encode-pipeline-genome-data/hg38/hg38.blacklist.bed.gz gensz hs tss_enrich /Users/liangzhu/data/encode-pipeline-genome-data/hg38/ataqc/hg38_gencode_tss_unique.bed.gz dnase /Users/liangzhu/data/encode-pipeline-genome-data/hg38/ataqc/reg2map_honeybadger2_dnase_all_p10_ucsc.hg19_to_hg38.bed.gz prom /Users/liangzhu/data/encode-pipeline-genome-data/hg38/ataqc/reg2map_honeybadger2_dnase_prom_p2.hg19_to_hg38.bed.gz enh /Users/liangzhu/data/encode-pipeline-genome-data/hg38/ataqc/reg2map_honeybadger2_dnase_enh_p2.hg19_to_hg38.bed.gz reg2map /Users/liangzhu/data/encode-pipeline-genome-data/hg38/ataqc/hg38_dnase_avg_fseq_signal_formatted.txt.gz reg2map_bed /Users/liangzhu/data/encode-pipeline-genome-data/hg38/ataqc/hg38_celltype_compare_subsample.bed.gz roadmap_meta /Users/liangzhu/data/encode-pipeline-genome-data/hg38/ataqc/hg38_dnase_avg_fseq_signal_metadata.txt

Here is the details of running progress:

Liangzhus-MacBook-Pro:atac-seq-pipeline-master liangzhu$ java -jar -Dconfig.file=backends/backend.conf cromwell-33.1.jar run atac.wdl -i input.json -o workflow_opts/docker.json [2018-07-12 22:19:25,53] [info] Running with database db.url = jdbc:hsqldb:mem:44c09415-ad4c-49eb-afa6-4fe998615f4e;shutdown=false;hsqldb.tx=mvcc [2018-07-12 22:19:33,01] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000 [2018-07-12 22:19:33,02] [info] [RenameWorkflowOptionsInMetadata] 100% [2018-07-12 22:19:33,14] [info] Running with database db.url = jdbc:hsqldb:mem:cc65dde9-9235-4758-87a9-0525a68f4b79;shutdown=false;hsqldb.tx=mvcc [2018-07-12 22:19:33,39] [warn] This actor factory is deprecated. Please use cromwell.backend.google.pipelines.v1alpha2.PipelinesApiLifecycleActorFactory for PAPI v1 or cromwell.backend.google.pipelines.v2alpha1.PipelinesApiLifecycleActorFactory for PAPI v2 [2018-07-12 22:19:33,39] [warn] Couldn't find a suitable DSN, defaulting to a Noop one. [2018-07-12 22:19:33,40] [info] Using noop to send events. [2018-07-12 22:19:33,99] [info] Slf4jLogger started [2018-07-12 22:19:34,12] [info] Workflow heartbeat configuration: { "cromwellId" : "cromid-fcd6307", "heartbeatInterval" : "2 minutes", "ttl" : "10 minutes", "writeBatchSize" : 10000, "writeThreshold" : 10000 } [2018-07-12 22:19:34,14] [info] Metadata summary refreshing every 2 seconds. [2018-07-12 22:19:34,18] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds. [2018-07-12 22:19:34,18] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds. [2018-07-12 22:19:34,19] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds. [2018-07-12 22:19:34,80] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds. [2018-07-12 22:19:34,82] [info] JES batch polling interval is 33333 milliseconds [2018-07-12 22:19:34,82] [info] JES batch polling interval is 33333 milliseconds [2018-07-12 22:19:34,82] [info] JES batch polling interval is 33333 milliseconds [2018-07-12 22:19:34,82] [info] PAPIQueryManager Running with 3 workers [2018-07-12 22:19:34,82] [info] SingleWorkflowRunnerActor: Submitting workflow [2018-07-12 22:19:34,87] [info] Unspecified type (Unspecified version) workflow e0ae37ea-6dc1-471e-825a-99d79d886bad submitted [2018-07-12 22:19:34,92] [info] SingleWorkflowRunnerActor: Workflow submitted e0ae37ea-6dc1-471e-825a-99d79d886bad [2018-07-12 22:19:34,92] [info] 1 new workflows fetched [2018-07-12 22:19:34,92] [info] WorkflowManagerActor Starting workflow e0ae37ea-6dc1-471e-825a-99d79d886bad [2018-07-12 22:19:34,93] [info] WorkflowManagerActor Successfully started WorkflowActor-e0ae37ea-6dc1-471e-825a-99d79d886bad [2018-07-12 22:19:34,93] [info] Retrieved 1 workflows from the WorkflowStoreActor [2018-07-12 22:19:34,94] [warn] SingleWorkflowRunnerActor: received unexpected message: Done in state RunningSwraData [2018-07-12 22:19:34,94] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes. [2018-07-12 22:19:35,00] [info] MaterializeWorkflowDescriptorActor [e0ae37ea]: Parsing workflow as WDL draft-2 [2018-07-12 22:20:05,53] [info] MaterializeWorkflowDescriptorActor [e0ae37ea]: Call-to-Backend assignments: atac.bam2ta -> Local, atac.idr_pr -> Local, atac.pool_ta -> Local, atac.reproducibility_overlap -> Local, atac.macs2_pooled -> Local, atac.macs2_pr2 -> Local, atac.reproducibility_idr -> Local, atac.bowtie2 -> Local, atac.pool_ta_pr1 -> Local, atac.overlap_pr -> Local, atac.xcor -> Local, atac.read_genome_tsv -> Local, atac.idr_ppr -> Local, atac.pool_ta_pr2 -> Local, atac.overlap -> Local, atac.spr -> Local, atac.idr -> Local, atac.macs2_ppr2 -> Local, atac.qc_report -> Local, atac.macs2 -> Local, atac.macs2_ppr1 -> Local, atac.ataqc -> Local, atac.macs2_pr1 -> Local, atac.overlap_ppr -> Local, atac.filter -> Local, atac.trim_adapter -> Local [2018-07-12 22:20:05,62] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,62] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,62] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,62] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,62] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,62] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,62] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,62] [warn] Local [e0ae37ea]: Key/s [preemptible, disks, cpu, time, memory] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,62] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,62] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,62] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:05,63] [warn] Local [e0ae37ea]: Key/s [cpu, memory, time, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-12 22:20:07,86] [info] WorkflowExecutionActor-e0ae37ea-6dc1-471e-825a-99d79d886bad [e0ae37ea]: Starting atac.read_genome_tsv [2018-07-12 22:20:07,86] [info] WorkflowExecutionActor-e0ae37ea-6dc1-471e-825a-99d79d886bad [e0ae37ea]: Condition met: '!align_only && !true_rep_only'. Running conditional section [2018-07-12 22:20:07,86] [info] WorkflowExecutionActor-e0ae37ea-6dc1-471e-825a-99d79d886bad [e0ae37ea]: Condition met: '!align_only && !true_rep_only && enable_idr'. Running conditional section [2018-07-12 22:20:07,86] [info] WorkflowExecutionActor-e0ae37ea-6dc1-471e-825a-99d79d886bad [e0ae37ea]: Condition met: '!true_rep_only'. Running conditional section [2018-07-12 22:20:07,86] [info] WorkflowExecutionActor-e0ae37ea-6dc1-471e-825a-99d79d886bad [e0ae37ea]: Condition met: 'enable_idr'. Running conditional section [2018-07-12 22:20:07,86] [info] WorkflowExecutionActor-e0ae37ea-6dc1-471e-825a-99d79d886bad [e0ae37ea]: Condition met: '!disable_xcor'. Running conditional section [2018-07-12 22:20:07,87] [info] WorkflowExecutionActor-e0ae37ea-6dc1-471e-825a-99d79d886bad [e0ae37ea]: Condition met: 'enable_idr'. Running conditional section [2018-07-12 22:20:09,31] [warn] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.read_genome_tsv:NA:1]: Unrecognized runtime attribute keys: disks, cpu, time, memory [2018-07-12 22:20:09,67] [info] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.read_genome_tsv:NA:1]: echo "Reading genome_tsv /cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-read_genome_tsv/inputs/-474501674/hg38.tsv ..." [2018-07-12 22:20:09,74] [info] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.read_genome_tsv:NA:1]: executing: # make sure there is no preexisting Docker CID file rm -f /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-read_genome_tsv/execution/docker_cid

run as in the original configuration without --rm flag (will remove later)

docker run
--cidfile /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-read_genome_tsv/execution/docker_cid
-i

--entrypoint /bin/bash
-v /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-read_genome_tsv:/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-read_genome_tsv
quay.io/encode-dcc/atac-seq-pipeline@sha256:97940d0951cf2c3df554584f18ee8e6f1ac9bd39aa39326c6a0f7110d8037398 /cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-read_genome_tsv/execution/script

get the return code (working even if the container was detached)

rc=$(docker wait cat /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-read_genome_tsv/execution/docker_cid)

remove the container after waiting

docker rm cat /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-read_genome_tsv/execution/docker_cid

return exit code

exit $rc [2018-07-12 22:20:11,96] [info] WorkflowExecutionActor-e0ae37ea-6dc1-471e-825a-99d79d886bad [e0ae37ea]: Starting atac.trim_adapter [2018-07-12 22:20:12,83] [warn] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.trim_adapter:0:1]: Unrecognized runtime attribute keys: disks, cpu, time, memory [2018-07-12 22:20:12,87] [info] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.trim_adapter:0:1]: python $(which encode_trim_adapter.py)
/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-trim_adapter/shard-0/execution/write_tsv_af6e37112f59ce68a451c1a28e5f9c68.tmp
--adapters /cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-trim_adapter/shard-0/execution/write_tsv_d41d8cd98f00b204e9800998ecf8427e.tmp
--paired-end
--auto-detect-adapter
--min-trim-len 5
--err-rate 0.1
--nth 2 [2018-07-12 22:20:12,88] [info] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.trim_adapter:0:1]: executing: # make sure there is no preexisting Docker CID file rm -f /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-trim_adapter/shard-0/execution/docker_cid

run as in the original configuration without --rm flag (will remove later)

docker run
--cidfile /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-trim_adapter/shard-0/execution/docker_cid
-i

--entrypoint /bin/bash
-v /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-trim_adapter/shard-0:/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-trim_adapter/shard-0
quay.io/encode-dcc/atac-seq-pipeline@sha256:97940d0951cf2c3df554584f18ee8e6f1ac9bd39aa39326c6a0f7110d8037398 /cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-trim_adapter/shard-0/execution/script

get the return code (working even if the container was detached)

rc=$(docker wait cat /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-trim_adapter/shard-0/execution/docker_cid)

remove the container after waiting

docker rm cat /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-trim_adapter/shard-0/execution/docker_cid

return exit code

exit $rc [2018-07-12 22:20:14,22] [info] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.trim_adapter:0:1]: job id: 6153 [2018-07-12 22:20:14,22] [info] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.read_genome_tsv:NA:1]: job id: 6143 [2018-07-12 22:20:14,22] [info] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.trim_adapter:0:1]: Status change from - to WaitingForReturnCodeFile [2018-07-12 22:20:14,22] [info] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.read_genome_tsv:NA:1]: Status change from - to Done [2018-07-12 22:24:05,85] [info] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.trim_adapter:0:1]: Status change from WaitingForReturnCodeFile to Done [2018-07-12 22:24:07,58] [info] WorkflowExecutionActor-e0ae37ea-6dc1-471e-825a-99d79d886bad [e0ae37ea]: Starting atac.bowtie2 [2018-07-12 22:24:07,81] [warn] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.bowtie2:0:1]: Unrecognized runtime attribute keys: preemptible, disks, cpu, time, memory [2018-07-12 22:24:07,83] [info] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.bowtie2:0:1]: python $(which encode_bowtie2.py)
/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-bowtie2/shard-0/inputs/-260578579/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.tar
/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-bowtie2/shard-0/inputs/-102408422/merge_fastqs_R1_SRR891275_1.trim.merged.fastq.gz /cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-bowtie2/shard-0/inputs/-102408422/merge_fastqs_R2_SRR891275_2.trim.merged.fastq.gz
--paired-end
--multimapping 0

--nth 4 [2018-07-12 22:24:07,84] [info] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.bowtie2:0:1]: executing: # make sure there is no preexisting Docker CID file rm -f /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-bowtie2/shard-0/execution/docker_cid

run as in the original configuration without --rm flag (will remove later)

docker run
--cidfile /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-bowtie2/shard-0/execution/docker_cid
-i

--entrypoint /bin/bash
-v /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-bowtie2/shard-0:/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-bowtie2/shard-0
quay.io/encode-dcc/atac-seq-pipeline@sha256:97940d0951cf2c3df554584f18ee8e6f1ac9bd39aa39326c6a0f7110d8037398 /cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-bowtie2/shard-0/execution/script

get the return code (working even if the container was detached)

rc=$(docker wait cat /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-bowtie2/shard-0/execution/docker_cid)

remove the container after waiting

docker rm cat /Users/liangzhu/atac-seq-pipeline-master/cromwell-executions/atac/e0ae37ea-6dc1-471e-825a-99d79d886bad/call-bowtie2/shard-0/execution/docker_cid

return exit code

exit $rc [2018-07-12 22:24:09,19] [info] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.bowtie2:0:1]: job id: 6163 [2018-07-12 22:24:09,19] [info] BackgroundConfigAsyncJobExecutionActor [e0ae37eaatac.bowtie2:0:1]: Status change from - to WaitingForReturnCodeFile

The pipeline is stuck on the last step and keep running. Is there any problem? Thank you in advance!

Jul 13 '18 14:07 llz-hiv

MacBook Pro doesn't have enough resources to run pipelines smoothly. Please try to add "atac.bowtie2.cpu" : 1, in your input JSON (see the resource section https://encode-dcc.github.io/wdl-pipelines/input_json_atac.html for details). Also try to reduce your sample size. Or try on Google Cloud or DNANexus. New pipeline supports these platforms.

Jul 13 '18 19:07 leepc12

Thank you, I will try. Another question: I want to run MySQL when I run the pipeline. So I run a mysql server with docker. do I need to change something in my input.json or backends files as well?

Jul 13 '18 20:07 llz-hiv

@llz-hiv : please take a look https://encode-dcc.github.io/wdl-pipelines/install.html#running-mysql-server-with-docker

Jul 13 '18 20:07 leepc12

Hi Jin,

I am trying to run the pipeline on google platform and DNANexus. I met problems on both platform.

For google-platform, the error message is as below: Caused by: com.google.api.client.http.HttpResponseException: 403 Forbidden { "error": { "errors": [ { "domain": "global", "reason": "forbidden", "message": "[email protected] does not have storage.objects.create access to your-bucket-name." } ],

In fact, I created a bucket named 'atacout' in my google project, shall I create a file under such directory? How? The command line I used is as below: $ java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=google -Dbackend.provider.google.config.project=liangzhu-atac2018 -Dbackend.provider.google.config.root=atacout/test cromwell-33.1.jar run atac.wdl -i input.json -o workflow_opts/docker.json

For DNAnexus platform: I created a directory named 'runatac' in my dx-project. the command line is as below: $ java -jar dxWDL-0.69.jar compile atac.wdl -f -folder /runatac -defaults input-dx.json -extras workflow_opts/docker.json

the error message is as below: Unsupported runtime attribute preemptible,we currently support Set(dx_instance_type, disks, docker, cpu, memory) Unsupported runtime attribute failOnStderr,we currently support Set(dx_instance_type, disks, docker, cpu, memory) Unsupported runtime attribute bootDiskSizeGb,we currently support Set(dx_instance_type, disks, docker, cpu, memory) Unsupported runtime attribute continueOnReturnCode,we currently support Set(dx_instance_type, disks, docker, cpu, memory) Unsupported runtime attribute noAddress,we currently support Set(dx_instance_type, disks, docker, cpu, memory) Unsupported runtime attribute zones,we currently support Set(dx_instance_type, disks, docker, cpu, memory) java.lang.Exception: project is unspecified at dxWDL.Main$.pathOptions(Main.scala:254) at dxWDL.Main$.compile(Main.scala:384) at dxWDL.Main$.dispatchCommand(Main.scala:588) at dxWDL.Main$.delayedEndpoint$dxWDL$Main$1(Main.scala:641) at dxWDL.Main$delayedInit$body.apply(Main.scala:12) at scala.Function0.apply$mcV$sp(Function0.scala:34) at scala.Function0.apply$mcV$sp$(Function0.scala:34) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App.$anonfun$main$1$adapted(App.scala:76) at scala.collection.immutable.List.foreach(List.scala:389) at scala.App.main(App.scala:76) at scala.App.main$(App.scala:74) at dxWDL.Main$.main(Main.scala:12) at dxWDL.Main.main(Main.scala)

Do I need to modify the docker.json file? or is there any problems in my input.json? my input-dx.json for DNANexus platform is as below: { "atac.pipeline_type" : "atac", "atac.genome_tsv" : "dx://Users/liangzhu/data/encode-pipeline-genome-data/hg38.tsv", "atac.fastqs" : [[ [ "dx://Users/liangzhu/git/atacomate/SRR891275/SRR891275_1.fastq.gz", "dx://Users/liangzhu/git/atacomate/SRR891275/SRR891275_2.fastq.gz" ]]],

"paired_end" : true,
"multimapping" : 4,

"trim_adapter.auto_detect_adapter" : true,

"bowtie2.cpu" : 4,
"bowtie2.mem_mb" : 20000,


"filter.cpu" : 2,
"filter.mem_mb" : 12000,


"macs2_mem_mb" : 16000,

"smooth_win" : 73,
"enable_idr" : true,
"idr_thresh" : 0.05,



"qc_report.name" : "SRR891275",
"qc_report.desc" : "hESC_ATAC"

}

Thanks a lot!

Jul 16 '18 14:07 llz-hiv

It's a Google Cloud bucket permission problem. Go to the Google Cloud Project console and click on the vertical ... at the right end of your bucket name and then give Storage Admin permission to yourself or your colleagues.

Also, don't forget to log in before running pipelines.

$ gcloud auth login --no-launch-browser
$ gcloud auth application-default login --no-launch-browser

Jul 16 '18 18:07 leepc12

Hi Jin, Thank you for your reply. I have already added the 'storage admin' permission to myself. But it does not work. Would you please help me to check the command line I used? I doubt that I input wrong bucket path (-Dbackend.provider.google.config.root=atacout (should it be: =Bucket/atacout? or =Bucket/atacout/sample?))

java -jar -Dconfig.file=backends/backend.conf -Dbackend.default=google -Dbackend.provider.google.config.project=liangzhu-atac2018 -Dbackend.provider.google.config.root=atacout cromwell-33.1.jar run atac.wdl -i input-google.json -o workflow_opts/docker.json

Below is the total information of process: [2018-07-16 16:43:05,96] [info] Running with database db.url = jdbc:hsqldb:mem:b61c6b54-bd1d-4753-ac93-9285ca82d1d6;shutdown=false;hsqldb.tx=mvcc [2018-07-16 16:43:13,37] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000 [2018-07-16 16:43:13,38] [info] [RenameWorkflowOptionsInMetadata] 100% [2018-07-16 16:43:13,49] [info] Running with database db.url = jdbc:hsqldb:mem:adf70a9a-31b4-4838-90e5-2b556269b424;shutdown=false;hsqldb.tx=mvcc [2018-07-16 16:43:13,75] [warn] This actor factory is deprecated. Please use cromwell.backend.google.pipelines.v1alpha2.PipelinesApiLifecycleActorFactory for PAPI v1 or cromwell.backend.google.pipelines.v2alpha1.PipelinesApiLifecycleActorFactory for PAPI v2 [2018-07-16 16:43:13,76] [warn] Couldn't find a suitable DSN, defaulting to a Noop one. [2018-07-16 16:43:13,76] [info] Using noop to send events. [2018-07-16 16:43:14,12] [info] Slf4jLogger started [2018-07-16 16:43:14,25] [info] Workflow heartbeat configuration: { "cromwellId" : "cromid-1989d5f", "heartbeatInterval" : "2 minutes", "ttl" : "10 minutes", "writeBatchSize" : 10000, "writeThreshold" : 10000 } [2018-07-16 16:43:14,28] [info] Metadata summary refreshing every 2 seconds. [2018-07-16 16:43:14,34] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds. [2018-07-16 16:43:14,34] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds. [2018-07-16 16:43:14,34] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds. [2018-07-16 16:43:14,96] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds. [2018-07-16 16:43:14,97] [info] JES batch polling interval is 33333 milliseconds [2018-07-16 16:43:14,97] [info] JES batch polling interval is 33333 milliseconds [2018-07-16 16:43:14,97] [info] JES batch polling interval is 33333 milliseconds [2018-07-16 16:43:14,98] [info] PAPIQueryManager Running with 3 workers [2018-07-16 16:43:14,98] [info] SingleWorkflowRunnerActor: Submitting workflow [2018-07-16 16:43:15,02] [info] Unspecified type (Unspecified version) workflow e6b20ecd-5059-4c69-b624-28bab6740e18 submitted [2018-07-16 16:43:15,06] [info] SingleWorkflowRunnerActor: Workflow submitted e6b20ecd-5059-4c69-b624-28bab6740e18 [2018-07-16 16:43:15,07] [info] 1 new workflows fetched [2018-07-16 16:43:15,07] [info] WorkflowManagerActor Starting workflow e6b20ecd-5059-4c69-b624-28bab6740e18 [2018-07-16 16:43:15,07] [info] WorkflowManagerActor Successfully started WorkflowActor-e6b20ecd-5059-4c69-b624-28bab6740e18 [2018-07-16 16:43:15,07] [info] Retrieved 1 workflows from the WorkflowStoreActor [2018-07-16 16:43:15,08] [warn] SingleWorkflowRunnerActor: received unexpected message: Done in state RunningSwraData [2018-07-16 16:43:15,09] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes. [2018-07-16 16:43:15,15] [info] MaterializeWorkflowDescriptorActor [e6b20ecd]: Parsing workflow as WDL draft-2 [2018-07-16 16:43:45,18] [info] MaterializeWorkflowDescriptorActor [e6b20ecd]: Call-to-Backend assignments: atac.read_genome_tsv -> google, atac.overlap_ppr -> google, atac.macs2_pr2 -> google, atac.macs2_pooled -> google, atac.macs2_ppr1 -> google, atac.idr_pr -> google, atac.pool_ta_pr1 -> google, atac.ataqc -> google, atac.pool_ta_pr2 -> google, atac.idr -> google, atac.overlap -> google, atac.macs2_pr1 -> google, atac.overlap_pr -> google, atac.pool_ta -> google, atac.reproducibility_overlap -> google, atac.xcor -> google, atac.spr -> google, atac.bam2ta -> google, atac.filter -> google, atac.trim_adapter -> google, atac.idr_ppr -> google, atac.bowtie2 -> google, atac.macs2_ppr2 -> google, atac.reproducibility_idr -> google, atac.macs2 -> google, atac.qc_report -> google [2018-07-16 16:43:45,27] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,27] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,27] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,27] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,27] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,27] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,27] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,28] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:45,29] [warn] google [e6b20ecd]: Key/s [time] is/are not supported by backend. Unsupported attributes will not be part of job executions. [2018-07-16 16:43:47,61] [info] WorkflowExecutionActor-e6b20ecd-5059-4c69-b624-28bab6740e18 [e6b20ecd]: Starting atac.read_genome_tsv [2018-07-16 16:43:47,62] [info] WorkflowExecutionActor-e6b20ecd-5059-4c69-b624-28bab6740e18 [e6b20ecd]: Condition met: '!true_rep_only'. Running conditional section [2018-07-16 16:43:47,62] [info] WorkflowExecutionActor-e6b20ecd-5059-4c69-b624-28bab6740e18 [e6b20ecd]: Condition met: '!align_only && !true_rep_only'. Running conditional section [2018-07-16 16:43:47,62] [info] WorkflowExecutionActor-e6b20ecd-5059-4c69-b624-28bab6740e18 [e6b20ecd]: Condition met: '!disable_xcor'. Running conditional section [2018-07-16 16:43:47,62] [info] WorkflowExecutionActor-e6b20ecd-5059-4c69-b624-28bab6740e18 [e6b20ecd]: Condition met: 'enable_idr'. Running conditional section [2018-07-16 16:43:47,62] [info] WorkflowExecutionActor-e6b20ecd-5059-4c69-b624-28bab6740e18 [e6b20ecd]: Condition met: 'enable_idr'. Running conditional section [2018-07-16 16:43:47,62] [info] WorkflowExecutionActor-e6b20ecd-5059-4c69-b624-28bab6740e18 [e6b20ecd]: Condition met: '!align_only && !true_rep_only && enable_idr'. Running conditional section [2018-07-16 16:43:48,23] [warn] PipelinesApiAsyncBackendJobExecutionActor [e6b20ecdatac.read_genome_tsv:NA:1]: Unrecognized runtime attribute keys: time [2018-07-16 16:43:48,41] [info] PipelinesApiAsyncBackendJobExecutionActor [e6b20ecdatac.read_genome_tsv:NA:1]: echo "Reading genome_tsv /cromwell_root/Users/liangzhu/data/encode-pipeline-genome-data/hg38.tsv ..." [2018-07-16 16:43:49,04] [error] PipelinesApiAsyncBackendJobExecutionActor [e6b20ecdatac.read_genome_tsv:NA:1]: Error attempting to Execute cromwell.core.CromwellFatalException: com.google.api.client.http.HttpResponseException: 403 Forbidden { "error": { "errors": [ { "domain": "global", "reason": "forbidden", "message": "[email protected] does not have storage.objects.create access to your-bucket-name." } ], "code": 403, "message": "[email protected] does not have storage.objects.create access to your-bucket-name." } }

at cromwell.core.CromwellFatalException$.apply(core.scala:18)
at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:38)
at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:37)
at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:413)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Caused by: com.google.api.client.http.HttpResponseException: 403 Forbidden { "error": { "errors": [ { "domain": "global", "reason": "forbidden", "message": "[email protected] does not have storage.objects.create access to your-bucket-name." } ], "code": 403, "message": "[email protected] does not have storage.objects.create access to your-bucket-name." } }

at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1070)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.open(HttpStorageRpc.java:606)
at com.google.cloud.storage.BlobWriteChannel$2.call(BlobWriteChannel.java:69)
at com.google.cloud.storage.BlobWriteChannel$2.call(BlobWriteChannel.java:66)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:91)
at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.storage.BlobWriteChannel.open(BlobWriteChannel.java:66)
at com.google.cloud.storage.BlobWriteChannel.<init>(BlobWriteChannel.java:37)
at com.google.cloud.storage.StorageImpl.writer(StorageImpl.java:490)
at com.google.cloud.storage.StorageImpl.writer(StorageImpl.java:485)
at com.google.cloud.storage.StorageImpl.writer(StorageImpl.java:79)
at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.newWriteChannel(CloudStorageFileSystemProvider.java:346)
at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.newByteChannel(CloudStorageFileSystemProvider.java:235)
at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
at java.nio.file.Files.newOutputStream(Files.java:216)
at java.nio.file.Files.write(Files.java:3292)
at better.files.File.writeByteArray(File.scala:270)
at better.files.File.write(File.scala:280)
at cromwell.core.path.BetterFileMethods.write(BetterFileMethods.scala:179)
at cromwell.core.path.BetterFileMethods.write$(BetterFileMethods.scala:178)
at cromwell.filesystems.gcs.GcsPath.write(GcsPathBuilder.scala:165)
at cromwell.engine.io.nio.NioFlow.$anonfun$write$1(NioFlow.scala:71)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:654)
at scala.util.Success.$anonfun$map$1(Try.scala:251)
at scala.util.Success.map(Try.scala:209)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:288)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
... 12 common frames omitted

[2018-07-16 16:43:49,69] [error] WorkflowManagerActor Workflow e6b20ecd-5059-4c69-b624-28bab6740e18 failed (during ExecutingWorkflowState): cromwell.core.CromwellFatalException: com.google.api.client.http.HttpResponseException: 403 Forbidden { "error": { "errors": [ { "domain": "global", "reason": "forbidden", "message": "[email protected] does not have storage.objects.create access to your-bucket-name." } ], "code": 403, "message": "[email protected] does not have storage.objects.create access to your-bucket-name." } }

at cromwell.core.CromwellFatalException$.apply(core.scala:18)
at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:38)
at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:37)
at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:413)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Caused by: com.google.api.client.http.HttpResponseException: 403 Forbidden { "error": { "errors": [ { "domain": "global", "reason": "forbidden", "message": "[email protected] does not have storage.objects.create access to your-bucket-name." } ], "code": 403, "message": "[email protected] does not have storage.objects.create access to your-bucket-name." } }

at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1070)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.open(HttpStorageRpc.java:606)
at com.google.cloud.storage.BlobWriteChannel$2.call(BlobWriteChannel.java:69)
at com.google.cloud.storage.BlobWriteChannel$2.call(BlobWriteChannel.java:66)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:91)
at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.storage.BlobWriteChannel.open(BlobWriteChannel.java:66)
at com.google.cloud.storage.BlobWriteChannel.<init>(BlobWriteChannel.java:37)
at com.google.cloud.storage.StorageImpl.writer(StorageImpl.java:490)
at com.google.cloud.storage.StorageImpl.writer(StorageImpl.java:485)
at com.google.cloud.storage.StorageImpl.writer(StorageImpl.java:79)
at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.newWriteChannel(CloudStorageFileSystemProvider.java:346)
at com.google.cloud.storage.contrib.nio.CloudStorageFileSystemProvider.newByteChannel(CloudStorageFileSystemProvider.java:235)
at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
at java.nio.file.Files.newOutputStream(Files.java:216)
at java.nio.file.Files.write(Files.java:3292)
at better.files.File.writeByteArray(File.scala:270)
at better.files.File.write(File.scala:280)
at cromwell.core.path.BetterFileMethods.write(BetterFileMethods.scala:179)
at cromwell.core.path.BetterFileMethods.write$(BetterFileMethods.scala:178)
at cromwell.filesystems.gcs.GcsPath.write(GcsPathBuilder.scala:165)
at cromwell.engine.io.nio.NioFlow.$anonfun$write$1(NioFlow.scala:71)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:654)
at scala.util.Success.$anonfun$map$1(Try.scala:251)
at scala.util.Success.map(Try.scala:209)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:288)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
... 12 more

[2018-07-16 16:43:49,69] [info] WorkflowManagerActor WorkflowActor-e6b20ecd-5059-4c69-b624-28bab6740e18 is in a terminal state: WorkflowFailedState [2018-07-16 16:43:58,27] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'. [2018-07-16 16:43:59,36] [info] Workflow polling stopped [2018-07-16 16:43:59,37] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds [2018-07-16 16:43:59,38] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds [2018-07-16 16:43:59,38] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds [2018-07-16 16:43:59,38] [info] Aborting all running workflows. [2018-07-16 16:43:59,38] [info] WorkflowStoreActor stopped [2018-07-16 16:43:59,38] [info] JobExecutionTokenDispenser stopped [2018-07-16 16:43:59,39] [info] WorkflowLogCopyRouter stopped [2018-07-16 16:43:59,39] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds [2018-07-16 16:43:59,39] [info] WorkflowManagerActor All workflows finished [2018-07-16 16:43:59,39] [info] WorkflowManagerActor stopped [2018-07-16 16:43:59,39] [info] Connection pools shut down [2018-07-16 16:43:59,40] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds [2018-07-16 16:43:59,40] [info] Shutting down JobStoreActor - Timeout = 1800 seconds [2018-07-16 16:43:59,40] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds [2018-07-16 16:43:59,40] [info] SubWorkflowStoreActor stopped [2018-07-16 16:43:59,40] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds [2018-07-16 16:43:59,40] [info] Shutting down DockerHashActor - Timeout = 1800 seconds [2018-07-16 16:43:59,40] [info] Shutting down IoProxy - Timeout = 1800 seconds [2018-07-16 16:43:59,40] [info] CallCacheWriteActor Shutting down: 0 queued messages to process [2018-07-16 16:43:59,40] [info] JobStoreActor stopped [2018-07-16 16:43:59,40] [info] CallCacheWriteActor stopped [2018-07-16 16:43:59,40] [info] WriteMetadataActor Shutting down: 0 queued messages to process [2018-07-16 16:43:59,40] [info] KvWriteActor Shutting down: 0 queued messages to process [2018-07-16 16:43:59,40] [info] DockerHashActor stopped [2018-07-16 16:43:59,40] [info] IoProxy stopped [2018-07-16 16:43:59,40] [info] ServiceRegistryActor stopped [2018-07-16 16:43:59,42] [info] Database closed [2018-07-16 16:43:59,42] [info] Stream materializer shut down Workflow e6b20ecd-5059-4c69-b624-28bab6740e18 transitioned to state Failed [2018-07-16 16:43:59,46] [info] Automatic shutdown of the async connection [2018-07-16 16:43:59,46] [info] Gracefully shutdown sentry threads. [2018-07-16 16:43:59,46] [info] Shutdown finished.

Jul 16 '18 21:07 llz-hiv

Hi Jin, For now, I created three buckets in my project:

'encode-pipeline-genome-data-1' for hg38 genome and h38_google.tsv.
'atacout' for storage of output data
'atac-samples' for sequencing raw data (fast.gz) I am sure I have set permission for all of these three buckets in my project.

when I run the pipeline on google platform, below information appeared:

[2018-07-17 10:17:08,43] [info] PipelinesApiAsyncBackendJobExecutionActor [8cd46ebfatac.read_genome_tsv:NA:1]: echo "Reading genome_tsv /cromwell_root/encode-pipeline-genome-data-1/hg38_google.tsv ..." [2018-07-17 10:17:09,36] [error] PipelinesApiAsyncBackendJobExecutionActor [8cd46ebfatac.read_genome_tsv:NA:1]: Error attempting to Execute

So what I am confusing is the path:"Reading genome_tsv /cromwell_root/encode-pipeline-genome-data-1/hg38_google.tsv ...". The "encode-pipeline-genome-data-1/hg38_google.tsv" is the correct path for my project, but the "Reading genome_tsv /cromwell_root/" is not. There is no such buckets in my project. Is it the reason that forbid my access to the bucket?

I am really exhausted about it...... Thank you for your patience.

Jul 17 '18 14:07 llz-hiv

Please try with -Dbackend.provider.google.config.root=gs://YOUR_BUCKET_NAME. Your bucket name should start with gs://.

You don't need to make your own genome data buckets. Pick a genome TSV file under gs://encode-pipeline-genome-data/. This bucket is open to public.

Jul 17 '18 20:07 leepc12

I tried and picked a genome TSV file under gs://encode-pipeline-genome-data/ in the input.json. But it still doesn't work.....

I used the command line gsutil iam ch -d user:[email protected]:objectCreator,objectViewer gs://my-bucket . The command successfully added access permission to the bucket. I don't know why I couldn't access the bucket (even your public bucket gs://encode-pipeline-genome-data/ ) when I was running the pipeline on google platform.

Shall I install a cromwell server on Google platform as showed in the test_workflow?

Fortunately, I am able to run the pipeline on the DNAnexus now. At the very beginning, I used the latest version dxWDL0.69, but it didn't work until I replaced it with the old version dxWDL-0.66.1.jar.

Jul 18 '18 03:07 llz-hiv

@llz-hiv, that dxWDL thing is a known issue. For the latest dxWDL, you need to add -project [NAME_OF_YOUR_DNANEXUS_PRJ] to the command line.

About the bucket permission problem... Does gsutil cp ... work for files on the public bucket and your pipeline output bucket?

Jul 20 '18 21:07 leepc12

Hi Jin,

I tried the command gsutily cp....
I can copy files from my computer to my bucket and also be able to copy file from my bucket to my computer. I can also copy files from public bucket, but can't write files to the public bucket.

Jul 24 '18 16:07 llz-hiv

As described in the instruction web page, did you run the following to store Google Cloud credential information on your computer?

$ gcloud auth login --no-launch-browser
$ gcloud auth application-default login --no-launch-browser

Jul 25 '18 23:07 leepc12

FYI, /cromwell_root/ is a directory for localized files in an instance. For every sub-task, cromwell copies files on buckets into an instance on this directory.

I created a new google account and tested and it worked fine...

Jul 26 '18 00:07 leepc12

Hi Jin,

I'm trying to run the pipeline locally on a Mac, but libgcc 5.2 is not available for OSX. I installed the pipeline with Docker to get around this issue, but now I can't build genome database because that requires me to install_dependencies.sh, which gives me the error about libgcc=5.2.0 is not available. Is there a way to work around this? Could I use a genome downloaded from somewhere? (I'm working with mm10.)

Thanks! Erin

Nov 09 '18 00:11 erin-a-clark

atac_dnase_pipelines
atac_dnase_pipelines copied to clipboard

Can I install this ATAC-pipeline in Mac?

run as in the original configuration without --rm flag (will remove later)

get the return code (working even if the container was detached)

remove the container after waiting

return exit code

run as in the original configuration without --rm flag (will remove later)

get the return code (working even if the container was detached)

remove the container after waiting

return exit code

run as in the original configuration without --rm flag (will remove later)

get the return code (working even if the container was detached)

remove the container after waiting

return exit code

atac_dnase_pipelines atac_dnase_pipelines copied to clipboard

Can I install this ATAC-pipeline in Mac?

run as in the original configuration without --rm flag (will remove later)

get the return code (working even if the container was detached)

remove the container after waiting

return exit code

run as in the original configuration without --rm flag (will remove later)

get the return code (working even if the container was detached)

remove the container after waiting

return exit code

run as in the original configuration without --rm flag (will remove later)

get the return code (working even if the container was detached)

remove the container after waiting

return exit code

atac_dnase_pipelines
atac_dnase_pipelines copied to clipboard