mtag
mtag copied to clipboard
JonJala
get stuck at Beginning MTAG calculations
Hi I'm running MTAG on 37 traits, it had took me eleven days but the log file still stop at "2019/06/01/03:58:29 PM Beginning MTAG calculations... ". seems the progress do nothing?
work.sh python mtag.py --sumstats trait1.sumstats.txt,trait2.sumstats.txt,...,trait37.sumstats.txt --cores 4 --n_min 0.0 --force --out outdir --stream_stdout
the end of logfile 2019/06/01/12:08:19 PM Warning: The mean chi2 statistic of trait 36 is less 1.02 - MTAG estimates may be unstable. 2019/06/01/12:08:38 PM Read in Trait 37 summary statistics (3830361 SNPs) from /zfssz2/ST_MCHRI/BIGDATA/USER/lanxianmei/metabolite_gwas_plink1.9_20190417/mtag/output/mtag_format/Aminoacid_Val.sumstats.txt ... 2019/06/01/12:08:38 PM <><><<>><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> 2019/06/01/12:08:38 PM Munging Trait 37 <><><<>><><><><><><><><><><><><><><><><><><><><><><><><><>< 2019/06/01/12:08:38 PM <><><<>><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> 2019/06/01/12:08:38 PM Interpreting column names as follows: 2019/06/01/12:08:38 PM snpid: Variant ID (e.g., rs number) n: Sample size a1: a1, interpreted as ref allele for signed sumstat. pval: p-Value a2: a2, interpreted as non-ref allele for signed sumstat. z: Directional summary statistic as specified by --signed-sumstats.
2019/06/01/12:08:38 PM Reading sumstats from provided DataFrame into memory 10000000 SNPs at a time. 2019/06/01/12:08:46 PM Read 3830361 SNPs from --sumstats file. Removed 0 SNPs with missing values. Removed 0 SNPs with INFO <= None. Removed 0 SNPs with MAF <= 0.01. Removed 0 SNPs with SE <0 or NaN values. Removed 0 SNPs with out-of-bounds p-values. Removed 0 variants that were not SNPs. Note: strand ambiguous SNPs were not dropped. 3830361 SNPs remain. 2019/06/01/12:08:51 PM Removed 0 SNPs with duplicated rs numbers (3830361 SNPs remain). 2019/06/01/12:08:52 PM Removed 0 SNPs with N < 0.0 (3830361 SNPs remain). 2019/06/01/12:09:56 PM Median value of SIGNED_SUMSTAT was 0.0, which seems sensible. 2019/06/01/12:09:56 PM Dropping snps with null values 2019/06/01/12:09:57 PM Metadata: 2019/06/01/12:09:57 PM Mean chi^2 = 1.002 2019/06/01/12:09:57 PM WARNING: mean chi^2 may be too small. 2019/06/01/12:09:58 PM Lambda GC = 1.009 2019/06/01/12:09:58 PM Max chi^2 = 22.866 2019/06/01/12:09:58 PM 0 Genome-wide significant SNPs (some may have been removed by filtering). 2019/06/01/12:09:58 PM Conversion finished at Sat Jun 1 12:09:58 2019 2019/06/01/12:09:58 PM Total time elapsed: 1.0m:19.57s 2019/06/01/12:10:08 PM <><><<>><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> 2019/06/01/12:10:08 PM Munging of Trait 37 complete. SNPs remaining: 3830361 2019/06/01/12:10:08 PM <><><<>><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
2019/06/01/12:10:09 PM Warning: The mean chi2 statistic of trait 37 is less 1.02 - MTAG estimates may be unstable. 2019/06/01/12:10:29 PM Dropped 0 SNPs due to strand ambiguity, 3830447 SNPs remain in intersection after merging trait1 2019/06/01/12:10:49 PM Dropped 0 SNPs due to strand ambiguity, 3829329 SNPs remain in intersection after merging trait2 2019/06/01/12:11:08 PM Dropped 0 SNPs due to strand ambiguity, 3828887 SNPs remain in intersection after merging trait3 2019/06/01/12:11:31 PM Dropped 0 SNPs due to strand ambiguity, 3828653 SNPs remain in intersection after merging trait4 2019/06/01/12:11:56 PM Dropped 0 SNPs due to strand ambiguity, 3828509 SNPs remain in intersection after merging trait5 2019/06/01/12:12:23 PM Dropped 0 SNPs due to strand ambiguity, 3828337 SNPs remain in intersection after merging trait6 2019/06/01/12:12:53 PM Dropped 0 SNPs due to strand ambiguity, 3828245 SNPs remain in intersection after merging trait7 2019/06/01/12:13:27 PM Dropped 0 SNPs due to strand ambiguity, 3828138 SNPs remain in intersection after merging trait8 2019/06/01/12:14:02 PM Dropped 0 SNPs due to strand ambiguity, 3827915 SNPs remain in intersection after merging trait9 2019/06/01/12:14:42 PM Dropped 0 SNPs due to strand ambiguity, 3827229 SNPs remain in intersection after merging trait10 2019/06/01/12:15:24 PM Dropped 0 SNPs due to strand ambiguity, 3827209 SNPs remain in intersection after merging trait11 2019/06/01/12:16:10 PM Dropped 0 SNPs due to strand ambiguity, 3827052 SNPs remain in intersection after merging trait12 2019/06/01/12:16:56 PM Dropped 0 SNPs due to strand ambiguity, 3826929 SNPs remain in intersection after merging trait13 2019/06/01/12:17:45 PM Dropped 0 SNPs due to strand ambiguity, 3826906 SNPs remain in intersection after merging trait14 2019/06/01/12:18:42 PM Dropped 0 SNPs due to strand ambiguity, 3826877 SNPs remain in intersection after merging trait15 2019/06/01/12:19:34 PM Dropped 0 SNPs due to strand ambiguity, 3826828 SNPs remain in intersection after merging trait16 2019/06/01/12:20:31 PM Dropped 0 SNPs due to strand ambiguity, 3826765 SNPs remain in intersection after merging trait17 2019/06/01/12:21:28 PM Dropped 0 SNPs due to strand ambiguity, 3826431 SNPs remain in intersection after merging trait18 2019/06/01/12:22:32 PM Dropped 0 SNPs due to strand ambiguity, 3826365 SNPs remain in intersection after merging trait19 2019/06/01/12:23:39 PM Dropped 0 SNPs due to strand ambiguity, 3826264 SNPs remain in intersection after merging trait20 2019/06/01/12:24:46 PM Dropped 0 SNPs due to strand ambiguity, 3826229 SNPs remain in intersection after merging trait21 2019/06/01/12:25:56 PM Dropped 0 SNPs due to strand ambiguity, 3826215 SNPs remain in intersection after merging trait22 2019/06/01/12:27:08 PM Dropped 0 SNPs due to strand ambiguity, 3826210 SNPs remain in intersection after merging trait23 2019/06/01/12:28:36 PM Dropped 0 SNPs due to strand ambiguity, 3826205 SNPs remain in intersection after merging trait24 2019/06/01/12:30:05 PM Dropped 0 SNPs due to strand ambiguity, 3826189 SNPs remain in intersection after merging trait25 2019/06/01/12:31:29 PM Dropped 0 SNPs due to strand ambiguity, 3826180 SNPs remain in intersection after merging trait26 2019/06/01/12:33:03 PM Dropped 0 SNPs due to strand ambiguity, 3826161 SNPs remain in intersection after merging trait27 2019/06/01/12:34:31 PM Dropped 0 SNPs due to strand ambiguity, 3826159 SNPs remain in intersection after merging trait28 2019/06/01/12:36:01 PM Dropped 0 SNPs due to strand ambiguity, 3826149 SNPs remain in intersection after merging trait29 2019/06/01/12:37:43 PM Dropped 0 SNPs due to strand ambiguity, 3826077 SNPs remain in intersection after merging trait30 2019/06/01/12:39:20 PM Dropped 0 SNPs due to strand ambiguity, 3826071 SNPs remain in intersection after merging trait31 2019/06/01/12:41:00 PM Dropped 0 SNPs due to strand ambiguity, 3826068 SNPs remain in intersection after merging trait32 2019/06/01/12:42:44 PM Dropped 0 SNPs due to strand ambiguity, 3826042 SNPs remain in intersection after merging trait33 2019/06/01/12:44:40 PM Dropped 0 SNPs due to strand ambiguity, 3826035 SNPs remain in intersection after merging trait34 2019/06/01/12:46:54 PM Dropped 0 SNPs due to strand ambiguity, 3826006 SNPs remain in intersection after merging trait35 2019/06/01/12:48:57 PM Dropped 0 SNPs due to strand ambiguity, 3825999 SNPs remain in intersection after merging trait36 2019/06/01/12:50:54 PM Dropped 0 SNPs due to strand ambiguity, 3825995 SNPs remain in intersection after merging trait37 2019/06/01/12:50:54 PM ... Merge of GWAS summary statistics complete. Number of SNPs: 3825995 2019/06/01/12:53:41 PM Using 3825995 SNPs to estimate Omega (0 SNPs excluded due to strand ambiguity) 2019/06/01/12:53:41 PM Estimating sigma.. 2019/06/01/03:51:49 PM Checking for positive definiteness .. 2019/06/01/03:51:49 PM Sigma hat: [[1.004 0.344 0.253 ... 0.103 0.144 0.225] [0.344 1.009 0.35 ... 0.233 0.245 0.305] [0.253 0.35 0.986 ... 0.317 0.393 0.478] ... [0.103 0.233 0.317 ... 0.984 0.462 0.404] [0.144 0.245 0.393 ... 0.462 0.987 0.588] [0.225 0.305 0.478 ... 0.404 0.588 0.998]] 2019/06/01/03:51:50 PM Mean chi^2 of SNPs used to estimate Omega is low for some SNPsMTAG may not perform well in this situation. 2019/06/01/03:51:52 PM Beginning estimation of Omega ... 2019/06/01/03:53:56 PM Using GMM estimator of Omega .. 2019/06/01/03:58:22 PM Checking for positive definiteness .. 2019/06/01/03:58:22 PM matrix is not positive definite, performing adjustment.. 2019/06/01/03:58:22 PM Warning: max number of iterations reached in adjustment procedure. Sigma matrix used is still non-positive-definite. 2019/06/01/03:58:29 PM Completed estimation of Omega ... 2019/06/01/03:58:29 PM Beginning MTAG calculations...
Hi @XianmeiLan ,
There is a similar discussion on using MTAG on a large number of traits previously in an issue here. As Patrick mentioned in this post, the Omega and Sigma matrices become extremely large objects in your case. That might be why it takes forever for it to run. It might be helpful if you can come up with a reasonable way to group the traits and run mtag sequentially.
Best, Hui
is there any update on this? exactly how much memory needed for 95 sumstats of 7 Mil markers?
