mtag
mtag copied to clipboard
JonJala
KeyError: 'A10'
Dear authors,
Here is the other and last error (I cross fingers) that I get when performing mtag with the meta_format option.
Of note, I used exactly the same sumstats to do mtag without using the meta_format option and it worked.
This is my script:
Calling ./mtag.py
--force
--stream-stdout
--n-min 0.0
--meta-format
--sumstats file1,file2,file3,file4
--out ./mtag_metaformat
And this is the error:
2020/08/20/05:36:58 PM Dropped 4242425 SNPs due to strand ambiguity, 24536849 SNPs remain in intersection after merging trait1
2020/08/20/05:39:33 PM Dropped 9783 SNPs due to inconsistent allele pairs from phenotype 2. 10055090 SNPs remain.
2020/08/20/05:39:45 PM Dropped 0 SNPs due to strand ambiguity, 10055090 SNPs remain in intersection after merging trait2
2020/08/20/05:41:55 PM Dropped 1318 SNPs due to inconsistent allele pairs from phenotype 3. 6057263 SNPs remain.
2020/08/20/05:42:00 PM Flipped the signs of of 2735528 SNPs to make them consistent with the effect allele orderings of the first trait.
2020/08/20/05:42:07 PM Dropped 0 SNPs due to strand ambiguity, 6057263 SNPs remain in intersection after merging trait3
2020/08/20/05:44:29 PM Flipped the signs of of 2711110 SNPs to make them consistent with the effect allele orderings of the first trait.
2020/08/20/05:44:36 PM Dropped 0 SNPs due to strand ambiguity, 5994594 SNPs remain in intersection after merging trait4
2020/08/20/05:47:07 PM Flipped the signs of of 2709916 SNPs to make them consistent with the effect allele orderings of the first trait.
2020/08/20/05:47:15 PM Dropped 0 SNPs due to strand ambiguity, 5991582 SNPs remain in intersection after merging trait5
2020/08/20/05:50:33 PM Dropped 3439 SNPs due to inconsistent allele pairs from phenotype 6. 5988119 SNPs remain.
2020/08/20/05:50:41 PM Flipped the signs of of 5988119 SNPs to make them consistent with the effect allele orderings of the first trait.
2020/08/20/05:50:49 PM Dropped 0 SNPs due to strand ambiguity, 5988119 SNPs remain in intersection after merging trait6
2020/08/20/05:54:01 PM Flipped the signs of of 2969644 SNPs to make them consistent with the effect allele orderings of the first trait.
2020/08/20/05:54:09 PM Dropped 0 SNPs due to strand ambiguity, 5935715 SNPs remain in intersection after merging trait7
2020/08/20/05:58:33 PM Dropped 377 SNPs due to inconsistent allele pairs from phenotype 8. 5935274 SNPs remain.
2020/08/20/05:58:43 PM Flipped the signs of of 5935274 SNPs to make them consistent with the effect allele orderings of the first trait.
2020/08/20/05:58:53 PM Dropped 0 SNPs due to strand ambiguity, 5935274 SNPs remain in intersection after merging trait8
2020/08/20/06:03:25 PM Flipped the signs of of 5878372 SNPs to make them consistent with the effect allele orderings of the first trait.
2020/08/20/06:03:34 PM Dropped 0 SNPs due to strand ambiguity, 5878372 SNPs remain in intersection after merging trait9
2020/08/20/06:08:16 PM Flipped the signs of of 2352415 SNPs to make them consistent with the effect allele orderings of the first trait.
2020/08/20/06:08:26 PM Dropped 0 SNPs due to strand ambiguity, 5178742 SNPs remain in intersection after merging trait10
2020/08/20/06:13:31 PM Flipped the signs of of 2352399 SNPs to make them consistent with the effect allele orderings of the first trait.
2020/08/20/06:13:41 PM Dropped 0 SNPs due to strand ambiguity, 5178705 SNPs remain in intersection after merging trait11
2020/08/20/06:13:41 PM ... Merge of GWAS summary statistics complete. Number of SNPs: 5178705
2020/08/20/06:15:41 PM Using 5178705 SNPs to estimate Omega (0 SNPs excluded due to strand ambiguity)
2020/08/20/06:15:41 PM Estimating sigma..
2020/08/20/06:32:02 PM Checking for positive definiteness ..
2020/08/20/06:32:02 PM Sigma hat:
[[ 1.056e+00 5.884e-01 1.599e-02 1.108e-02 1.591e-02 4.894e-02 4.984e-02 3.509e-02 2.426e-01 9.309e-02 6.564e-03]
[ 5.884e-01 1.025e+00 6.942e-03 -4.352e-03 2.667e-04 4.382e-02 4.603e-02 4.404e-02 1.367e-01 9.841e-02 4.644e-03]
[ 1.599e-02 6.942e-03 1.104e+00 3.013e-01 8.250e-01 -1.250e-02 2.742e-02 1.848e-03 1.358e-02 -1.056e-02 6.200e-03]
[ 1.108e-02 -4.352e-03 3.013e-01 1.054e+00 8.883e-01 3.360e-02 2.217e-02 -1.209e-02 1.396e-02 -3.231e-03 -1.243e-03]
[ 1.591e-02 2.667e-04 8.250e-01 8.883e-01 1.099e+00 1.596e-02 2.980e-02 -8.236e-03 1.721e-02 -9.154e-03 3.263e-03]
[ 4.894e-02 4.382e-02 -1.250e-02 3.360e-02 1.596e-02 1.081e+00 3.998e-02 8.516e-02 5.277e-02 9.268e-03 -1.015e-02]
[ 4.984e-02 4.603e-02 2.742e-02 2.217e-02 2.980e-02 3.998e-02 8.547e-01 2.031e-02 5.720e-02 1.801e-02 -2.329e-03]
[ 3.509e-02 4.404e-02 1.848e-03 -1.209e-02 -8.236e-03 8.516e-02 2.031e-02 1.042e+00 3.529e-02 1.700e-02 -1.295e-02]
[ 2.426e-01 1.367e-01 1.358e-02 1.396e-02 1.721e-02 5.277e-02 5.720e-02 3.529e-02 1.005e+00 2.281e-01 1.560e-03]
[ 9.309e-02 9.841e-02 -1.056e-02 -3.231e-03 -9.154e-03 9.268e-03 1.801e-02 1.700e-02 2.281e-01 1.014e+00 4.786e-03]
[ 6.564e-03 4.644e-03 6.200e-03 -1.243e-03 3.263e-03 -1.015e-02 -2.329e-03 -1.295e-02 1.560e-03 4.786e-03 9.980e-01]]
2020/08/20/06:32:02 PM Mean chi^2 of SNPs used to estimate Omega is low for some SNPsMTAG may not perform well in this situation.
2020/08/20/06:32:03 PM Beginning estimation of Omega ...
2020/08/20/06:32:08 PM Using GMM estimator of Omega ..
2020/08/20/06:32:21 PM Checking for positive definiteness ..
2020/08/20/06:32:21 PM matrix is not positive definite, performing adjustment..
2020/08/20/06:32:21 PM Completed in 11 iterations
2020/08/20/06:32:21 PM Completed estimation of Omega ...
2020/08/20/06:33:39 PM There are 3166887 SNPs present in Traits [0 6 8]
2020/08/20/06:33:43 PM Beginning MTAG calculations...
2020/08/20/06:33:52 PM ... Completed MTAG calculations.
2020/08/20/06:33:55 PM There are 28763 SNPs present in Traits [ 0 1 2 5 6 7 8 9 10]
2020/08/20/06:33:55 PM 'A10'
Traceback (most recent call last):
File "/home/pjordab/projects/def-rtadros/tools/mtag/mtag.py", line 1567, in
Many thanks in advance for your help!
Paloma
Hmm. I think this might be an error due to the way we index the traits when there are more than 10 input files. How many files are you trying to combine at once? Do you get this error when there are only 9 input files?
On Thu, Aug 20, 2020 at 10:45 PM pjordab [email protected] wrote:
Dear authors,
Here is the other and last error (I cross fingers) that I get when performing mtag with the meta_format option.
Of note, I used exactly the same sumstats to do mtag without using the meta_format option and it worked.
This is my script:
Calling ./mtag.py --force --stream-stdout --n-min 0.0 --meta-format --sumstats file1,file2,file3,file4 --out ./mtag_metaformat
And this is the error:
2020/08/20/05:36:58 PM Dropped 4242425 SNPs due to strand ambiguity, 24536849 SNPs remain in intersection after merging trait1 2020/08/20/05:39:33 PM Dropped 9783 SNPs due to inconsistent allele pairs from phenotype 2. 10055090 SNPs remain. 2020/08/20/05:39:45 PM Dropped 0 SNPs due to strand ambiguity, 10055090 SNPs remain in intersection after merging trait2 2020/08/20/05:41:55 PM Dropped 1318 SNPs due to inconsistent allele pairs from phenotype 3. 6057263 SNPs remain. 2020/08/20/05:42:00 PM Flipped the signs of of 2735528 SNPs to make them consistent with the effect allele orderings of the first trait. 2020/08/20/05:42:07 PM Dropped 0 SNPs due to strand ambiguity, 6057263 SNPs remain in intersection after merging trait3 2020/08/20/05:44:29 PM Flipped the signs of of 2711110 SNPs to make them consistent with the effect allele orderings of the first trait. 2020/08/20/05:44:36 PM Dropped 0 SNPs due to strand ambiguity, 5994594 SNPs remain in intersection after merging trait4 2020/08/20/05:47:07 PM Flipped the signs of of 2709916 SNPs to make them consistent with the effect allele orderings of the first trait. 2020/08/20/05:47:15 PM Dropped 0 SNPs due to strand ambiguity, 5991582 SNPs remain in intersection after merging trait5 2020/08/20/05:50:33 PM Dropped 3439 SNPs due to inconsistent allele pairs from phenotype 6. 5988119 SNPs remain. 2020/08/20/05:50:41 PM Flipped the signs of of 5988119 SNPs to make them consistent with the effect allele orderings of the first trait. 2020/08/20/05:50:49 PM Dropped 0 SNPs due to strand ambiguity, 5988119 SNPs remain in intersection after merging trait6 2020/08/20/05:54:01 PM Flipped the signs of of 2969644 SNPs to make them consistent with the effect allele orderings of the first trait. 2020/08/20/05:54:09 PM Dropped 0 SNPs due to strand ambiguity, 5935715 SNPs remain in intersection after merging trait7 2020/08/20/05:58:33 PM Dropped 377 SNPs due to inconsistent allele pairs from phenotype 8. 5935274 SNPs remain. 2020/08/20/05:58:43 PM Flipped the signs of of 5935274 SNPs to make them consistent with the effect allele orderings of the first trait. 2020/08/20/05:58:53 PM Dropped 0 SNPs due to strand ambiguity, 5935274 SNPs remain in intersection after merging trait8 2020/08/20/06:03:25 PM Flipped the signs of of 5878372 SNPs to make them consistent with the effect allele orderings of the first trait. 2020/08/20/06:03:34 PM Dropped 0 SNPs due to strand ambiguity, 5878372 SNPs remain in intersection after merging trait9 2020/08/20/06:08:16 PM Flipped the signs of of 2352415 SNPs to make them consistent with the effect allele orderings of the first trait. 2020/08/20/06:08:26 PM Dropped 0 SNPs due to strand ambiguity, 5178742 SNPs remain in intersection after merging trait10 2020/08/20/06:13:31 PM Flipped the signs of of 2352399 SNPs to make them consistent with the effect allele orderings of the first trait. 2020/08/20/06:13:41 PM Dropped 0 SNPs due to strand ambiguity, 5178705 SNPs remain in intersection after merging trait11 2020/08/20/06:13:41 PM ... Merge of GWAS summary statistics complete. Number of SNPs: 5178705 2020/08/20/06:15:41 PM Using 5178705 SNPs to estimate Omega (0 SNPs excluded due to strand ambiguity) 2020/08/20/06:15:41 PM Estimating sigma.. 2020/08/20/06:32:02 PM Checking for positive definiteness .. 2020/08/20/06:32:02 PM Sigma hat: [[ 1.056e+00 5.884e-01 1.599e-02 1.108e-02 1.591e-02 4.894e-02 4.984e-02 3.509e-02 2.426e-01 9.309e-02 6.564e-03] [ 5.884e-01 1.025e+00 6.942e-03 -4.352e-03 2.667e-04 4.382e-02 4.603e-02 4.404e-02 1.367e-01 9.841e-02 4.644e-03] [ 1.599e-02 6.942e-03 1.104e+00 3.013e-01 8.250e-01 -1.250e-02 2.742e-02 1.848e-03 1.358e-02 -1.056e-02 6.200e-03] [ 1.108e-02 -4.352e-03 3.013e-01 1.054e+00 8.883e-01 3.360e-02 2.217e-02 -1.209e-02 1.396e-02 -3.231e-03 -1.243e-03] [ 1.591e-02 2.667e-04 8.250e-01 8.883e-01 1.099e+00 1.596e-02 2.980e-02 -8.236e-03 1.721e-02 -9.154e-03 3.263e-03] [ 4.894e-02 4.382e-02 -1.250e-02 3.360e-02 1.596e-02 1.081e+00 3.998e-02 8.516e-02 5.277e-02 9.268e-03 -1.015e-02] [ 4.984e-02 4.603e-02 2.742e-02 2.217e-02 2.980e-02 3.998e-02 8.547e-01 2.031e-02 5.720e-02 1.801e-02 -2.329e-03] [ 3.509e-02 4.404e-02 1.848e-03 -1.209e-02 -8.236e-03 8.516e-02 2.031e-02 1.042e+00 3.529e-02 1.700e-02 -1.295e-02] [ 2.426e-01 1.367e-01 1.358e-02 1.396e-02 1.721e-02 5.277e-02 5.720e-02 3.529e-02 1.005e+00 2.281e-01 1.560e-03] [ 9.309e-02 9.841e-02 -1.056e-02 -3.231e-03 -9.154e-03 9.268e-03 1.801e-02 1.700e-02 2.281e-01 1.014e+00 4.786e-03] [ 6.564e-03 4.644e-03 6.200e-03 -1.243e-03 3.263e-03 -1.015e-02 -2.329e-03 -1.295e-02 1.560e-03 4.786e-03 9.980e-01]] 2020/08/20/06:32:02 PM Mean chi^2 of SNPs used to estimate Omega is low for some SNPsMTAG may not perform well in this situation. 2020/08/20/06:32:03 PM Beginning estimation of Omega ... 2020/08/20/06:32:08 PM Using GMM estimator of Omega .. 2020/08/20/06:32:21 PM Checking for positive definiteness .. 2020/08/20/06:32:21 PM matrix is not positive definite, performing adjustment.. 2020/08/20/06:32:21 PM Completed in 11 iterations 2020/08/20/06:32:21 PM Completed estimation of Omega ... 2020/08/20/06:33:39 PM There are 3166887 SNPs present in Traits [0 6 8] 2020/08/20/06:33:43 PM Beginning MTAG calculations... 2020/08/20/06:33:52 PM ... Completed MTAG calculations. 2020/08/20/06:33:55 PM There are 28763 SNPs present in Traits [ 0 1 2 5 6 7 8 9 10] 2020/08/20/06:33:55 PM 'A10' Traceback (most recent call last): File "/home/pjordab/projects/def-rtadros/tools/mtag/mtag.py", line 1567, in mtag(args) File "/home/pjordab/projects/def-rtadros/tools/mtag/mtag.py", line 1410, in mtag Zs, Ns, Fs, resid_cols, _, N_raw = extract_gwas_sumstats(snp_df, args, tl) File "/home/pjordab/projects/def-rtadros/tools/mtag/mtag.py", line 604, in extract_gwas_sumstats results_template.loc[:,col] = DATA[col+str(t0[0])] File "/project/6005813/tools/Python_ENV/lib/python2.7/site-packages/pandas/core/frame.py", line 2927, in getitem indexer = self.columns.get_loc(key) File "/project/6005813/tools/Python_ENV/lib/python2.7/site-packages/pandas/core/indexes/base.py", line 2658, in get_loc return self._engine.get_loc(self._maybe_cast_indexer(key)) File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 1601, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 1608, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'A10' 2020/08/20/06:33:55 PM Analysis terminated from error at Thu Aug 20 18:33:55 2020 2020/08/20/06:33:55 PM Total time elapsed: 1.0h:53.0m:51.7s
Many thanks in advance for your help!
Paloma
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/JonJala/mtag/issues/108, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFBUB5PKN62PCTCITFKKORDSBXNV3ANCNFSM4QG2A67Q .
Do you get the error when you use just 9 of the traits?
On Fri, Aug 21, 2020 at 3:32 PM pjordab [email protected] wrote:
Thank you very much for the quick answer. I’m using up to 16 traits...
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/JonJala/mtag/issues/108#issuecomment-678456182, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFBUB5P5HOQBMCCF6QSMZC3SB3DVDANCNFSM4QG2A67Q .
Hi Patrick, You are right. This error (A10) is not observed when only using 9 traits. We are still struggling with another error (see issue #107 posted by Paloma). Thanks for your help, Rafik
Great! This bug is going to be a little more complicated to fix, but what you might try in the meantime is run MTAG on half of the cohorts at a time and then running MTAG on the resulting two sets of summary statistics. This should be approximately (but not exactly) equivalent to running it all at one. If you want to verify that this works, you might compare the results from this split version to the results for a handful of SNPs that remain when you don't use the meta-format options.
On Sat, Aug 22, 2020 at 9:41 AM rafiktadros [email protected] wrote:
Hi Patrick, You are right. This error (A10) is not observed when only using traits. We are still struggling with another error (see issue #107 https://github.com/JonJala/mtag/issues/107 posted by Paloma). Thanks for your help, Rafik
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/JonJala/mtag/issues/108#issuecomment-678642691, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFBUB5L2IMJ5BVZJCNEVBK3SB7DHJANCNFSM4QG2A67Q .
Hi Patrick, I would like to know if there is any update about the --meta-format option and if it is feasible to use it with 10 or more traits. Thank you very much. Paloma
HI Paloma,
We tried looking at this, but we didn't code it up in a very robust way, unfortunately. We have it on our list of things we need to fix, but it's not something that we will be able to resolve very quickly. Sorry about that.
Patrick
On Tue, Sep 21, 2021 at 12:45 PM pjordab @.***> wrote:
Hi Patrick, I would like to know if there is any update about the --meta-format option and if it is feasible to use it with 10 or more traits. Thank you very much. Paloma
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/JonJala/mtag/issues/108#issuecomment-924166116, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFBUB5MWWJDRXV6J2W4RJ4TUDCZBTANCNFSM4QG2A67Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.