mtag
mtag copied to clipboard
JonJala
Saving Omega and Sigma calculated by MTAG
Hi, I wanted to run mtag for 87 traits. It was able to compute Sigma and Omega successfully but it terminated at the MTAG calculations.
2023/03/08/11:33:50 PM Completed estimation of Omega ... 2023/03/08/11:33:50 PM Beginning MTAG calculations...
Is there any way to get the Sigma_hat and Omega_hat if MTAG terminated after calculating Omega and Sigma?
Thank you so much.
I don't believe it saves it out to a file. Did it terminate with an error? Getting to look at the log file would help.
On Wed, Mar 15, 2023 at 1:37 PM MeghSubedi @.***> wrote:
Hi, I wanted to run mtag for 87 traits. It was able to compute Sigma and Omega successfully but it terminated at the MTAG calculations.
2023/03/08/11:33:50 PM Completed estimation of Omega ... 2023/03/08/11:33:50 PM Beginning MTAG calculations...
Is there any way to get the Sigma_hat and Omega_hat if MTAG terminated after calculating Omega and Sigma?
Thank you so much.
— Reply to this email directly, view it on GitHub https://github.com/JonJala/mtag/issues/176, or unsubscribe https://github.com/notifications/unsubscribe-auth/APIOF52R5Q6R5QF3DDAMBWLW4H44PANCNFSM6AAAAAAV4FE47Y . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Hi JonJala,
Thank you for the quick response. It did stop without throwing any error. This is what it shows in the log file before terminating the program.
2023/03/08/11:30:33 PM Genetic Correlation: 1.001 (0.004) Z-score: 260.338 P: 0.
2023/03/08/11:30:35 PM [[ 9.972e-01 -1.299e-03 3.434e-03 ... 6.886e-03 -2.316e-03 8.860e-04] [-1.299e-03 1.004e+00 -1.315e-02 ... -2.939e-03 3.489e-03 -2.099e-03] [ 3.434e-03 -1.315e-02 9.980e-01 ... -3.372e-03 -1.710e-03 1.222e-03] ... [ 6.886e-03 -2.939e-03 -3.372e-03 ... 9.979e-01 8.608e-04 2.167e-03] [-2.316e-03 3.489e-03 -1.710e-03 ... 8.608e-04 9.933e-01 1.157e-02] [ 8.860e-04 -2.099e-03 1.222e-03 ... 2.167e-03 1.157e-02 9.993e-01]] 2023/03/08/11:30:38 PM Checking for positive definiteness .. 2023/03/08/11:30:38 PM Sigma hat: [[ 9.972e-01 -1.299e-03 3.434e-03 ... 6.886e-03 -2.316e-03 8.860e-04] [-1.299e-03 1.004e+00 -1.315e-02 ... -2.939e-03 3.489e-03 -2.099e-03] [ 3.434e-03 -1.315e-02 9.980e-01 ... -3.372e-03 -1.710e-03 1.222e-03] ... [ 6.886e-03 -2.939e-03 -3.372e-03 ... 9.979e-01 8.608e-04 2.167e-03] [-2.316e-03 3.489e-03 -1.710e-03 ... 8.608e-04 9.933e-01 1.157e-02] [ 8.860e-04 -2.099e-03 1.222e-03 ... 2.167e-03 1.157e-02 9.993e-01]] 2023/03/08/11:30:39 PM Mean chi^2 of SNPs used to estimate Omega is low for some SNPsMTAG may not perform well in this situation. 2023/03/08/11:30:39 PM Beginning estimation of Omega ... 2023/03/08/11:31:28 PM Using GMM estimator of Omega .. 2023/03/08/11:33:48 PM Checking for positive definiteness .. 2023/03/08/11:33:48 PM matrix is not positive definite, performing adjustment.. 2023/03/08/11:33:50 PM Warning: max number of iterations reached in adjustment procedure. Sigma matrix used is still non-positive-definite. 2023/03/08/11:33:50 PM Completed estimation of Omega ... 2023/03/08/11:33:50 PM Beginning MTAG calculations...
Hello,
Just checking that you didn't see any kind of exception being raised on the command line? And you're sure the process actually died and wasn't just taking some time?
On Wed, Mar 15, 2023 at 2:00 PM MeghSubedi @.***> wrote:
Hi JonJala,
Thank you for the quick response. It did stop without throwing any error. This is what it shows in the log file before terminating the program.
2023/03/08/11:30:33 PM Genetic Correlation: 1.001 (0.004) Z-score: 260.338 P: 0.
2023/03/08/11:30:35 PM [[ 9.972e-01 -1.299e-03 3.434e-03 ... 6.886e-03 -2.316e-03 8.860e-04] [-1.299e-03 1.004e+00 -1.315e-02 ... -2.939e-03 3.489e-03 -2.099e-03] [ 3.434e-03 -1.315e-02 9.980e-01 ... -3.372e-03 -1.710e-03 1.222e-03] ... [ 6.886e-03 -2.939e-03 -3.372e-03 ... 9.979e-01 8.608e-04 2.167e-03] [-2.316e-03 3.489e-03 -1.710e-03 ... 8.608e-04 9.933e-01 1.157e-02] [ 8.860e-04 -2.099e-03 1.222e-03 ... 2.167e-03 1.157e-02 9.993e-01]] 2023/03/08/11:30:38 PM Checking for positive definiteness .. 2023/03/08/11:30:38 PM Sigma hat: [[ 9.972e-01 -1.299e-03 3.434e-03 ... 6.886e-03 -2.316e-03 8.860e-04] [-1.299e-03 1.004e+00 -1.315e-02 ... -2.939e-03 3.489e-03 -2.099e-03] [ 3.434e-03 -1.315e-02 9.980e-01 ... -3.372e-03 -1.710e-03 1.222e-03] ... [ 6.886e-03 -2.939e-03 -3.372e-03 ... 9.979e-01 8.608e-04 2.167e-03] [-2.316e-03 3.489e-03 -1.710e-03 ... 8.608e-04 9.933e-01 1.157e-02] [ 8.860e-04 -2.099e-03 1.222e-03 ... 2.167e-03 1.157e-02 9.993e-01]] 2023/03/08/11:30:39 PM Mean chi^2 of SNPs used to estimate Omega is low for some SNPsMTAG may not perform well in this situation. 2023/03/08/11:30:39 PM Beginning estimation of Omega ... 2023/03/08/11:31:28 PM Using GMM estimator of Omega .. 2023/03/08/11:33:48 PM Checking for positive definiteness .. 2023/03/08/11:33:48 PM matrix is not positive definite, performing adjustment.. 2023/03/08/11:33:50 PM Warning: max number of iterations reached in adjustment procedure. Sigma matrix used is still non-positive-definite. 2023/03/08/11:33:50 PM Completed estimation of Omega ... 2023/03/08/11:33:50 PM Beginning MTAG calculations...
— Reply to this email directly, view it on GitHub https://github.com/JonJala/mtag/issues/176#issuecomment-1470503882, or unsubscribe https://github.com/notifications/unsubscribe-auth/APIOF53MFHOE6VW34CZG4RLW4H7SXANCNFSM6AAAAAAV4FE47Y . You are receiving this because you commented.Message ID: @.***>
Hi, Yes the program died during the MTAG calculation without raising any error. The process died all of the sudden at the beginning of MTAG calculations. Sometimes it get killed while estimating Omega without raising an error. This might be the reason of memory issue. While I was using the computer with 32GB ram, mtag ran successfully with 17 traits having 1 mil SNPs . When I use 18 traits with 1 mil SNPs the process get killed at the Omega estimation process. Later I tried with computer of ram of size 320 Gb mtag successively handled 58 traits with 1 mil SNPs. While I ran mtag for 87 traits with 1 mil SNPs each, program died with-out raising any error. 2023/03/08/11:33:50 PM Beginning MTAG calculations... This was the last line I could see in the log file.
That does sound like a possible memory issue, then. I would have expected any other issues to have raised an exception. Even running out of memory can raise an OutOfMemoryError, but if it's inside of a package like numpy or pandas, not sure that still happens with certainty.
I think it's possible to run MTAG on two disjoint subsets of traits (which should cut the size of the analysis) and then metaanalyze those together with MTAG, but that would be more of an issue for Professor Turley to comment on.
On Thu, Mar 16, 2023, 1:17 PM MeghSubedi @.***> wrote:
Hi, Yes the program died during the MTAG calculation without raising any error. The process died all of the sudden at the beginning of MTAG calculations. Sometimes it get killed while estimating Omega without raising an error. This might be the reason of memory issue. While I was using the computer with 32GB ram, mtag ran successfully with 17 traits having 1 mil SNPs . When I use 18 traits with 1 mil SNPs the process get killed at the Omega estimation process. Later I tried with computer of ram of size 320 Gb mtag successively handled 58 traits with 1 mil SNPs. While I ran mtag for 87 traits with 1 mil SNPs each, program died with-out raising any error. 2023/03/08/11:33:50 PM Beginning MTAG calculations... This was the last line I could see in the log file.
— Reply to this email directly, view it on GitHub https://github.com/JonJala/mtag/issues/176#issuecomment-1472387180, or unsubscribe https://github.com/notifications/unsubscribe-auth/APIOF57A3Q5H6RCR2NEBFRTW4NDLZANCNFSM6AAAAAAV4FE47Y . You are receiving this because you commented.Message ID: @.***>
Hi!
I don't know why Jon called me Professor Turley, but yes. One option would be to run MTAG on half the traits including your target trait, then take the MTAG summary statistics for the target trait and use MTAG again on the second half of the traits and the MTAG output from the first step. It's not equivalent to doing it all at once (unless you are assuming perfect genetic correlation or something), but it should still increase power.
That said, we haven't tested this approach, so it may be that it introduces issues that I'm unaware of. It just seems like it should work theoretically in my head. You would need to be sure to document that you used this piecemeal approach in your manuscript rather than the standard full MTAG approach.
On Thu, Mar 16, 2023 at 12:24 PM MeghSubedi @.***> wrote:
Thank you so much JonJala..
— Reply to this email directly, view it on GitHub https://github.com/JonJala/mtag/issues/176#issuecomment-1472619427, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFBUB5KL62Z2O7GEZEIH4ATW4NSHZANCNFSM6AAAAAAV4FE47Y . You are receiving this because you are subscribed to this thread.Message ID: @.***>
