Failed to open file output.bam

[yuanqm55]

I follow the instructions(install dependencies, set paths, download data and index the reference sequence), and everything's going well. Then I run run_vifi.py using the test data(test_R1.fq.gz and test_R2.fq.gz) and ran into a problem: Failed to open file output.bam.

[namphuon]

Can you let me know what's in the folder ./test_result? I am wondering if it might be some path issue. I don't know if this would impact it, but maybe try giving the full path to the -o option. If that does work, I'll fix my code so that full path is not needed.

On Tue, Sep 11, 2018 at 2:06 AM, 阿克苏地 [email protected] wrote:

I follow the instructions(install dependencies, set paths, download data and index the reference sequence), and everything's going well. Then I run run_vifi.py using the test data(test_R1.fq.gz and test_R2.fq.gz) and ran into a problem: Failed to open file output.bam. [image: helpme] https://user-images.githubusercontent.com/34777741/45349803-dd040800-b5e4-11e8-8490-0de1ecaab050.png

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/namphuon/ViFi/issues/10, or mute the thread https://github.com/notifications/unsubscribe-auth/AA5tNMnz20mY5FbqTb4a-0U87WSI7l77ks5uZ3z7gaJpZM4Wi5HR .

-- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093-0404

[yuanqm55]

Can you let me know what's in the folder ./test_result? I am wondering if it might be some path issue. I don't know if this would impact it, but maybe try giving the full path to the -o option. If that does work, I'll fix my code so that full path is not needed. … On Tue, Sep 11, 2018 at 2:06 AM, 阿克苏地 @.***> wrote: I follow the instructions(install dependencies, set paths, download data and index the reference sequence), and everything's going well. Then I run run_vifi.py using the test data(test_R1.fq.gz and test_R2.fq.gz) and ran into a problem: Failed to open file output.bam. [image: helpme] https://user-images.githubusercontent.com/34777741/45349803-dd040800-b5e4-11e8-8490-0de1ecaab050.png — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#10>, or mute the thread https://github.com/notifications/unsubscribe-auth/AA5tNMnz20mY5FbqTb4a-0U87WSI7l77ks5uZ3z7gaJpZM4Wi5HR . -- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093-0404

Here's the items in the folder ./test_result: And the program seems to work after I gave the full path to the -o option, as you suggested. However, I run into a new issue:TypeError: AlignmentHeader does not support item assignment (use header.to_dict() I'm a newcomer to bioinformatics. Thanks for helping!

[namphuon]

Hi,

I believe this issue was due to ViFi not being compatible with the latest version of pysam. I've updated the code. Can you try the latest version and let me know if you still have an issue? Thanks!

Regards, Nam

On Sun, Sep 16, 2018 at 10:04 PM, 阿克苏地 [email protected] wrote:

Can you let me know what's in the folder ./test_result? I am wondering if it might be some path issue. I don't know if this would impact it, but maybe try giving the full path to the -o option. If that does work, I'll fix my code so that full path is not needed. … <#m_1651321738927992081_> On Tue, Sep 11, 2018 at 2:06 AM, 阿克苏地 @.***> wrote: I follow the instructions(install dependencies, set paths, download data and index the reference sequence), and everything's going well. Then I run run_vifi.py using the test data(test_R1.fq.gz and test_R2.fq.gz) and ran into a problem: Failed to open file output.bam. [image: helpme] https://user-images.githubusercontent.com/34777741/45349803-dd040800- b5e4-11e8-8490-0de1ecaab050.png — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub < #10 https://github.com/namphuon/ViFi/issues/10>, or mute the thread https://github.com/notifications/unsubscribe-auth/AA5tNMnz20mY5FbqTb4a- 0U87WSI7l77ks5uZ3z7gaJpZM4Wi5HR . -- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093 https://maps.google.com/?q=9500+Gilman+Drive,+La+Jolla,+CA+92093&entry=gmail&source=g -0404

Here's the items in the folder ./test_result: [image: what s in] https://user-images.githubusercontent.com/34777741/45605761-1bc91080-ba72-11e8-8f3e-4966cf37d879.png And the program seems to work after I gave the full path to the -o option, as you suggested. However, I run into a new issue:TypeError: AlignmentHeader does not support item assignment (use header.to_dict() [image: newproblem] https://user-images.githubusercontent.com/34777741/45606533-736a7a80-ba78-11e8-8fed-dd141a53b623.png I'm a newcomer to bioinformatics. Thanks for helping!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/namphuon/ViFi/issues/10#issuecomment-421891110, or mute the thread https://github.com/notifications/unsubscribe-auth/AA5tND86dvZIGOiqI6rDNw9ZIzDsb8jPks5uby1wgaJpZM4Wi5HR .

-- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093-0404

[yuanqm55]

Hi, I believe this issue was due to ViFi not being compatible with the latest version of pysam. I've updated the code. Can you try the latest version and let me know if you still have an issue? Thanks! Regards, Nam … On Sun, Sep 16, 2018 at 10:04 PM, 阿克苏地 @.> wrote: Can you let me know what's in the folder ./test_result? I am wondering if it might be some path issue. I don't know if this would impact it, but maybe try giving the full path to the -o option. If that does work, I'll fix my code so that full path is not needed. … <#m_1651321738927992081_> On Tue, Sep 11, 2018 at 2:06 AM, 阿克苏地 @.> wrote: I follow the instructions(install dependencies, set paths, download data and index the reference sequence), and everything's going well. Then I run run_vifi.py using the test data(test_R1.fq.gz and test_R2.fq.gz) and ran into a problem: Failed to open file output.bam. [image: helpme] https://user-images.githubusercontent.com/34777741/45349803-dd040800- b5e4-11e8-8490-0de1ecaab050.png — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub < #10 <#10>>, or mute the thread https://github.com/notifications/unsubscribe-auth/AA5tNMnz20mY5FbqTb4a- 0U87WSI7l77ks5uZ3z7gaJpZM4Wi5HR . -- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093 https://maps.google.com/?q=9500+Gilman+Drive,+La+Jolla,+CA+92093&entry=gmail&source=g -0404 Here's the items in the folder ./test_result: [image: what s in] https://user-images.githubusercontent.com/34777741/45605761-1bc91080-ba72-11e8-8f3e-4966cf37d879.png And the program seems to work after I gave the full path to the -o option, as you suggested. However, I run into a new issue:TypeError: AlignmentHeader does not support item assignment (use header.to_dict() [image: newproblem] https://user-images.githubusercontent.com/34777741/45606533-736a7a80-ba78-11e8-8fed-dd141a53b623.png I'm a newcomer to bioinformatics. Thanks for helping! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AA5tND86dvZIGOiqI6rDNw9ZIzDsb8jPks5uby1wgaJpZM4Wi5HR . -- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093-0404

Hi, After I tried the latest version, pysam seemed to work normally.

However, the program stalled here for over 2 hours when running the test data:

and finally the program seemed to killed itself.

Note that output.clusters.txt and output.clusters.txt.range is empty. In addition, when the program stalled, memory was uesed 99%.

Thanks for helping!

[namphuon]

Thanks for the summary. Is it correct that you have 16 GB of RAM on your machine? I will look into what caused the issue and will get back to you.

Regards, Nam

On Thu, Sep 27, 2018 at 7:34 AM, 阿克苏地 [email protected] wrote:

Hi, I believe this issue was due to ViFi not being compatible with the latest version of pysam. I've updated the code. Can you try the latest version and let me know if you still have an issue? Thanks! Regards, Nam … <#m_-3372050740237915504_> On Sun, Sep 16, 2018 at 10:04 PM, 阿克苏地 @.> wrote: Can you let me know what's in the folder ./test_result? I am wondering if it might be some path issue. I don't know if this would impact it, but maybe try giving the full path to the -o option. If that does work, I'll fix my code so that full path is not needed. … <#m_1651321738927992081_> On Tue, Sep 11, 2018 at 2:06 AM, 阿克苏地 @.> wrote: I follow the instructions(install dependencies, set paths, download data and index the reference sequence), and everything's going well. Then I run run_vifi.py using the test data(test_R1.fq.gz and test_R2.fq.gz) and ran into a problem: Failed to open file output.bam. [image: helpme] https://user-images. githubusercontent.com/34777741/45349803-dd040800- b5e4-11e8-8490-0de1ecaab050.png — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub < #10 https://github.com/namphuon/ViFi/issues/10 <#10 https://github.com/namphuon/ViFi/issues/10>>, or mute the thread https://github.com/notifications/unsubscribe-auth/AA5tNMnz20mY5FbqTb4a- 0U87WSI7l77ks5uZ3z7gaJpZM4Wi5HR . -- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093 https://maps.google.com/?q=9500+Gilman+Drive,+La+Jolla,+CA+92093&entry=gmail&source=g https://maps.google.com/?q=9500+Gilman+Drive,+La+Jolla,+ CA+92093&entry=gmail&source=g -0404 Here's the items in the folder ./test_result: [image: what s in] https://user-images. githubusercontent.com/34777741/45605761-1bc91080- ba72-11e8-8f3e-4966cf37d879.png And the program seems to work after I gave the full path to the -o option, as you suggested. However, I run into a new issue:TypeError: AlignmentHeader does not support item assignment (use header.to_dict() [image: newproblem] https://user-images. githubusercontent.com/34777741/45606533-736a7a80- ba78-11e8-8fed-dd141a53b623.png I'm a newcomer to bioinformatics. Thanks for helping! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10 (comment) https://github.com/namphuon/ViFi/issues/10#issuecomment-421891110>, or mute the thread https://github.com/notifications/unsubscribe-auth/ AA5tND86dvZIGOiqI6rDNw9ZIzDsb8jPks5uby1wgaJpZM4Wi5HR . -- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093 https://maps.google.com/?q=9500+Gilman+Drive,+La+Jolla,+CA+92093&entry=gmail&source=g -0404

Hi, After I tried the latest version, pysam seemed to work normally. [image: 1] https://user-images.githubusercontent.com/34777741/46153070-012e3d00-c2a5-11e8-97e7-ccdb51e94c33.png

However, the program stalled here for over 2 hours when running the test data: [image: 2] https://user-images.githubusercontent.com/34777741/46153064-fb385c00-c2a4-11e8-8a34-2516f4c3c9fd.png

and finally the program seemed to killed itself. [image: kill2] https://user-images.githubusercontent.com/34777741/46153048-f4114e00-c2a4-11e8-920e-28144d07510c.png

Note that output.clusters.txt and output.clusters.txt.range is empty. In addition, when the program stalled, memory was uesed 99%. [image: default] https://user-images.githubusercontent.com/34777741/46153036-e8258c00-c2a4-11e8-9de7-94f389902801.png

Thanks for helping!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/namphuon/ViFi/issues/10#issuecomment-425116114, or mute the thread https://github.com/notifications/unsubscribe-auth/AA5tNJIc6gQL_dNerTfZ41Z5F-3j72ujks5ufOHkgaJpZM4Wi5HR .

-- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093-0404

[yuanqm55]

Thanks for the summary. Is it correct that you have 16 GB of RAM on your machine? I will look into what caused the issue and will get back to you. Regards, Nam … On Thu, Sep 27, 2018 at 7:34 AM, 阿克苏地 @.> wrote: Hi, I believe this issue was due to ViFi not being compatible with the latest version of pysam. I've updated the code. Can you try the latest version and let me know if you still have an issue? Thanks! Regards, Nam … <#m_-3372050740237915504_> On Sun, Sep 16, 2018 at 10:04 PM, 阿克苏地 @.> wrote: Can you let me know what's in the folder ./test_result? I am wondering if it might be some path issue. I don't know if this would impact it, but maybe try giving the full path to the -o option. If that does work, I'll fix my code so that full path is not needed. … <#m_1651321738927992081_> On Tue, Sep 11, 2018 at 2:06 AM, 阿克苏地 @.***> wrote: I follow the instructions(install dependencies, set paths, download data and index the reference sequence), and everything's going well. Then I run run_vifi.py using the test data(test_R1.fq.gz and test_R2.fq.gz) and ran into a problem: Failed to open file output.bam. [image: helpme] https://user-images. githubusercontent.com/34777741/45349803-dd040800- b5e4-11e8-8490-0de1ecaab050.png — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub < #10 <#10> <#10 <#10>>>, or mute the thread https://github.com/notifications/unsubscribe-auth/AA5tNMnz20mY5FbqTb4a- 0U87WSI7l77ks5uZ3z7gaJpZM4Wi5HR . -- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093 https://maps.google.com/?q=9500+Gilman+Drive,+La+Jolla,+CA+92093&entry=gmail&source=g https://maps.google.com/?q=9500+Gilman+Drive,+La+Jolla,+ CA+92093&entry=gmail&source=g -0404 Here's the items in the folder ./test_result: [image: what s in] https://user-images. githubusercontent.com/34777741/45605761-1bc91080- ba72-11e8-8f3e-4966cf37d879.png And the program seems to work after I gave the full path to the -o option, as you suggested. However, I run into a new issue:TypeError: AlignmentHeader does not support item assignment (use header.to_dict() [image: newproblem] https://user-images. githubusercontent.com/34777741/45606533-736a7a80- ba78-11e8-8fed-dd141a53b623.png I'm a newcomer to bioinformatics. Thanks for helping! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10 (comment) <#10 (comment)>>, or mute the thread https://github.com/notifications/unsubscribe-auth/ AA5tND86dvZIGOiqI6rDNw9ZIzDsb8jPks5uby1wgaJpZM4Wi5HR . -- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093 https://maps.google.com/?q=9500+Gilman+Drive,+La+Jolla,+CA+92093&entry=gmail&source=g -0404 Hi, After I tried the latest version, pysam seemed to work normally. [image: 1] https://user-images.githubusercontent.com/34777741/46153070-012e3d00-c2a5-11e8-97e7-ccdb51e94c33.png However, the program stalled here for over 2 hours when running the test data: [image: 2] https://user-images.githubusercontent.com/34777741/46153064-fb385c00-c2a4-11e8-8a34-2516f4c3c9fd.png and finally the program seemed to killed itself. [image: kill2] https://user-images.githubusercontent.com/34777741/46153048-f4114e00-c2a4-11e8-920e-28144d07510c.png Note that output.clusters.txt and output.clusters.txt.range is empty. In addition, when the program stalled, memory was uesed 99%. [image: default] https://user-images.githubusercontent.com/34777741/46153036-e8258c00-c2a4-11e8-9de7-94f389902801.png Thanks for helping! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AA5tNJIc6gQL_dNerTfZ41Z5F-3j72ujks5ufOHkgaJpZM4Wi5HR . -- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093-0404

Hi, it is correct that I have 16 GB of RAM on my machine. Thanks in advance :)

[namphuon]

Sorry for the long delay in reply. I've looked into this and can't figure out why it would hang/take that much memory at that step. The most memory intensive/CPU intensive steps are the mapping and HMM search steps. After that, it should be fast.

Can you let me know whether or not the following bam files are empty or their sizes by running:

ls -lht /home/yuanqm/ViFi/test_result/*.bam

If there is no file labeled:

/home/yuanqm/ViFi/test_result/output.fixed.trans.bam

Then the process got killed at the merging step. If you do have the this file and it is non-empty, then the very last script failed.

The merged file can be created using:

cd /home/yuanqm/ViFi/test_result/output python /home/yuanqm/ViFi/scripts/merge_viral_reads.py --unknown /home/yuanqm/ViFi/test_result/output.unknown.bam --trans /home/yuanqm/ViFi/test_result/output.trans.bam --reduced tmp/temp/reduced.csv --map tmp/temp/unmapped.map --output /home/yuanqm/ViFi/test_result/output.fixed.trans.bam

If you could first try running that command and seeing if it eats up your CPU, let me know. If it does not, then it's been isolated to the very last script.

Regards, Nam

On Thu, Sep 27, 2018 at 10:08 AM 阿克苏地 [email protected] wrote:

Thanks for the summary. Is it correct that you have 16 GB of RAM on your machine? I will look into what caused the issue and will get back to you. Regards, Nam … <#m_4466952601121542937_> On Thu, Sep 27, 2018 at 7:34 AM, 阿克苏地 @.> wrote: Hi, I believe this issue was due to ViFi not being compatible with the latest version of pysam. I've updated the code. Can you try the latest version and let me know if you still have an issue? Thanks! Regards, Nam … <#m_-3372050740237915504_> On Sun, Sep 16, 2018 at 10:04 PM, 阿克苏地 @.> wrote: Can you let me know what's in the folder ./test_result? I am wondering if it might be some path issue. I don't know if this would impact it, but maybe try giving the full path to the -o option. If that does work, I'll fix my code so that full path is not needed. … <#m_1651321738927992081_> On Tue, Sep 11, 2018 at 2:06 AM, 阿克苏地 @.***> wrote: I follow the instructions(install dependencies, set paths, download data and index the reference sequence), and everything's going well. Then I run run_vifi.py using the test data(test_R1.fq.gz and test_R2.fq.gz) and ran into a problem: Failed to open file output.bam. [image: helpme] https://user-images. githubusercontent.com/34777741/45349803-dd040800- b5e4-11e8-8490-0de1ecaab050.png — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub < #10 https://github.com/namphuon/ViFi/issues/10 <#10 https://github.com/namphuon/ViFi/issues/10> <#10 https://github.com/namphuon/ViFi/issues/10 <#10 https://github.com/namphuon/ViFi/issues/10>>>, or mute the thread https://github.com/notifications/unsubscribe-auth/AA5tNMnz20mY5FbqTb4a- 0U87WSI7l77ks5uZ3z7gaJpZM4Wi5HR . -- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093 https://maps.google.com/?q=9500+Gilman+Drive,+La+Jolla,+CA+92093&entry=gmail&source=g https://maps.google.com/?q=9500+Gilman+Drive,+La+Jolla,+ CA+92093&entry=gmail&source=g -0404 Here's the items in the folder ./test_result: [image: what s in] https://user-images. githubusercontent.com/34777741/45605761-1bc91080- ba72-11e8-8f3e-4966cf37d879.png And the program seems to work after I gave the full path to the -o option, as you suggested. However, I run into a new issue:TypeError: AlignmentHeader does not support item assignment (use header.to_dict() [image: newproblem] https://user-images. githubusercontent.com/34777741/45606533-736a7a80- ba78-11e8-8fed-dd141a53b623.png I'm a newcomer to bioinformatics. Thanks for helping! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10 https://github.com/namphuon/ViFi/issues/10 (comment) <#10 (comment) https://github.com/namphuon/ViFi/issues/10#issuecomment-421891110>>, or mute the thread https://github.com/notifications/unsubscribe-auth/ AA5tND86dvZIGOiqI6rDNw9ZIzDsb8jPks5uby1wgaJpZM4Wi5HR . -- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093 https://maps.google.com/?q=9500+Gilman+Drive,+La+Jolla,+CA+92093&entry=gmail&source=g -0404 Hi, After I tried the latest version, pysam seemed to work normally. [image: 1] https://user-images.githubusercontent.com/34777741/46153070-012e3d00-c2a5-11e8-97e7-ccdb51e94c33.png However, the program stalled here for over 2 hours when running the test data: [image: 2] https://user-images.githubusercontent.com/34777741/46153064-fb385c00-c2a4-11e8-8a34-2516f4c3c9fd.png and finally the program seemed to killed itself. [image: kill2] https://user-images.githubusercontent.com/34777741/46153048-f4114e00-c2a4-11e8-920e-28144d07510c.png Note that output.clusters.txt and output.clusters.txt.range is empty. In addition, when the program stalled, memory was uesed 99%. [image: default] https://user-images.githubusercontent.com/34777741/46153036-e8258c00-c2a4-11e8-9de7-94f389902801.png Thanks for helping! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10 (comment) https://github.com/namphuon/ViFi/issues/10#issuecomment-425116114>, or mute the thread https://github.com/notifications/unsubscribe-auth/AA5tNJIc6gQL_dNerTfZ41Z5F-3j72ujks5ufOHkgaJpZM4Wi5HR . -- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093-0404

Hi, it is correct that I have 16 GB of RAM on my machine. Thanks in advance :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/namphuon/ViFi/issues/10#issuecomment-425169989, or mute the thread https://github.com/notifications/unsubscribe-auth/AA5tNPRgq35CtseORUFi-CuGsf3VgxfTks5ufQXwgaJpZM4Wi5HR .

-- Nam-phuong Nguyen, PhD Computer Science and Engineering University of California, San Diego 9500 Gilman Drive, La Jolla, CA 92093-0404

[ibqcarlos]

Hi Nam, I have a similar problem and I think it's with the cluster_trans_new.py script. I attach some images of the problem.

Thank you, Carlos