FragPipe
FragPipe copied to clipboard
Nesvilab
Philosopher cannot create report files due to unknown reason. It also needs more than 2 TB memory with 280 GB interact.pep.xml files.
Hi FragPipe,
I am trying to create a spectral library with DIA-Umpire_SpecLib workflow. While running Philosopher Report, an Cannot write file error occurred.
Here are some details:
- Raw data: About 2500 DIA files and 350 DDA files
- Workflow: DIA_DIA-Umpire_SpecLib_Quant
- Version: FragPipe 17.1, MSFragger 3.4 and Philosopher 4.1.1
In addition, the DIA-Umpire, MSFragger, PeptideProphet and ProteinProphet were run on one server. But the Filter and Report were run on another server since a 2 TB memory requirement. But I am not supposed it is the cause of this issue. Because similar processes have already been performed normally with less files.
I am sure that the disk space is sufficient. And it does not seem like a permission issue. Is it possible due to too large .bin files or something, since the size of psm.bin is up to 113 GB? Or how could we deal with it?
Best, Wenhao
@Wenhhao try the latest release of Philosopher, which will reduce the file sizes significantly. https://github.com/Nesvilab/philosopher/tags
Thanks @guoci . So is that means I could start from Philosopher Report? And could we get rid of this issue with the lateset release? Hundreds of GB is not that big, but dozens of GB is the most I have got in the runs before.
@Wenhhao You need to rerun all the philosopher steps because the binary files are no longer compatible. For non-TMT data, the new version should only use about 15% of the storage of that version of Philosopher you are currently using.
Hi @guoci ,
But why would the size be an issue? They have enough space and memory. Do you think with the new Philosopher and smaller bin files, this issue will be gone?
Best,
Fengchao
@guoci Do you think the Cannot write file error is due to big binary file size? Since I am not sure about it, just my guess.
I am not sure, why not try it out. Or do you need to pay for the compute resources?
I can try it, but it will take more than a month. So it would be better if there were a definite solution.
@Wenhhao try the latest release of Philosopher, which will reduce the file sizes significantly. https://github.com/Nesvilab/philosopher/tags Can the latest Philosopher reduce the resources we required, since wenhao mentioned we need 2TB memory in one server which is not easy for us.
May I ask in which command Philosopher needs 2TB memory?
Thanks,
Fengchao
On Sun, May 8, 2022 at 1:15 AM XiaoQi @.***> wrote:
@Wenhhao https://github.com/Wenhhao try the latest release of Philosopher, which will reduce the file sizes significantly. https://github.com/Nesvilab/philosopher/tags Can the latest Philosopher reduce the resources we required, since wenhao mentioned we need 2TB memory in one server which is not easy for us.
— Reply to this email directly, view it on GitHub https://github.com/Nesvilab/FragPipe/issues/669#issuecomment-1120352102, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU27WZGRRP5EKN6J2EK6GLVI5EXXANCNFSM5VKVCGWA . You are receiving this because you commented.Message ID: @.***>
-- Dr. Fengchao Yu Research Investigator University of Michigan
@XiaoQiiiii both the memory and storage will be reduced, and the processing speed will also be improved. @Wenhhao we cannot provide a definite solution. There is no way to prove it will run on your data successfully until you do it.
May I ask in which command Philosopher needs 2TB memory? Thanks, Fengchao On Sun, May 8, 2022 at 1:15 AM XiaoQi @.> wrote: @Wenhhao https://github.com/Wenhhao try the latest release of Philosopher, which will reduce the file sizes significantly. https://github.com/Nesvilab/philosopher/tags Can the latest Philosopher reduce the resources we required, since wenhao mentioned we need 2TB memory in one server which is not easy for us. — Reply to this email directly, view it on GitHub <#669 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABU27WZGRRP5EKN6J2EK6GLVI5EXXANCNFSM5VKVCGWA . You are receiving this because you commented.Message ID: @.> -- Dr. Fengchao Yu Research Investigator University of Michigan
In command Philosopher Filter. Precisely was 1.5-2.0 TB memory.
@guoci @fcyu Thank you very much. We are going to try again use the latest Philosopher.
Besides, I still have a question for the files genetated by DIAumpire. To save time, we plan to use Q1 and Q2 and abandon Q3, do you think it is resonable?
You can use just Q1. This is what I did with large datasets myself in the last Alexey
Get Outlook for iOShttps://aka.ms/o0ukef
From: XiaoQi @.> Sent: Monday, May 9, 2022 2:03:56 AM To: Nesvilab/FragPipe @.> Cc: Subscribed @.***> Subject: Re: [Nesvilab/FragPipe] Philosopher error: cannot create report file (Issue #669)
External Email - Use Caution
@guocihttps://github.com/guoci @fcyuhttps://github.com/fcyu Thank you very much. We are going to try again use the latest Philosopher.
Besides, I still have a question for the files genetated by DIAumpire. To save time, we plan to use Q1 and Q2 and abandon Q3, do you think it is resonable?
— Reply to this email directly, view it on GitHubhttps://github.com/Nesvilab/FragPipe/issues/669#issuecomment-1120677829, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIIMM66C2NMX7GLRADVKIPTVJCTEZANCNFSM5VKVCGWA. You are receiving this because you are subscribed to this thread.Message ID: @.***>
Electronic Mail is not secure, may not be read every day, and should not be used for urgent or sensitive issues
@Wenhhao Could you give me an estimate on how much data you're trying to process? Please sum the size in GBs of all your .pepXML files
@Wenhhao Could you give me an estimate on how much data you're trying to process? Please sum the size in GBs of all your .pepXML files
We have more than 15,000 pep.xml files but only 280 GB in size.