IPED
IPED copied to clipboard
sepinf-inc
Improve LocalConfig.txt options loading on different machines
@patrickdalla just reported a "Too few available memory per processing thread" warning while generating a report. The issue was: numThreads was fixed as 40 in processing machine, but the analysis machine has much less processing cores. A related issue could happen with hashesDB. indexTemp already has a special handling to be used just if it exists, but maybe that is not desired by the user. But if the same machine is used for processing and analysis, those options should be loaded. Maybe we can store them in local user OS temp like other options.
We had the same issue a while ago, with numThreads=60 that came from cases processing in our server.
@lfcnassif, I believe we talked about this in the past, and recently I observed again an issue caused by the indexTemp location of a case processed in one machine and acessed by another user/machine.
If the configured indexTemp path can't be acessed, IPED will use the default configuration, right?
In the particular case, "G:\TempIPED" was used during processing. When opening the case "G:" was a pen drive, so a "\TempIPED" was created. Everything worked, but much slower that it should be. Probably, it could ran into disk space issues...
Even when the indexTemp point to a folder like "C:\IPED-TEMP", it will work in other machines, but the user may find strange that such folder was created in the root level of his/her C:.
What about reseting some configurations in LocalConfig.txt, when the output is a report or "--portable" parameter is used? I am talking about:
- indexTemp
- numThreads
- hashesDB
If the configured
indexTemppath can't be acessed, IPED will use the default configuration, right?
Right, that is the current intended behavior.
In the particular case, "G:\TempIPED" was used during processing. When opening the case "G:" was a pen drive, so a "\TempIPED" was created. Everything worked, but much slower that it should be. Probably, it could ran into disk space issues...
Yes, "no space left on device" when opening the case was reported recently to me because of this.
What about reseting some configurations in LocalConfig.txt, when the output is a report or "--portable" parameter is used? I am talking about:
Seems reasonable, I like the reset if report approach. Maybe that won't cover all situations... About the --portable option, I was planning to remove it, since I think it is useless today, the application always try to derivate relative paths to source evidences if they are in the same volume as the case.
Another option would be to save last LocalConfig.txt params used in processing in user.home/.iped folder. If that is found (examiner opening the processed case), use it. If not (investigator opening a case/report), use the defaults.
Maybe that won't cover all situations...
Like sending the complete case to an investigator or if the examiner opens the report.
Another option would be to save last LocalConfig.txt params used in processing in user.home/.iped folder. If that is found (examiner opening the processed case), use it. If not (investigator opening a case/report), use the defaults.
Since it is a local machine configuration, saving it in the local machine and trying to reuse it if saved/configured seems good to me. We could even just use the user.home/.iped/ipedRoot.txt file to find the LocalConfig.txt of current IPED "installation" and use it for all opened cases.
Another option would be to save last LocalConfig.txt params used in processing in user.home/.iped folder. If that is found (examiner opening the processed case), use it. If not (investigator opening a case/report), use the defaults.
That seems to be a good option.
Another option would be to save last LocalConfig.txt params used in processing in user.home/.iped folder. If that is found (examiner opening the processed case), use it. If not (investigator opening a case/report), use the defaults.
Since it is a
local machine configuration, saving it in the local machine and trying to reuse it if saved/configured seems good to me. We could even just use the user.home/.iped/ipedRoot.txt file to find the LocalConfig.txt of current IPED "installation" and use it for all opened cases.
Thinking a bit more about this, using the LocalConfig.txt from the current "IPED installation" is a simple/practical solution, but may have some disadvantages.
Would LocalConfig.txt still be copied to the processed case folder? I guess having the file there, but using the configuration from another location may be confusing. So, it would be probably better to not copy this file. But that may be inconvenient in situations when you want to change some configuration for a specific case. Another concern about not having the file, would be the 'locale' parameter. Using the default locale in another machine, probably won't be the desired behavior.
Seems reasonable, I like the reset if report approach. Maybe that won't cover all situations... About the --portable option, I was planning to remove it, since I think it is useless today, the application always try to derivate relative paths to source evidences if they are in the same volume as the case.
Even in the first case processed it would be useful to restore such properties in LocalConfig.txt. A scenario is where we process the UFDR with --portable flag and deliver the case with UFDR in a media. Another scenario is where the case is delivered in a server.
Even in the first case processed it would be useful to restore such properties in
LocalConfig.txt. A scenario is where we process the UFDR with--portableflag and deliver the case with UFDR in a media. Another scenario is where the case is delivered in a server.
Yes, that's why I said the reset if report approach would not cover all cases.
process the UFDR with
--portableflag
This has no effect, --portable just applies with evidences processed by TSK, replacing absolute paths in TSK database by relative ones. Even if not used, and if the evidence is not found, IPED already tries to find the evidence (the heuristic in detailed in the Wiki/Manual). For other evidence types, a relative path is already used internally. So I think that option is useless today...
So we have 2 approaches for now:
- keep the --portable option instead of removing it (it is useless today) and reset the LocalConfig.txt (except locale) if it is used or if output is a report;
- move the
localeparam to another file (IPEDConfig.txt or a new LocaleConfig.txt), stop copying the LocalConfig.txt to output folder and reuse the LocalConfig.txt from "installation folder" if found or the default values if it doesn't exist;
Is there another solution? Votes?
OK, I'm implementing the first approach.
- indexTemp
- numThreads
- hashesDB
I'll also reset indexTempOnSSD param. What about outputOnSSD, should it also be reseted?
And should a blind (profile) data extraction case also reset those params?
I'll also reset
indexTempOnSSDparam. What aboutoutputOnSSD, should it also be reseted?
I think these are less critical, but in my own cases (before sending to someone else) I am resetting both to false.
And should a blind (profile) data extraction case also reset those params?
Maybe this is not needed, users could use the --portable option.
By the way, just remembered --portable can be used to #863 (in a distant future 5.0.0 version)
Shouldnt IPEDConfig be reset too? Or enableAudioTranscription be put in LocalConfig ? We tried to generate a report with few selected itens in other machine and got some python erros because enableAudioTranscription was set true and we have no python modules on that machine.
We tried to generate a report with few selected itens in other machine and got some python erros because enableAudioTranscription was set true and we have no python modules on that machine.
What version triggered the error? This should have been fixed on 4.1.1, see #1589
4.1.2. Setting false it worked fine. IPED-2023-06-29-11-36-15.log
Thanks for reporting, I was able to reproduce the issue. Seems it wasn't completely fixed, I'll reopen #1589 and apply the fix there.