pefile
pefile copied to clipboard
erocarrera
Missing testfiles in tarball
Hello, there are missing testfiles in the release tarballs. Could you please include them and maybe bump a new version if you have new features? Thanks!
Some of test files are malware and are encrypted, See https://github.com/erocarrera/pefile/issues/82#issuecomment-192018385
FYI to support the effort by @Rafiot to get pefile to work better with Python 3 in @viper-framework , I committed this repo of non-encrypted test files for pefile https://github.com/viper-framework/pefile-tests including a set of excellent warped PEs by @angea (some of them that make pefile choke alright were moved to the "failing" directory) and including some PE with undocumented origins (these will be documented eventually)
Any tests that relied on files not reproducible were deleted.
It would be best if @erocarrera you could get some of your test files made public .... (including for instance your highly coveted empty_file that surely does not deserve military-grade encryption ;) )
@erocarrera is there anything special about the files that you have in your encrypted test set and that are referenced in these places:
- MSVBVM60.DLL here and elsewhere https://github.com/erocarrera/pefile/blob/aa44f2d96d1a7e7e9c84ec994efdcc068d6f6e3f/tests/pefile_test.py#L140
- mfc40.dll and kernel32.dll at https://github.com/erocarrera/pefile/blob/aa44f2d96d1a7e7e9c84ec994efdcc068d6f6e3f/tests/pefile_test.py#L158
- cmd.exe here https://github.com/erocarrera/pefile/blob/aa44f2d96d1a7e7e9c84ec994efdcc068d6f6e3f/tests/pefile_test.py#L175 (that I verified externall to be a vanilla cmd.exe from Win7 64.
Because if these are the problematic test files, there can easily be replaced with vanilla equivalent PEs. And @angea set of malformed PEs are otherwise great (and fairly small)
Beyond these:
-
in https://github.com/erocarrera/pefile/blob/aa44f2d96d1a7e7e9c84ec994efdcc068d6f6e3f/tests/pefile_test.py#L171 66c74e4c9dbd1d33b22f63cd0318b72dea88f9dbb4d36a3383d3da20b037d42e is likely a sha256 for a version of msrd2x35.dll at https://virustotal.com/en/file/66c74e4c9dbd1d33b22f63cd0318b72dea88f9dbb4d36a3383d3da20b037d42e/analysis/
-
in https://github.com/erocarrera/pefile/blob/aa44f2d96d1a7e7e9c84ec994efdcc068d6f6e3f/tests/pefile_test.py#L446 is likely: https://virustotal.com/en/file/075356de51afac92d1c20ba53c966fa145172897a96cfdb1b3bb369edb376a77/analysis/
... and so on. The point is that there are unlikely cases that could not be reprodcued eitherr from public sources and or with carefully crafted PEs like suggestedd in https://github.com/erocarrera/pefile/issues/82#issuecomment-182138771 by @JonathonReinhart
@RyPeck Of course it's used for malware. But you can absolutely test a library like pefile without having actual, live malware checked into the repo. If there's some technique being used by malware that pefile has been modified to test, then a PE file should be created that mimics just that technique.
A good way to avoid having large binaries checked-in, just for the sake of unit testing, is to have scripts/source code which generates those input files.
For the records, this is the current status of the testings:
- https://travis-ci.org/viper-framework/pefile/builds/228798579
- https://codecov.io/gh/viper-framework/pefile/
I'll try to fix it in the coming days.
@Rafiot it sounds convenient to support Python 3 versions before 3.6 ..... because it is installed on some distros (such as some still maintained Ubuntu LTS 14) ... Yet, I feel the overall usage of Py3.4 is confidential at best. And installing a Python 3.6 from sources or from a "backport" package repo is rather easy...
On my side I would be 100% happy if the minimum version of Py3 that would be supported by pefile (and viper) would be Python 3.6 only. This would make several issues of the port just go away.... @erocarrera what's your take there: e.g. support both 2.7+ and 3.6+ but none of 3 <3.6 ?
@erocarrera Any feedback or comment? Would you be OK to have an open test suite based on binaries that have a known open source provenance and license?
FWIW, I updated https://github.com/pombredanne/pefile-tests with a new set of PE tests files from lief. This makes it one of the largest PE test suite I know of. Short of may be the closed tests suite encrypted in this repo..... :angel:
And I also adopted a few extra refinements for tests using pytest with its parametric fixtures feature for simpler code.
Thank you @pombredanne for the amazing work of putting together such a test suite. I am now using it as part of a GitHub Action to run tests, something I had pending for a long time since I had dropped Travis. I still want to look into adding any files from the encrypted blob, as you found out, for many it probably makes little sense to keep them there.
