dissect.target Add Unix and Windows application plugins

This PR adds Unix (snapd and .desktop) and Windows (registry) application plugins to dissect.

Fixes #884, fixes #885 and fixes #886.

Sep 17 '24 09:09 JSCU-CNI

As mentioned in #852. Please also add the record field documentation to their respective exported functions

In case you missed it, could you please add this to the exported functions still?

Sep 26 '24 10:09 Horofic

In case you missed it, could you please add this to the exported functions still?

I missed that, thanks for reminding me. Added in f354b97.

Sep 30 '24 07:09 JSCU-CNI

Codecov Report

Attention: Patch coverage is 89.21569% with 11 lines in your changes missing coverage. Please review.

Project coverage is 76.85%. Comparing base (d0fa985) to head (b989f78). Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...issect/target/plugins/os/unix/linux/debian/snap.py	78.94%	8 Missing :warning:
...ect/target/plugins/os/windows/regf/applications.py	91.30%	2 Missing :warning:
dissect/target/plugins/os/unix/applications.py	96.77%	1 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #851      +/-   ##
==========================================
+ Coverage   76.75%   76.85%   +0.10%     
==========================================
  Files         315      318       +3     
  Lines       27130    27231     +101     
==========================================
+ Hits        20823    20929     +106     
+ Misses       6307     6302       -5

Flag	Coverage Δ
unittests	`76.85% <89.21%> (+0.10%)`	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Oct 03 '24 08:10 codecov[bot]

The tests seem to be failing on the Windows applications plugin. Looks like an invalid timestamp is passed. Can you try fix this?

Oct 03 '24 08:10 Horofic

The tests seem to be failing on the Windows applications plugin. Looks like an invalid timestamp is passed. Can you try fix this?

I cannot reproduce that locally, could you share stdout/err?

Oct 07 '24 09:10 JSCU-CNI

Upon looking a little closer the the Windows application plugin fails from Python (and PyPy) version 3.9 up to and including 3.10 on Ubuntu and Windows.

Errors across Python versions and OSes are similar of nature (PyPy 3.10 on Windows):

cls = <class 'flow.record.fieldtypes.datetime'>, date_string = '20240301'

    @classmethod
    def fromisoformat(cls, date_string):
        """Construct a datetime from the output of datetime.isoformat()."""
        if not isinstance(date_string, str):
            raise TypeError('fromisoformat: argument must be str')
    
        # Split this at the separator
        dstr = date_string[0:10]
        tstr = date_string[11:]
    
        try:
>           date_components = _parse_isoformat_date(dstr)

C:\hostedtoolcache\windows\PyPy\3.10.14\x86\Lib\datetime.py:1752: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

dtstr = '20240301'

    def _parse_isoformat_date(dtstr):
        # It is assumed that this function will only be called with a
        # string of length exactly 10, and (though this is not used) ASCII-only
        if len(dtstr) < 10:
>           raise ValueError('isoformat expects a string of length 10')
E           ValueError: isoformat expects a string of length 10

C:\hostedtoolcache\windows\PyPy\3.10.14\x86\Lib\datetime.py:275: ValueError

During handling of the above exception, another exception occurred:

target_win_users = <Target D:\a\dissect.target\dissect.target\.tox\pypy3.10\tmp\test_windows_applications0\MockTarget-yc3dbmav>
hive_hklm = <VirtualHive>

    def test_windows_applications(target_win_users: Target, hive_hklm: VirtualHive) -> None:
        """test if windows applications are detected correctly in the registry"""
    
        firefox_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\Mozilla Firefox 123.0.1 (x64 nl)"
        firefox_key = VirtualKey(hive_hklm, firefox_name)
        firefox_key.add_value("Comments", "Mozilla Firefox 123.0.1 (x64 nl)")
        firefox_key.add_value("DisplayIcon", "C:\\Program Files\\Mozilla Firefox\\firefox.exe,0")
        firefox_key.add_value("DisplayName", "Mozilla Firefox (x64 nl)")
        firefox_key.add_value("DisplayVersion", "123.0.1")
        firefox_key.add_value("EstimatedSize", 238271)
        firefox_key.add_value("HelpLink", "https://support.mozilla.org/")
        firefox_key.add_value("InstallLocation", "C:\\Program Files\\Mozilla Firefox")
        firefox_key.add_value("NoModify", 1)
        firefox_key.add_value("NoRepair", 1)
        firefox_key.add_value("Publisher", "Mozilla")
        firefox_key.add_value("URLInfoAbout", "https://www.mozilla.org/")
        firefox_key.add_value("URLUpdateInfo", "https://www.mozilla.org/firefox/123.0.1/releasenotes")
        firefox_key.add_value("UninstallString", '"C:\\Program Files\\Mozilla Firefox\\uninstall\\helper.exe"')
        hive_hklm.map_key(firefox_name, firefox_key)
    
        chrome_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\{47FB91DD-98F3-3C87-A963-357B14EAC7C9}"
        chrome_key = VirtualKey(hive_hklm, chrome_name)
        chrome_key.add_value("DisplayVersion", "122.0.6261.95")
        chrome_key.add_value("InstallDate", "20240301")
        chrome_key.add_value("InstallLocation", "")
        chrome_key.add_value("InstallSource", "C:\\Users\\user\\Desktop\\GoogleChromeEnterpriseBundle64\\Installers\\")
        chrome_key.add_value("ModifyPath", "MsiExec.exe /X{47FB91DD-98F3-3C87-A963-357B14EAC7C9}")
        chrome_key.add_value("NoModify", 1)
        chrome_key.add_value("Publisher", "Google LLC")
        chrome_key.add_value("EstimatedSize", 113725)
        chrome_key.add_value("UninstallString", "MsiExec.exe /X{47FB91DD-98F3-3C87-A963-357B14EAC7C9}")
        chrome_key.add_value("VersionMajor", 70)
        chrome_key.add_value("VersionMinor", 29)
        chrome_key.add_value("WindowsInstaller", 1)
        chrome_key.add_value("Version", 1176322143)
        chrome_key.add_value("Language", 1033)
        chrome_key.add_value("DisplayName", "Google Chrome")
        hive_hklm.map_key(chrome_name, chrome_key)
    
        addressbook_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\AddressBook"
        addressbook_key = VirtualKey(hive_hklm, addressbook_name)
        addressbook_key.timestamp = datetime(2024, 12, 31, 13, 37, 0, tzinfo=timezone.utc)
        hive_hklm.map_key(addressbook_name, addressbook_key)
    
        msvc_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\{D5D19E2F-7189-42FE-8103-92CD1FA457C2}"
        msvc_key = VirtualKey(hive_hklm, msvc_name)
        msvc_key.add_value("DisplayName", "Microsoft Visual C++ 2022 X64 Minimum Runtime - 14.36.32532")
        msvc_key.add_value("InstallDate", "20240301")
        msvc_key.add_value("DisplayVersion", "14.36.32532")
        msvc_key.add_value("Publisher", "Microsoft Corporation")
        msvc_key.add_value(
            "InstallSource",
            "C:\\ProgramData\\Package Cache\\{D5D19E2F-7189-42FE-8103-92CD1FA457C2}v14.36.32532\\packages\\vcRuntimeMinimum_amd64\\",  # noqa: E501
        )
        msvc_key.add_value("SystemComponent", 1)
        hive_hklm.map_key(msvc_name, msvc_key)
    
        target_win_users.add_plugin(WindowsApplicationsPlugin)
>       results = sorted(list(target_win_users.applications()), key=lambda r: r.name)

tests\plugins\os\windows\regf\test_applications.py:68: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
dissect\target\plugins\os\windows\regf\applications.py:49: in applications
    yield WindowsApplicationRecord(
dissect\target\helpers\record.py:78: in __call__
    return super().__call__(*args, **kwargs)
.tox\pypy3.10\lib\site-packages\flow\record\base.py:595: in __call__
    return self.recordType(*args, **kwargs)
<string>:26: in __init__
    ???
.tox\pypy3.10\lib\site-packages\flow\record\base.py:195: in __setattr__
    v = field_type(v)
.tox\pypy3.10\lib\site-packages\flow\record\fieldtypes\__init__.py:323: in __new__
    obj = cls.fromisoformat(arg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cls = <class 'flow.record.fieldtypes.datetime'>, date_string = '20240301'

    @classmethod
    def fromisoformat(cls, date_string):
        """Construct a datetime from the output of datetime.isoformat()."""
        if not isinstance(date_string, str):
            raise TypeError('fromisoformat: argument must be str')
    
        # Split this at the separator
        dstr = date_string[0:10]
        tstr = date_string[11:]
    
        try:
            date_components = _parse_isoformat_date(dstr)
        except ValueError:
>           raise ValueError(f'Invalid isoformat string: {date_string!r}')
E           ValueError: Invalid isoformat string: '20240301'

C:\hostedtoolcache\windows\PyPy\3.10.14\x86\Lib\datetime.py:1754: ValueError

Python 3.9 on Ubuntu:

target_win_users = <Target /home/runner/work/dissect.target/dissect.target/.tox/3.9/tmp/test_windows_applications0/MockTarget-kj40a8z8>
hive_hklm = <VirtualHive>

    def test_windows_applications(target_win_users: Target, hive_hklm: VirtualHive) -> None:
        """test if windows applications are detected correctly in the registry"""
    
        firefox_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\Mozilla Firefox 123.0.1 (x64 nl)"
        firefox_key = VirtualKey(hive_hklm, firefox_name)
        firefox_key.add_value("Comments", "Mozilla Firefox 123.0.1 (x64 nl)")
        firefox_key.add_value("DisplayIcon", "C:\\Program Files\\Mozilla Firefox\\firefox.exe,0")
        firefox_key.add_value("DisplayName", "Mozilla Firefox (x64 nl)")
        firefox_key.add_value("DisplayVersion", "123.0.1")
        firefox_key.add_value("EstimatedSize", 238271)
        firefox_key.add_value("HelpLink", "https://support.mozilla.org/")
        firefox_key.add_value("InstallLocation", "C:\\Program Files\\Mozilla Firefox")
        firefox_key.add_value("NoModify", 1)
        firefox_key.add_value("NoRepair", 1)
        firefox_key.add_value("Publisher", "Mozilla")
        firefox_key.add_value("URLInfoAbout", "https://www.mozilla.org/")
        firefox_key.add_value("URLUpdateInfo", "https://www.mozilla.org/firefox/123.0.1/releasenotes")
        firefox_key.add_value("UninstallString", '"C:\\Program Files\\Mozilla Firefox\\uninstall\\helper.exe"')
        hive_hklm.map_key(firefox_name, firefox_key)
    
        chrome_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\{47FB91DD-98F3-3C87-A963-357B14EAC7C9}"
        chrome_key = VirtualKey(hive_hklm, chrome_name)
        chrome_key.add_value("DisplayVersion", "122.0.6261.95")
        chrome_key.add_value("InstallDate", "20240301")
        chrome_key.add_value("InstallLocation", "")
        chrome_key.add_value("InstallSource", "C:\\Users\\user\\Desktop\\GoogleChromeEnterpriseBundle64\\Installers\\")
        chrome_key.add_value("ModifyPath", "MsiExec.exe /X{47FB91DD-98F3-3C87-A963-357B14EAC7C9}")
        chrome_key.add_value("NoModify", 1)
        chrome_key.add_value("Publisher", "Google LLC")
        chrome_key.add_value("EstimatedSize", 113725)
        chrome_key.add_value("UninstallString", "MsiExec.exe /X{47FB91DD-98F3-3C87-A963-357B14EAC7C9}")
        chrome_key.add_value("VersionMajor", 70)
        chrome_key.add_value("VersionMinor", 29)
        chrome_key.add_value("WindowsInstaller", 1)
        chrome_key.add_value("Version", 1176322143)
        chrome_key.add_value("Language", 1033)
        chrome_key.add_value("DisplayName", "Google Chrome")
        hive_hklm.map_key(chrome_name, chrome_key)
    
        addressbook_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\AddressBook"
        addressbook_key = VirtualKey(hive_hklm, addressbook_name)
        addressbook_key.timestamp = datetime(2024, 12, 31, 13, 37, 0, tzinfo=timezone.utc)
        hive_hklm.map_key(addressbook_name, addressbook_key)
    
        msvc_name = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall\\{D5D19E2F-7189-42FE-8103-92CD1FA457C2}"
        msvc_key = VirtualKey(hive_hklm, msvc_name)
        msvc_key.add_value("DisplayName", "Microsoft Visual C++ 2022 X64 Minimum Runtime - 14.36.32532")
        msvc_key.add_value("InstallDate", "20240301")
        msvc_key.add_value("DisplayVersion", "14.36.32532")
        msvc_key.add_value("Publisher", "Microsoft Corporation")
        msvc_key.add_value(
            "InstallSource",
            "C:\\ProgramData\\Package Cache\\{D5D19E2F-7189-42FE-8103-92CD1FA457C2}v14.36.32532\\packages\\vcRuntimeMinimum_amd64\\",  # noqa: E501
        )
        msvc_key.add_value("SystemComponent", 1)
        hive_hklm.map_key(msvc_name, msvc_key)
    
        target_win_users.add_plugin(WindowsApplicationsPlugin)
>       results = sorted(list(target_win_users.applications()), key=lambda r: r.name)

tests/plugins/os/windows/regf/test_applications.py:68: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
dissect/target/plugins/os/windows/regf/applications.py:49: in applications
    yield WindowsApplicationRecord(
dissect/target/helpers/record.py:78: in __call__
    return super().__call__(*args, **kwargs)
.tox/3.9/lib/python3.9/site-packages/flow/record/base.py:595: in __call__
    return self.recordType(*args, **kwargs)
<string>:26: in __init__
    ???
.tox/3.9/lib/python3.9/site-packages/flow/record/base.py:195: in __setattr__
    v = field_type(v)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'flow.record.fieldtypes.datetime'>, args = ('20240301',)
kwargs = {}, arg = '20240301', tstr = '20240301', tzstr = '', tzsearch = ''
tzpos = 0, microsecond_pos = 0

    def __new__(cls, *args, **kwargs):
        if len(args) == 1 and not kwargs:
            arg = args[0]
            if isinstance(arg, bytes_type):
                arg = arg.decode("utf-8")
            if isinstance(arg, string_type):
                # If we are on Python 3.11 or newer, we can use fromisoformat() to parse the string (fast path)
                #
                # Else we need to do some manual parsing to fix some issues with the string format:
                # - Python 3.10 and older do not support nanoseconds in fromisoformat()
                # - Python 3.10 and older do not support Z as timezone info in fromisoformat()
                # - Python 3.10 and older do not support +0200 as timezone info in fromisoformat()
                # - Python 3.10 and older requires "T" between date and time in fromisoformat()
                #
                # There are other incompatibilities, but we don't care about those for now.
                if not PY_311_OR_HIGHER:
                    # Convert Z to +00:00 so that fromisoformat() works correctly on Python 3.10 and older
                    if arg[-1] == "Z":
                        arg = arg[:-1] + "+00:00"
    
                    # Find timezone info after the date part. Possible formats, so we use the longest one:
                    #
                    # YYYYmmdd      length: 8
                    # YYYY-mm-dd    length: 10
                    tstr = arg
                    tzstr = ""
                    tzsearch = arg[10:]
                    if tzpos := tzsearch.find("+") + 1 or tzsearch.find("-") + 1:
                        tzstr = arg[10 + tzpos - 1 :]
                        tstr = arg[: 10 + tzpos - 1]
    
                    # Convert +0200 to +02:00 so that fromisoformat() works correctly on Python 3.10 and older
                    if len(tzstr) == 5 and tzstr[3] != ":":
                        tzstr = tzstr[:3] + ":" + tzstr[3:]
    
                    # Python 3.10 and older do not support nanoseconds in fromisoformat()
                    if microsecond_pos := arg.rfind(".") + 1:
                        microseconds = arg[microsecond_pos:]
                        tstr = arg[: microsecond_pos - 1]
                        if tzpos := (microseconds.find("+") + 1 or microseconds.find("-") + 1):
                            microseconds = microseconds[: tzpos - 1]
                        # Pad microseconds to 6 digits, truncate if longer
                        microseconds = microseconds.ljust(6, "0")[:6]
                        arg = tstr + "." + microseconds + tzstr
                    else:
                        arg = tstr + tzstr
    
>               obj = cls.fromisoformat(arg)
E               ValueError: Invalid isoformat string: '20240301'

.tox/3.9/lib/python3.9/site-packages/flow/record/fieldtypes/__init__.py:323: ValueError

Oct 07 '24 13:10 Horofic

This should now be fixed in ae15c0a.

Oct 23 '24 10:10 JSCU-CNI