elf support

[williballenthin]

[mr-tz]

Test candidates: A47FD9031D7C45AB61F9E8C68458BA81, 1D081F79D698DBDE8FBD8B6C857ECCA8, and 164EF16A5257311859822765778383D0 (all from the same malware family).

[mr-tz]

here is a good example of an obfuscated elf binary: a0cd554c35dee3fed3d1607dc18debd1296faaee29b5bd77ff83ab6956a6f9d6 https://www.intezer.com/blog/research/new-linux-threat-symbiote/

• https://github.com/mandiant/flare-floss/issues/276#issuecomment-1168244301

[symbolicvoid]

I was wondering about this issue. @mr-tz told me a little bit about this on an e-mail and pointed me to the capa PR that added ELF support, and it seems easy. Majority of the changes there seem to be related to functionality of capa, so is there something in FLOSS that works for PE but might not work for ELF files? Or could this be as simple as just allowing FLOSS to analyze ELF files?

[williballenthin]

i think enabling the support via vivisect is probably pretty easy, as you've found. the bigger concern is being able to prove its working, which involves finding enough test samples to demonstrate that FLOSS does something useful. if we can find 5-10 samples/techniques, then i'd be happy to merge this support.

[mr-tz]

The testfiles repo should contain (or we can easily add them) the test samples generated from source.

[symbolicvoid]

i think enabling the support via vivisect is probably pretty easy, as you've found. the bigger concern is being able to prove its working, which involves finding enough test samples to demonstrate that FLOSS does something useful. if we can find 5-10 samples/techniques, then i'd be happy to merge this support.

By that, do you mean demonstrating ELF files have obfuscated strings, or demonstrating that FLOSS can extract them from ELF files? Or do we just want more ELF malware samples to test for obfuscated strings to prove that ELF support is actually useful?

The testfiles repo should contain (or we can easily add them) the test samples generated from source.

Do you mean we could use the test samples in the testfiles repo for this as well?

[mr-tz]

If we can prove this works for the testfiles (e.g. https://github.com/mandiant/flare-floss-testfiles/tree/master/src/decode-base64/bin) I'd be pretty happy already.

[williballenthin]

@c-urly this is the issue that you're hoping to work on. please review this thread and propose a plan 😁

the other threads you commented on were specifically for (1) QUANTUMSTRAND, an experimental rendering mode, and (2) Go language string support.

this thread is about deobfuscating strings found in ELF files.

[c-urly]

Thanks, @williballenthin will go through it and will ask any questions i will have on this thread.

[c-urly]

@williballenthin So here is what I have understood so far, Please point me out if I am wrong somewhere.

1. We currently only support PE files as SUPPORTED_FILE_MAGIC only has PE header.
2. I think first I will have to check that by adding ELF in SUPPORTED_FILE _MAGIC macro, the code deobfuscates the elf malware or not, if there is any problem I will have to dig deeper into each string deobfuscation And I will have to compare floss performance with strings (let me know if there is any other way to check floss' performance).
3. I will also have to add code to load the elf file in the Vivisect tool, I am still checking this.

[c-urly]

Does this look fine? Curently I have made these changes and tested on some flare elf testfiles.

diff --git a/floss/const.py b/floss/const.py index 3369880..24140f7 100644 --- a/floss/const.py +++ b/floss/const.py @@ -3,7 +3,7 @@ KILOBYTE = 1024 MEGABYTE = 1024 * KILOBYTE MAX_FILE_SIZE = 16 * MEGABYTE -SUPPORTED_FILE_MAGIC = {b"MZ"} +SUPPORTED_FILE_MAGIC = {b"MZ",b"\x7fELF"} MIN_STRING_LENGTH = 4 MAX_STRING_LENGTH = 2048

diff --git a/floss/main.py b/floss/main.py index a49edea..5157ff3 100644 --- a/floss/main.py +++ b/floss/main.py @@ -356,10 +356,12 @@ def is_supported_file_type(sample_file_path: Path): :return: True if file type is supported, False otherwise """ with sample_file_path.open("rb") as f:

•    magic = f.read(2)
  

•    magic = f.read(4)
  

• print(magic[:2]) if magic in SUPPORTED_FILE_MAGIC: return True
• elif magic[:2] in SUPPORTED_FILE_MAGIC:

•    return True
  

  else: return False

$ floss test-decode-split-stackstrings INFO: floss: extracting static strings .. analyzing program finding decoding function features: 100%|██████████████████████████████████████████████████████████████| 15/15 [00:00<00:00, 488.13 functions/s, skipped 0 library functions] INFO: floss.stackstrings: extracting stackstrings from 11 functions INFO: floss.results: goodbye INFO: floss.results: goodbye world INFO: floss.results: goodbye moon extracting stackstrings: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:00<00:00, 97.40 functions/s] INFO: floss.tightstrings: extracting tightstrings from 0 functions... extracting tightstrings: 0 functions [00:00, ? functions/s] INFO: floss.string_decoder: decoding strings emulating function 0x400418 (call 1/1): 100%|████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:01<00:00, 10.08 functions/s] INFO: floss: finished execution after 14.08 seconds INFO: floss: rendering results

FLARE FLOSS RESULTS (version 3.0.1)

+------------------------+------------------------------------------------------------------------------------+ | file path | test-decode-split-stackstrings | | identified language | unknown | | extracted strings | | | static strings | 79 (1043 characters) | | language strings | 0 ( 0 characters) | | stack strings | 3 | | tight strings | 0 | | decoded strings | 0 | +------------------------+------------------------------------------------------------------------------------+

─────────────────────────── FLOSS STATIC STRINGS (79) ───────────────────────────

+----------------------------------+ | FLOSS STATIC STRINGS: ASCII (78) | +----------------------------------+

/lib64/ld-linux-x86-64.so.2 libc.so.6 printf rand _libc_start_main gmon_start GLIBC_2.2.5 UH-H UH-H []A\A]A^A ;*3$" GCC: (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4 Ubuntu clang version 3.4-1ubuntu3 (tags/RELEASE_34/final) (based on LLVM 3.4) .symtab .strtab .shstrtab .interp .note.ABI-tag .note.gnu.build-id .gnu.hash .dynsym .dynstr .gnu.version .gnu.version_r .rela.dyn .rela.plt .init .text .fini .rodata .eh_frame_hdr .eh_frame .init_array .fini_array .jcr .dynamic .got .got.plt .data .bss .comment crtstuff.c JCR_LIST deregister_tm_clones register_tm_clones __do_global_dtors_aux completed.6973 __do_global_dtors_aux_fini_array_entry frame_dummy __frame_dummy_init_array_entry test-decode-split-stackstrings.c FRAME_END JCR_END __init_array_end _DYNAMIC __init_array_start GLOBAL_OFFSET_TABLE __libc_csu_fini _ITM_deregisterTMCloneTable data_start _edata _fini printf@@GLIBC_2.2.5 __libc_start_main@@GLIBC_2.2.5 __data_start gmon_start __dso_handle _IO_stdin_used __libc_csu_init _end _start __bss_start main _Jv_RegisterClasses TMC_END _ITM_registerTMCloneTable _init rand@@GLIBC_2.2.5

+------------------------------------+ | FLOSS STATIC STRINGS: UTF-16LE (1) | +------------------------------------+

@8\t@

───────────────────────── FLOSS STACK STRINGS (3) ─────────────────────────

goodbye goodbye world goodbye moon

───────────────────────── FLOSS TIGHT STRINGS (0) ─────────────────────────

─────────────────────────── FLOSS DECODED STRINGS (0) ───────────────────────────

[c-urly]

Hi @williballenthin , @mr-tz , How to run all the test cases in flare-floss-testfiles with pytest?

[williballenthin]

pytest tests/

you can also check out the CI config here for a more detailed walkthrough: https://github.com/mandiant/flare-floss/blob/master/.github/workflows/tests.yml

[c-urly]

I have not checked this on actual malware. I would appreciate any help from you all to download the given malware. All test files under flare-floss-test files/src/ passed for Linux and Windows.

[mr-tz]

If the test files pass that's a great start. Can you clarify what you mean for the actual malware samples (i.e. how to download the samples mentioned in https://github.com/mandiant/flare-floss/issues/40#issuecomment-286215095)?

[c-urly]

Yes, how to download sample mentioned in the comments.

[mr-tz]

Hm, it seems the ones I've mentioned are private-only samples. Let's focus on the test samples generated from source for now.

[c-urly]

@mr-tz Pl check the PR in the test repo, https://github.com/mandiant/flare-floss-testfiles/pull/17