when analysis a firmware, crash with ValueError: A string literal cannot contain NUL (0x00) characters.

[emorasoul]

when analysis a firmware, crash with Under info. looks like some bin file contain NUL (0x00) characters.

Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/icsl/FACT_core/src/helperFunctions/process.py", line 56, in run raise exception File "/home/icsl/FACT_core/src/helperFunctions/process.py", line 51, in run Process.run(self) File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/icsl/FACT_core/src/scheduler/analysis.py", line 426, in _result_collector self.post_analysis(fw.uid, plugin_name, fw.processed_analysis[plugin_name]) File "/home/icsl/FACT_core/src/storage/db_interface_backend.py", line 69, in add_analysis self.insert_analysis(uid, plugin, analysis_dict) File "/home/icsl/FACT_core/src/storage/db_interface_backend.py", line 99, in insert_analysis session.add(analysis) File "/usr/lib/python3.8/contextlib.py", line 120, in exit next(self.gen) File "/home/icsl/FACT_core/src/storage/db_interface_base.py", line 52, in get_read_write_session session.commit() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1435, in commit self._transaction.commit(_to_root=self.future) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit self._prepare_impl() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl self.session.flush() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3367, in flush self._flush(objects) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3507, in flush transaction.rollback(capture_exception=True) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in exit compat.raise( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise raise exception File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3467, in _flush flush_context.execute() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute rec.execute(self) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute util.preloaded.orm_persistence.save_obj( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj _emit_insert_statements( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements c = connection._execute_20( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 return meth(self, args_10style, kwargs_10style, execution_options) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection return connection._execute_clauseelement( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement ret = self._execute_context( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context self.handle_dbapi_exception( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2047, in handle_dbapi_exception util.raise(exc_info[1], with_traceback=exc_info[2]) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise raise exception File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context self.dialect.do_execute( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute cursor.execute(statement, parameters) ValueError: A string literal cannot contain NUL (0x00) characters.

[dorpvom]

Hi emorasoul,

I've encountered a similar error and traced it to the cpu_architecture plugin. I'll attach a patch that you can check in the short term and we'll create a bugfix PR soon.

[dorpvom]

diff --git a/src/plugins/analysis/architecture_detection/internal/dt.py b/src/plugins/analysis/architecture_detection/internal/dt.py
index 9d9a0fa4..e823daa5 100644
--- a/src/plugins/analysis/architecture_detection/internal/dt.py
+++ b/src/plugins/analysis/architecture_detection/internal/dt.py
@@ -81,7 +81,7 @@ def construct_result(file_object):
 
         result.update(
             {
-                compatible_entry: 'DeviceTree',
+                compatible_entry.replace('\x00', '\\x00'): 'DeviceTree',
             },
         )
 
diff --git a/src/plugins/analysis/architecture_detection/internal/elf.py b/src/plugins/analysis/architecture_detection/internal/elf.py
index 8ab5f85b..3e837869 100644
--- a/src/plugins/analysis/architecture_detection/internal/elf.py
+++ b/src/plugins/analysis/architecture_detection/internal/elf.py
@@ -39,8 +39,11 @@ def _get_arm_isa(elffile):
 
     result = ''
 
-    # Some how the section does not appear in arm64 binarys
+    # Somehow the section does not appear in arm64 binarys
     sec = elffile.get_section_by_name('.ARM.attributes')
+    if not sec:
+        return result
+
     for sub_sec in sec.iter_subsections():
         for sub_sub_sec in sub_sec.iter_subsubsections():
             for attribute in sub_sub_sec.iter_attributes():
diff --git a/src/storage/db_interface_backend.py b/src/storage/db_interface_backend.py
index c7ffdf4e..bebb25e3 100644
--- a/src/storage/db_interface_backend.py
+++ b/src/storage/db_interface_backend.py
@@ -72,6 +72,9 @@ class BackendDbInterface(DbInterfaceCommon, ReadWriteDbInterface):
                               f' it is not JSON-serializable: {uid}\n{analysis_dict}')
         except DbInterfaceError as error:
             logging.error(f'Could not store analysis result: {str(error)}')
+        except ValueError:
+            logging.error(f'Bad value in analysis result of {plugin} on {uid}.\n{analysis_dict}')
+            raise
 
     def analysis_exists(self, uid: str, plugin: str) -> bool:
         with self.get_read_only_session() as session:

[emorasoul]

thanks，it's OK now.

[emorasoul]

patch should commit to master

[dorpvom]

Yes, I agree. We'll keep this open until patch is merged.

[jstucke]

@emorasoul I just merged #861 and it should hopefully fix the problems. Could you please confirm this (i.e. by pulling the master branch and reanalyzing the problematic firmware)?

[emorasoul]

I pull the new code，but problem still happen. {'architectures': {'arm,cortex-a76arm,armv8': 'DeviceTree'}, 'summary': ['arm,cortex-a76\x00arm,armv8'], 'analysis_date': 1664247561.237713, 'plugin_version': '0.4.0'} Process ExceptionSafeProcess-61: Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/home/icsl/FACT_core/src/helperFunctions/process.py", line 56, in run raise exception File "/home/icsl/FACT_core/src/helperFunctions/process.py", line 51, in run Process.run(self) File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/icsl/FACT_core/src/scheduler/analysis.py", line 426, in _result_collector self.post_analysis(fw.uid, plugin_name, fw.processed_analysis[plugin_name]) File "/home/icsl/FACT_core/src/storage/db_interface_backend.py", line 69, in add_analysis self.insert_analysis(uid, plugin, analysis_dict) File "/home/icsl/FACT_core/src/storage/db_interface_backend.py", line 102, in insert_analysis session.add(analysis) File "/usr/lib/python3.8/contextlib.py", line 120, in exit next(self.gen) File "/home/icsl/FACT_core/src/storage/db_interface_base.py", line 52, in get_read_write_session session.commit() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1435, in commit self._transaction.commit(_to_root=self.future) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit self._prepare_impl() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl self.session.flush() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3367, in flush self._flush(objects) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3507, in flush transaction.rollback(capture_exception=True) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in exit compat.raise( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise raise exception File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3467, in _flush flush_context.execute() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute rec.execute(self) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute util.preloaded.orm_persistence.save_obj( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj _emit_insert_statements( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements c = connection._execute_20( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 return meth(self, args_10style, kwargs_10style, execution_options) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection return connection._execute_clauseelement( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement ret = self._execute_context( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context self.handle_dbapi_exception( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2047, in handle_dbapi_exception util.raise(exc_info[1], with_traceback=exc_info[2]) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise raise exception File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context self.dialect.do_execute( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute cursor.execute(statement, parameters) ValueError: A string literal cannot contain NUL (0x00) characters. [pid: 121627|app: 0|req: 48/243] 70.254.77.155 () {34 vars in 668 bytes} [Tue Sep 27 10:59:23 2022] GET /ajax/system_health => generated 2833 bytes in 18 msecs (HTTP/1.1 200) 3 headers in 87 bytes (1 switches on core 0) [2022-09-27 10:59:23][process][ERROR]: Exception in Scheduler process: Traceback (most recent call last): File "/home/icsl/FACT_core/src/helperFunctions/process.py", line 51, in run Process.run(self) File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/icsl/FACT_core/src/scheduler/analysis.py", line 426, in _result_collector self.post_analysis(fw.uid, plugin_name, fw.processed_analysis[plugin_name]) File "/home/icsl/FACT_core/src/storage/db_interface_backend.py", line 69, in add_analysis self.insert_analysis(uid, plugin, analysis_dict) File "/home/icsl/FACT_core/src/storage/db_interface_backend.py", line 102, in insert_analysis session.add(analysis) File "/usr/lib/python3.8/contextlib.py", line 120, in exit next(self.gen) File "/home/icsl/FACT_core/src/storage/db_interface_base.py", line 52, in get_read_write_session session.commit() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1435, in commit self._transaction.commit(_to_root=self.future) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit self._prepare_impl() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl self.session.flush() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3367, in flush self._flush(objects) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3507, in flush transaction.rollback(capture_exception=True) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in exit compat.raise( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise raise exception File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3467, in _flush flush_context.execute() File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute rec.execute(self) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute util.preloaded.orm_persistence.save_obj( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj _emit_insert_statements( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements c = connection._execute_20( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 return meth(self, args_10style, kwargs_10style, execution_options) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection return connection._execute_clauseelement( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement ret = self._execute_context( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context self.handle_dbapi_exception( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2047, in handle_dbapi_exception util.raise(exc_info[1], with_traceback=exc_info[2]) File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 207, in raise raise exception File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context self.dialect.do_execute( File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute cursor.execute(statement, parameters) ValueError: A string literal cannot contain NUL (0x00) characters.

[2022-09-27 10:59:23][fact_base][INFO]: Shutting down components of FACT Backend [2022-09-27 10:59:24][back_end_binding][WARNING]: InterCom down [2022-09-27 10:59:24][comparison_scheduler][INFO]: Comparison Scheduler offline

[emorasoul]

I found a patch not submit in master. I am testing after patch that. diff --git a/src/plugins/analysis/architecture_detection/internal/dt.py b/src/plugins/analysis/architecture_detection/internal/dt.py index 9d9a0fa4..e823daa5 100644 --- a/src/plugins/analysis/architecture_detection/internal/dt.py +++ b/src/plugins/analysis/architecture_detection/internal/dt.py @@ -81,7 +81,7 @@ def construct_result(file_object):

     result.update(
         {

•            compatible_entry: 'DeviceTree',
  

•            compatible_entry.replace('\x00', '\\x00'): 'DeviceTree',
         },
     )
  

[dorpvom]

Hi emorasoul, would it be possible for you to provide @jstucke the firmware that leads to the crash? Though I encountered similar issues I did not directly find one of the affected images and he could not reproduce during patching.

[emorasoul]

I found the problem is 'summary': ['arm,cortex-a76\x00arm,armv8']，when sanitize skip the meta ‘summary’.

I am sorry i can't provide the firmware,it violating company regulation.

[2022-09-27 10:59:21][db_interface_backend][ERROR]: Bad value in analysis result of cpu_architecture on 5c69f7721aab627c0f65007f7df17f417c0f5c2e17309607ee4a9fe85ac05003_267392: A string literal cannot contain NUL (0x00) characters. {'architectures': {'arm,cortex-a76arm,armv8': 'DeviceTree'}, 'summary': ['arm,cortex-a76\x00arm,armv8'], 'analysis_date': 1664247561.237713, 'plugin_version': '0.4.0'}

def get_analysis_without_meta(analysis_data: dict) -> dict: analysis_without_meta = { key: value for key, value in analysis_data.items() if key not in META_KEYS } sanitize(analysis_without_meta) return analysis_without_meta

[emorasoul]

I subjuct sanitize action should be doing in summary、plugin_version...but not only get_analysis_without_meta() func，to prevent the NUL exception.

def create_analysis_entries(file_object: FileObject, fo_backref: FileObjectEntry) -> List[AnalysisEntry]: return [ AnalysisEntry( uid=file_object.uid, plugin=plugin_name, plugin_version=analysis_data.get('plugin_version'), system_version=analysis_data.get('system_version'), analysis_date=analysis_data.get('analysis_date'), summary=analysis_data.get('summary'), tags=analysis_data.get('tags'), result=get_analysis_without_meta(analysis_data), file_object=fo_backref, ) for plugin_name, analysis_data in file_object.processed_analysis.items() ]

[jstucke]

#861 only fixed NULL bytes in result keys but it seems they can also appear in summaries. #862 should fix those cases

[jstucke]

Since #862 is merged now and has hopefully fixed all remaining problems, could you please retry it and report your findings?

[emorasoul]

Since #862 is merged now and has hopefully fixed all remaining problems, could you please retry it and report your findings? It’s OK now~!Thanks!