[DeepDive] Reverse engineering, hardware hacking, firmware extraction, javascript deobfuscation, etc

[0xdevalias]

Reverse Engineering

• https://ghidra-sre.org/
  • https://github.com/NationalSecurityAgency/ghidra
  • http://ghidra-server.org/
• https://binary.ninja/
  • https://binary.ninja/demo/
• https://www.hex-rays.com/products/ida/
  • https://www.hex-rays.com/products/ida/support/download_freeware.shtml

Hardware Hacking / Firmware Hacking

• https://github.com/ReFirmLabs/binwalk
  • https://www.refirmlabs.com/binwalk-pro/
• https://www.hackster.io/rayburne/avr-firmware-duplicator-115d8f
• http://wiki.infosectcbr.com.au/index.php/Busside#Connecting_the_GPIO_pins
• https://github.com/sigmike/avrdude
  • https://learn.adafruit.com/usbtinyisp/avrdude
    • https://learn.adafruit.com/usbtinyisp

Javascript Reverse Engineering / Deobfuscation

• http://www.nice2predict.org/
  • https://github.com/eth-sri/UnuglifyJS
  • https://github.com/eth-sri/Nice2Predict
• http://www.jsnice.org/
  • https://github.com/brettlangdon/jsnice
• https://beautifier.io/
  • https://github.com/beautify-web/js-beautify
• https://lelinhtinh.github.io/de4js/
  • https://github.com/lelinhtinh/de4js
• https://www.unminify2.com/
• https://cloudfour.com/thinks/module_bunders_part_1/

  • Unpacking Module Bundlers Part 1: What is a module?

• https://www.joesecurity.org/blog/4297261482537891261

  • Generic Unpacking of Javascript with Microsoft AMSI

Abstract Syntax Tree's (AST's)

• https://github.com/benjamn/recast

  • JavaScript syntax tree transformer, nondestructive pretty-printer, and automatic source map generator

  • https://github.com/benjamn/recast#using-a-different-parser

    • By default, Recast uses the Esprima JavaScript parser when you call recast.parse(code). While Esprima supports almost all modern ECMAScript syntax, you may want to use a different parser to enable TypeScript or Flow syntax, or just because you want to match other compilation tools you might be using.

    • To take some of the guesswork out of configuring common parsers, Recast provides several preconfigured parsers

      • https://github.com/benjamn/recast/tree/master/parsers

        • acorn, babel, babylon, esprima, flow, typescript

  • https://github.com/benjamn/recast#source-maps

    • One of the coolest consequences of tracking and reusing original source code during reprinting is that it's pretty easy to generate a high-resolution mapping between the original code and the generated code—completely automatically!

    • All you have to think about is how to manipulate the syntax tree, and Recast will give you a source map in exchange for specifying the names of your source file(s) and the desired name of the map

    • Note that you are free to mix and match syntax trees parsed from different source files, and the resulting source map will automatically keep track of the separate file origins for you.

  • https://github.com/benjamn/ast-types
• https://github.com/github/semantic

  • Parsing, analyzing, and comparing source code across many languages

  • https://github.com/github/semantic#language-support
  • https://github.com/tree-sitter/tree-sitter

    • An incremental parsing system for programming tools

    • https://tree-sitter.github.io/tree-sitter/

      • Tree-sitter is a parser generator tool and an incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited.

      • https://tree-sitter.github.io/tree-sitter/#available-parsers
    • https://github.com/tree-sitter/node-tree-sitter
    • https://github.com/tree-sitter/tree-sitter-javascript
• https://github.com/estree/estree

  • Once upon a time, an unsuspecting Mozilla engineer created an API in Firefox that exposed the SpiderMonkey engine's JavaScript parser as a JavaScript API. Said engineer documented the format it produced, and this format caught on as a lingua franca for tools that manipulate JavaScript source code.

• https://esprima.org/

  • Esprima is a high performance, standard-compliant ECMAScript parser written in ECMAScript (also popularly known as JavaScript).

  • Once the full syntax tree is obtained, various static code analysis can be applied to give an insight to the code: syntax visualization, code validation, editing autocomplete (with type inferencing) and many others.

  • Regenerating the code from the syntax tree permits a few different types of code transformation, from a simple rewriting (with specific formatting) to a more complicated minification.

  • https://esprima.readthedocs.io/en/latest/
    • https://esprima.readthedocs.io/en/latest/getting-started.html#using-node-js-to-play-with-esprima
  • https://github.com/jquery/esprima
• https://github.com/acornjs/acorn

  • A small, fast, JavaScript-based JavaScript parser

• https://github.com/sourcegraph/sourcegraph
  • https://github.com/sourcegraph/astannotate.js

    • astannotate is a node.js package that supports querying a JavaScript AST annotated with comments. It is useful for testing libraries that perform JavaScript source introspection because it lets you define test expectations inline in source code instead of having to put them in separate test files.

  • https://github.com/sourcegraph/acorn-walkall

    • acorn-walkall provides a custom walker for Marijn Haverbeke's acorn JavaScript parser that traverses all AST nodes.

  • https://github.com/sourcegraph/javascript-idents

    • Walks a JavaScript AST and collects Identifier nodes

  • https://github.com/sourcegraph/defnode.js

    • defnode is a node.js package that maps JavaScript Identifier AST nodes to/from their corresponding definition nodes.

  • https://github.com/sourcegraph/node-idast

    • Walks a JavaScript AST (parsed by acorn), giving each AST node a unique and meaningful ID.

  • https://docs.sourcegraph.com/extensions
• https://github.com/babel/babel/tree/master/packages#core-packages

  • @babel/core is the Babel compiler itself; it exposes the babel.transform method, where transformedCode = transform(src).code.

  • The compiler can be broken down into 3 parts:

    • The parser: @babel/parser

    • The transformer[s]: All the plugins/presets

      • These all use @babel/traverse to traverse through the AST

    • The generator: @babel/generator

    • The flow goes like this: input string -> @babel/parser parser -> AST -> transformer[s] -> AST -> @babel/generator -> output string

  • https://babeljs.io/docs/en/babel-parser
    • https://github.com/babel/babel/tree/master/packages/babel-parser
  • https://github.com/babel/babel/tree/master/packages/babel-types

    • Babel Types is a Lodash-esque utility library for AST nodes

• http://lisperator.net/uglifyjs/ast
  • http://lisperator.net/uglifyjs/transform
• https://github.com/prettier/prettier/
  • https://github.com/prettier/prettier/blob/master/package.json
    • https://github.com/babel/babel/tree/master/packages/babel-parser
      • https://babeljs.io/docs/en/next/babel-parser
    • https://github.com/babel/babel/tree/master/packages/babel-code-frame

      • a standalone package used to generate errors that print the source code and point to error locations

      • https://babeljs.io/docs/en/next/babel-code-frame.html
    • https://github.com/facebook/flow/tree/master/packages/flow-parser
    • etc
• https://ternjs.net/

  • Tern is a stand-alone code-analysis engine for JavaScript. It is intended to be used with a code editor plugin to enhance the editor's support for intelligent JavaScript editing.

  • https://github.com/ternjs/tern
• https://astexplorer.net/
• https://github.com/v8/v8/blob/master/src/ast/ast.h
• https://github.com/danbev/learning-v8#abstract-syntax-tree-ast
• https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/Parser_API
• https://medium.com/@drag13dev/https-medium-com-drag13dev-how-to-get-javascript-bytecode-from-nodejs-7bd396805d30

Questions, Blogs, Etc

• https://stackoverflow.com/questions/29752326/how-to-get-the-ast-of-the-current-chrome-tab-from-an-extension

  • No, Chrome's V8 does not expose any internals as part of Extension APIs, or, for that reason, remote debugger protocol.

• https://stackoverflow.com/questions/9451067/access-the-abstract-syntax-tree-of-v8-engine

  • This is pretty old but maybe the answer helps someone stumbling upon this. The answer is yes, assuming you are willing to modify V8 and compile your own version of it.

• https://blog.sessionstack.com/how-javascript-works-parsing-abstract-syntax-trees-asts-5-tips-on-how-to-minimize-parse-time-abfcf7e8a0c8
• • https://stackoverflow.com/questions/5067532/editing-in-the-chrome-debugger

DevTools / Debugger

• https://github.com/WICG/devtools-protocol
• https://chromedevtools.github.io/devtools-protocol/
• https://github.com/ChromeDevTools/devtools-protocol

Source Maps

• https://github.com/mozilla/source-map
  • https://github.com/mozilla/source-map#generating-a-source-map

Modules, bundlers, packers, etc

• https://webpack.js.org/
• https://rollupjs.org/
• https://parceljs.org/
• https://github.com/umdjs/umd

Redux DevTools

• It would be cool to figure out how we could consistently inject redux devtools into arbitrary production builds..
  • https://github.com/zalmoxisus/redux-devtools-extension#usage
  • https://github.com/zalmoxisus/redux-devtools-extension/blob/master/docs/API/Arguments.md