Add Permanent Database Support

[ricekot]

Here is an overview of the changes in this PR (click to expand):

Datanucleus

Datanucleus allows mapping java classes to tables in a database. This makes it convenient to interact with data simply by interacting with java objects.

Files affected

• addOns/addOns.gradle.kts: The datanucleus gradle plugin is added here. Datanucleus modifies Java bytecode to add methods to classes with appropriate annotations. For this, it provides a tool called an "enhancer" that does this. The unofficial gradle plugin used here adds enhancement tasks to all the add-ons. While they can be called manually, annotated classes are automatically enhanced during building which is convenient.

Flyway

Flyway is a tool that allows database migrations. We will be using this programmatically. For example, if the permanent database does not exist in the user's ZAP home directory, it will be created.

Flyway also provides the flexibility to have multiple SQL scripts for various databases. This will be useful when we want to use databases other than HSQLDB like MySQL or PostgreSQL.

Changes to commonlib

• addOns/commonlib/commonlib.gradle.kts:
  • Datanucleus is added as a dependency to the commonlib add-on. This means that any add-on that wants to use the permanent db will have to depend on commonlib. Note that the datanucleus requires its jar files to be present in the classpath so they are added as bundledLibs.
  • Flyway is also added as a dependency.
• addOns/commonlib/src/main/java/org/zaproxy/addon/commonlib/db/Database.java: An abstract class that should be overriden by database implementations. Some common methods like persistEntity are defined here.
• addOns/commonlib/src/main/java/org/zaproxy/addon/commonlib/db/PermanentDatabase.java: Extends the Database class and provides implementation details for the permanent database. These include the DB type (hsql), connection URL, username, password, etc.

Changes to oast

• addOns/oast/oast.gradle.kts:
  • commonlib is added as a dependency
  • the datanucleus gradle plugin is configured.
• addOns/oast/src/main/resources/META-INF/package-hsql.orm: Mapping for datanucleus between the tables in the permanent database and the oast entity classes.
• addOns/oast/src/main/resources/META-INF/persistence.xml: Configuration for datanucleus. Tells datanucleus which classes must be enhanced.
• addOns/oast/src/main/resources/db/migration/V1__Create_tables.sql: Flyway SQL v1 script for creating the oast tables in the permanent db.
• BoastEntity, InteractshEntity: PersistenceCapable annotated classes. Each class maps to a table in the permanent DB and each instance variable of these classes is a column in the tables. They both override OastEntity for convenience.
• OastParam.java, GeneralOastOptionsPanelTab: A new option is added for the user to select whether they want to use the permanent DB or not.
• OastState.java: The events emitted by OAST actions are updated to include the event types (e.g. REGISTERED, POLLED, UNREGISTERED). This allows for a clean implementation where the services are (mostly) oblivious about the permanent DB. ExtensionOast is responsible for actually persisting / retrieving entities from the database. This was done mainly to keep the services independent of the extension.
• If the "Use permanent DB" option is enabled, registered OAST servers from the DB are added to a list in the OAST services when the session is changed. They are polled according to the set interval in their options.

Miscellaneous

Permanent DB Location

The permanent DB is created under zapHome/db/permanent.

SQL Workbench Screenshots

Tables

BOAST

Column Definitions

Example Data

Interactsh

Column Definitions

Example Data

What's pending?

• Testing that everything works as expected.
• Are all (or most) DB best practices being followed in this implementation?

What's next?

• Persist Interactsh payloads.
• Adding an option to pick the "age" of servers that are polled or deleted.
• Alternative DB implementations like MySQL or PostgreSQL.
• Integration tests with DbUnit.

Signed-off-by: ricekot [email protected]

[ricekot]

Have updated the PR description with an overview of the changes in this PR. Also rebased on top of latest changes. Keeping the WIP for now because there are certain behaviours I want to test.

[ricekot]

This PR is ready for reviews :)

[ricekot]

Rebased and reworked to use the database add-on instead of commonlib.

[thc202]

Thank you!