Config key annotation to auto generate docs

[pcadabam-zz]

Changes

This pr creates a way to automatically generate documentation for configuration keys used in gobblin. Every configuration key defined should be annotated with @ConfigKey annotation. The tool automatically reads the Javadoc associated with this key. All the config keys and docs are finally written to an HTML file. The tool uses java's doclet APIs to get Javadoc as compiled jars do not have comments or Javadocs. It extends gradle's Javadoc task with custom doclet. The task name is generateConfigDocs defined in gobblin-distribution. The HTML is generated using Apache FreeMarker templates

Supported attributes

A ConfigKey has the following attributes and all of them are optional.

• Doc - Javadoc
• IsRequired - true/false
• Default - Default value for this config key
• Group - The group this key belongs to. Like Retention, Distcp etc.

Files

• ConfigKey is the annotation that needs to be added to every private static String KEY_NAME="key.something" that has to be documented
• ConfigWikiGenerator is the custom doclet class that scans all classes for fields annotated with ConfigKey and writes them to an HTML file (currently /tmp/configDocs.html) -``configDocs.ftl` is the Freemarker html template

Examples

The ConfigWikiGenerator has some example configuration keys that are annotated with ConfigKey. You can see those keys show up in the HTML below.

  /**
   * Root directory where task state files are stored
   */
  @ConfigKey
  public static final String STATE_STORE_ROOT_DIR_KEY = "state.store.dir";
  /**
   * File system URI for file-system-based task store *
   */
  @ConfigKey
  public static final String STATE_STORE_FS_URI_KEY = "state.store.fs.uri";

  /**
   * Enable / disable state store
   */
  @ConfigKey(def = "false")
  public static final String STATE_STORE_ENABLED = "state.store.enabled";

  /**
   * Set this to only log and not delete data
   */
  @ConfigKey(required = false, def = "false", group = "Retention")
  private static final String RETENTION_SIMULATE_KEY = "gobblin.retention.simulate";

  /**
   * Set his to skip trash and delete data
   */
  @ConfigKey(required = false, def = "false", group = "Retention")
  private static final String RETENTION_SKIP_TRASH_KEY = "gobblin.retention.skiptrash";

  /**
   * Name of the destination db
   */
  @ConfigKey(required = true, group = "Hive Conversion")
  private static final String CONVERSION_DESTINATION_DB_NAME_KEY = "gobblin.conversion.destination.dbName";

  /**
   * Name of the destination table
   */
  @ConfigKey(required = true, group = "Hive Conversion")
  private static final String CONVERSION_DESTINATION_TABLE_NAME_KEY = "gobblin.conversion.destination.tableName";

  /**
   * List of tables to be whitelisted of the form [dbpattern.tablepattern1|tablepattern2|...]
   */
  @ConfigKey(required = true, def = "*.*", group = "Hive Distcp")
  private static final String HIVE_DATASET_FINDER_WHITELIST = "hive.dataset.whitelist";

  /**
   * List of tables to blacklisted of the form [dbpattern.tablepattern1|tablepattern2|...]
   */
  @ConfigKey(required = false, def = "-", group = "Hive Distcp")
  private static final String HIVE_DATASET_FINDER_BLACKLIST = "hive.dataset.blacklist";

Generated HTML

The task can be invoked by running ./gradlew gobblin-distribution:generateConfigDocs. Later we expect this task to always run with ./gradlew build

@chavdar @shirshanka @abti please have a look

[abti]

https://issues.apache.org/jira/browse/GOBBLIN-93

Please update your PR title with following prefix: [GOBBLIN-93]