[MD] Logging and Auditing

[zhongnansu]

Task breakdown

• [x] Research notes
• [ ] TODO

Research Notes

Some questions we need to answer

• What's the different between logging and auditing?
  • Auditing is used to answer the question "Who did what?" and possibly why.
  • Logging is more focussed on what's happening.
• How's the current logging and auditing of OSD visualization/search/dashboards?
  • Logging: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/src/core/server/logging/README.md
  • Auditing: https://opensearch.org/docs/latest/security-plugin/audit-logs/index/
  • See below section for detail
• What's the requirement for logging and auditing for data source?
  • which user
  • which datasource
  • what query
  • at what time
  • Error

Logging

Data source logging will log datasource, query, time, and error, with correct logging setting and client settings in osd.yml

Similar to what we currently have with default single opensearch cluster. It makes use of the event emitter provided by opensearch-js client lib, that hook into internal events, such as request and response. Doc Reference

Current logging

https://github.com/opensearch-project/OpenSearch-Dashboards/blob/5fb4143b11a0a0292ca5cc96391d192a7ef643b3/src/core/server/opensearch/client/configure_client.ts#L51-L72

Auditing

Security Plugin Audit Log feature

• Auditing in Opensearch is achieved by opensearch security plugin audit log. By correct configuration, it can monitor any REST/transport API request with info of user, request params/body, and timestamp. See below for an example of audit log generated by opening a visualization from OSD.
• As for CRUD operation around datasource and credential manager, it will also be logged because they are all saved objects.
• There's some limitation that we can't get all info in one audit log line, because by connecting to datasource, it sends request to external opensearch endpoints. Currently security plugin only logs API request to the default single cluster. We don't want to make changes to security plugin at this stage. But it can be an enhancement in the future.
• Summary: We'll consider other option - OSD audit service, for data source logging and auditing.

[Proposed Solution] OSD Audit Service + Logging service

• Core has an audit trail service, Plugins can get scoped Auditor from the core service to add events to introspect. Plugin can register some audit trail clients that implements the Audit interfaces in core, and make use of the logging service to write output to file by configuring "logging -> custom appender". The audit service can get the authenticated user info, then we can enrich that with datasouce, timestamp, query, error to create single audit log line, and saved to some file on disk
• For detail, it's interesting to take a look at how Kibana x-pack -> audit_trail plugin is using core - audit trail service in this PR. Similarly, we can integrate with core - audit_service from data_source plugin.

core - audit service

// @public
export interface AuditableEvent {
    // (undocumented)
    message: string;
    // (undocumented)
    type: string;
}
// @public
export interface Auditor {
    add(event: AuditableEvent): void;
    withAuditScope(name: string): void;
}
// @public
export interface AuditorFactory {
    // (undocumented)
    asScoped(request: OSDRequest): Auditor;
}
// Warning: (ae-missing-release-tag) "AuditTrailSetup" is exported by the package, but it is missing a release tag (@alpha, @beta, @public, or @internal)
//
// @public (undocumented)
export interface AuditTrailSetup {
    register(auditor: AuditorFactory): void;
}

data source plugin -> audit trail client

export class AuditTrailClient implements Auditor {
  private scope?: string;
  constructor(
    private readonly request: OSDRequest,
    private readonly event$: Subject<AuditEvent>,
    private readonly deps: Deps
  ) {}

  public withAuditScope(name: string) {
    if (this.scope !== undefined) {
      throw new Error(`Audit scope is already set to: ${this.scope}`);
    }
    this.scope = name;
  }

  public add(event: AuditableEvent) {
    const user = this.deps.getCurrentUser(this.request);
    // doesn't use getSpace since it's async operation calling ES
    const spaceId = this.deps.getSpaceId ? this.deps.getSpaceId(this.request) : undefined;

    this.event$.next({
      message: event.message,
      type: event.type,
      user: user?.username,
      space: spaceId,
      scope: this.scope,

data source plugin -> plugin.ts

const configSchema = schema.object({
  enabled: schema.boolean({ defaultValue: false }),
  appender: schema.maybe(coreConfig.logging.appenders),
  logger: schema.object({
    enabled: schema.boolean({ defaultValue: false }),
  }),
});

export type AuditTrailConfigType = TypeOf<typeof configSchema>;

export class DataSource implements Plugin {
  private readonly logger: Logger;
  private readonly config$: Observable<AuditTrailConfigType>;
  private readonly event$ = new Subject<AuditEvent>();

  constructor(private readonly context: PluginInitializerContext) {
    this.logger = this.context.logger.get();
    this.config$ = this.context.config.create();
  }

  public setup(core: CoreSetup, deps: DepsSetup) {
    const depsApi = {
      getCurrentUser:
    };

    core.auditTrail.register({
      asScoped: (request: OSDRequest) => {
        return new AuditTrailClient(request, this.event$, depsApi);
      },
    });

    core.logging.configure(
      this.config$.pipe<LoggerContextConfigInput>(
        map((config) => ({
          appenders: {
            auditTrailAppender: this.getAppender(config),
          },
          loggers: [
            {
              // plugins.auditTrail prepended automatically
              context: '',
              level: config.logger.enabled ? 'debug' : 'off',
              appenders: ['auditTrailAppender'],
            },
          ],
        }))
      )
    );
  }

[zhongnansu]

Reference

1. Kibana Audit Service

https://github.com/elastic/kibana/issues/52125 Original PR to initialize Audit service: https://github.com/elastic/kibana/pull/69278

2. Grafana Audit logs for datasource management

Data sources management

Action	Distinguishing fields
Create datasource	{"action": "create", "resources": [{"type": "datasource"}]}
Update datasource	{"action": "update", "resources": [{"type": "datasource"}]}
Delete datasource	{"action": "delete", "resources": [{"type": "datasource"}]}
Enable permissions for datasource	{"action": "enable-permissions", "resources": [{"type": "datasource"}]}
Disable permissions for datasource	{"action": "disable-permissions", "resources": [{"type": "datasource"}]}
Grant datasource permission to role, team, or user	{"action": "create", "resources": [{"type": "datasource"}, {"type": "dspermission"}]}*
Remove datasource permission	{"action": "delete", "resources": [{"type": "datasource"}, {"type": "dspermission"}]}
Enable caching for datasource	{"action": "enable-cache", "resources": [{"type": "datasource"}]}
Disable caching for datasource	{"action": "disable-cache", "resources": [{"type": "datasource"}]}
Update datasource caching configuration	{"action": "update", "resources": [{"type": "datasource"}]}

* resources may also contain a third item with "type": set to "user" or "team".

3. OpenSearch security plugin audit log example

{
    "audit_cluster_name": "841677925608:new-m6",
    "audit_node_name": "8b168deaa71e1e5da322cc20de7b812b",
    "audit_request_initiating_user": "admin",
    "audit_rest_request_method": "POST",
    "audit_category": "AUTHENTICATED",
    "audit_request_origin": "REST",
    "audit_request_body": "{\"docs\":[{\"_id\":\"visualization:2edf78b0-5395-11e8-99bf-1ba7b1bdaa61\",\"_index\":\".kibana\"}]}",
    "audit_node_id": "xfijqJVHS7SOD9KZVBYdcQ",
    "audit_request_layer": "REST",
    "audit_rest_request_path": "/_mget",
    "@timestamp": "2022-08-03T07:14:18.431+00:00",
    "audit_request_effective_user_is_admin": false,
    "audit_format_version": 4,
    "audit_request_remote_address": "24.19.168.206",
    "audit_rest_request_headers": {
        "Connection": [
            "keep-alive"
        ],
        "User-Agent": [
            "elasticsearch-js/7.10.0-rc.1 (linux 5.4.117-58.216.amzn2.aarch64-arm64; Node.js v10.24.1)"
        ],
        "x-opaque-id": [
            "30c9fa0c-c416-44d8-841a-11e31dccb235"
        ],
        "Host": [
            "localhost:9200"
        ],
        "x-opensearch-product-origin": [
            "opensearch-dashboards"
        ],
        "Content-Length": [
            "90"
        ],
        "Content-Type": [
            "application/json"
        ]
    },
    "audit_request_effective_user": "admin"
}

[zhongnansu]

@zengyan-amazon @seraphjiang Any thoughts?