log injection - dd.trace_id, dd.span_id, dd.env, dd.service, dd.version are not getting injected to tomcat access logs

[umarkhan17]

We are routing access logs to datadog, following standard mechanism, stdout -> datadog-agent -> datadog.

There are two types of logs that we are writing, application log and access log. Application logs are being written by log4j2 and access logs are provided by tomcat and written using AccessLogValve.

Now, dd-trace-java is able to inject dd specific attributes(dd.trace_id, dd.span_id, dd.env, dd.service, dd.version) to application logs, but it's not able to inject those attributes to access logs.

This is the AccessLogValve that writes the log to console.

private AccessLogValve createAccessLogValveForConsole() {
    Preconditions.checkNotNull(applicationName);

    AccessLogValve logValve = new AccessLogValve();
    logValve.setPattern(LogConfigHelper.getLogFormat(loggingType));
    logValve.setBuffered(false);
    logValve.setRequestAttributesEnabled(true);
    logValve.setDirectory("/dev");
    logValve.setPrefix("stdout");
    logValve.setSuffix("");
    logValve.setFileDateFormat("");
    return logValve;
  }

---

and this is the access log format,

StringBuilder formatBuilder = new StringBuilder();
    formatBuilder.append('{');
    formatBuilder.append("\"timestamp\":\"%{yyyy-MM-dd'T'HH:mm:ss.SSSX}t\",");

    // the localhost hostname in all the log entries
    formatBuilder.append("\"hostname\":\"");
    formatBuilder.append(localHostname);
    formatBuilder.append("\",");

    formatBuilder.append("\"agent\":\"%{User-Agent}i\",");
    formatBuilder.append("\"code\":\"%s\",");
    formatBuilder.append("\"path\":\"%U\",");
    formatBuilder.append("\"referer\":\"%{Referer}i\",");
    formatBuilder.append("\"method\":\"%m\",");
    formatBuilder.append("\"size\":\"%B\",");
    formatBuilder.append("\"response-time\":\"%D\",");
    formatBuilder.append("\"user\":\"%u\",");
    formatBuilder.append('}');

    return formatBuilder.toString();

---

We have enabled dd.log.injection in opts. jvmProps.add("-Ddd.logs.injection=true");

am i missing something? How can dd attributes(dd.span_id, dd.trace_id,..etc) be injected in tomcat access logs?

[devinsba]

Unfortunately for access logs, they are usually written after the spans/traces have been finished. So they aren't active when the log is written, in order for us to support Tomcat access logs we would need to add support for it specifically anyway.

From the user point of view, would it make sense to you if we made our attributes behave the same way as %{User-Agent} etc? ie: you would add a line like formatBuilder.append("\"dd.trace_id\":\"%{dd.trace_id}i\","); to your config

[umarkhan17]

I tried exactly this and it didn't work, now it makes sense why it didn't.

Yes, it would be great if it worked like that.

[ksiczek]

Isn't it a duplicate of https://github.com/DataDog/dd-trace-java/issues/1214 ?