komu.engineer
komu.engineer copied to clipboard
Discussion: The complete guide to OpenTelemetry in Golang.
This is the place for comments/discussion regarding the article: The complete guide to OpenTelemetry in Golang
Please be civil.
Hi 👋 Thanks for this excellent article 🙇 I am a newbie to OTel, and your example is extremely easy to follow. I got the following problem and would to share my solution.
Description
I tried to run the TLS setup script but it didn't on my local environment:
- OS:
Debian 11
-
OpenSSL 1.1.1n
Got this error message: req: Unrecognized flag CA
when trying to create and sign the certificates for Servers and Clients
Suggested Solution
-
I looked through the man page for
openssl
for usage ofreq
andx509
and some other examples of using the tool:- https://www.openssl.org/docs/man1.1.1/man1/openssl-req.html
- https://www.openssl.org/docs/man1.1.1/man1/openssl-x509.html
-
I found that we can break creating certificates and signing into 2 separate commands. The following commands work for me:
...
{ # create server certs.
openssl \
req \
-new \
-newkey rsa:2048 \
-days 372 \
-nodes \
-keyout confs/server.key \
-out confs/server.cst \
-subj "/C=US/ST=CA/O=MyOrg/CN=myOrgCA" \
-addext "subjectAltName=DNS:example.com,DNS:example.net,DNS:otel_collector,DNS:localhost" \
# sign the cert
openssl \
x509 \
-req \
-in confs/server.cst -CA confs/rootCA.crt -CAkey confs/rootCA.key -CAcreateserial -out confs/server.crt
}
{ # create client certs.
openssl \
req \
-new \
-newkey rsa:2048 \
-days 372 \
-nodes \
-keyout confs/client.key \
-out confs/client.cst \
-subj "/C=US/ST=CA/O=MyOrg/CN=myOrgCA" \
-addext "subjectAltName=DNS:example.com,DNS:example.net,DNS:otel_collector,DNS:localhost" \
# sign the cert
openssl \
x509 \
-req \
-in confs/client.cst -CA confs/rootCA.crt -CAkey confs/rootCA.key -CAcreateserial -out confs/client.crt
}
...
Thanks a lot for the excellent article!! thanks to you I have now a better idea about what to do. Thanks!
@meobilivang Hello,
Yeah the openssl script I had shared is not cross-platform. I had only shared what works for my machine and as you found out, it does not work across all OSes/openSSL versions.
I'm glad to hear that you however found something that works for you.
Thanks a lot for the excellent article!! thanks to you I have now a better idea about what to do. Thanks!
@xmarlem Thanks and you are welcome.
This is the article that got me started in Telemetry in Golang earlier this year, so it's always a pleasure to re-read it 😊. Once more thank you.
I usually use zerolog but as you highlighted elsewhere, zerolog's hooks don't let us access the attributes when correlating with traces/spans. This shouldn't be a problem if log aggregators had some endpoint that lets us retrieve logs with a given TraceID/SpanID (aka plain-old SQL left joins). Alternatively, one could use zerolog's Logger.Output(w io.Writer) -> Logger
to derive a child logger that tees the output to both the original destination (eg. std.Out) and to the Span (as an attribute)
OS: macOS Ventura OpenSSL 3.1.2 1 Aug 2023 (Library: OpenSSL 3.1.2 1 Aug 2023)
#!/bin/bash
set -x # have bash print command been ran
set -e # fail if any command fails
setup_certs(){
{ # clean
rm -rf confs/*.key
rm -rf confs/*.crt
rm -rf confs/*.cst
}
{ # create CA.
openssl \
req \
-new -verbose \
-newkey rsa:4096 \
-nodes \
-x509 \
-subj "/C=US/ST=CA/O=MyOrg/CN=myOrgCA" \
-keyout confs/rootCA.key \
-out confs/rootCA.crt
}
{ # create server certs.
openssl \
req \
-new -verbose \
-subj "/C=US/ST=CA/O=MyOrg/CN=myOrgCA" -subject \
-addext "subjectAltName = DNS:otel_collector,DNS:otel-col,DNS:otelcol,DNS:localhost,DNS:otlp,IP:127.0.0.1" \
-addext "extendedKeyUsage = serverAuth" \
-newkey rsa:2048 \
-nodes -verify \
-keyout confs/server.key \
-out confs/server.cst \
# sign the cert
openssl \
x509 \
-req \
-copy_extensions copy \
-in confs/server.cst -CA confs/rootCA.crt -CAkey confs/rootCA.key -CAcreateserial -out confs/server.crt
}
{ # create client certs.
openssl \
req \
-new -verbose \
-newkey rsa:2048 \
-nodes -verify \
-keyout confs/client.key \
-out confs/client.cst \
-subj "/C=US/ST=CA/O=MyOrg/CN=myOrgCA" \
-addext "subjectAltName = DNS:otel_collector,DNS:otel-col,DNS:otelcol,DNS:localhost,DNS:otlp,IP:127.0.0.1" \
-addext "extendedKeyUsage = serverAuth"
# sign the cert
openssl \
x509 \
-req \
-copy_extensions copyall \
-in confs/client.cst -CA confs/rootCA.crt -CAkey confs/rootCA.key -CAcreateserial -out confs/client.crt
}
{ # clean
rm -rf confs/*.csr
rm -rf confs/*.srl
chmod 666 confs/server.crt confs/server.key confs/rootCA.crt
}
{ # verify
openssl x509 -in confs/server.crt -text -noout
}
}
setup_certs
I had 2 versions of openssl installed: one from conda
another from homebrew
. I used the one from homebrew. The -copy_extensions copyall
argument was especially important. Without it, the certificates didn't have the
X509v3 extensions:
X509v3 Subject Alternative Name:
portion.
It's very likely some DNS entries in subjectAltName above can be removed.
Hi @gcsfred2 , yes the openSSL commands are bound to have small differences across different OSes and/or across different openssl versions. The script included was meant to serve as an illustration.
I just wanted to say, this was a great article. I haven't implemented in my own code yet, but I can't wait to do that.
Thank you immensely for the insightful article! It was a game-changer for me, as I successfully implemented OpenTelemetry with Golang seamlessly for the first time. Your content made the process exceptionally clear and accessible. Grateful for your expertise!
@komuw Thank you very much for your article, it has been of great help to me.
About logs, I have another question that I would like to ask you. Now I want to create service observabilty with grafana , grafna loki , grafana tempo and prometheus . In my local machine, with golang service, I want to sent logs to loki with opentelemetry collector : app -> otel-collector -> loki I don't know if it can be implemented in this route. I search lots of articles, but I don't get the result.
In grafana website doc Send logs to Loki with Loki receiver , the route is: app/service -> logfile <- promtail -> otel-collector -> loki
I don't want to use promtail, because it will scrape container logs(not service log) when service deploy in docker or k8s.
@komuw Thank you very much for your article, it has been of great help to me.
About logs, I have another question that I would like to ask you. Now I want to create service observabilty with grafana , grafna loki , grafana tempo and prometheus . In my local machine, with golang service, I want to sent logs to loki with opentelemetry collector : app -> otel-collector -> loki I don't know if it can be implemented in this route. I search lots of articles, but I don't get the result.
In grafana website doc Send logs to Loki with Loki receiver , the route is: app/service -> logfile <- promtail -> otel-collector -> loki
I don't want to use promtail, because it will scrape container logs(not service log) when service deploy in docker or k8s.
Have you seen; https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/lokiexporter/README.md ? I haven't tried out loki, so I unfortunately do not have a lot of useful information to offer.
@komuw Thank you very much for your article, it has been of great help to me. About logs, I have another question that I would like to ask you. Now I want to create service observabilty with grafana , grafna loki , grafana tempo and prometheus . In my local machine, with golang service, I want to sent logs to loki with opentelemetry collector : app -> otel-collector -> loki I don't know if it can be implemented in this route. I search lots of articles, but I don't get the result. In grafana website doc Send logs to Loki with Loki receiver , the route is: app/service -> logfile <- promtail -> otel-collector -> loki I don't want to use promtail, because it will scrape container logs(not service log) when service deploy in docker or k8s.
Have you seen; https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/lokiexporter/README.md ? I haven't tried out loki, so I unfortunately do not have a lot of useful information to offer.
Okay, I got it. Thank you very much.
Just wanted to add here that global.SetMeterProvider(mp)
doesn't seem to work anymore. You have to use otel.SetMeterProvider(mp)
Just wanted to add here that
global.SetMeterProvider(mp)
doesn't seem to work anymore. You have to useotel.SetMeterProvider(mp)
The main branch of https://github.com/komuw/otero usually has upto date code that is similar in shape to the blogpost.