helix leverages OpenTelemetry to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software's performance and behavior.
By being configured at its core, OpenTelemetry on helix services brings strong observability consistency across integrations and services.
helix relies on the following required environment variables:
ENVIRONMENT represents the environment the service is currently running in. When value is one of local, localhost, dev, development, the logger handles logs at debug level and higher. Otherwise, the logger handles logs at info level and higher.
OTEL_TRACES_ENDPOINT sets the target endpoint the trace exporter will connect to. See example using Grafana Agent below for more details.
When possible, it's strongly advised to leverage Event propagation within distributed tracing.
If the context provided contains a span then the newly-created span will be a child of that span, otherwise it will be a root span.
In the example below, we pass the HTTP request's context. The REST router integration automatically handles tracing. Therefore, the custom span created will be a child of the HTTP request's span, with no additional work on your end.
import("net/http""go.nunchi.studio/helix/telemetry/trace")router.GET("/path",func(rw http.ResponseWriter, req *http.Request){// ..._, span := trace.Start(req.Context(), trace.SpanKindServer,"Custom Span")defer span.End()if2+2==4{ span.RecordError("this is a demo error based on a dummy condition", errors.New("any error"))}// ...})
Logs are JSON formatted and have by default the following keys:
message: The log's message.
timestamp: Timetamp of the log, formatted for RFC-3339 with nano precision.
trace_id and span_id: Trace details, if log is part of a trace.
Logs levels are debug, info, warn, error, fatal.
If environment variable ENVIRONMENT is one of local, localhost, dev, development, the logger handles logs at debug level and higher. Otherwise, the logger handles logs at info level and higher.
By passing a Go context, the logger is aware if the log is part of a trace/span. If so, trace_id and span_id are added to the log so it can be linked to the respective trace/span.
In the example below, we pass the HTTP request's context. The REST router integration automatically handles tracing. Therefore, the log will be associated to the trace/span with no additional work on your end.
import("net/http""go.nunchi.studio/helix/telemetry/log")router.GET("/path",func(rw http.ResponseWriter, req *http.Request){// ... log.Warn(req.Context(),"this is a warning")// ...})
You will find below examples for configuring Grafana Agent with appropriate log labels (using Loki) and trace attributes (using Tempo) when running helix services.
Is something missing?
If you notice something we've missed or could be improved on, please follow this link and submit a pull request to the repository. Once we merge it, the changes will be reflected on the website the next time it is deployed. Thank you for your contributions!