Add OTLP stdout exporter

[zeitlinger]

Towards #3817

Changes

Add OTLP Stdout exporter.

For non-trivial changes, follow the change proposal process.

• Related issues Is OTLP file exporter required for SDKs? #3817
• Related OTEP(s) #
• Links to the prototypes: OTLP Stdout exporter opentelemetry-java#6632
• CHANGELOG.md file updated for non-trivial changes
• spec-compliance-matrix.md updated if necessary

[MrAlias]

How does this relate to the existing console/stdout¹ and file exporters? This seems at best overlapping, but at worst duplicative of already specified exporters.

Footnotes

1. https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk_exporters/stdout.md ↩

[pellared]

What about standard error and and file as output destinations?

[zeitlinger]

How does this relate to the existing console/stdout and file exporters? This seems at best overlapping, but at worst duplicative of already specified exporters.

• console/stdout: is for debugging purposes - it doesn't have a specified format, which you need for https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/otlpjsonconnector
• file: it is exactly the stdout part of that, I failed to link that. in the spec meeting, we discussed that it would be easier to start with stdout as a subset to keep it simple

[zeitlinger]

@pellared @MrAlias can you check again?

[cijothomas]

Fixes #3817

The linked issue is about OTLP File exporter, but this seems to talking about stdout exporter... Did I misread this?

[zeitlinger]

Fixes #3817

The linked issue is about OTLP File exporter, but this seems to talking about stdout exporter... Did I misread this?

otlp-stdout is part of the file exporter

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[zeitlinger]

@pellared @MrAlias can you check again?

[zeitlinger]

@cijothomas can you check again?

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[zeitlinger]

still working on the pr

[cyrille-leclerc]

As discussed with @zeitlinger, this capability to export standardized OTel logs in files that can be seamlessly ingested by the OTel Collector is a major improvement for OTel logs.

That being said, I would be interested in understanding better the developer experience, how this OTEL_LOGS_EXPORTER=otlp/stdout config option will co-exist with the config of logging framework like Logback in Java?

Taking the example of a Java Spring Boot developer, I'll typically use the default logs configuration that emits unstructured logs in stdout (example below) and, if I want to modify this, I'll modify this config through Spring's application.properties and logback-spring.xml files (see reference docs here).

If I want to modify the stdout output of my Spring Boot application to adopt OTel JSON logging to replace the unstructured logs,

1. I'll expect to have to modify these Spring Boot config files for logging: application.properties and logback-spring.xml
2. I'll have to modify these Spring Boot config files to disable the unstructured log messages that are outputted in stdout.

I think that Elastic has done this very well with the Elastic ECS Logging framework. In addition to a manual config, Elastic has an experimental config flag log_ecs_reformatting do seamlessly change the logger config and disable the pre-existing config in favor of the ECS/JSON one. Maybe @AlexanderWert could share some lessons learned.

I'm wondering if we could start with implementing a few extensions of popular logging frameworks (LogBack, Log4j, Pino...) that developers would manually use in their logs config.This could be a natural extension of the OTLP gRPC and HTTP appenders we have implemented so far (eg [here](Appender Instrumentation for Logback version 1.0 and higher
))

I would imagine to the following developer experience for Spring Boot app:

• export OTEL_LOGS_EXPORTER=none to disable the otlp http output
• Edit logback-spring.xml to:

<configuration>
  <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
     <!-- below is just an idea, don't copy paste this fragment -->
     <encoder class="io.opentelemetry.....OtlpJsonEncoder">
  </appender>
  ...
  <root level="INFO">
    <appender-ref ref="console"/>
  </root>

</configuration>

Example Spring Boot console logs (Spring docs here):

2024-08-22T16:59:59.387Z  INFO 109360 --- [myapp] [           main] o.s.b.d.f.logexample.MyApplication       : Starting MyApplication using Java 17.0.12 with PID 109360 (/opt/apps/myapp.jar started by myuser in /opt/apps/)
2024-08-22T16:59:59.410Z  INFO 109360 --- [myapp] [           main] o.s.b.d.f.logexample.MyApplication       : No active profile set, falling back to 1 default profile: "default"
2024-08-22T17:00:02.105Z  INFO 109360 --- [myapp] [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8080 (http)
2024-08-22T17:00:02.133Z  INFO 109360 --- [myapp] [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-08-22T17:00:02.133Z  INFO 109360 --- [myapp] [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.28]
2024-08-22T17:00:02.296Z  INFO 109360 --- [myapp] [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2024-08-22T17:00:02.301Z  INFO 109360 --- [myapp] [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2647 ms
2024-08-22T17:00:03.219Z  INFO 109360 --- [myapp] [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8080 (http) with context path '/'
2024-08-22T17:00:03.250Z  INFO 109360 --- [myapp] [           main] o.s.b.d.f.logexample.MyApplication       : Started MyApplication in 5.793 seconds (process running for 7.164)

[martinjt]

I feel this is a bad direction for OpenTelemetry in general.

stdout for debugging is a valid usecase, but we should strongly encourage people to use OTLP and push the data to an endpoint (ideally a collector). We should not be promoting push data to the console/stdout and using another tool to pull that data and ingest it. I understand that K8s has done it this way in the past, and that 12 factor apps popularised this approach, but now we have a standardised protocol and we don't need that now.

From inefficiencies in some languages, to issues with polling for that data, it's just not something that we should push for when we have a lot of alternatives like OTLP and collector, that are part of the OpenTelemetry project.

All the requests I've seen from this in the .NET world have been from people not wanting to move from other technologies like fluent etc. and not that it's a better way to work.

I can understand a desire to have a console logging output that formats the log line with the template to produce the human readable string, but not one who's purpose is to provide a format for other tools to use. This will fragment the ecosystem further than it already is.

Right now, we have a path that we push people down. Use the OTLP exporter to push data to an endpoint that accepts OTLP. If we push down this route as a production usecase, that will become diluted with vendors saying "use our agent and configure OpenTelemetry to push to stdout", and still claim that they support OpenTelemetry. Especially if this is then supported for Metrics and Traces too. That will actively harm OpenTelemetry as a project, when we're trying make the developer experience better.

[alessandrobologna]

I would maybe rephrase it to 'situations where OTLP push is infeasible' rather than 'legacy'. For instance, serverless platforms like Lambda have a fairly robust 'take console output and write it somewhere', I can see this being very useful for those situations. While I'd love to wave a magic wand and have everyone just use OTLP push, I think stuff like that will take a while, so this seems like a good solution until then.

Completely agree with the above: right now adopting OTEL on serverless (at least AWS Lambda) adds a consistent overhead, both in terms of extra latency in cold starts and execution time, and also memory requirements, if using the ADOT layer. Buffering is not very viable because of the lambda lifecycle, with the execution suspended at the end of the invocation, without a corresponding increase in billed execution time.
Finally, limiting the exporting options only to network protocols also requires exposing the collector to the internet, for a non VPC connected lambda, with the extra headaches that come with that. Having a mechanism that would allow to consistently export telemetry to stdout (and so to cloudwatch logs), and ingest them in the collector (possibly using the live tail API, or polling) would be ideal.

[cijothomas]

We see occasional asks about the capability to export to stdout with well defined format. I see no harm in making this part of the spec.

eg:
open-telemetry/opentelemetry-dotnet#5844
open-telemetry/opentelemetry-dotnet#5036, open-telemetry/opentelemetry-dotnet#5036 (comment)
open-telemetry/opentelemetry-dotnet#4548 (comment)

[pellared]

#3817 status has been changed to accepted by @open-telemetry/technical-committee based on community feedback. Therefore, I think we can continue reviewing this PR.

[joshhayes-sheen-vt]

Chiming in that this would be helpful for onboarding serverless/lambda use cases incrementally, I'd like to instrument my lambda deployments, then add https://github.com/open-telemetry/opentelemetry-lambda but continue routing my telemetry through cloudwatch (which in that environment is stdout) until I have the rest of my infrastructure deployed and tested, then redirect that stripped down collector to other otlp targets (collectors in k8s/ecs/ec2/whatever) for a more advanced processing pipeline