Why Tempo returns a 501 Not Implemented when sent a opentelemetry-dotnet HttpProtobuf request?

[rgl]

I'm trying to use the opentelemetry-dotnet library to send traces to Tempo.

I've configured Tempo to enable the otlp http protocol as (the full config is at my playground repo):

distributor:
  receivers:
    otlp:
      protocols:
        http:

I've configured opentelemetry-dotnet as (the full code is at my playground repo):

builder.Services.AddOpenTelemetryTracing(tpb => tpb
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(builder.Configuration.GetValue<string>("Otlp:ServiceName")))
    .AddOtlpExporter(eo =>
        {
            eo.Endpoint = new Uri(builder.Configuration.GetValue<string>("Otlp:Endpoint"));
            eo.Protocol = OtlpExportProtocol.HttpProtobuf;
        }
    )
    .AddAspNetCoreInstrumentation()
);

But Tempo is returning a 501 Not Implemented as can be seen at this wireshark text capture:

POST / HTTP/1.1
Host: localhost:55681
Transfer-Encoding: chunked
Content-Type: application/x-protobuf


..
]
.
.service.name..

Quotes.Api
=
.service.instance.id.&
$00991fd3-9b07-426d-9bc7-e35fce7403f5...
3
(OpenTelemetry.Instrumentation.AspNetCore..1.0.0.0...
...B".N..FE....N......	..!~*./0.9|.h.T...AL...U...J.
	http.host..
.localhost:8000J.
.http.method..
.GETJ.
.http.target..
./J$
.http.url..
.http://localhost:8000/J}
.http.user_agent.j
hMozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36J.
.http.status_code.....z.....

HTTP/1.1 501 Not Implemented
Content-Type: application/x-protobuf
Date: Sun, 12 Dec 2021 19:41:57 GMT
Content-Length: 19

....Not Implemented

Do you have any idea why? And what should I do to make it work?

[joe-elliott]

Tempo only supports ingestion of trace telelmetry data. By default I think that an OTel client will try to send metrics, traces and logs to the configured endpoint.

Can you confirm that logs/metrics data is turned off? The current env vars are here:

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/sdk-environment-variables.md#exporter-selection

OTEL_LOGS_EXPORTER=none
OTEL_MERICS_EXPORTER=none

This has changed though so you may want to double check your client. Another option would be to point your app at the OTel Collector which supports all telemetry ingestion and then have the collector send to Tempo.

[joe-elliott]

To add one more possibility. We used to be on a quite old version of our OTel dependency but have recently updated to latest. If you try the latest Tempo images you may find that it works.

[rgl]

Using the gRPC protocol seems to work without any changes (I can see the traces in Tempo), and without apparent errors. Do you known if that is normal?

I wasn't aware of the OTEL environment variables (thx for the link!); they do not seem to work with the dotnet library thou. I will check the library source code to see whether they are actually looking into them.

PS: The actual source code does not look for any of the OTEL_*_EXPORTER environment variables. Thou, it seems to only send Traces, so its not trying to send Metrics nor Logs.

With latest Tempo images you mean after 1.2.1? For example, the latest from the main branch?

PS: Tried with grafana/tempo:main-9f0a021, same result.