otlp logs interface mixes attributes with resource attributes

[abh]

I tried upgrading our log pipeline from "write to disk / buffer -> promtail -> loki" to "write to otelcol -> loki" to better align metadata between traces and logs.

Other benefits: avoid the buffer through the kube logging system for some applications that log a lot; log directly to existing otelcol infrastructure from applications not running inside the kube cluster.

It's working great, except the loki logs in grafana don't show the attributes provided with the logs. Previously we'd have either JSON or a very human readable version like

msg="some message" attrA=a attrB=b

Now the log explorer just says

some message

I looked at how Grafana could do this better and realized that with the otlp loki logs it's basically impossible because the ingestion flattens the resource attributes with the "log attributes".

So ... hence the bug / feature request: To be useful for native OTLP logs I believe Loki needs to separate out the resource attributes / labels with the message specific attributes / labels.

I realize you all probably know this already, but I didn't see another issue for it so I figured it might be worth logging.