explore - loki log Histogram is not coming up, when we query for large interval

[Uday599]

We have grafana and loki deployed in EKS Cluster -

Histogram log are not coming up when we query for large interval ~more than 30mins. throwing gateway timedout.
would be helpfull if we know how this histogram is built

Error:
No logs volume available
No volume information available for the current queries and time range.

Grafana Version - grafana-8.4.8
Loki Version - loki-6.19.0

Please assist.

[mveitas]

@Uday599 if you are using a reverse proxy such as nginx, make sure to check the configuration settings to allow for longer requests. We recently ran into this situation and had to set the annotation to configure the ingress configuration: nginx.ingress.kubernetes.io/proxy-read-timeout.

If you run the same sort of query that Grafana is doing via the CLI directly against Loki, you'll see that it most likely succeeds.

Grafana (browser) => kubernetes nginx ingress (this is where the error was) => Grafana => Loki

[Uday599]

Hi @mveitas , thank you very much for responding.

I have configured AWS load ingress load balancer,
grafana (browser) => ALB Ingress Controller => Grafana => Loki.

I tried increasing load balancer idle connection , still not able to see histogram,

error -
Failed to load log volume for this query

<title>504 Gateway Time-out</title>

504 Gateway Time-out

FYI- we were able to see log volume histogram when we query for like about 3hrs, 12hrs (though it takes time). but failing when we query for about 24hrs and more.

we were able to see complete logs, but histogram is not loading.
This query will process approximately 363.7 GiB for last 24hrs.

Would be helpfull, how actually this histogram is working in backend.

Please let me know if you need any info.