Main/Unreleased

All Changes

Loki

Enhancements

9797 chaudum: Add new loki_index_gateway_requests_total counter metric to observe per-tenant RPS
9710 chaudum: Add shuffle sharding to index gateway
9573 CCOLLOT: Lambda-Promtail: Add support for AWS CloudFront log ingestion.
9497 CCOLLOT: Lambda-Promtail: Add support for AWS CloudTrail log ingestion.
8886 MichelHollands: Add new logql template function unixToTime
8067 CCOLLOT: Lambda-Promtail: Add support for AWS CloudTrail log ingestion.
9515 MichelHollands: Fix String() on vector aggregation LogQL expressions that contain without ().
8067 DylanGuedes: Distributor: Add auto-forget unhealthy members support.
9175 MichelHollands: Ingester: update the prepare_shutdown endpoint so it supports GET and DELETE and stores the state on disk.
8953 dannykopping: Querier: block queries by hash.
8851 jeschkies: Introduce limit to require a set of labels for selecting streams.
9016 kavirajk: Change response type of format_query handler to application/json
8972 salvacorts Index stat requests are now cached in the results cache.
9177 salvacorts Index stat cache can be enabled or disabled with the new cache_index_stats_results flag. Disabled by default.
9096 salvacorts: Compute proportional TSDB index stats for chunks that doesn't fit fully in the queried time range.
8939 Suruthi-G-K: Loki: Add support for trusted profile authentication in COS client.
8852 wtchangdm: Loki: Add route_randomly to Redis options.
8848 dannykopping: Ruler: add configurable rule evaluation jitter.
8826 amankrsingh2000: Loki: Add support for IBM cloud object storage as storage client.
8752 chaudum: Add query fairness control across actors within a tenant to scheduler, which can be enabled by passing the X-Loki-Actor-Path header to the HTTP request of the query.
8786 DylanGuedes: Ingester: add new /ingester/prepare_shutdown endpoint.
8744 dannykopping: Ruler: remote rule evaluation.
8670 salvacorts Introduce two new limits to refuse log and metric queries that would read too much data.
8918 salvacorts Introduce limit to require at least a number label matchers on metric and log queries.
8909 salvacorts Requests to /loki/api/v1/index/stats are split in 24h intervals.
8732 abaguas: azure: respect retry config before cancelling the context
9206 dannykopping: Ruler: log rule evaluation detail.
9184 periklis: Bump dskit to introduce IPv6 support for memberlist
9357 Indransh: Add HTTP API to change the log level at runtime
9431 dannykopping: Add more buckets to loki_memcache_request_duration_seconds metric; latencies can increase if using memcached with NVMe
8684 oleksii-boiko-ua: Helm: Add hpa templates for read, write and backend components.
9535 salvacorts Index stats cache can be configured independently of the results cache. If it's not configured, but it's enabled, it will use the results cache configuration.
9604 dannykopping: Querier: configurable writeback queue bytes size
9626 ashwanthgoli logfmt: add --strict flag to enable strict parsing, perform nostrict parsing by default
9672 zeitlinger: Add alignLeft and alignRight line formatting functions.
9693 salvacorts Add keep stage to LogQL.
7447 ashwanthgoli compactor: multi-store support.
7754 ashwanthgoli index-shipper: add support for multiple stores.
8662 liguozhong: LogQL: Introduce distinct
9813 jeschkies: Enable Protobuf encoding via content negotiation between querier and query frontend.

Fixes

8979 slim-bean: Fix the case where a logs query with start time == end time was returning logs when none should be returned.
9099 salvacorts: Fix the estimated size of chunks when writing a new TSDB file during compaction.
9130 salvacorts: Pass LogQL engine options down to the split by range, sharding, and query size limiter middlewares.
9252 jeschkies: Use un-escaped regex literal for string matching.
9176 DylanGuedes: Fix incorrect association of per-stream rate limit when sharding is enabled.
9463 Totalus: Fix OpenStack Swift client object listing to fetch all the objects properly.
9471 sandeepsukhani: query-scheduler: fix query distribution in SSD mode.
9495 thampiotr: Promtail: Fix potential goroutine leak in file tailer.
9650 ashwanthgoli: Config: ensure storage config defaults apply to named stores.
9629 periklis: Fix duplicate label values from ingester streams.
9757 sandeepsukhani: Frontend Caching: Fix a bug in negative logs results cache causing Loki to unexpectedly send empty/incorrect results.
9754 ashwanthgoli: Fixes an issue with indexes becoming unqueriable if the index prefix is different from the one configured in the latest period config.
9763 ssncferreira: Fix the logic of the offset operator for downstream queries on instant query splitting of (range) vector aggregation expressions containing an offset.
9773 ssncferreira: Fix instant query summary statistic's splits corresponding to the number of subqueries a query is split into based on split_queries_by_interval.

Changes

9857 DylanGuedes: Stop emitting spans for every AWS.S3 or Azure.Blob call.
9212 trevorwhitney: Rename UsageReport to Analytics. The only external impact of this change is a change in the -list-targets output.

Promtail

Enhancements

8474 andriikushch: Promtail: Add a new target for the Azure Event Hubs
8874 rfratto: Promtail: Support expoential backoff when polling unchanged files for logs.
9508 farodin91: Promtail: improve behavior of partial lines.

Fixes

8988 darxriggs: Promtail: Prevent logging errors on normal shutdown.
9155 farodin91: Promtail: Break on iterate journal failure.
8987 darxriggs: Promtail: Fix file descriptor leak.
9863 ashwanthgoli: Promtail: Apply defaults to HTTP client config. This ensures follow_redirects is set to true.

LogCLI

Fixes

9597 vlad-diachenko: Set TSDB shipper mode to ReadOnly and disabled indexGatewayClient during local query run and changed index downloading timeout from 5s to 1m.
8566 ndrpnt: Allow queries to start with negative filters (!= and !~) when omitting stream selector with --stdin flag

Mixins

Enhancements

Fixes

9684 thampiotr: Mixins: Fix promtail cluster template not finding all clusters.
8995 dannykopping: Mixins: Fix Jsonnet RUNTIME ERROR that occurs when you try to use the mixins with use_boltdb_shipper: false.

FluentD

Enhancements

LOG-4012 jcantril: fluent-plugin-grapha-loki: Add config to support tls: ciphers, min_versio

Jsonnet

9790 manohar-koukuntla: Add TSDB equivalent of use_boltdb_shipper flag to be able to configure tsdb_shipper section.
8855 JoaoBraveCoding: Add gRPC port to loki compactor mixin
8880 JoaoBraveCoding: Normalize headless service name for query-frontend/scheduler

2.8.2 (2023-05-03)

Loki

Security

9370 dannykopping: upgrade to go1.20.4

Promtail

Enhancements

8994 DylanGuedes: Promtail: Add new decompression configuration to customize the decompressor behavior.

2.8.1 (2023-04-24)

Loki

Fixes

9156 ashwanthgoli: Expiration: do not drop index if period is a zero value.
8971 dannykopping: Stats: fix Cache.Chunk.BytesSent statistic and loki_chunk_fetcher_fetched_size_bytes metric with correct chunk size.
9185 dannykopping: Prevent redis client from incorrectly choosing cluster mode with local address.

Changes

9106 trevorwhitney: Update go to 1.20.3.

Build

9264 trevorwhitney: Update build and other docker image to alpine 3.16.5.

Promtail

Fixes

9095 JordanRushing Fix journald support in amd64 binary build.

2.8.0 (2023-04-04)

Loki

Enhancements

8824 periklis: Expose optional label matcher for label values handler
8727 cstyan jeschkies: Propagate per-request limit header to querier.
8682 dannykopping: Add fetched chunk size distribution metric loki_chunk_fetcher_fetched_size_bytes.
8532 justcompile: Adds Storage Class option to S3 objects
7951 MichelHollands: Add a count template function to line_format and label_format.
7380 liguozhong: metrics query: range vector support streaming agg when no overlap.
7731 bitkill: Add healthchecks to the docker-compose example.
7759 kavirajk: Improve error message for loading config with ENV variables.
7785 dannykopping: Add query blocker for queries and rules.
7817 kavirajk: fix(memcached): panic on send on closed channel.
7916 ssncferreira: Add doc-generator tool to generate configuration flags documentation.
7964 slim-bean: Add a since query parameter to allow querying based on relative time.
7989 liguozhong: logql support sort and sort_desc.
7997 kavirajk: fix(promtail): Fix cri tags extra new lines when joining partial lines
7975 adityacs: Support drop labels in logql
7946 ashwanthgoli config: Add support for named stores
8027 kavirajk: chore(promtail): Make batchwait and batchsize config explicit with yaml tags
7978 chaudum: Shut down query frontend gracefully to allow inflight requests to complete.
8047 bboreham: Dashboards: add k8s resource requests to CPU and memory panels.
8061 kavirajk: Remove circle from Loki OSS
8092 dannykopping: add rule-based sharding to ruler.
8131 jeschkies: Compile Promtail ARM and ARM64 with journald support.
8212 kavirajk: ingester: Add ingester_memory_streams_labels_bytes metric for more visibility of size of metadata of in-memory streams.
8271 kavirajk: logql: Support urlencode and urldecode template functions
8259 mar4uk: Extract push.proto from the logproto package to the separate module.
7906 kavirajk: Add API endpoint that formats LogQL expressions and support new fmt subcommand in logcli to format LogQL query.
6675 btaani: Add logfmt expression parser for selective extraction of labels from logfmt formatted logs
8474 farodin91: Add support for short-lived S3 session tokens
8774 slim-bean: Add new logql template functions bytes, duration, unixEpochMillis, unixEpochNanos, toDateInZone, b64Enc, and b64Dec

Fixes

7784 isodude: Fix default values of connect addresses for compactor and querier workers to work with IPv6.
7880 sandeepsukhani: consider range and offset in queries while looking for schema config for query sharding.
7937 ssncferreira: Deprecate CLI flag -ruler.wal-cleaer.period and replace it with -ruler.wal-cleaner.period.
7966 sandeepsukhani: Fix query-frontend request load balancing when using k8s service.
8251 sandeepsukhani index-store: fix indexing of chunks overlapping multiple schemas.
8151 sandeepsukhani fix log deletion with line filters.
8448 chaudum: Fix bug in LogQL parser that caused certain queries that contain a vector expression to fail.
8775 sandeepsukhani: index-gateway: fix failure in initializing index gateway when boltdb-shipper is not being used.
8448 sandeepsukhani: deletion: fix issue in processing delete requests with tsdb index
8753 slim-bean A zero value for retention_period will now disable retention.
8959 periklis: Align common instance_addr with memberlist advertise_addr

Changes

8315 thepalbi Relicense and export pkg/ingester WAL code to be used in Promtail's WAL.
8761 slim-bean Remove "subqueries" from the metrics.go log line and instead provide splits and shards
8887 3deep5me Helm: Removed support for PodDisruptionBudget in policy/v1alpha1 and upgraded it to policy/v1.

Build

Promtail

Enhancements

8231 CCOLLOT: Lambda-promtail: add support for AWS SQS message ingestion.
7619 cadrake: Add ability to pass query params to heroku drain targets for relabelling.
7973 chodges15: Add configuration to drop rate limited batches in Loki client and new metric label for drop reason.
8153 kavirajk: promtail: Add max-line-size limit to drop on client side
8096 kavirajk: doc(promtail): Doc about how log rotate works with promtail
8233 nicoche: promtail: Add max-line-size-truncate limit to truncate too long lines on client side
7462 MarNicGit: Allow excluding event message from Windows Event Log entries.
7597 redbaron: allow ratelimiting by label
3493 adityacs Support geoip stage.
8382 kelnage: Promtail: Add event log message stage

Fixes

8231 CCOLLOT: Lambda-promtail: fix flushing behavior of batches, leading to a significant increase in performance.

Changes

LogCLI

Enhancement

8413 chaudum: Try to load tenant-specific schemaconfig-{orgID}.yaml when using --remote-schema argument and fallback to global schemaconfig.yaml.
8537 jeschkies: Allow fetching all entries with --limit=0.

Fluent Bit

Loki Canary

Enhancements

8024 jijotj: Support passing loki address as environment variable

Jsonnet

7923 manohar-koukuntla: Add zone aware ingesters in jsonnet deployment

Fixes

8247 Whyeasy fix usage of cluster label within Mixin.

Build

7938 ssncferreira: Add DroneCI pipeline step to validate configuration flags documentation generation.

Notes

Dependencies

2.7.5 (2023-03-28)

Loki

Fixes

7924 jeschkies: Flush buffered logger on exit

2.7.4 (2023-02-24)

Loki

Fixes

8531 garrettlish: logql: fix panics when cloning a special query
8120 ashwanthgoli: fix panic on hitting /scheduler/ring when ring is disabled.
7988 ashwanthgoli: store: write overlapping chunks to multiple stores.
7925 sandeepsukhani: Fix bugs in logs results caching causing query-frontend to return logs outside of query window.

Build

8575 MichelHollands: Update build image to go 1.20.1 and alpine 3.16.4.
8583 MichelHollands: Use 0.28.1 build image and update go and alpine versions.

Promtail

Enhancements

Fixes

8497 kavirajk: Fix cri tags treating different streams as the same
7771 GeorgeTsilias: Handle nil error on target Details() call.
7461 MarNicGit: Promtail: Fix collecting userdata field from Windows Event Log

2.7.3 (2023-02-01)

Loki

Fixes

8340 MasslessParticle Fix bug in compactor that caused panics when startTime and endTime of a delete request are equal.

Build

8232 TaehyunHwang Fix build issue that caused --version to show wrong version for Loki and Promtail binaries.

2.7.2 (2023-01-25)

Loki

Fixes

7926 MichelHollands: Fix bug in validation of pattern and regexp parsers where missing or empty parameters caused panics.
7720 sandeepsukhani: Fix bugs in processing delete requests with line filters.
7708 DylanGuedes: Fix bug in multi-tenant querying.

Notes

This release was created from a branch starting at commit 706c22e9e40b0156031f214b63dc6ed4e210abc1 but it may also contain backported changes from main.

Check the history of the branch release-2.7.x.

Dependencies

Go version: 1.19.5

2.7.1 (2022-12-09)

Loki

Enhancements

6360 liguozhong: Hide error message when context timeout occurs in s3.getObject
7602 vmax: Add decolorize filter to easily parse colored logs.
7804 sandeepsukhani: Use grpc for communicating with compactor for query time filtering of data requested for deletion.
7684 kavirajk: Add missing embedded-cache config under cache_config reference documentation.

Fixes

7453 periklis: Add single compactor http client for delete and gennumber clients

Changes

7877A trevorwhitney: Due to a known bug with experimental new delete mode feature, the default delete mode has been changed to filter-only.

Promtail

Enhancements

7602 vmax: Add decolorize stage to Promtail to easily parse colored logs.

Fixes

Changes

7587 mar4uk: Add go build tag promtail_journal_enabled to include/exclude Promtail journald code from binary.

2.7.0

Loki

Enhancements

7436 periklis: Expose ring and memberlist handlers through internal server listener
7227 Red-GV: Add ability to configure tls minimum version and cipher suites
7179 vlad-diachenko: Add ability to use Azure Service Principals credentials to authenticate to Azure Blob Storage.
7063 kavirajk: Add additional push mode to Loki canary that can directly push logs to given Loki URL.
7069 periklis: Add support for custom internal server listener for readiness probes.
7023 liguozhong: logql engine support exec vector(0) grammar.
6983 slim-bean: __timestamp__ and __line__ are now available in the logql label_format query stage.
6821 kavirajk: Introduce new cache type embedded-cache which is an in-process cache system that runs loki without the need for an external cache (like memcached, redis, etc). It can be run in two modes distributed: false (default, and same as old fifocache) and distributed: true which runs cache in distributed fashion sharding keys across peers if Loki is run in microservices or SSD mode.
6691 dannykopping: Update production-ready Loki cluster in docker-compose
6317 dannykoping: General: add cache usage statistics
6444 aminesnow Add TLS config to query frontend.
6179 chaudum: Add new HTTP endpoint to delete ingester ring token file and shutdown process gracefully
5997 simonswine: Querier: parallize label queries to both stores.
5406 ctovena: Revise the configuration parameters that configure the usage report to grafana.com.
7264 bboreham: Chunks: decode varints directly from byte buffer, for speed.
7263 bboreham: Dependencies: klauspost/compress package to v1.15.11; improves performance.
7270 wilfriedroset: Add support for username to redis cache configuration.
6952 DylanGuedes: Experimental: Introduce a new feature named stream sharding.

Fixes

7426 periklis: Add missing compactor delete client tls client config
7238 periklis: Fix internal server bootstrap for query frontend
7288 ssncferreira: Fix query mapping in AST mapper rangemapper to support the new VectorExpr expression.
7040 bakunowski: Remove duplicated loki_boltdb_shipper prefix from tables_upload_operation_total metric.
6937 ssncferreira: Fix topk and bottomk expressions with parameter <= 0.
6780 periklis: Attach the panic recovery handler on all HTTP handlers
6358 taharah: Fixes sigv4 authentication for the Ruler's remote write configuration by allowing both a global and per tenant configuration.
6375 dannykopping: Fix bug that prevented users from using the json parser after a line_format pipeline stage.
6505 dmitri-lerko Fixes failed to receive pubsub messages error with promtail GCPLog client.
6372 splitice: Add support for numbers in JSON fields.

Changes

6726 kavirajk: upgrades go from 1.17.9 -> 1.18.4
6415 salvacorts: Evenly spread queriers across kubernetes nodes.
6349 simonswine: Update the default HTTP listen port from 80 to 3100. Make sure to configure the port explicitly if you are using port 80.
6835 DylanGuedes: Add new per-tenant query timeout configuration and remove engine query timeout.
7212 Juneezee: Replaces deprecated io/ioutil with io and os.
7292 jmherbst: Add string conversion to value based drops to more intuitively match numeric fields. String conversion failure will result in no lines being dropped.
7361 szczepad: Renames metric loki_log_messages_total to loki_internal_log_messages_total
7416 mstrzele: Use the stable HorizontalPodAutoscaler v2, if possible, when installing using Helm
7510 slim-bean: Limited queries (queries without filter expressions) will now be split and sharded.
5400 BenoitKnecht: promtail/server: Disable profiling by default

Promtail

7470 Jack-King: Add configuration for adding custom HTTP headers to push requests

Enhancements

7593 chodges15: Promtail: Add tenant label to client drop metrics and logs
7101 liguozhong: Promtail: Add support for max stream limit.
7247 liguozhong: Add config reload endpoint / signal to promtail.
6708 DylanGuedes: Add compressed files support to Promtail.
5977 juissi-t lambda-promtail: Add support for Kinesis data stream events
6828 alexandre1984rj Add the BotScore and BotScoreSrc fields once the Cloudflare API returns those two fields on the list of all available log fields.
6656 carlospeon: Allow promtail to add matches to the journal reader
7401 thepalbi: Add timeout to GCP Logs push target
7414 thepalbi: Add basic tracing support

Fixes

7394 liguozhong: Fix issue with the Cloudflare target that caused it to stop working after it received an error in the logpull request as explained in issue grafana#6150
6766 kavirajk: fix(logql): Make LabelSampleExtractor ignore processing the line if it doesn't contain that specific label. Fixes unwrap behavior explained in the issue grafana#6713
7016 chodges15: Fix issue with dropping logs when a file based SD target's labels are updated

Changes

quodlibetor: Change Docker target discovery log level from Error to Info

Logcli

7325 dbirks: Document setting up command completion
8518 SN9NV: Add parallel flags

Fluent Bit

Loki Canary

7398 verejoel: Allow insecure TLS connections

Jsonnet

6189 irizzant: Add creation of a ServiceMonitor object for Prometheus scraping through configuration parameter create_service_monitor. Simplify mixin usage by adding (https://github.com/prometheus-operator/kube-prometheus) library.
6662 Whyeasy: Fixes memberlist error when using a stateful ruler.

Notes

This release was created from a branch starting at commit 706c22e9e40b0156031f214b63dc6ed4e210abc1 but it may also contain backported changes from main.

Check the history of the branch release-2.7.x.

Dependencies

Go Version: FIXME

2.6.1 (2022/07/18)

All Changes

6658 Updated the versions of dskit and memberlist to allow configuring cluster labels for memberlist. Cluster labels prevent mixing the members between two consistent hash rings of separate applications that are run in the same Kubernetes cluster.
6681 Fixed an HTTP connection leak between the querier and the compactor when the log entry deletion feature is enabled.
6583 Fixed noisy error messages when the log entry deletion feature is disabled for a tenant.

2.6.0 (2022/07/08)

All Changes

Here is the list with the changes that were produced since the previous release.

Loki

Enhancements

5662 ssncferreira chaudum Improve performance of instant queries by splitting range into multiple subqueries that are executed in parallel.
5848 arcosx: Add Baidu AI Cloud as a storage backend choice.
6410 MichelHollands: Add support for per tenant delete API access enabling.
5879 MichelHollands: Remove lines matching delete request expression when using "filter-and-delete" deletion mode.
5984 dannykopping and salvacorts: Improve query performance by preventing unnecessary querying of ingesters when the query data is old enough to be in object storage.
5971 kavirajk: Extend the metrics.go recording of statistics about metadata queries to include labels and series queries.
6136 periklis: Add support for alertmanager header authorization.
6163 jburnham: LogQL: Add a default sprig template function in LogQL label/line formatter.

Fixes

6152 slim-bean: Fixes unbounded ingester memory growth when live tailing under specific circumstances.
5685 chaudum: Fix bug in push request parser that allowed users to send arbitrary non-string data as "log line".
5799 cyriltovena Fix deduping issues when multiple entries with the same timestamp exist. !hide or not hide (bugfix Loki)
5888 Papawy Fix common configuration block net interface name when overwritten by ring common configuration.

Changes

6361 chaudum: Sum values in unwrapped rate aggregation instead of treating them as counter.
6412 chaudum: Add new unwrapped range aggregation rate_counter() to LogQL
6042 slim-bean: Add a new configuration to allow fudging of ingested timestamps to guarantee sort order of duplicate timestamps at query time.
6120 KMiller-Grafana: Rename configuration parameter fudge_duplicate_timestamp to be increment_duplicate_timestamp.
5777 tatchiuleung: storage: make Azure blobID chunk delimiter configurable
5650 cyriltovena: Remove more chunkstore and schema version below v9
5643 simonswine: Introduce a ChunkRef type as part of logproto
6435 MichelHollands: Remove the whole-stream-deletion mode.
5899 simonswine: Update go image to 1.17.9.

Promtail

Enhancements

6105 rutgerke Export metrics for the Promtail journal target.
5943 tpaschalis: Add configuration support for excluding configuration files when instantiating Promtail.
5790 chaudum: Add UDP support for Promtail's syslog target.
6102 timchenko-a: Add multi-tenancy support to lambda-promtail.
6099 cstyan: Drop lines with malformed JSON in Promtail JSON pipeline stage.
5715 chaudum: Allow promtail to push RFC5424 formatted syslog messages
6395 DylanGuedes: Add encoding support

Fixes

6034 DylanGuedes: Promtail: Fix symlink tailing behavior.

Changes

6371 witalisoft: BREAKING: Support more complex match based on multiple extracted data fields in drop stage
5686 ssncferreira: Move promtail StreamLagLabels config to upper level config.Config
5839 marctc: Add ActiveTargets method to promtail
5661 masslessparticle: Invalidate caches on deletes

Fluent Bit

5711 MichelHollands: Update fluent-bit output name

Loki Canary

6310 chodges15: Add support for client-side TLS certs in loki-canary for Loki connection

Notes

This release was created from a branch starting at commit 1794a766134f07b54386b1a431b58e1d44e6d7f7 but it may also contain backported changes from main.

Check the history of the branch release-2.6.x.

Dependencies

Go Version: 1.17.9

2.5.0 (2022/04/07)

Release notes for 2.5.0 can be found on the release notes page

All Changes

Here is a list of all significant changes, in the past we have included all changes but with over 500 PR's merged since the last release we decided to curate the list to include only the most relevant.

Loki

Enhancements

5542 bboreham: regexp filter: use modified package with optimisations
5318 jeschkies: Speed up EntrySortIterator by 20%.
5317 owen-d: Logql/parallel binop
5315 bboreham: filters: use faster regexp package
5311 vlad-diachenko: Removed redundant memory allocations in parsers
5291 owen-d: less opaque chunk keys on fs with v12
5275 SasSwart: Parse duration expressions in accordance with promql
5249 3JIou-home: Push: add deflate compression in post requests
5160 sandeepsukhani: add objects list caching for boltdb-shipper index store to reduce object storage list api calls
5148 chaudum: Auto-expire old items from FIFO cache
5093 liguozhong: [enhancement] querier : Add "query_memory_only" to make loki have option to rely only on memory availability.
5078 ssncferreira: Loki: Implement custom /config handler (#4785)
5054 JordanRushing: new v12 schema optimized to better handle S3 prefix rate limits
5013 liguozhong: [new feature] logql: extrapolate unwrapped rate function
4947 siavashs: Support Redis Cluster Configuration Endpoint
4938 DylanGuedes: Add distributor ring page
4879 cyriltovena: LogQL: add line function to | line_format template
4858 sandy2008: feat(): add ManagedIdentity in Azure Blob Storage

Main

5789 bboreham: Production config: add dot to some DNS address to reduce lookups.
5780 simonswine: Update alpine image to 3.15.4.
5715 chaudum Add option to push RFC5424 syslog messages from Promtail in syslog scrape target.
5696 paullryan don't block scraping of new logs from cloudflare within promtail if an error is received from cloudflare about too early logs.
5685 chaudum Fix bug in push request parser that allowed users to send arbitrary non-string data as "log line".
5707 franzwong Promtail: Rename config name limit_config to limits_config.
5626 jeschkies Apply query limits to multi-tenant queries by choosing the most restrictive limit from the set of tenant limits.
5622 chaudum: Fix bug in query splitter that caused interval query parameter to be ignored and therefore returning more logs than expected.
5521 cstyan: Move stream lag configuration to top level clients config struct and refactor stream lag metric, this resolves a bug with duplicate metric collection when a single Promtail binary is running multiple Promtail clients.
5568 afayngelerindbx: Fix canary panics due to concurrent execution of confirmMissing
5552 jiachengxu: Loki mixin: add DiskSpaceUtilizationPanel
5541 bboreham: Queries: reject very deeply nested regexps which could crash Loki.
5536 jiachengxu: Loki mixin: make labelsSelector in loki chunks dashboards configurable
5535 jiachengxu: Loki mixins: use labels selector for loki chunks dashboard
5507 MichelHollands: Remove extra param in call for inflightRequests metric.
5481 MichelHollands: Add a DeletionMode config variable to specify the delete mode and validate match parameters.
5356 jbschami: Enhance lambda-promtail to support adding extra labels from an environment variable value
5409 ldb: Enable best effort parsing for Syslog messages
5392 MichelHollands: Etcd credentials are parsed as secrets instead of plain text now.
5361 ctovena: Add usage report to grafana.com.
5354 tlinhart: Add support for ARM64 to lambda-promtail drone build job.
5289 ctovena: Fix deduplication bug in queries when mutating labels.
5302 MasslessParticle Update azure blobstore client to use new sdk.
5243 ssncferreira: Update Promtail to support duration string formats.
5266 jeschkies: Write Promtail position file atomically on Unix.
5280 jeschkies: Fix Docker target connection loss.
5243 owen-d: moves querier.split-queries-by-interval to limits code only.
5139 DylanGuedes: Drop support for legacy configuration rules format.
5262 MichelHollands: Remove the labelFilter field
4911 jeschkies: Support Docker service discovery in Promtail.
5107 chaudum Fix bug in fluentd plugin that caused log lines containing non UTF-8 characters to be dropped.
5148 chaudum Add periodic task to prune old expired items from the FIFO cache to free up memory.
5187 aknuds1 Rename metric cortex_experimental_features_in_use_total to loki_experimental_features_in_use_total and metric log_messages_total to loki_log_messages_total.
5170 chaudum Fix deadlock in Promtail caused when targets got removed from a target group by the discovery manager.
5163 chaudum Fix regression in fluentd plugin introduced with #5107 that caused NoMethodError when parsing non-string values of log lines.
5144 dannykopping Ruler: fix remote write basic auth credentials.
5091 owen-d: Changes ingester.concurrent-flushes default to 32
5031 liguozhong: Promtail: Add global read rate limiting.
4879 cyriltovena: LogQL: add line function to | line_format template.
5081 SasSwart: Add the option to configure memory ballast for Loki
5085 aknuds1: Upgrade Cortex to e0807c4eb487 and Prometheus to 692a54649ed7
5067 cstyan: Add a metric to Azure Blob Storage client to track total egress bytes
5065 AndreZiviani: lambda-promtail: Add ability to ingest logs from S3
4950 DylanGuedes: Implement common instance addr/net interface
4949 ssncferreira: Add query queueTime metric to statistics and metrics.go
4938 DylanGuedes: Implement ring status page for the distributor
5023 ssncferreira: Move querier.split-queries-by-interval to a per-tenant configuration
4993 thejosephstevens: Fix parent of wal and wal_cleaner in loki ruler config docs
4933 jeschkies: Support matchers in series label values query.
4926 thejosephstevens: Fix comment in Loki module loading for accuracy
4920 chaudum: Add -list-targets command line flag to list all available run targets
4860 cyriltovena: Add rate limiting and metrics to hedging
4865 taisho6339: Fix duplicate registry.MustRegister call in Promtail Kafka
4845 chaudum Return error responses consistently as JSON
4826 cyriltovena: Adds the ability to hedge storage requests.
4785 DylanGuedes: Loki: Print current config by calling /config
4775 jeschkies: Make * and + non-greedy to double regex filter speed.
4769 cyriltovena: Improve LogQL format stages requireLabel
4731 cyriltovena: Improve heap iterators.
4394 cyriltovena: Improve case insensitive search to avoid allocations.

Fixes

5768 slim-bean: Loki: Increase flush_op_timeout default from 10s to 10m
5761 slim-bean: Promtil: Fix a panic when using the loki push api target.
5622 chaudum: Preserve interval parameter when splitting queries by time
5541 bboreham: Queries: update package to reject very deeply nested regexps which could crash Loki
5527 liguozhong: [bugfix] fix nil pointer
5474 cyriltovena: Disable sharding of count/avg when labels are mutated
5472 MasslessParticle: Fix potential deadlock in the table manager
5444 cyriltovena: Do not insert missing point when sharding
5425 cyriltovena: Do not use WaitGroup context for StepEvaluator
5423 cyriltovena: Correctly sets hash value for headblock iterator
5418 RangerCD: Fix two remote_timeout configs in ingester_client block
5413 MasslessParticle: Fix a deadlock in the Azure Blob client
5399 MasslessParticle: Fix Azure issue where 404 not recognized
5362 gotjosh: Ruler: Rule group not found API message
5342 sandeepsukhani: Fix apply retention issue
5334 kavirajk: Makes tailer.droppedStreams slice bounded.
5324 owen-d: Release entryBufferPool once
5303 owen-d: Better logic for when to shard wrt disabled lookback
5298 sandeepsukhani: fix a panic in index-gateway caused by double closing of a channel
5297 vlad-diachenko: Changed logic of handling RPC error with code Cancelled
5289 cyriltovena: Fixes log deduplication when mutating Labels using LogQL
5261 sandeepsukhani: use default retention period to check user index may have expired chunks when user does not have custom retention
5234 RangerCD: Ignore missing stream while querying from ingester
5168 kavirajk: Add nil check for Ruler BasicAuth config.
5144 dannykopping: Ruler: Fix remote write basic auth credentials
5113 kavirajk: Fix cancel issue between Query Frontend and Query Schdeduler
5080 kavirajk: Handle context cancellation in some of the querier downstream requests
5075 cyriltovena: Fixes a possible cancellation issue in the frontend
5063 cyriltovena: Fix deadlock in disconnecting querier
5060 cyriltovena: Fix race conditions in frontend_scheduler_worker.
5006 sandeepsukhani: fix splitting of queries when step is larger than split interval
4904 bboreham: ingester: use consistent set of instances to avoid panic
4902 cyriltovena: Fixes 500 when query is outside of max_query_lookback
4828 chaudum: Set correct Content-Type header in query response
4761 slim-bean: Loki: Set querier worker max concurrent regardless of run configuration.
4741 sandeepsukhani: index cleanup fixes while applying retention

Changes

5544 ssncferreira: Update vectorAggEvaluator to fail for expressions without grouping
5543 cyriltovena: update loki go version to 1.17.8
5450 BenoitKnecht: pkg/ruler/base: Add external_labels option
5522 liguozhong: chunk backend: Integrate Alibaba Cloud oss
5484 sandeepsukhani: Add support for per user index query readiness with limits overrides
5719 kovaxur: Loki can use both basic-auth and tenant-id
5358 DylanGuedes: Add RingMode support to IndexGateway
5435 slim-bean: set match_max_concurrent true by default
5361 cyriltovena: Add usage report into Loki.
5243 owen-d: Refactor/remove global splitby
5229 chaudum: Return early if push payload does not contain data
5217 sandeepsukhani: step align start and end time of the original query while splitting it
5204 trevorwhitney: Default max_outstanding_per_tenant to 2048
5181 sandeepsukhani: align metric queries by step and other queries by split interval
5178 liguozhong: Handle context cancellation in some of the querier store.index-cache-read.
5172 cyriltovena: Avoid splitting large range vector aggregation.
5125 sasagarw: Remove split-queries-by-interval validation
5091 owen-d: better defaults for flush queue parallelism
5083 liguozhong: [enhancement] querier cache: WriteBackCache should be off query path
5081 SasSwart: Add the option to configure memory ballast for Loki
5077 trevorwhitney: improve default config values
5067 cstyan: Add an egress bytes total metric to the azure client.
5026 sandeepsukhani: compactor changes for building per user index files in boltdb shipper
5023 ssncferreira: Move querier.split-queries-by-interval to a per-tenant configuration
5022 owen-d: adds instrumentation to azure object client
4942 cyriltovena: Allow to disable http2 for GCS.
4891 liguozhong: [optimization] cache prometheus : fix "loki_cache_request_duration_seconds_bucket" ‘status_code’ label always equals "200"
4737 owen-d: ensures components with required SRV lookups use the correct port
4736 sandeepsukhani: allow applying retention at different interval than compaction with a config
4656 ssncferreira: Fix dskit/ring metric with 'cortex_' prefix

Promtail

Enhancements

5359 JBSchami: Lambda-promtail: Enhance lambda-promtail to support adding extra labels from an environment variable value
5290 ssncferreira: Update promtail to support duration string formats
5051 liguozhong: [new] promtail pipeline: Promtail Rate Limit stage #5048
5031 liguozhong: [new] promtail: add readline rate limit
4911 jeschkies: Provide Docker target and discovery in Promtail.
4813 cyriltovena: Promtail pull cloudflare logs
4744 cyriltovena: Add GELF support for Promtail.
4663 taisho6339: Add SASL&mTLS authentication support for Kafka in Promtail

Fixes

5497 MasslessParticle: Fix orphaned metrics in the file tailer
5409 ldb: promtail/targets/syslog: Enable best effort parsing for Syslog messages
5246 rsteneteg: Promtail: skip glob search if filetarget path is an existing file and not a directory
5238 littlepangdi: Promtail: fix TargetManager.run() not exit after stop is called
4874 Alan01252: Promtail: Fix replace missing adjacent capture groups
4832 taisho6339: Use http prefix path correctly in promtail
4716 cyriltovena: Fixes Promtail User-Agent.
5698 paullryan: Promtail: Fix retry/stop when erroring for out of cloudflare retention range (e.g. over 168 hours old)

Changes

5377 slim-bean: Promtail: Remove promtail_log_entries_bytes_bucket histogram
5266 jeschkies: Write Promtail position file atomically.
4794 taisho6339: Aggregate inotify watcher to file target manager
4745 taisho6339: Expose Kafka message key in labels

Logcli

5477 atomic77: logcli: Remove port from TLS server name when provided in --addr
4667 jeschkies: Package logcli as rpm and deb.
4606 kavirajk: Execute Loki queries on raw log data piped to stdin

Lambda-Promtail

5065 AndreZiviani: lambda-promtail: Add ability to ingest logs from S3
7632 changhyuni: lambda-promtail: Add kinesis data stream to use in terraform

Fluent Bit

5223 cyriltovena: fluent-bit: Attempt to unmarshal nested json.

FluentD

6240 taharah: Add the feature flag include_thread_label to allow the fluentd_thread label included when using multiple threads for flushing to be configurable
5107 chaudum: fluentd: Fix bug that caused lines to be dropped when containing non utf-8 characters
5163 chaudum: Fix encoding error in fluentd client

Notes

This release was created from a branch starting at commit 614912181e6f3988b2b22791053278cfb64e169c but it may also contain backported changes from main.

Check the history of the branch release-2.5.x.

Dependencies

Go Version: 1.17.8

2.4.1 (2021/11/07)

Release notes for 2.4.1 can be found on the release notes page

All Changes

4687 owen-d: overrides checks for nil tenant limits on AllByUserID
4683 owen-d: Adds replication_factor doc to common config
4681 slim-bean: Loki: check new Read target when initializing boltdb-shipper store

2.4.0 (2021/11/05)

Release notes for 2.4.0 can be found on the release notes page

All Changes

Here is a list of all changes included in 2.4.0.

Loki

4649 cstyan: Instrument s3 client DeleteObject requests.
4643 trevorwhitney: compactor depends on memberlist for memberlist ring option
4642 slim-bean: Loki: fix handling of tail requests when using target all or read
4641 ssncferreira: Migration to dskit/ring
4638 DylanGuedes: Loki: Revert distributor defaulting to inmemory
4635 owen-d: dont try to use the scheduler ring when a downstream url is configured
4630 chaudum: Allow HTTP POST requests on ring pages
4627 slim-bean: Loki: Explicitly define allowed HTTP methods on HTTP endpoints
4625 sandeepsukhani: Logs deletion fixes
4617 trevorwhitney: Add common ring configuration
4615 owen-d: uses ring.Write instead of ring.WriteNoExtend for compactor ring checks
4614 slim-bean: Loki: query scheduler should send shutdown to frontends when ReplicationSet changes
4608 trevorwhitney: default ingester final sleep to 0 unless otherwise specified
4607 owen-d: improves scheduler & compactor ringwatcher checks
4603 garrettlish: add date time sprig template functions in logql label/line formatter
4598 kavirajk: Fix ip matcher lexer to differentiate filter from identifier
4596 owen-d: Ignore validity window during wal replay
4595 owen-d: Cleans up redundant setting of stream.unorderedWrites=true during replay
4594 owen-d: Enable unordered_writes by default
4593 taisho6339: Respect gRPC context error when handling errors
4592 owen-d: introduces "entry too far behind" instrumentation for unordered writes
4589 owen-d: replaces fallthrough statement in InitFrontend
4586 dannykopping: Configuring query-frontend interface names with loopback device
4585 sandeepsukhani: set wal dir to /loki/wal in docker config
4577 taisho6339: Respect shard number in series api
4574 slim-bean: Loki: Add a ring to the compactor used to control concurrency when not running standalone
4573 sandeepsukhani: validate default limits config with other configs at startup
4570 DylanGuedes: Loki: Append loopback to ingester net interface default list
4569 DylanGuedes: Config: Change default RejectOldSamplesMaxAge from 14d to 7d
4563 cyriltovena: Fixes the Series function to handle properly sharding.
4554 cyriltovena: Fixes a panic in the labels API when no parameters are supplied.
4550 cyriltovena: Fixes an edge case in the batch chunk iterator.
4546 slim-bean: Loki: Apply the ingester ring config to all other rings (distributor, ruler, query-scheduler)
4545 trevorwhitney: Fix race condition in Query Scheduler ring with frontend/worker
4543 trevorwhitney: Change a few default config values and improve application of common storage config
4542 owen-d: only exports tenant limits which differ from defaults and export defa…
4531 JordanRushing: Add quick nil check in TenantLimits for runtime_config
4529 owen-d: correctly sets subservicesWatcher on scheduler
4525 owen-d: Safely checks read ring for potentially nil scheduler
4524 dannykopping: Clarify error message when no valid target scrape config is defined for promtail job
4520 JordanRushing: Introduce overrides-exporter module to Loki
4519 DylanGuedes: Loki: Enable FIFO cache by default
4518 slim-bean: Loki: Fix bug where items are returned to a sync.Pool incorrectly
4510 lingpeng0314: add group_{left,right} to LogQL
4508 trevorwhitney: Apply better defaults when boltdb shipper is being used
4498 trevorwhitney: Feature: add virtual read and write targets
4487 cstyan: Update go.mod to go 1.17
4484 dannykopping: Replacing go-kit/kit/log with go-kit/log
4482 owen-d: always expose loki_build_info
4479 owen-d: restores for state at seconds(now-forDuration)
4478 replay: Update cortex to newer version
4473 trevorwhitney: Configuration: add a common config section for object storage
4457 kavirajk: Fix return values of Matrix and Vector during query range in QueryShardingMiddleware
4453 liguozhong: [querier] s3: add getObject retry
4446 garrettlish: make LogQL syntax scope from private to public
4443 DylanGuedes: Loki: Change how push API checks for contentType
4440 DylanGuedes: Loki: Override distributor's default ring KV store
4437 dannykopping: Ruler: Do not clear remote-write HTTP client config
4436 JordanRushing: Add metric prefix changes for chunk store and runtime config to upgrading.md
4435 trevorwhitney: Change default values for two GRPC setting we have to set so the queriers can connect to a frontend or scheduler
4433 trevorwhitney: Add more tests around config parsing changes from common config PR
4432 owen-d: tests checkpoints immediately and gives more of a time buffer
4431 dannykopping: Ruler: Overwrite instead of merge remote-write headers
4429 dannykopping: Ruler: Refactoring remote-write config overrides
4424 slim-bean: Loki: Add a ring to the query scheduler to allow discovery via the ring as an alternative to DNS
4421 owen-d: Safe per tenant overrides loading
4415 DylanGuedes: Loki: Change default limits to common values
4413 trevorwhitney: add compactor working dir to auto-configured file paths
4411 slim-bean: Loki: Bug: frontend waiting on results which would never come
4400 trevorwhitney: auto-apply memberlist ring config when join_members provided
4391 garrettlish: add on and ignoring clauses in binOpExpr
4388 trevorwhitney: default chunk target size to ~~1MB~~ 1.5MB
4367 owen-d: removes deprecated duplicate per stream rate limit fields
4364 dannykopping: Ruler: improve control over marshaling relabel.Config
4354 dannykopping: Ruler: adding pkg/metrics from agent
4349 JordanRushing: Add recovery middleware to Ingester; re-add recovery middleware to Querier when not running in standalone mode
4348 trevorwhitney: allow ingester and distributor to run on same instance
4347 slim-bean: Loki: Common Config
4344 dannykopping: Ruler: per-tenant WAL
4327 aknuds1: Chore: Use dskit/limiter
4322 owen-d: Hotfix #4308 into k62
4321 owen-d: Hotfix #4308 into k61
4313 aknuds1: Chore: Use middleware package from dskit
4312 aknuds1: Chore: Use dskit/grpcclient
4308 cyriltovena: Fixes the pattern parser validation.
4304 aknuds1: Chore: Reformat Go files
4302 cyriltovena: Fixes a bug in the block cache code.
4301 trevorwhitney: Feature: allow querier and query frontend targets to run on same process
4295 aknuds1: Chore: Upgrade dskit
4289 kavirajk: Add custom UnmarshalJSON for bytesize type
4282 chaudum: Chore: Update Cortex and use kv package from grafana/dskit
4276 chaudum: Export MemberlistKV field on Loki struct
4272 taisho6339: Add count to 'loki_ingester_memory_chunks' when recovery from wal
4265 owen-d: remove empty streams after wal replay
4255 owen-d: replaces old cortex_chunk_store prefix with loki_chunk_store
4253 JordanRushing: Change prefix for runtimeconfig metrics from cortex_ to loki_
4251 dannykopping: Runtime config: do not validate nil limits
4246 JordanRushing: Add missing Inc() to correctly increment the dropStage.dropCount metric on valid dropped log line; update related docs
4240 bboreham: Simplify Distributor.push
4238 liguozhong: [fix] distributor: fix goroutine leak
4236 owen-d: better per stream rate limits configuration options
4228 owen-d: bumps per stream default rate limits
4227 aknuds1: Chore: Use runtimeconfig from dskit
4225 aknuds1: Flagext: Use flagext package from dskit
4213 owen-d: Refactor per stream rate limit
4212 owen-d: WAL replay discard metrics
4211 BenoitKnecht: pkg/storage/chunk/aws: Add s3.http.ca-file option
4207 cstyan: Improve error message for stream rate limit.
4196 56quarters: Chore: Use services and modules from grafana/dskit
4193 owen-d: adds loki_ingester_wal_replay_active metric and records this more acc…
4192 owen-d: Cleanup/unordered writes ingester config
4191 cstyan: [ingester/stream]: Add a byte stream rate limit.
4188 aknuds1: Chore: Upgrade to latest Cortex
4185 sandeepsukhani: Canary: allow setting tenant id for querying logs from loki
4181 owen-d: initiate grpc health check always
4176 sokoide: Authc/z: Enable grpc_client_config to allow mTLS
4172 sandeepsukhani: Retention speedup
4160 owen-d: safely close nonOverlapping iterators
4155 owen-d: Auth followup - Remove unused
4153 owen-d: uses more fleshed out cortex auth utility & adds new auth-ignored routes
4149 owen-d: add unordered writes to local config
4141 dannykopping: Ruler: write meaningful logs when remote-write is disabled or is misconfigured
4135 slim-bean: Build: Fix build version info
4132 owen-d: Promote/ruler api
4130 owen-d: Tenant/unordered
4128 sandeepsukhani: add a storage client for boltdb-shipper which would do all the object key management for storage operations
4126 cstyan: Allow for loki-canary to generate a percentage of out of order log lines
4114 owen-d: Stream iterators account for unordered data
4111 owen-d: ingester.index-shards config
4107 sandeepsukhani: fix finding tables which would have out of retention data
4104 owen-d: Discard/ooo
4071 jeschkies: Support frontend V2 with query scheduler.

Promtail

4599 rsteneteg: [Promtail] resolve issue with promtail not scraping target if only path changed in a simpler way that dont need mutex to sync threads
4588 owen-d: regenerates assets from current vfsgen dependency
4568 cyriltovena: Promtail Kafka target
4567 cyriltovena: Refactor client configs in Promtail.
4556 james-callahan: promtail: no need for GCP promtail_instance label now that loki supports out-of-order writes
4516 lizzzcai: promtail: update promtail base image to debian:bullseye-slim
4507 dannykopping: Promtail: allow for customisable stream lag labels
4495 sankalp-r: Promtail: add static labels in stage
4461 rsteneteg: Promtail: fix filetarget to not be stuck if no files was detected on startup
4346 sandeepsukhani: add logfmt promtail stage to be able to extract data from logfmt formatted log
4336 ldb: clients/promtail: Add ndjson and plaintext formats to loki_push
4235 kavirajk: Add metrics for gcplog scrape.
3907 johanfleury: promtail: add support for TLS/mTLS in syslog receiver

Logcli

4303 cyriltovena: Allow to run local boltdb queries with logcli.
4242 chaudum: cli: Register configuration option store.max-look-back-period as CLI argument
4203 invidian: cmd/logcli: add --follow flag as an alias for --tail

Build

4639 slim-bean: Build: simplify how protos are built
4609 slim-bean: Build: Update CODEOWNERS to put Karen back in charge of the docs
4541 cstyan: Fix drone ECR publish.
4481 cstyan: Update golang and loki-build-image image versions.
4480 cstyan: Add drone build job for lambda-promtail images.
4462 cstyan: Update loki-build-image to drone 1.4.0
4373 jeschkies: Instruct how to sign drone.yml.
4358 JordanRushing: Add DroneCI pipeline stage to validate loki example configs; create example configuration files
4353 dannykopping: CI: Fixing linter deprecations
4286 slim-bean: Build: Tweak stalebot message
4252 slim-bean: Build: update stalebot message to be more descriptive and friendlier
4226 aknuds1: Makefile: Add format target
4220 slim-bean: Build: Add github action backport workflow
4189 mathew-fleisch: Makefile: Add darwin/arm64 build to release binaries

Project

4535 carlpett: Fix branch reference in PR template
4604 kavirajk: Update PR template to include changelog update in the checklist
4494 cstyan: Add a a parameter to keep/drop the stream label from cloudwatch.
4315 cstyan: Rewrite lambda-promtail to use subscription filters.

Dashboards

4634 cyriltovena: Fixes the operational dashboard using an old metric.
4618 cstyan: loki-mixin: fix label selectors + logs dashboard
4575 dannykopping: Adding recording rules dashboard
4441 owen-d: Revert "loki-mixin: use centralized configuration for dashboard matchers / selectors"
4438 dannykopping: Dashboards: adding "logs" into regex
4423 cstyan: Add tag/link fix to operational dashboard and promtail mixin dashboard.
4401 cstyan: Minor dashboard fixes

Docker-driver

4396 owen-d: Removes docker driver empty log line message
4190 jeschkies: Document known Docker driver issues.

FluentD

4261 MrWong99: FluentD output plugin: Remove an unused variable when processing chunks

Docs

4646 KMiller-Grafana: Docs: revise modes of operation section
4631 kavirajk: Add changelog and upgrade guide for #4556
4616 owen-d: index-gw sts doc fix. closes #4583
4612 surdaft: Docs: Fix typo in docs
4611 KMiller-Grafana: Docs: revise incendiary language added in PR 4507
4601 mustafacansevinc: docs: fix promtail docs links in loki installation page
4597 owen-d: a few doc fixes in preparation for 2.4
4590 owen-d: improves grouping docs examples
4579 DylanGuedes: Docs: Modify modes of operation image
4576 DylanGuedes: Rename hybrid mode to simple scalable mode
4566 dannykopping: Documenting recording rules per-tenant WAL
4565 DylanGuedes: Docs: Add virtual targets docs
4559 chri2547: docs: Update curl POST example in docs
4548 cstyan: Improve lambda-promtail docs based on Owens review.
4540 JordanRushing: Update CHANGELOG.md and /docs with info on new overrides-exporter module for Loki
4539 cstyan: Modify lambda-promtail docs based on rewrite.
4527 yangkb09: Docs: add missing quote to log_queries.md
4521 owen-d: brings storage architecture up to date
4499 vdm: Docs: Remove ListObjects S3 permission
4493 DylanGuedes: Docs: Move rule storages configs to their own sections
4486 KMiller-Grafana: Docs: correct the page parameter in the Grafana Cloud advertisement
4485 DylanGuedes: Document the common config section
4422 KMiller-Grafana: Docs: revise wording of Grafana Cloud advertisement
4417 KMiller-Grafana: Docs: remove empty section "Generic placeholders"
4416 KMiller-Grafana: Docs: correctly represent product name
4403 KMiller-Grafana: Docs: introduce a fundamentals section
4399 KMiller-Grafana: Docs: prominently advertise free Grafana Cloud availability
4374 KMiller-Grafana: Docs: clarify distinction between single binary and microservices.
4363 KMiller-Grafana: Docs: Remove wording like "As of version 1.6, you can..."
4361 JasonGiedymin: fix(docs): spelling mistake
4357 carehart: Correct typo
4345 pr0PM: Deduplicating the compactor docs
4342 KMiller-Grafana: Docs: Organize and edit the LogQL section
4324 lingenavd: Docs: Update _index.md to add value boltdb-shipper for the key store
4320 KMiller-Grafana: Docs: improve spelling, grammar, and formatting.
4310 dannykopping: Correcting documentation example for /api/prom/query
4309 GneyHabub: Docs: Fix a link
4294 mr-karan: docs: (logs-deletion.md) URL Encode curl command
4293 Birdi7: docs: fix link to Promtail documentation
4283 SeriousM: Correct the indention for azure storage configuration
4277 ivanahuckova: Update example for /series endpoint in _index.md
4247 KMiller-Grafana: Docs: inject newlines for configuration section readability
4245 KMiller-Grafana: Docs: revise max_query_lookback knob definition
4244 JordanRushing: Update limits_config docs to include querier.max_query_lookback flag
4237 KMiller-Grafana: Docs: first draft, Loki accepts out-of-order writes
4231 Aletor93: doc: fix typo on loki-external-labels for docker client labels
4222 KMiller-Grafana: Docs: minor improvements to Loki Canary docs
4208 cstyan: Update tanka installation docs to refer to tanka section about jb
4206 jeschkies: Link Kubernetes service discovery configuration.
4199 owen-d: fixes typo
4184 mcdeck: Update docker.md
4175 KMiller-Grafana: Docs: correct path to Promtail configuration file
4163 smuth4: Docs: Update docker install to work out of the box
4152 charles-woshicai: Docs: example about using azure storage account as storage
4147 KMiller-Grafana: Docs: fluentd client phrasing and formatting
4145 KMiller-Grafana: Docs: improve LogQL section
4134 KMiller-Grafana: Docs: revise section header (out of order writes)
4131 owen-d: updates unordered writes config docs
4125 owen-d: Initial out of order docs
4122 yasharne: update boltdb-shipper index period
4120 vitaliyf: Docs: Fix broken "Upgrading" link
4113 KMiller-Grafana: Docs: Fix typos and grammar. Inject newlines for readability.
4112 slim-bean: Docs: updated changelog and references to 2.3
4100 jeschkies: Document operation with the query scheduler.
4088 KMiller-Grafana: Update Loki README with better links and descriptions
3880 timothydlister: Update fluent-plugin-loki documentation URLs

Jsonnet

4629 owen-d: Default wal to enabled in jsonnet lib
4624 chaudum: Disable chunk transfers in jsonnet lib
4530 owen-d: Jsonnet/overrides exporter
4496 jeschkies: Use different metrics for PromtailFileLagging.
4405 jdbaldry: fix: Correct grafana-token creation command
4279 kevinschoonover: loki-mixin: use centralized configuration for dashboard matchers / selectors
4259 eamonryan: Jsonnet: Update license path argument name
4217 Duologic: fix(rules): upstream recording rule switched to sum_irate
4182 owen-d: fine tune grpc configs jsonnet
4180 owen-d: corrects query scheduler image
4165 jdbaldry: Jsonnet: Add Grafana Enterprise Logs library
4154 owen-d: updates scheduler libsonnet
4102 jeschkies: Define ksonnet lib for query scheduler.

Notes

This release was created from a branch starting at commit e95d193acf1633a6ec33a328b8a4a3d844e8e5f9 but it may also contain backported changes from main.

Check the history of the branch release-2.4.

Dependencies

Go Version: 1.17.2
Cortex Version: 3f329a21cad432325268717eecf2b77c8d95150f

2.3.0 (2021/08/06)

Release notes for 2.3.0 can be found on the release notes page

All Changes

Loki

4048 dannykopping: Ruler: implementing write relabelling on recording rule samples
4091 cyriltovena: Fixes instant queries in the frontend.
4087 cyriltovena: Fixes unaligned shards between ingesters and storage.
4047 cyriltovena: Add min_sharding_lookback limits to the frontends
4027 jdbaldry: fix: Restore /config endpoint and correct handlerFunc for buildinfo
4020 simonswine: Restrict path segments in TenantIDs (CVE-2021-36156 CVE-2021-36157)
4019 cyriltovena: Improve decoding of JSON responses.
4018 sandeepsukhani: Compactor improvements
4017 aknuds1: Chore: Upgrade Prometheus and Cortex
3996 owen-d: fixes a badly referenced variable name in StepEvaluator code
3995 owen-d: Headblock interop
3992 MichelHollands: Update Cortex version
3991 periklis: Add LogQL AST walker
3990 cyriltovena: Intern label keys for LogQL parser.
3986 kavirajk: Ip matcher for LogQL
3984 jeschkies: Filter instant queries and shard them.
3983 cyriltovena: Reject labels with invalid runes when using implicit extraction parser.
3981 owen-d: fixes chunk size method in facade
3979 MichelHollands: Add a chunk filterer field to the config
3977 sandeepsukhani: add a metric for counting number of failures in opening existing active index files
3976 sandeepsukhani: fix flaky retention tests
3974 owen-d: WAL Replay counter
3973 56quarters: Use the Cortex wrapper for getting tenant ID from a context
3972 jeschkies: Return build info under /loki/api/v1/status/buildinfo.
3970 sandeepsukhani: log name of the file failed to open during startup by ingester
3969 sandeepsukhani: add some tests in compactor and fix a bug in IntervalHasExpiredChunks check in retention with tests
3968 cyriltovena: Improve head chunk allocations when reading samples.
3967 sandeepsukhani: fix a panic in compactor when retention is not enabled
3966 sandeepsukhani: fix panic in compactor when retention is not enabled
3957 owen-d: Unordered head block
3949 cyriltovena: Allow no overrides config for tenants.
3946 cyriltovena: Improve marker file current time metrics.
3934 sandeepsukhani: optimize table retetion
3932 Timbus: Parser: Allow literal control chars in logfmt decoder
3929 sandeepsukhani: remove boltdb files from ingesters on startup which do not have a index bucket
3928 dannykopping: Querier/Ingester: Fixing json expression parser bug
3919 github-vincent-miszczak: Add ingester.autoforget-unhealthy-timeout opt-in feature
3888 kavirajk: Make overrides configmap names and mount path as variables.
3871 kavirajk: Add explict syntax for using pattern parser
3865 sandeepsukhani: feat: index-gateway for boltdb-shipper index store
3856 cyriltovena: Shards Series API.
3852 cyriltovena: Shard ingester queries.
3849 cyriltovena: Logs ingester and store queries boundaries.
3840 cyriltovena: Add retention label to loki_distributor_bytes_received_total metrics
3837 cyriltovena: LogQL: Pattern Parser
3835 sesky4: lz4: update lz4 version to v4.1.7 to avoid possibly panic
3833 cyriltovena: Fixes a flaky retention test.
3827 sandeepsukhani: Logs deletion fixes
3816 dannykopping: Extracting queue interface
3807 dannykopping: Loki: allow for multiple targets
3797 dannykopping: Exposing remote writer for use in integration tests
3792 MichelHollands: Add a QueryFrontendTripperware module
3785 sandeepsukhani: just log a warning when a store type other than boltdb-shipper is detected when custom retention is enabled
3772 sandeepsukhani: initialize retention and deletion components only when they are enabled
3771 sandeepsukhani: revendor cortex to latest master
3769 sandeepsukhani: reduce allocs in delete requests manager by reusing slice for tracing non-deleted intervals for chunks
3766 dannykopping: Ruler: Recording Rules
3763 cyriltovena: Fixes parser labels hint for grouping.
3762 cyriltovena: Improve mark file processing.
3758 owen-d: exposes loki codec
3746 sandeepsukhani: Boltdb shipper deletion fixes
3743 cyriltovena: Replace satori.uuid with gofrs/uuid
3736 cyriltovena: Add fromJson to the template stage.
3733 cyriltovena: Fixes a goroutine leak in the store when doing cancellation.
3706 cyriltovena: Improve retention mark files.
3700 slim-bean: Loki: Add a flag for queriers to run standalone and only query store
3693 cyriltovena: Removes file sync syscall for compaction.
3688 sandeepsukhani: Logs deletion support for boltdb-shipper store
3687 cyriltovena: Use model.Duration for easy yaml/json marshalling.
3686 cyriltovena: Fixes a panic with the frontend when use with downstream URL.
3677 cyriltovena: Deprecate max_look_back_period in the chunk storage.
3673 cyriltovena: Pass in the now value to the retention.
3672 cyriltovena: Use pgzip in the compactor.
3665 cyriltovena: Trigger compaction prior retention.
3664 owen-d: revendor compatibility: various prom+k8s+cortex
3643 cyriltovena: Rejects push requests with streams without labels.
3642 cyriltovena: Custom Retention
3641 owen-d: removes naming collision
3632 kavirajk: replace time.Duration -> model.Duration for Limits.
3628 kavirajk: Add json struct tags to limits.
3627 MichelHollands: Update cortex to 1.8
3623 slim-bean: Loki/Promtail: Client Refactor
3619 liguozhong: [ui] add '/config' page
3618 MichelHollands: Add interceptor override and make ingester and cfg public
3605 sandeepsukhani: cleanup boltdb files failing to open during loading tables which are possibly corrupt
3603 cyriltovena: Adds chunk filter hook for ingesters.
3602 MichelHollands: Loli: Make the store field public
3595 owen-d: locks trailers during iteration
3594 owen-d: adds distributor replication factor metric
3573 cyriltovena: Fixes a race when using specific tenant and multi-client.
3569 cyriltovena: Add a chunk filter hook in the store.
3566 cyriltovena: Properly release the ticker in Loki client.
3564 cyriltovena: Improve matchers validations.
3563 sandeepsukhani: ignore download of missing boltdb files possibly removed during compaction
3562 cyriltovena: Fixes a test from #3216.
3553 cyriltovena: Add a target to reproduce fuzz testcase
3550 cyriltovena: Fixes a bug in MatrixStepper when sharding queries.
3549 MichelHollands: LBAC changes
3544 alrs: single import of jsoniter in logql subpackages
3540 cyriltovena: Support for single step metric query.
3532 MichelHollands: Loki: Update cortex version and fix resulting changes
3530 sandeepsukhani: split series api queries by day in query-frontend
3517 cyriltovena: Fixes a race introduced by #3434.
3515 cyriltovena: Add sprig text/template functions to template stage.
3509 sandeepsukhani: fix live tailing of logs from Loki
3572 slim-bean: Loki: Distributor log message bodySize should always reflect the compressed size
3496 owen-d: reduce replay flush threshold
3491 sandeepsukhani: make prefix for keys of objects created by boltdb-shipper configurable
3487 cyriltovena: Set the byte slice cap correctly when unsafely converting string.
3471 cyriltovena: Set a max size for the logql parser to 5k.
3470 cyriltovena: Fixes Issue 28593: loki:fuzz_parse_expr: Timeout in fuzz_parse_expr.
3469 cyriltovena: Fixes out-of-memory fuzzing issue.
3466 pracucci: Upgrade Cortex
3455 garrettlish: Implement offset modifier for range vector aggregation in LogQL
3434 adityacs: support math functions in line_format and label_format
3216 sandeepsukhani: check for stream selectors to have atleast one equality matcher
3050 cyriltovena: first_over_time and last_over_time

Docs

4031 KMiller-Grafana: Docs: add weights to YAML metadata to order the LogQL subsections
4029 bearice: Docs: Update S3 permissions list
4026 KMiller-Grafana: Docs: correct fluentbit config value for DqueSync
4024 KMiller-Grafana: Docs: fix bad links
4016 lizzzcai: :fix typo in remote debugging docs
4012 KMiller-Grafana: Revise portions of the docs LogQL section
3998 owen-d: Fixes regexReplaceAll docs
3980 KMiller-Grafana: Docs: Revise/update the overview section.
3965 mamil: fix typos
3962 KMiller-Grafana: Docs: added new target (docs-next) to the docs' Makefile.
3956 sandeepsukhani: add config and documentation about index-gateway
3938 seiffert: Doc: List 'compactor' as valid value for target option
3936 lukahartwig: Fix typo
3921 KMiller-Grafana: Docs: revise the LogCLI subsection
3911 KMiller-Grafana: Docs: Make identification of experimental items consistent and obvious
3910 KMiller-Grafana: Docs: add structure for a release notes section
3909 kavirajk: Sync main branch docs to next folder
3899 KMiller-Grafana: Docs: correct “ and ” with " and same with single quote mark.
3897 kavirajk: Update steps to release versioned docs
3882 KMiller-Grafana: Docs: improve section on building from source
3876 ivanahuckova: Documentation: Unify spelling of backtick in documentation
3873 KMiller-Grafana: Docs: remove duplicated arch info from the overview section
3875 kavirajk: Add missing -querier.max-concurrent config in the doc
3868 sanadhis: docs: http_path_prefix as correct item of server_config
3860 KMiller-Grafana: Docs: Correct capitalization and formatting of "Promtail"
3851 dannykopping: Ruler: documentation for recording rules
3846 crockk: Docs: Minor syntax tweaks for consistency
3843 azuwis: multiline: Add regex stage example and note
3829 arempter: Add oauth2 docs options for promtail client
3828 julienduchesne: Fix broken link in Windows Event Log scraping docs
3826 sandeepsukhani: docs for logs deletion feature
3824 KMiller-Grafana: Docs: add and order missing design docs
3823 KMiller-Grafana: Docs: updates
3815 paketb0te: Docs: fixed typo in "Loki compared to other log systems" (levels -> labels)
3810 alegmal: documentation: corrected double "the the" in index.md
3799 bt909: docs: Add memached_client parameter "addresses" list
3798 bt909: docs: Change redis configuration value for enabling TLS to correct syntax
3790 KMiller-Grafana: Docs: remove unnecessary lists of sections
3775 cyriltovena: Retention doc
3764 slim-bean: Docs: fix makefile
3757 fionaliao: [docs] Remove unnecessary backtick from example
3756 fredrikekre: [docs] add LokiLogger.jl to unofficial clients.
3723 oddlittlebird: Docs: Update _index.md
3720 fredrikekre: [docs/clients] fix header for "Unofficial clients" and add a reference to said section.
3715 jaddqiu: Update troubleshooting.md
3714 kavirajk: Fluent-bit git repo link fix
3713 cyriltovena: Add a target to find dead links in our documentation.
3690 atxviking: API Documentation: Fix document links for /loki/api/v1/push example
3655 trevorwhitney: Documentation: add note about wildcard log patterns and log rotation
3648 Ruppsn: Update labels.md in Loki Docs
3647 3Xpl0it3r: fix the promtail-default-config download link in doc
3644 periklis: Add Red Hat to adopters
3633 osg-grafana: Fix wget link.
3596 timazet: documentation: typo correction
3578 liguozhong: [doc] mtric -> metric
3576 sergeykranga: Promtail documentation: fix template example for regexReplaceAll function
3568 MichelHollands: docs: some small docs fixes
3559 klausenbusk: Doc: Remove removed --ingester.recover-from-wal option and fix out-of-date defaults
3555 samjewell: LogQL Docs: Remove key-value pair missing from logfmt output
3552 lkokila: Update README.md
3551 cyriltovena: Fixes doc w/r/t grpc compression.
3542 kavirajk: Remove memberlist config from ring config.
3529 Whyeasy: Added docs for GCP internal labels.
3525 robbymilo: docs: add title to Lambda Promtail
3516 cyriltovena: Fixes broken link in the documentation.
3513 owen-d: fixes broken link
3543 owen-d: compactor docs
3526 wardbekker: Added Architecture Diagram
3518 wardbekker: fix spelling in doc
3503 cyriltovena: Update README.md
3484 thomasrockhu: Add Codecov badge to README
3478 chancez: docs/upgrading: Fix typo
3477 slim-bean: Jsonnet/Docs: update for 2.2 release
3472 aronisstav: Docs: Fix markdown for promtail's output stage
3464 camilleryr: Documentation: Update boltdb-shipper.md to fix typo
3442 owen-d: adds deprecation notice for chunk transfers
3430 kavirajk: doc(gcplog): Add note on scraping multiple GCP projects

Promtail

4011 dannykopping: Promtail: adding pipeline stage inspector
4006 dannykopping: Promtail: output timestamp with nanosecond precision in dry-run mode
3971 cyriltovena: Fixes negative gauge in Promtail.
3834 trevorwhitney: Promtail: add consul agent service discovery
3711 3Xpl0it3r: add debug information for extracted data
3683 kbudde: promtail: added timezone to logger in dry-run mode #3679"
3654 cyriltovena: Adds the ability to provide a tripperware to Promtail client.
3587 rsteneteg: Promtail: Remove non-ready filemanager targets
3501 kavirajk: Add unique promtail_instance id to labels for gcptarget
3457 nmiculinic: Promtail: Added path information to deleted tailed file
3400 adityacs: support max_message_length configuration for syslog parser

Logcli

3879 vyzigold: logcli: Add retries to unsuccessful log queries
3749 dbluxo: logcli: add support for bearer token authentication
3739 rsteneteg: correct logcli instant query timestamp param name
3678 cyriltovena: Add the ability to wrap the roundtripper of the logcli client.

Build

4034 aknuds1: loki-build-image: Fix building
4028 aknuds1: loki-build-image: Upgrade golangci-lint and Go
4007 dannykopping: Adding @grafana/loki-team as default CODEOWNERS
3997 owen-d: aligns rule path in docker img with bundled config. closes #3952
3950 julienduchesne: Sign drone.yml file
3944 jeschkies: Lint script files.
3941 cyriltovena: Development Docker Compose Setup
3935 ecraven: Makefile: Only set PROMTAIL_CGO if CGO_ENABLED is not 0.
3832 julienduchesne: Add step to identify windows Drone runner
3731 cyriltovena: Fix website branch to trigger update.
3708 julienduchesne: Deploy loki with Drone plugin instead of CircleCI job
3703 darkn3rd: Update docker.md for 2.2.1
3625 slim-bean: Build: Update CI for branch rename to main
3624 slim-bean: Build: Fix drone dependencies on manifest step
3615 slim-bean: Remove codecov
3481 slim-bean: Update Go and Alpine versions

Jsonnet

4030 cyriltovena: Improve the sweep lag panel in the retention dashboard.
3917 jvrplmlmn: refactor(production/ksonnet): Remove kausal from the root element
3893 sandeepsukhani: update uid of loki-deletion dashboard
3891 sandeepsukhani: add index-gateway to reads and reads-resources dashboards
3877 sandeepsukhani: Fix jsonnet for index-gateway
3854 cyriltovena: Fixes Loki reads dashboard.
3848 kavirajk: Add explicit main to pull loki and promtail to install it via Tanka
3794 sandeepsukhani: add a dashboard for log deletion requests in loki
3697 owen-d: better operational dashboard annotations via diff logger
3658 cyriltovena: Add a dashboard for retention to the loki-mixin.
3601 owen-d: Dashboard/fix operational vars
3584 sandeepsukhani: add loki resource usage dashboard for read and write path

Project

3963 rfratto: Remove Robert Fratto from list of team members
3926 cyriltovena: Add Danny Kopping to the Loki Team.
3732 dannykopping: Issue Templates: Improve wording and add warnings
3722 oddlittlebird: Update CODEOWNERS
3951 owen-d: update sizing calc
3931 owen-d: Hackathon/cluster
3920 owen-d: adds replication & deduping into cost
3630 slim-bean: Re-license to AGPLv3

Docker-driver

3814 kavirajk: Update the docker-driver doc about default labels
3727 3Xpl0it3r: docker-driver: remove duplicated code
3709 cyriltovena: Fixes docker driver that would panic when closed.

Notes

This release was created from revision 8012362674568379a3871ff8c4a2bfd1ddba7ad1 (Which was PR 3460)

Dependencies

Go Version: 1.16.2
Cortex Version: 485474c9afb2614fb89af3f48803c37d016bbaed

2.2.1 (2021/04/05)

2.2.1 fixes several important bugs, it is recommended everyone running 2.2.0 upgrade to 2.2.1

2.2.1 also adds the labelallow pipeline stage in Promtail which lets an allowlist be created for what labels will be sent by Promtail to Loki.

3468 adityacs: Support labelallow stage in Promtail
3502 cyriltovena: Fixes a bug where unpack would mutate log line.
3540 cyriltovena: Support for single step metric query.
3550 cyriltovena: Fixes a bug in MatrixStepper when sharding queries.
3566 cyriltovena: Properly release the ticker in Loki client.
3573 cyriltovena: Fixes a race when using specific tenant and multi-client.

2.2.0 (2021/03/10)

With over 200 PR's 2.2 includes significant features, performance improvements, and bug fixes!

The most upvoted issue for Loki was closed in this release! Issue 74 requesting support for handling multi-line logs in Promtail was implemented in PR 3024. Thanks @jeschkies!

Other exciting news for Promtail, PR 3246 by @cyriltovena introduces support for reading Windows Events!

Switching to Loki, @owen-d has added a write ahead log to Loki! PR 2981 was the first of many as we have spent the last several months using and abusing our write ahead logs to flush out any bugs!

A significant number of the PR's in this release have gone to improving the features introduced in Loki 2.0. @cyriltovena overhauled the JSON parser in PR 3163 (and a few other PR's), to provide both a faster and smarter parsing to only extract JSON content which is used in the query output. The newest Loki squad member @dannykopping fine tuned the JSON parser options in PR 3280 allowing you to specific individual JSON elements, including support now for accessing elements in an array. Many, many other additional improvements have been made, as well as several fixes to the new LogQL features added some months ago, this upgrade should have everyone seeing improvements in their queries.

@cyriltovena also set his PPROF skills loose on the Loki write path which resulted in about 8x less memory usage on our distributors and a much more stable memory usage when ingesters are flushing a lot of chunks at the same time.

There are many other noteworthy additions and fixes, too many to list, but we should call out one more feature all you Google Cloud Platform users might be excited about: in PR 3083 @kavirajk added support to Promtail for listening on Google Pub/Sub topics, letting you setup log sinks for your GCP logs to be ingested by Promtail and sent to Loki!

Thanks to everyone for another exciting Loki release!!

Please read the Upgrade Guide before upgrading for a smooth experience.

TL;DR Loki 2.2 changes the internal chunk format which limits what versions you can downgrade to, a bug in how many queries were allowed to be scheduled per tenant was fixed which might affect your max_query_parallelism and max_outstanding_per_tenant settings, and we fixed a mistake related scrape_configs which do not have a pipeline_stages defined. If you have any Promtail scrape_configs which do not specify pipeline_stages you should go read the upgrade notes!

All Changes

Loki

3460 slim-bean: Loki: Per Tenant Runtime Configs
3459 cyriltovena: Fixes split interval for metrics queries.
3432 slim-bean: Loki: change ReadStringAsSlice to ReadString so it doesn't parse quotes inside the packed _entry
3429 cyriltovena: Improve the parser hint tests.
3426 cyriltovena: Only unpack entry if the key _entry exist.
3424 cyriltovena: Add fgprof to Loki and Promtail.
3423 cyriltovena: Add limit and line_returned in the query log.
3420 cyriltovena: Introduce a unpack parser.
3417 cyriltovena: Fixes a race in the scheduling limits.
3416 ukolovda: Distributor: append several tests for HTTP parser.
3411 slim-bean: Loki: fix alignment of atomic 64 bit to work with 32 bit OS
3409 gotjosh: Instrumentation: Add histogram for request duration on gRPC client to Ingesters
3408 jgehrcke: distributor: fix snappy-compressed protobuf POST request handling (#3407)
3388 owen-d: prevents duplicate log lines from being replayed. closes #3378
3383 cyriltovena: Fixes head chunk iterator direction.
3380 slim-bean: Loki: Fix parser hint for extracted labels which collide with stream labels
3372 cyriltovena: Fixes a panic with whitespace key.
3350 cyriltovena: Fixes ingester stats.
3348 cyriltovena: Fixes 500 in the querier when returning multiple errors.
3347 cyriltovena: Fixes a tight loop in the Engine with LogQL parser.
3344 cyriltovena: Fixes some 500 returned by querier when storage cancellation happens.
3342 cyriltovena: Bound parallelism frontend
3340 owen-d: Adds some flushing instrumentation/logs
3339 owen-d: adds Start() method to WAL interface to delay checkpointing until aft…
3338 sandeepsukhani: dedupe index on all the queries for a table instead of query batches
3326 owen-d: removes wal recover flag
3307 slim-bean: Loki: fix validation error and metrics
3306 cyriltovena: Add finalizer to zstd.
3300 sandeepsukhani: increase db retain period in ingesters to cover index cache validity period as well
3299 owen-d: Logql/absent label optimization
3295 jtlisi: chore: update cortex to latest and fix refs
3291 ukolovda: Distributor: Loki API can receive gzipped JSON
3280 dannykopping: LogQL: Simple JSON expressions
3279 cyriltovena: Fixes logfmt parser hints.
3278 owen-d: Testware/ rate-unwrap-multi
3274 liguozhong: [ingester_query] change var "clients" to "reps"
3267 jeschkies: Update vendored Cortex to 0976147451ee
3263 cyriltovena: Fix a bug with metric queries and label_format.
3261 sandeepsukhani: fix broken json logs push path
3256 jtlisi: update vendored cortex and add new replace overrides
3251 cyriltovena: Ensure we have parentheses for bin ops.
3249 cyriltovena: Fixes a bug where slice of Entries where not zeroed
3241 cyriltovena: Allocate entries array with correct size while decoding WAL entries.
3237 cyriltovena: Fixes unmarshalling of tailing responses.
3236 slim-bean: Loki: Log a crude lag metric for how far behind a client is.
3234 cyriltovena: Fixes previous commit not using the new sized body.
3233 cyriltovena: Re-introduce grafana#3178.
3228 MichelHollands: Add config endpoint
3218 owen-d: WAL backpressure
3217 cyriltovena: Rename checkpoint proto package to avoid conflict with cortex.
3215 cyriltovena: Cortex update pre 1.7
3211 cyriltovena: Fixes tail api marshalling for v1.
3210 cyriltovena: Reverts flush buffer pooling.
3201 sandeepsukhani: limit query range in async store for ingesters when query-ingesters-within flag is set
3200 cyriltovena: Improve ingester flush memory usage.
3195 owen-d: Ignore flushed chunks during checkpointing
3194 cyriltovena: Fixes unwrap expressions from last optimization.
3193 cyriltovena: Improve checkpoint series iterator.
3188 cyriltovena: Improves checkpointerWriter memory usage
3180 owen-d: moves boltdb flags to config file
3178 cyriltovena: Logs PushRequest data.
3163 cyriltovena: Uses custom json-iter decoder for log entries.
3159 MichelHollands: Make httpAuthMiddleware field public
3153 cyriltovena: Improve wal entries encoding.
3152 AllenzhLi: update github.com/gorilla/websocket to fixes a potential denial-of-service (DoS) vector
3146 owen-d: More semantically correct flush shutdown
3143 cyriltovena: Fixes absent_over_time to work with all log selector.
3141 owen-d: Swaps mutex for atomic in ingester's OnceSwitch
3137 owen-d: label_format no longer shardable and introduces the Shardable() metho…
3136 owen-d: Don't fail writes due to full WAL disk
3134 cyriltovena: Improve distributors validation and apply in-place filtering.
3132 owen-d: Integrates label replace into sharding code
3131 MichelHollands: Update cortex to 1 6
3126 dannykopping: Implementing line comments
3122 owen-d: Self documenting pipeline process interface
3117 owen-d: Wal/recover corruption
3114 owen-d: Disables the stream limiter until wal has recovered
3092 liguozhong: lru cache logql.ParseLabels
3090 cyriltovena: Improve tailer matching by using the index.
3087 MichelHollands: feature: make server publicly available
3080 cyriltovena: Improve JSON parser and add labels parser hints.
3077 MichelHollands: Make the moduleManager field public
3065 cyriltovena: Optimizes SampleExpr to remove unnecessary line_format.
3064 cyriltovena: Add zstd and flate compressions algorithms.
3053 cyriltovena: Add absent_over_time
3048 cyriltovena: Support rate for unwrapped expressions.
3047 cyriltovena: Add function label_replace.
3030 cyriltovena: Allows by/without to be empty and available for max/min_over_time
3025 cyriltovena: Fixes a swallowed context err in the batch storage.
3013 owen-d: headblock checkpointing up to v3
3008 cyriltovena: Fixes the ruler storage with the boltdb store.
3000 owen-d: Introduces per stream chunks mutex
2981 owen-d: Adds WAL support (experimental)
2960 sandeepsukhani: fix table deletion in table client for boltdb-shipper

Promtail

3422 kavirajk: Modify script to accept inclusion and exclustion filters as variables
3404 dannykopping: Remove default docker pipeline stage
3401 slim-bean: Promtail: Add pack stage
3381 adityacs: fix nested captured groups indexing in replace stage
3332 cyriltovena: Embed timezone data in Promtail.
3304 kavirajk: Use project-id from the variables. Remove hardcoding
3303 cyriltovena: Increase the windows bookmark buffer.
3302 cyriltovena: Fixes races in multiline stage and promtail.
3298 gregorybrzeski: Promtail: fix typo in config variable name - BookmarkPath
3285 kavirajk: Make incoming labels from gcp into Loki internal labels.
3284 kavirajk: Avoid putting all the GCP labels into loki labels
3246 cyriltovena: Windows events
3224 veltmanj: Fix(pkg/promtail) CVE-2020-11022 JQuery vulnerability
3207 cyriltovena: Fixes panic when using multiple clients
3191 rfratto: promtail: pass registerer to gcplog
3175 rfratto: Promtail: pass a prometheus registerer to promtail components
3083 kavirajk: Gcplog targetmanager
3024 jeschkies: Collapse multiline logs based on a start line.
3015 cyriltovena: Add more information about why a tailer would stop.
2996 cyriltovena: Asynchronous Promtail stages
2898 kavirajk: fix(docker-driver): Propagate promtail's client.Stop properly

Logcli

3325 cyriltovena: Fixes step encoding in logcli.
3271 chancez: Refactor logcli Client interface to use time objects for LiveTailQueryConn
3270 chancez: logcli: Fix handling of logcli query using --since/--from and --tail
3229 dethi: logcli: support --include-label when not using --tail

Jsonnet

3447 owen-d: Use better memory metric on operational dashboard
3439 owen-d: simplifies jsonnet sharding
3357 owen-d: Libsonnet/better sharding parallelism defaults
3356 owen-d: removes sharding queue math after global concurrency PR
3329 sandeepsukhani: fix config for statefulset rulers when using boltdb-shipper
3297 owen-d: adds stateful ruler clause for boltdb shipper jsonnet
3254 hairyhenderson: ksonnet: Remove invalid hostname from default promtail configuration
3181 owen-d: remaining sts use parallel mgmt policy
3179 owen-d: Ruler statefulsets
3156 slim-bean: Jsonnet: Changes ingester PVC from 5Gi to 10Gi
3139 owen-d: Less confusing jsonnet error message when using boltdb shipper defaults.
3079 rajatvig: Fix ingester PVC data declaration to use configured value
3074 c0ffeec0der: Ksonnet: Assign appropriate pvc size and class to compactor and ingester
3062 cyriltovena: Remove regexes in the operational dashboard.
3014 owen-d: loki wal libsonnet
3010 cyriltovena: Fixes promtail mixin dashboard.

fluentd

3358 fpob: Fix fluentd plugin when kubernetes labels were missing

fluent bit

3240 sbaier1: fix fluent-bit output plugin generating invalid JSON

Docker Logging Driver

3331 cyriltovena: Add pprof endpoint to docker-driver.
3225 Le0tk0k: (fix: cmd/docker-driver): Insert a space in the error message

Docs

5934 johgsc: Docs: revise modes of operation section
3437 caleb15: docs: add note about regex
3421 kavirajk: doc(gcplog): Advanced log export filter example
3419 suitupalex: docs: promtail: Fix typo w/ windows_events hyperlink.
3418 dannykopping: Adding upgrade documentation for promtail pipeline_stages change
3385 paaacman: Documentation: enable environment variable in configuration
3373 StMarian: Documentation: Fix configuration description
3371 owen-d: Distributor overview docs
3370 tkowalcz: documentation: Add Tjahzi to the list of unofficial clients
3352 kavirajk: Remove extra space between broken link
3351 kavirajk: Add some operation details to gcplog doc
3316 kavirajk: docs(fix): Make best practices docs look better
3292 wapmorgan: Patch 2 - fix link to another documentation files
3265 sandeepsukhani: Boltdb shipper doc fixes
3239 owen-d: updates tanka installation docs
3235 scoof: docs: point to latest release for docker installation
3220 liguozhong: [doc] fix err. "loki_frontend" is invalid
3212 nvtkaszpir: Fix: Update docs for logcli
3190 kavirajk: doc(gcplog): Fix titles for Cloud provisioning for GCP logs
3165 liguozhong: [doc] fix:querier do not handle "/flush" api
3164 owen-d: updates alerting docs post 2.0
3162 huikang: Doc: Add missing wal in configuration
3148 huikang: Doc: add missing type supported by table manager
3147 marionxue: Markdown Code highlighting
3138 jeschkies: Give another example for multiline.
3128 cyriltovena: Fixes LogQL documentation links.
3124 wujie1993: fix time duration unit
3123 scoren-gl: Update getting-in-touch.md
3115 valmack: Docs: Include instruction to enable variable expansion
3109 nileshcs: Documentation fix for downstream_url
3102 slim-bean: Docs: Changelog: fix indentation and add helm repo url
3094 benjaminhuo: Fix storage guide links
3088 cyriltovena: Small fixes for the documentation.
3084 ilpianista: Update reference to old helm chart repo
3078 kavirajk: mention the use of config.expand-env flag in the doc.
3049 vitalets: [Docs] Clarify docker-driver configuration options
3039 jdbaldry: doc: logql formatting fixes
3035 unguiculus: Fix multiline docs
3033 RichiH: docs: Create ADOPTERS.md
3032 oddlittlebird: Docs: Update _index.md
3029 jeschkies: Correct multiline documentation.
3027 nop33: Fix docs header inconsistency
3026 owen-d: wal docs
3017 jdbaldry: doc: Cleanup formatting
3009 jdbaldry: doc: Fix query-frontend typo
3002 keyolk: Fix typo
2991 jontg: Documentation: Add a missing field to the extended config s3 example
2956 owen-d: Updates chunkenc doc for V3

Build

3412 rfratto: Remove unneeded prune-ci-tags drone job
3390 wardbekker: Don't auto-include pod labels as loki labels as a sane default
3321 owen-d: defaults promtail to 2.1.0 in install script
3277 kavirajk: Add step to version Loki docs during public release process.
3243 chancez: dist: Build promtail for windows/386 to support 32 bit windows hosts
3206 kavirajk: Terraform script to automate GCP provisioning for gcplog
3149 jlosito: Allow dependabot to keep github actions up-to-date
3072 slim-bean: Helm: Disable CI
3031 AdamKorcz: Testing: Introduced continuous fuzzing
3006 huikang: Fix the docker image version in compose deployment

Tooling

3377 slim-bean: Tooling: Update chunks-inspect to understand the new chunk format as well as new compression algorithms
3151 slim-bean: Loki migrate-tool

Notes

This release was created from revision 8012362674568379a3871ff8c4a2bfd1ddba7ad1 (Which was PR 3460)

Dependencies

Go Version: 1.15.3
Cortex Version: 7dac81171c665be071bd167becd1f55528a9db32

2.1.0 (2020/12/23)

Happy Holidays from the Loki team! Please enjoy a new Loki release to welcome in the New Year!

2.1.0 Contains a number of fixes, performance improvements and enhancements to the 2.0.0 release!

Notable changes

Helm users read this!

The Helm charts have moved!

2720 torstenwalter: Deprecate Charts as they have been moved

This was done to consolidate Grafana's helm charts for all Grafana projects in one place: https://github.com/grafana/helm-charts/

From now moving forward, please use the new Helm repo url: https://grafana.github.io/helm-charts

The charts in the Loki repo will soon be removed so please update your Helm repo to the new URL and submit your PR's over there as well

Special thanks to @torstenwalter, @unguiculus, and @scottrigby for their initiative and amazing work to make this happen!

Also go check out the microservices helm chart contributed by @unguiculus in the new repo!

Fluent bit plugin users read this!

Fluent bit officially supports Loki as an output plugin now! WoooHOOO!

However this created a naming conflict with our existing output plugin (the new native output uses the name loki) so we have renamed our plugin.

2974 hedss: fluent-bit: Rename Fluent Bit plugin output name.

In time our plan is to deprecate and eliminate our output plugin in favor of the native Loki support. However until then you can continue using the plugin with the following change:

Old:

[Output]
    Name loki

New:

[Output]
    Name grafana-loki

Fixes

A lot of work went into 2.0 with a lot of new code and rewrites to existing, this introduced and uncovered some bugs which are fixed in 2.1:

2807 cyriltovena: Fix error swallowed in the frontend.
2805 cyriltovena: Improve pipeline stages ast errors.
2824 owen-d: Fix/validate compactor config
2830 sandeepsukhani: fix panic in ingester when not running with boltdb shipper while queriers does
2850 owen-d: Only applies entry limits to non-SampleExprs.
2855 sandeepsukhani: fix query intervals when running boltdb-shipper in single binary
2895 shokada: Fix error 'Unexpected: ("$", "$") while parsing field definition'
2902 cyriltovena: Fixes metric query issue with no grouping.
2901 cyriltovena: Fixes a panic with the logql.NoopPipeline.
2913 cyriltovena: Fixes logql.QueryType.
2917 cyriltovena: Fixes race condition in tailer since logql v2.
2960 sandeepsukhani: fix table deletion in table client for boltdb-shipper

Enhancements

A number of performance and resource improvements have been made as well!

2911 sandeepsukhani: Boltdb shipper query readiness
2875 cyriltovena: Labels computation LogQLv2
2927 cyriltovena: Improve logql parser allocations.
2926 cyriltovena: Cache label strings in ingester to improve memory usage.
2931 cyriltovena: Only append tailed entries if needed.
2973 cyriltovena: Avoid parsing labels when tailer is sending from a stream.
2959 cyriltovena: Improve tailer matcher function.
2876 jkellerer: LogQL: Add unwrap bytes() conversion function

Notable mentions

Thanks to @timbyr for adding an often requested feature, the ability to support environment variable expansion in config files!

2837 timbyr: Configuration: Support environment expansion in configuration

Thanks to @huikang for adding a new docker-compose file for running Loki as microservices!

2740 huikang: Deploy: add docker-compose cluster deployment file

All Changes

Loki

2988 slim-bean: Loki: handle faults when opening boltdb files
2984 owen-d: adds the ability to read chunkFormatV3 while writing v2
2983 slim-bean: Loki: recover from panic opening boltdb files
2975 cyriltovena: Fixes vector grouping injection.
2972 cyriltovena: Add ProcessString to Pipeline.
2962 cyriltovena: Implement io.WriteTo by chunks.
2951 owen-d: bumps rules-action ref to logqlv2+ version
2946 cyriltovena: Fixes the Stringer of the byte label operator.
2945 cyriltovena: Fixes iota unexpected behaviour with bytes for chunk encoding.
2941 jeschkies: Test label filter for bytes.
2934 owen-d: chunk schema v3
2930 cyriltovena: Fixes all in one grpc registrations.
2929 cyriltovena: Cleanup labels parsing.
2922 codewithcheese: Distributor registers logproto.Pusher service to receive logs via GRPC
2918 owen-d: Includes delete routes for ruler namespaces
2903 cyriltovena: Limit series for metric queries.
2892 cyriltovena: Improve the chunksize test.
2891 sandeepsukhani: fix flaky load tables test for boltdb-shipper uploads table-manager
2836 andir: tests: fix quoting issues in test output when building with Go 1.15
2831 sandeepsukhani: fix flaky tests in boltdb-shipper
2822 cyriltovena: LogQL: Improve template format
2794 sandeepsukhani: Revendor cortex to latest master
2764 owen-d: WAL/marshalable chunks
2751 jeschkies: Logging: Log throughput and total bytes human readable.

Helm

2986 cyriltovena: Move CI to helm3.
2967 czunker: Remove helm init
2965 czunker: [Helm Chart Loki] Add needed k8s objects for alerting config
2940 slim-bean: Helm: Update logstash to new chart and newer version
2835 tracyde: Iss2734
2789 bewiwi: Allows service targetPort modificaion
2651 scottrigby: helm chart: Fix broken logo

Jsonnet

2976 beorn7: Improve promtail alerts to retain the namespace label
2961 sandeepsukhani: add missing ingester query routes in loki reads and operational dashboard
2899 halcyondude: gateway: fix regression in tanka jsonnet
2873 Duologic: fix(loki-mixin): refer to super.annotations
2852 chancez: production/ksonnet: Add config_hash annotation to gateway deployment based on gateway configmap
2820 owen-d: fixes promtail libsonnet tag. closes #2818
2718 halcyondude: parameterize PVC storage class (ingester, querier, compactor)

Docs

2969 simonswine: Add community forum to README.md
2968 yuichi10: logcli: Fix logcli logql document URL
2942 hedss: Docs: Corrects Fluent Bit documentation link to build the plugin.
2933 oddlittlebird: Update CODEOWNERS
2909 fredr: Docs: Add max_cache_freshness_per_query to limit_config
2890 dfang: Fix typo
2888 oddlittlebird: Update CODEOWNERS
2879 zhanghjster: documentation: add tail_proxy_url option to query_frontend_config section
2869 nehaev: documentation: Add loki4j to the list of unofficial clients
2853 RangerCD: Fix typos in promtail
2848 dminca: documentation: fix broken link in Best Practices section
2833 siavashs: Docs: -querier.split-queries-by-day deprecation
2819 owen-d: updates docs with delete permissions notice
2817 scoof: Documentation: Add S3 IAM policy to be able to run Compactor
2811 slim-bean: Docs: improve the helm upgrade section
2810 hedss: CHANGELOG: Update update document links to point to the right place.
2704 owen-d: WAL design doc
2636 LTek-online: promtail documentation: changing the headers of the configuration docu to reflect configuration code

Promtail

2957 slim-bean: Promtail: Update debian image and use a newer libsystemd
2928 cyriltovena: Skip journald bad message.
2914 chancez: promtail: Add support for using syslog message timestamp
2910 rfratto: Expose underlying promtail client

Logcli

2948 tomwilkie: Add a few more instructions to logcli --help.

Build

2877 cyriltovena: Update to go 1.15
2814 torkelo: Stats: Adding metrics collector GitHub action

Fluentd

2825 cyriltovena: Bump fluentd plugin
2434 andsens: fluent-plugin: Improve escaping in key_value format

Notes

This release was created from revision ae9c4b82ec4a5d21267da50d6a1a8170e0ef82ff (Which was PR 2960) and the following PR's were cherry-picked

2984 owen-d: adds the ability to read chunkFormatV3 while writing v2
2974 hedss: fluent-bit: Rename Fluent Bit plugin output name.

Dependencies

Go Version: 1.15.3
Cortex Version: 85942c5703cf22b64cecfd291e7e7c42d1b8c30c

2.0.1 (2020/12/10)

2.0.1 is a special release, it only exists to add the v3 support to Loki's chunk format.

There is no reason to upgrade from 2.0.0 to 2.0.1

This chunk version is internal to Loki and not configurable, and in a future version v3 will become the default (Likely 2.2.0).

We are creating this to enable users to roll back from a future release which was writing v3 chunks, back as far as 2.0.0 and still be able to read chunks.

This is mostly a safety measure to help if someone upgrades from 2.0.0 and skips versions to a future version which is writing v3 chunks and they encounter an issue which they would like to roll back. They would be able to then roll back to 2.0.1 and still read v3 chunks.

It should be noted this does not help anyone upgrading from a version older than 2.0.0, that is you should at least upgrade to 2.0.0 before going to a newer version if you are on a version older than 2.0.0.

2.0.0 (2020/10/26)

2.0.0 is here!!

We are extremely excited about the new features in 2.0.0, unlocking a whole new world of observability of our logs.

Thanks again for the many incredible contributions and improvements from the wonderful Loki community, we are very excited for the future!

Important Notes

Please Note There are several changes in this release which require your attention!

Anyone using a docker image please go read the upgrade guide!! There is one important consideration around a potentially breaking schema change depending on your configuration.
MAJOR changes have been made to the boltdb-shipper index, breaking changes are not expected but extra precautions are highly recommended, more details in the upgrade guide.
The long deprecated entry_parser config in Promtail has been removed, use pipeline_stages instead.

Check the upgrade guide for detailed information on all these changes.

2.0!!!!

There are too many PR's to list individually for the major improvements which we thought justified a 2.0 but here is the high level:

Significant enhancements to the LogQL query language!
Parse your logs to extract labels at query time.
Filter on query time extracted labels.
Format your log lines any way you please!
Graph the contents of your log lines as metrics, including support for many more of your favorite PromQL functions.
Generate prometheus alerts directly from your logs!
Create alerts using the same prometheus alert rule syntax and let Loki send alerts directly to your Prometheus Alertmanager!
boltdb-shipper is now production ready!
This is it! Now Loki only needs a single object store (S3,GCS,Filesystem...) to store all the data, no more Cassandra, DynamoDB or Bigtable!

We are extremely excited about these new features, expect some talks, webinars, and blogs where we explain all this new functionality in detail.

Notable mention

This is a small change but very helpful!

2737 dlemel8: cmd/loki: add "verify-config" flag

Thank you @dlemel8 for this PR! Now you can start Loki with -verify-config to make sure your config is valid and Loki will exit with a status code 0 if it is!

All Changes

Loki

2804 slim-bean: Loki: log any chunk fetch failure
2803 slim-bean: Update local and docker default config files to use boltdb-shipper with a few other config changes
2796 cyriltovena: Fixes a bug that would add error label incorrectly.
2793 cyriltovena: Improve the way we reverse iterator for backward queries.
2790 sandeepsukhani: Boltdb shipper metrics changes
2788 sandeepsukhani: add a metric in compactor to record timestamp of last successful run
2786 cyriltovena: Logqlv2 pushes groups down to edge
2778 cyriltovena: Logqv2 optimization
2774 cyriltovena: Handle panic in the store goroutine.
2773 cyriltovena: Fixes race conditions in the batch iterator.
2770 sandeepsukhani: Boltdb shipper query performance improvements
2769 cyriltovena: LogQL: Labels and Metrics Extraction
2768 cyriltovena: Fixes all lint errors.
2761 owen-d: Service discovery refactor
2755 owen-d: Revendor Cortex
2752 kavirajk: fix: Remove depricated entry_parser from scrapeconfig
2741 owen-d: better tenant logging in ruler memstore
2737 dlemel8: cmd/loki: add "verify-config" flag
2735 cyriltovena: Fixes the frontend logs to include org_id.
2732 sandeepsukhani: set timestamp in instant query done by canaries
2726 dvrkps: hack: clean getStore
2711 owen-d: removes r/w pools from block/chunk types
2709 cyriltovena: Bypass sharding middleware when a query can't be sharded.
2671 alrs: pkg/querier: fix dropped error
2665 cnbailian: Loki: Querier APIs respond JSON Content-Type
2663 owen-d: improves numeric literal stringer impl
2662 owen-d: exposes rule group validation fn
2661 owen-d: Enable local rules backend & disallow configdb.
2656 sandeepsukhani: run multiple queries per table at once with boltdb-shipper
2655 sandeepsukhani: fix store query bug when running loki in single binary mode with boltdb-shipper
2650 owen-d: Adds prometheus ruler routes
2647 arl: pkg/chunkenc: fix test using string(int) conversion
2645 arl: Tests: fix issue 2356: distributor_test.go fails when the system has no interface name in [eth0, en0, lo0]
2642 sandeepsukhani: fix an issue with building loki
2640 sandeepsukhani: improvements for boltdb-shipper compactor
2637 owen-d: Ruler docs + single binary inclusion
2627 sandeepsukhani: revendor cortex to latest master
2620 alrs: pkg/storage/stores/shipper/uploads: fix test error
2614 cyriltovena: Improve lz4 compression
2613 sandeepsukhani: fix a panic when trying to stop boltdb-shipper multiple times using sync.once
2610 slim-bean: Loki: Fix query-frontend ready handler
2601 sandeepsukhani: rpc for querying ingesters to get chunk ids from its store
2589 owen-d: Ruler/loki rule validator
2582 yeya24: Add _total suffix to ruler counter metrics
2580 owen-d: strict rule unmarshaling
2578 owen-d: exports grouploader
2576 owen-d: Better rule loading
2574 sandeepsukhani: fix closing of compressed file from boltdb-shipper compactor
2572 adityacs: Validate max_query_length in Labels API
2564 owen-d: Error on no schema configs
2559 sandeepsukhani: fix dir setup based on which mode it is running
2558 sandeepsukhani: cleanup boltdb files in queriers during startup/shutdown
2552 owen-d: fixes batch metrics help text & corrects bucketing
2550 sandeepsukhani: fix a flaky test in boltdb shipper
2548 sandeepsukhani: add some metrics for monitoring compactor
2546 sandeepsukhani: register boltdb shipper compactor cli flags
2543 sandeepsukhani: revendor cortex to latest master
2534 owen-d: Consistent chunk metrics
2530 sandeepsukhani: minor fixes and improvements for boltdb shipper
2526 sandeepsukhani: compactor for compacting boltdb files uploaded by shipper
2510 owen-d: adds batch based metrics
2507 sandeepsukhani: compress boltdb files to gzip while uploading from shipper
2458 owen-d: Feature/ruler (take 2)
2487 sandeepsukhani: upload boltdb files from shipper only when they are not expected to be modified or during shutdown

Docs

2797 cyriltovena: Logqlv2 docs
2772 DesistDaydream: reapir Retention Example Configuration
2762 PabloCastellano: fix: typo in upgrade.md
2750 owen-d: fixes path in prom rules api docs
2733 owen-d: Removes wrong capitalizations
2728 vishesh92: Docs: Update docs for redis
2725 dvrkps: fix some misspells
2724 MadhavJivrajani: DOCS: change format of unordered lists in technical docs
2716 huikang: Doc: fixing parameter name in configuration
2705 owen-d: shows cortextool lint command for loki in alerting docs
2702 huikang: Doc: fix broken links in production/README.md
2699 sandangel: docs: use repetitive numbering
2698 bemasher: Doc: Vague link text.
2697 owen-d: updates alerting docs with new cortex tool loki linting support
2692 philnichol: Docs: Corrected incorrect instances of (setup|set up)
2691 UniqueTokens: Update metrics.md
2689 pgassmann: docker plugin documentation update
2686 demon: docs: Fix link to code of conduct
2657 owen-d: fixes ruler docs & includes ruler configs in cmd/configs + docker img
2622 sandeepsukhani: add compactor details and other boltdb-shipper doc improvments
2621 cyriltovena: Fixes links in aws tutorials.
2606 cyriltovena: More template stage examples.
2605 Decad: Update docs to use raw link
2600 slim-bean: Docs: Fix broken links on generated site
2597 nek-00-ken: Fixup: url to access promtail config sample
2595 sh0rez: docs: fix broken links
2594 wardbekker: Update README.md
2592 owen-d: fixes some doc links
2591 woodsaj: Docs: fix links in installation docs
2586 ms42Q: Doc fixes: remove typos and long sentence
2579 oddlittlebird: Update CODEOWNERS
2566 owen-d: Website doc link fixes
2528 owen-d: Update tanka.md with steps for using k8s-alpha lib
2512 palemtnrider: Documentation: Fixes install and getting-started links in the readme
2508 owen-d: memberlist correct yaml path. closes #2499
2506 ferdikurniawan: Docs: fix dead link
2505 sh0rez: doc: close code block
2501 tivvit: fix incorrect upgrade link
2500 oddlittlebird: Docs: Update README.md

Helm

2746 marcosartori: helm/fluentbit K8S-Logging.Exclude & and Mem_Buf_Limit toggle
2742 steven-sheehy: Fix linting errors and use of deprecated repositories
2659 rskrishnar: [Promtail] enables configuring psp in helm chart
2554 alexandre-allard-scality: production/helm: add support for PV selector in Loki statefulset

FluentD

2739 jgehrcke: FluentD loki plugin: add support for bearer_token_file parameter

Fluent Bit

2568 zjj2wry: fluent-bit plugin support TLS

Promtail

2723 carlpett: Promtail: Add counter promtail_batch_retries_total
2717 slim-bean: Promtail: Fix deadlock on tailer shutdown.
2710 slim-bean: Promtail: (and also fluent-bit) change the max batch size to 1MB
2708 Falco20019: Promtail: Fix timestamp parser for short year format
2658 slim-bean: Promtail: do not mark the position if the file is removed
2618 slim-bean: Promtail: Add a stream lagging metric
2615 aminjam: Add fallback_formats for timestamp stage
2603 rfratto: Expose UserAgent and fix User-Agent version source
2575 unguiculus: Promtail: Fix docker-compose.yaml
2571 rsteneteg: Promtail: adding pipeline stage for dropping labels
2570 slim-bean: Promtail: Fix concurrent map iteration when using stdin
2565 carlpett: Add a counter for empty syslog messages
2542 slim-bean: Promtail: implement shutdown for the no-op server
2532 slim-bean: Promtail: Restart the tailer if we fail to read and upate current position

Ksonnet

2719 halcyondude: nit: fix formatting for ksonnet/loki
2677 sandeepsukhani: fix jsonnet for memcached-writes when using boltdb-shipper
2617 periklis: Add config options for loki dashboards
2612 fredr: Dashboard: typo in Loki Operational dashboard
2599 sandeepsukhani: fix closing bracket in dashboards from loki-mixin
2584 sandeepsukhani: Read, Write and operational dashboard improvements
2560 owen-d: Jsonnet/ruler
2547 sandeepsukhani: jsonnet for running loki using boltdb-shipper
2525 Duologic: fix(ksonnet): don't depend on specific k8s version
2521 charandas: fix: broken links in Tanka documentation
2503 owen-d: Ksonnet docs
2494 primeroz: Jsonnet Promtail: Change function for mounting configmap in promtail daemonset

Logstash

2607 adityacs: Logstash cpu usage fix

Build

2602 sandeepsukhani: add support for building querytee
2561 tharun208: Added logcli docker image
2549 simnv: Ignore .exe files build for Windows
2527 owen-d: Update docker-compose.yaml to use 1.6.0

Docker Logging Driver

2459 RaitoBezarius: Docker logging driver: Add a keymod for the extra attributes from the Docker logging driver

Dependencies

Go Version: 1.14.2
Cortex Version: 85942c5703cf22b64cecfd291e7e7c42d1b8c30c

1.6.1 (2020-08-24)

This is a small release and only contains two fixes for Promtail:

2542 slim-bean: Promtail: implement shutdown for the no-op server
2532 slim-bean: Promtail: Restart the tailer if we fail to read and upate current position

The first only applies if you are running Promtail with both --stdin and --server.disabled=true flags.

The second is a minor rework to how Promtail handles a very specific error when attempting to read the size of a file and failing to do so.

Upgrading Promtail from 1.6.0 to 1.6.1 is only necessary if you have logs full of msg="error getting tail position and/or size", the code changed in this release has been unchanged for a long time and we suspect very few people are seeing this issue.

No changes to any other components (Loki, Logcli, etc) are included in this release.

1.6.0 (2020-08-13)

It's the second thursday of the eighth month of the year which means it's time for another Loki Release!!

Before we highlight important features and changes, congratulations to @adityacs, who is the newest member of the Loki team! Aditya has been regularly contributing to the Loki project for the past year, with each contribution better than the last. Many of the items on the following list were thanks to his hard work. Thank you, Aditya, and welcome to the team!

I think we might have set a new record with 189 PR's in this release!

Important Notes

Please Note There are several changes in this release which might require your attention!

The NET_BIND_SERVICE capability was removed from the Loki process in the docker image, it's no longer possible to run Loki with the supplied image on a port less than 1024
If you run microservices, there is an important rollout sequence to prevent query errors.
Scrape configs have changed for Promtail in both Helm and Ksonnet affecting two labels: instance -> pod and container_name -> container.
Almost all of the Loki Canary metrics were renamed.
A few command line flags where changed (although they are likely not commonly used)
If you use ksonnet and run on GCS and Bigtable you may see an error in your config as a default value was removed.
If you are using boltdb-shipper, you will likekly need to add a new schema_config entry.

Check the upgrade guide for detailed information on all these changes.

Notable Features and Fixes

Query language enhancements

2150 introduces bytes_rate, which calculates the per second byte rate of a log stream, and bytes_over_time, which returns the byte size of a log stream.
2182 introduces a long list of comparison operators, which will let you write queries like count_over_time({foo="bar"}[1m]) > 10. Check out the PR for a more detailed description.

Loki performance improvements

2216, 2218, and 2219 all improve how memory is allocated and reused for queries.
2239 is a huge improvement for certain cases in which a query covers a large number of streams that all overlap in time. Overlapping data is now internally cached while Loki works to sort all the streams into the proper time order.
2293 was a big refactor to how Loki internally processes log queries vs. metric queries, creating separate code paths to further optimize metric queries. Metric query performance is now 2 to 10 times faster.

If you are using the query-frontend:

2441 improves how label queries can be split and queried in parallel
2123 allows queries to the series API to be split by time and parallelized; and last but most significant
1927 allows for a much larger range of queries to be sharded and performed in parallel. Query sharding is a topic in itself, but as a rough summary, this type of sharding is not time dependent and leverages how data is already stored by Loki to be able to split queries up into 16 separate pieces to be queried at the same time.

Promtail

2296 allows Promtail to expose the Loki Push API. With this, you can push from any client to Promtail as if it were Loki, and Promtail can then forward those logs to another Promtail or to Loki. There are some good use cases for this with the Loki Docker Logging Driver; if you want an easier way to configure pipelines or expose metrics collection, point your Docker drivers at a Promtail instance.
2282 contains an example Amazon Lambda where you can use a fan-in approach and ingestion timestamping in Promtail to work around out of order issues with multiple Lambdas processing the same log stream. This is one way to get logs from a high-cardinality source without adding a high-cardinality label.
2060 introduces the Replace stage, which lets you find and replace or remove text inside a log line. Combined with 2422 and 2480, you can now find and replace sensitive data in a log line like a password or email address and replace it with ****, or hash the value to prevent readability, while still being able to trace the value through your logs. Last on the list of pipeline additions,
2496 adds a Drop pipeline stage, which lets you drop log lines based on several criteria options including regex matching content, line length, or the age of the log line. The last two are useful to prevent sending to Loki logs that you know would be rejected based on configured limits in the Loki server.

Logstash output plugin

1822 added a Logstash output plugin for Loki. If you have an existing Logstash install, you can now use this plugin to send your logs to Loki to make it easier to try out, or use Loki alongside an existing logging installation.

Loki Canary

2344 improved the canaries capabilities for checking for data integrity, including spot checking for logs over a longer time window and running metric queries to verify count_over_time accuracy.

Logcli

2470 allows you to color code your log lines based on their stream labels for a nice visual indicator of streams.
2497 expands on the series API query to Loki with the--analyze-labels flag, which can show you a detailed breakdown of your label key and value combinations. This is very useful for finding improper label usage in Loki or labels with high cardinality.
2482, in which LogCLI will automatically batch requests to Loki to allow making queries with a --limit= far larger than the server side limit defined in Loki. LogCLI will dispatch the request in a series of queries configured by the --batch= parameter (which defaults to 1000) until the requested limit is reached!

Misc

2453 improves the error messages when a query times out, as Context Deadline Exceeded wasn’t the most intuitive.
2336 provides two new flags that will print the entire Loki config object at startup. Be warned there are a lot of config options, and many won’t apply to your setup (such as storage configs you aren’t using), but this can be a really useful tool when troubleshooting. Sticking with the theme of best for last,
2224 and 2288 improve support for running Loki with a shared Ring using memberlist while not requiring Consul or Etcd. We need to follow up soon with some better documentation or a blog post on this!

Dependencies

Go Version: 1.14.2
Cortex Version: 7014ff11ed70d9d59ad29d0a95e73999c436c47c

All Changes

Loki

2484 slim-bean: Loki: fix batch iterator error when all chunks overlap and chunk time ranges are greater than query time range
2483 sandeepsukhani: download boltdb files parallelly during reads
2472 owen-d: series endpoint uses normal splits
2466 owen-d: BatchIter edge cases
2463 sandeepsukhani: revendor cortex to latest master
2457 adityacs: Fix panic in cassandra storage while registering metrics
2453 slim-bean: Loki: Improve error messages on query timeout or cancel
2450 adityacs: Fixes panic in runtime_config
2449 jvrplmlmn: Replace usage of sync/atomic with uber-go/atomic
2441 cyriltovena: Split label names queries in the frontend.
2427 owen-d: Revendor cortex
2392 owen-d: avoid mutating config while parsing -config.file
2346 cyriltovena: Fixes LogQL grouping
2336 slim-bean: Loki: add -print-config-stderr flag to dump loki's runtime config to stderr
2330 slim-bean: Loki: Use a new context to update the ring state after a failed chunk transfer
2328 slim-bean: Loki: Transfer one chunk at a time per series during chunk transfers
2327 adityacs: Fix data race in ingester
2323 cyriltovena: Improve object key parsing for boltdb shipper.
2306 cyriltovena: Fixes buffered iterator skipping very long lines.
2302 cyriltovena: Improve entry deduplication.
2294 cyriltovena: Remove NET_BIND_SERVICE capability requirement.
2293 cyriltovena: Improve metric queries by computing samples at the edges.
2288 periklis: Add support for memberlist dns-based discovery
2268 owen-d: lock fix for flaky test
2266 cyriltovena: Update to latest cortex.
2264 adityacs: Fix ingester results for series query
2261 sandeepsukhani: create smaller unique files from boltdb shipper and other code refactorings
2254 slim-bean: Loki: Series API will return all series with no match or empty matcher
2252 owen-d: avoids further time splitting in querysharding mware
2250 slim-bean: Loki: Remove redundant log warning
2249 owen-d: avoids recording stats in the sharded engine
2248 cyriltovena: Add performance profile flags for logcli.
2239 cyriltovena: Cache overlapping blocks
2224 periklis: Replace memberlist service in favor of cortex provided service
2223 adityacs: Add Error method for step evaluators
2219 cyriltovena: Reuse slice for the range vector allocations.
2218 cyriltovena: Reuse buffer for hash computation in the engine.
2216 cyriltovena: Improve point allocations for each steps in the logql engine.
2211 sandeepsukhani: query tee proxy with support for comparison of responses
2206 sandeepsukhani: disable index dedupe when rf > 1 and current or upcoming index type is boltdb-shipper
2204 owen-d: bumps cortex & fixes conflicts
2191 periklis: Add flag to disable tracing activation
2189 owen-d: Fix vector-scalar comparisons
2182 owen-d: Logql comparison ops
2178 cyriltovena: Fixes path prefix in the querier.
2166 sandeepsukhani: enforce requirment for periodic config for index tables to be 24h when using boltdb shipper
2161 cyriltovena: Fix error message for max tail connections.
2156 sandeepsukhani: boltdb shipper download failure handling and some refactorings
2150 cyriltovena: Bytes aggregations
2136 cyriltovena: Fixes Iterator boundaries
2123 adityacs: Fix Series API slowness
1927 owen-d: Feature/querysharding ii
2032 tivvit: Added support for tail to query frontend

Promtail

2496 slim-bean: Promtail: Drop stage
2475 slim-bean: Promtail: force the log level on any Loki Push API target servers to match Promtail's log level.
2474 slim-bean: Promtail: use --client.external-labels for all clients
2471 owen-d: Fix/promtail yaml config
2464 slim-bean: Promtail: Bug: loki push api, clone labels before handling
2438 rfratto: pkg/promtail: propagate a logger rather than using util.Logger globally
2432 pyr0hu: Promtail: Allow empty replace values for replace stage
2422 wardbekker: Template: Added a sha256 template function for obfuscating / anonymize PII data in e.g. the replace stage
2414 rfratto: Add RegisterFlagsWithPrefix to config structs
2386 cyriltovena: Add regex function to promtail template stage.
2345 adityacs: Refactor Promtail target manager code
2301 flixr: Promtail: support unix timestamps with fractional seconds
2296 slim-bean: Promtail: Loki Push API
2282 owen-d: Lambda-Promtail
2242 carlpett: Set user agent on outgoing http requests
2196 cyriltovena: Adds default -config.file for the promtail docker images.
2127 bastjan: Update go-syslog to accept non-UTF8 encoding in syslog message
2111 adityacs: Fix Promtail journal seeking known position
2105 fatpat: promtail: Add Entry variable to template
1118 shuttie: promtail: fix high CPU usage on large kubernetes clusters.
2060 adityacs: Feature: Replace stage in pipeline
2087 adityacs: Set JournalTarget Priority value to keyword

Logcli

2497 slim-bean: logcli: adds --analyize-labels to logcli series command and changes how labels are provided to the command
2482 slim-bean: Logcli: automatically batch requests
2470 adityacs: colored labels output for logcli
2235 pstibrany: logcli: Remove single newline from the raw line before printing.
2126 cyriltovena: Validate local storage config for the logcli
2083 adityacs: Support querying labels on time range in logcli

Docs

2473 owen-d: fixes lambda-promtail relative doc link
2454 oddlittlebird: Create CODEOWNERS
2439 till: Docs: updated "Upgrading" for docker driver
2437 wardbekker: DOCS: clarified globbing behaviour of path of the doublestar library
2431 endu: fix dead link
2425 RichiH: Change conduct contact email address
2420 petuhovskiy: Fix docker driver doc
2418 cyriltovena: Add logstash to clients page with FrontMatter.
2402 cyriltovena: More fixes for the website.
2400 tontongg: Fix URL to LogQL documentation
2398 robbymilo: Docs - update links, readme
2397 coderanger: 📝 Note that entry_parser is deprecated.
2396 dnsmichi: Docs: Fix Fluentd title (visible in menu)
2391 cyriltovena: Update fluentd docs and fixes links for the website.
2390 cyriltovena: Fluent bit docs
2389 cyriltovena: Docker driver doc
2385 abowloflrf: Update logo link in README.md
2378 robbymilo: Sync docs to website
2360 owen-d: Makes timestamp parsing docs clearer
2358 rille111: Documentation: Add example for having separate pvc for loki, using helm
2357 owen-d: Storage backend examples
2338 cyriltovena: Add a complete tutorial on how to ship logs from AWS EKS.
2335 cyriltovena: Improve documentation of the metric stage.
2331 cyriltovena: Add a tutorial to forward AWS ECS logs to Loki.
2321 cyriltovena: Tutorial to run Promtail on AWS EC2
2318 adityacs: Configuration documentation improvements
2317 owen-d: remove DynamoDB chunk store doc
2308 wardbekker: Added a link to the replace parsing stage
2305 rafaelpissolatto: Fix schema_config store value
2285 adityacs: Fix local.md doc
2284 owen-d: Update local.md
2279 Fra-nk: Documentation: Refine LogQL documentation
2273 RichiH: Fix typo
2247 carlpett: docs: Fix missing quotes
2233 vyzigold: docs: Add readmes to individual helm charts
2220 oddlittlebird: Docs: Local install edits
2217 fredr: docs: BoltDB typo
2215 fredr: docs: Correct loki address for docker-compose
2172 cyriltovena: Update old link for pipeline stages.
2163 slim-bean: docs: fix an error in the example log line and byte counter metrics
2160 slim-bean: Fix some errors in the upgrade guide to 1.5.0 and add some missing notes discovered by users.
2152 eamonryan: Fix typo in promtail ClusterRole
2139 adityacs: Fix configuration docs
2137 RichiH: Propose new governance
2136 cyriltovena: Fixes Iterator boundaries
2125 theMercedes: Update logql.md
2112 nileshcs: Documentation: Outdated fluentd image name, UID details, link update
2092 i-takizawa: docs: make visible

Build

2467 slim-bean: Update Loki build image

Ksonnet

2460 Duologic: refactor: use $.core.v1.envVar
2452 slim-bean: ksonnet: Reduce querier parallelism to a more sane default value and remove the default setting for storage_backend
2377 Duologic: refactor: moved jaeger-agent-mixin
2373 slim-bean: Ksonnet: Add a Pod Disruption Budget to Loki Ingesters
2185 cyriltovena: Refactor mixin routes and add series API.
2162 slim-bean: ksonnet: Fix up datasources and variables in Loki Operational
2091 beorn7: Keep scrape config in line with the new Prometheus scrape config

Docker logging driver

2435 cyriltovena: Add more precisions on the docker driver installed on the daemon.
2343 jdfalk: loki-docker-driver: Change "ignoring empty line" to debug logging
2295 cyriltovena: Remove mount in the docker driver.
2199 cyriltovena: Docker driver relabeling
2116 cyriltovena: Allows to change the log driver mode and buffer size.

Logstash output plugin

2415 cyriltovena: Set service values via --set for logstash.
2410 adityacs: logstash code refactor and doc improvements
1822 adityacs: Loki Logstash Plugin

Loki canary

2413 slim-bean: Loki-Canary: Backoff retries on query failures, add histograms for query performance.
2369 slim-bean: Loki Canary: One more round of improvements to query for missing websocket entries up to max-wait
2350 slim-bean: Canary tweaks
2344 slim-bean: Loki-Canary: Add query spot checking and metric count checking
2259 ombre8: Canary: make stream configurable

Fluentd

2407 cyriltovena: bump fluentd version to release a new gem.
2399 tarokkk: fluentd: Make fluentd version requirements permissive
2179 takanabe: Improve fluentd plugin development experience
2171 takanabe: Add server TLS certificate verification

Fluent Bit

2375 cyriltovena: Fixes the fluentbit batchwait backward compatiblity.
2367 dojci: fluent-bit: Add more loki client configuration options
2365 dojci: fluent-bit: Fix fluent-bit exit callback when buffering is enabled
2290 cyriltovena: Fixes the lint issue merged to master.
2286 adityacs: Fix fluent-bit newline and tab characters
2142 dojci: Add FIFO queue persistent buffering for fluent bit output plugin
2089 FrederikNS: Allow configuring more options for output configuration

Helm

2406 steven-sheehy: Helm: Fix regression in chart name
2379 StevenReitsma: production/helm: Add emptyDir volume type to promtail PSP
2366 StevenReitsma: production/helm: Add projected and downwardAPI volume types to PodSecurityPolicy (#2355)
2258 Synehan: helm: add annotations to service monitor
2241 chauffer: Kubernetes manifests: Remove namespace from cluster-wide resources
2238 vhrosales: helm: Add loadBalancerIP option to loki chart
2205 joschi36: BUG: add missing namespace in ingress object
2197 cyriltovena: Render loki datasources even if Grafana is disabled.
2141 cyriltovena: Adds the ability to have a pull secrets for Promtail.
2099 allout58: helm/loki-stack: Support Prometheus on a sub-path in Grafana config
2086 osela: helm/loki-stack: render loki datasource only if grafana is enabled
2091 beorn7: Keep scrape config in line with the new Prometheus scrape config

Build

2371 cyriltovena: Fixes helm publish that needs now to add repo.
2341 slim-bean: Build: Fix CI helm test
2309 cyriltovena: Test again arm32 on internal ci.
2307 cyriltovena: Removes arm32 for now as we're migrating the CI.
2287 wardbekker: Change the Grafana image to latest
2212 roidelapluie: Remove unhelpful/problematic term in circleci.yml

1.5.0 (2020-05-20)

It's been a busy month and a half since 1.4.0 was released, and a lot of new improvements have been added to Loki since!

Be prepared for some configuration changes that may cause some bumps when upgrading, we apologize for this but are always striving to reach the right compromise of code simplicity and user/operating experience.

In this case we opted to keep a simplified configuration inline with Cortex rather than a more complicated and error prone internal config mapping or difficult to implement support for multiple config names for the same feature.

This does result in breaking config changes for some configurations, however, these will fail fast and with the list of diffs from the Cortex project should be quick to fix.

Important Notes

Be prepared for breaking config changes. Loki 1.5.0 vendors cortex v1.0.1-0.20200430170006-3462eb63f324, there were substantial breaking config changes in Cortex 1.0 which standardized config options, and fixed typos.

The Loki docker image user has changed to no longer be root

Check the upgrade guide for more detailed information on these changes.

Notable Features and Fixes

There are quite a few we want to mention listed in order they were merged (mostly)

1837 sandeepsukhani: flush boltdb to object store

This is perhaps the most exciting feature of 1.5.0, the first steps in removing a dependency on a separate index store! This feature is still very new and experimental, however, we want this to be the future for Loki. Only requiring just an object store.

If you want to test this new feature, and help us find any bugs, check out the docs to learn more and get started.

2073 slim-bean: Loki: Allow configuring query_store_max_look_back_period when running a filesystem store and boltdb-shipper

This is even more experimental than the previous feature mentioned however also pretty exciting for Loki users who use the filesystem storage. We can leverage changes made in 1837 to now allow Loki to run in a clustered mode with individual filesystem stores!

Please check out the last section in the filesystem docs for more details on how this works and how to use it!

2095 cyriltovena: Adds backtick for the quoted string token lexer.

This will come as a big win to anyone who is writing complicated reqular expressions in either their Label matchers or Filter Expressions. Starting now you can use the backtick to encapsulate your regex and not have to do any escaping of special characters!!

Examples:

{name="cassandra"} |~ `error=\w+`
{name!~`mysql-\d+`}

2055 aknuds1: Chore: Fix spelling of per second in code

This is technically a breaking change for anyone who wrote code to processes the new statistics output in the query result added in 1.4.0, we apologize to anyone in this situation but if we don't fix this kind of error now it will be there forever. And at the same time we didn't feel it was appropriate to make any major api revision changes for such a new feature and simple change. We are always trying to use our best judgement in cases like this.

2031 cyriltovena: Improve protobuf serialization

Thanks @cyriltovena for another big performance improvement in Loki, this time around protbuf's!

2021 slim-bean: Loki: refactor validation and improve error messages
2012 slim-bean: Loki: Improve logging and add metrics to streams dropped by stream limit

These two changes standardize the metrics used to report when a tenant hits a limit, now all discarded samples should be reported under loki_discarded_samples_total and you no longer need to also reference cortex_discarded_samples_total. Additionally error messages were improved to help clients take better action when hitting limits.

1970 cyriltovena: Allow to aggregate binary operations.

Another nice improvement to the query language which allows queries like this to work now:

sum by (job) (count_over_time({namespace="tns"}[5m] |= "level=error") / count_over_time({namespace="tns"}[5m]))

1713 adityacs: Log error message for invalid checksum

In the event something went wrong with a stored chunk, rather than fail the query we ignore the chunk and return the rest.

2066 slim-bean: Promtail: metrics stage can also count line bytes

This is a nice extension to a previous feature which let you add a metric to count log lines per stream, you can now count log bytes per stream.

Check out this example to configure this in your promtail pipelines.

1935 cyriltovena: Support stdin target via flag instead of automatic detection.

Third times a charm! With 1.4.0 we allowed sending logs directly to promtail via stdin, with 1.4.1 we released a patch for this feature which wasn't detecting stdin correctly on some operating systems. Unfortunately after a few more bug reports it seems this change caused some more undesired side effects so we decided to not try to autodetect stdin at all, instead now you must pass the --stdin flag if you want Promtail to listen for logs on stdin.

2076 cyriltovena: Allows to pass inlined pipeline stages to the docker driver.
1906 cyriltovena: Add no-file and keep-file log option for docker driver.

The docker logging driver received a couple very nice updates, it's always been challenging to configure pipeline stages for the docker driver, with the first PR there are now a few easier ways to do this! In the second PR we added config options to control keeping any log files on the host when using the docker logging driver, allowing you to run with no disk access if you would like, as well as allowing you to control keeping log files available after container restarts.

1864 cyriltovena: Sign helm package with GPG.

We now GPG sign helm packages!

All Changes

Loki

2097 owen-d: simplifies/updates some of our configuration examples
2095 cyriltovena: Adds backtick for the quoted string token lexer.
2093 cyriltovena: Fixes unit in stats request log.
2088 slim-bean: Loki: allow no encoding/compression on chunks
2078 owen-d: removes yolostring
2073 slim-bean: Loki: Allow configuring query_store_max_look_back_period when running a filesystem store and boltdb-shipper
2064 cyriltovena: Reverse entry iterator pool
2059 cyriltovena: Recover from panic in http and grpc handlers.
2058 cyriltovena: Fix a bug in range vector skipping data.
2055 aknuds1: Chore: Fix spelling of per second in code
2046 gouthamve: Fix bug in logql parsing that leads to crash.
2050 aknuds1: Chore: Correct typo "per seconds"
2034 sandeepsukhani: some metrics for measuring performance and failures in boltdb shipper
2031 cyriltovena: Improve protobuf serialization
2030 adityacs: Update loki to cortex master
2023 cyriltovena: Support post requests in the frontend queryrange handler.
2021 slim-bean: Loki: refactor validation and improve error messages
2019 slim-bean: make loki_ingester_memory_streams Gauge per tenant.
2012 slim-bean: Loki: Improve logging and add metrics to streams dropped by stream limit
2010 cyriltovena: Update lz4 library to latest to ensure deterministic output.
2001 sandeepsukhani: table client for boltdb shipper to enforce retention
1995 sandeepsukhani: make boltdb shipper singleton and some other minor refactoring
1987 slim-bean: Loki: Add a missing method to facade which is called by the metrics storage client in cortex
1982 cyriltovena: Update cortex to latest.
1977 cyriltovena: Ensure trace propagation in our logs.
1976 slim-bean: incorporate some better defaults into table-manager configs
1975 slim-bean: Update cortex vendoring to latest master
1970 cyriltovena: Allow to aggregate binary operations.
1965 slim-bean: Loki: Adds an interval paramater to query_range queries allowing a sampling of events to be returned based on the provided interval
1964 owen-d: chunk bounds metric now records 8h range in 1h increments
1963 cyriltovena: Improve the local config to work locally and inside docker.
1961 jpmcb: [Bug] Workaround for broken etcd gomod import
1958 owen-d: chunk lifespan histogram
1956 sandeepsukhani: update cortex to latest master
1953 jpmcb: Go mod: explicit golang.org/x/net replace
1950 cyriltovena: Fixes case handling in regex simplification.
1949 SerialVelocity: [Loki]: Cleanup dockerfile
1946 slim-bean: Loki Update the cut block size counter when creating a memchunk from byte slice
1939 owen-d: adds config validation, similar to cortex
1916 cyriltovena: Add cap_net_bind_service linux capabilities to Loki.
1914 owen-d: only fetches one chunk per series in /series
1875 owen-d: support match[] encoding
1869 pstibrany: Update Cortex to latest master
1846 owen-d: Sharding optimizations I: AST mapping
1838 cyriltovena: Move default port for Loki to 3100 everywhere.
1837 sandeepsukhani: flush boltdb to object store
1834 Mario-Hofstaetter: Loki/Change local storage directory to /loki/ and fix permissions (#1833)
1819 cyriltovena: Adds a counter for total flushed chunks per reason.
1816 sdojjy: loki can not be started with loki-local-config.yaml
1810 cyriltovena: Optimize empty filter queries.
1809 cyriltovena: Test stats memchunk
1804 pstibrany: Convert Loki modules to services
1799 pstibrany: loki: update Cortex to master
1798 adityacs: Support configurable maximum of the limits parameter
1713 adityacs: Log error message for invalid checksum
1706 cyriltovena: Non-root user docker image for Loki.

Logcli

2027 pstibrany: logcli: Query needs to be stored into url.RawQuery, and not url.Path
2000 cyriltovena: Improve URL building in the logcli to strip trailing /.
1922 bavarianbidi: logcli: org-id/tls-skip-verify set via env var
1861 yeya24: Support series API in logcli
1850 chrischdi: BugFix: Fix logcli client to use OrgID in LiveTail
1814 cyriltovena: Logcli remote storage.
1712 rfratto: clarify logcli commands and output

Promtail

2069 slim-bean: Promtail: log at debug level when nothing matches the specified path for a file target
2066 slim-bean: Promtail: metrics stage can also count line bytes
2049 adityacs: Fix promtail client default values
2075 cyriltovena: Fixes a panic in dry-run when using external labels.
2026 adityacs: Targets not required in promtail config
2004 cyriltovena: Adds config to disable HTTP and GRPC server in Promtail.
1935 cyriltovena: Support stdin target via flag instead of automatic detection.
1920 alexanderGalushka: feat: tms readiness check bypass implementation
1894 cyriltovena: Fixes possible panic in json pipeline stage.
1865 adityacs: Fix flaky promtail test
1815 adityacs: Log error message when source does not exist in extracted values
1627 rfratto: Proposal: Promtail Push API

Docker Driver

2076 cyriltovena: Allows to pass inlined pipeline stages to the docker driver.
2054 bkmit: Docker driver: Allow to provision external pipeline files to plugin
1906 cyriltovena: Add no-file and keep-file log option for docker driver.
1903 cyriltovena: Log docker driver config map.

Fluentd

2074 osela: fluentd plugin: support placeholders in tenant field
2006 Skeen: fluent-plugin-loki: Restructuring and CI
1909 jgehrcke: fluentd loki plugin README: add note about labels
1853 wardbekker: bump gem version
1811 JamesJJ: Error handling: Show data stream at "debug" level, not "warn"

Fluent Bit

2040 avii-ridge: Add extraOutputs variable to support multiple outputs for fluent-bit
1915 DirtyCajunRice: Fix fluent-bit metrics
1890 dottedmag: fluentbit: JSON encoding: avoid base64 encoding of []byte inside other slices
1791 cyriltovena: Improve fluentbit logfmt.

Ksonnet

1980 cyriltovena: Log slow query from the frontend by default in ksonnet.

Mixins

2080 beorn7: mixin: Accept suffixes to pod name in instance labels
2044 slim-bean: Dashboards: fixes the cpu usage graphs
2043 joe-elliott: Swapped to container restarts over terminated reasons
2041 slim-bean: Dashboard: Loki Operational improvements
1934 tomwilkie: Put loki-mixin and promtail-mixin dashboards in a folder.
1913 tomwilkie: s/dashboards/grafanaDashboards.

Helm

2038 oke-py: Docs: update Loki Helm Chart document to support Helm 3
2015 etashsingh: Change image tag from 1.4.1 to 1.4.0 in Helm chart
1981 sshah90: added extraCommandlineArgs in values file
1967 rdxmb: helm chart: add missing line feed
1898 stefanandres: [helm loki/promtail] make UpdateStrategy configurable
1871 stefanandres: [helm loki/promtail] Add systemd-journald example with extraMount, extraVolumeMount
1864 cyriltovena: Sign helm package with GPG.
1825 polar3130: Helm/loki-stack: refresh default grafana.image.tag to 6.7.0
1817 bclermont: Helm chart: Prevent prometheus to scrape both services

Loki Canary

1891 joe-elliott: Addition of a /suspend endpoint to Loki Canary

Docs

2056 cyriltovena: Update api.md
2014 jsoref: Spelling
1999 oddlittlebird: Docs: Added labels content
1974 rfratto: fix stores for chunk and index in documentation for period_config
1966 oddlittlebird: Docs: Update docker.md
1951 cstyan: Move build from source instructions to root readme.
1945 FlorianLudwig: docs: version pin the docker image in docker-compose
1925 wardbekker: Clarified that the api push path needs to be specified.
1905 sshah90: updating typo for end time parameter in api docs
1888 slim-bean: docs: cleaning up the comments for the cache_config, default_validity option
1887 slim-bean: docs: Adding a config change in release 1.4 upgrade doc, updating readme with new doc links
1881 cyriltovena: Add precision about the range notation for LogQL.
1879 slim-bean: docs: update promtail docs for backoff
1873 owen-d: documents frontend worker
1870 ushuz: Docs: Keep plugin install command example in one line
1856 slim-bean: docs: tweak the doc section of the readme a little
1852 slim-bean: docs: clean up schema recommendations
1843 vishesh92: Docs: Update configuration docs for redis

Build

2042 rfratto: Fix drone
2009 cyriltovena: Adds :delegated flags to speed up build experience on MacOS.
1942 owen-d: delete tag script filters by prefix instead of substring
1918 slim-bean: build: This Dockerfile is a remnant from a long time ago, not needed.
1911 slim-bean: build: push images for k branches
1849 cyriltovena: Pin helm version in circle-ci helm testing workflow.

1.4.1 (2020-04-06)

We realized after the release last week that piping data into promtail was not working on Linux or Windows, this should fix this issue for both platforms:

1893 cyriltovena: Removes file size check for pipe, not provided by linux.

Also thanks to @dottedmag for providing this fix for Fluent Bit!

1890 dottedmag: fluentbit: JSON encoding: avoid base64 encoding of []byte inside other slices

1.4.0 (2020-04-01)

Over 130 PR's merged for this release, from 40 different contributors!! We continue to be humbled and thankful for the growing community of contributors and users of Loki. Thank you all so much.

Important Notes

Really, this is important

Before we get into new features, version 1.4.0 brings with it the first (that we are aware of) upgrade dependency.

We have created a dedicated page for upgrading Loki in the operations section of the docs

The docker image tag naming was changed, the starting in 1.4.0 docker images no longer have the v prefix: grafana/loki:1.4.0

Also you should be aware we are now pruning old master-xxxxx docker images from docker hub, currently anything older than 90 days is removed. We will never remove released versions of Loki

Notable Features

1661 cyriltovena: Frontend & Querier query statistics instrumentation.

The API now returns a plethora of stats into the work Loki performed to execute your query, eventually this will be displayed in some form in Grafana to help users better understand how "expensive" their queries are. Our goal here initially was to better instrument the recent work done in v1.3.0 on query parallelization and to better understand the performance of each part of Loki. In the future we are looking at additional ideas to provide feedback to users to tailor their queries for better performance.

1652 cyriltovena: --dry-run Promtail.
1649 cyriltovena: Pipe data to Promtail

This is a long overdue addition to Promtail which can help setup and debug pipelines, with these new features you can do this to feed a single log line into Promtail:

echo -n 'level=debug msg="test log (200)"' | cmd/promtail/promtail -config.file=cmd/promtail/promtail-local-config.yaml --dry-run -log.level=debug 2>&1 | sed 's/^.*stage/stage/g'

-log.level=debug 2>&1 | sed 's/^.*stage/stage/g are added to enable debug output, direct the output to stdout, and a sed filter to remove some noise from the log lines.

The stdin functionality also works without --dry-run allowing you to feed any logs into Promtail via stdin and send them to Loki

1677 owen-d: Literal Expressions in LogQL
1662 owen-d: Binary operators in LogQL

These two extensions to LogQL now let you execute queries like this:

* `sum(rate({app="foo"}[5m])) * 2`
* `sum(rate({app="foo"}[5m]))/1e6`

1678 slim-bean: promtail: metrics pipeline count all log lines

Now you can get per-stream line counts as a metric from promtail, useful for seeing which applications log the most

- metrics:
    line_count_total:
      config:
        action: inc
        match_all: true
      description: A running counter of all lines with their corresponding
        labels
      type: Counter

1558 owen-d: ingester.max-chunk-age
1572 owen-d: Feature/query ingesters within

These two configs let you set the max time a chunk can stay in memory in Loki, this is useful to keep memory usage down as well as limit potential loss of data if ingesters crash. Combine this with the query_ingesters_within config and you can have your queriers skip asking the ingesters for data which you know won't still be in memory (older than max_chunk_age).

NOTE Do not set the max_chunk_age too small, the default of 1h is probably a good point for most people. Loki does not perform well when you flush many small chunks (such as when your logs have too much cardinality), setting this lower than 1h risks flushing too many small chunks.

1581 slim-bean: Add sleep to canary reconnect on error

This isn't a feature but it's an important fix, this is the second time our canaries have tried to DDOS our Loki clusters so you should update to prevent them from trying to attack you. Aggressive little things these canaries...

1840 slim-bean: promtail: Retry 429 rate limit errors from Loki, increase default retry limits
1845 wardbekker: throw exceptions on HTTPTooManyRequests and HTTPServerError so Fluentd will retry

These two PR's change how 429 HTTP Response codes are handled (Rate Limiting), previously these responses were dropped, now they will be retried for these clients

* Promtail
* Docker logging driver
* Fluent Bit
* Fluentd

This pushes the failure to send logs to two places. First is the retry limits. The defaults in promtail (and thus also the Docker logging driver and Fluent Bit, which share the same underlying code) will retry 429s (and 500s) on an exponential backoff for up to about 8.5 mins on the default configurations. (This can be changed; see the config docs for more info.)

The second place would be the log file itself. At some point, most log files roll based on size or time. Promtail makes an attempt to read a rolled log file but will only try once. If you are very sensitive to lost logs, give yourself really big log files with size-based rolling rules and increase those retry timeouts. This should protect you from Loki server outages or network issues.

All Changes

There are many other important fixes and improvements to Loki, way too many to call out in individual detail, so take a look!

Loki

1810 cyriltovena: Optimize empty filter queries.
1809 cyriltovena: Test stats memchunk
1807 pracucci: Enable global limits by default in production mixin
1802 cyriltovena: Add a test for duplicates count in the heap iterator and fixes it.
1799 pstibrany: loki: update Cortex to master
1797 cyriltovena: Use ingester client GRPC call options from config.
1794 pstibrany: loki: Convert module names to string
1793 johncming: pkg/chunkenc: fix leak of pool.
1789 adityacs: Fix loki exit on jaeger agent not being present
1787 cyriltovena: Regexp simplification
1785 pstibrany: Update Cortex to master
1758 cyriltovena: Query range should not support date where start == end.
1750 talham7391: Clearer error response from push endpoint when labels are malformed
1746 cyriltovena: Update cortex vendoring to include frontend status code improvement.
1745 cyriltovena: Refactor querier http error handling.
1736 adityacs: Add /ready endpoint to table-manager
1733 cyriltovena: This logs queries with latency tag when recording stats.
1730 adityacs: Fix nil pointer dereference in ingester client
1719 cyriltovena: Expose QueryType function.
1718 cyriltovena: Better logql metric status code.
1708 cyriltovena: Increase discarded samples when line is too long.
1704 owen-d: api support for scalars
1686 owen-d: max line lengths (component + tenant overrides)
1684 cyriltovena: Ensure status codes are set correctly in the frontend.
1677 owen-d: Literal Expressions in LogQL
1662 owen-d: Binary operators in LogQL
1661 cyriltovena: Frontend & Querier query statistics instrumentation.
1651 owen-d: removes duplicate logRangeExprExt grammar
1636 cyriltovena: Fixes stats summary computation.
1630 owen-d: adds stringer methods for all ast expr types
1626 owen-d: compiler guarantees for logql exprs
1616 owen-d: cache key cant be reused when an interval changes
1615 cyriltovena: Add statistics to query_range and instant_query API.
1612 owen-d: bumps cortex to 0.6.1 master
1605 owen-d: Decouple logql engine/AST from execution context
1582 slim-bean: Change new stats names
1579 rfratto: Disable transfers in loki-local-config.yaml
1572 owen-d: Feature/query ingesters within
1677 owen-d: Introduces numeric literals in LogQL
1569 owen-d: refactors splitby to not require buffered channels
1567 owen-d: adds span metadata for split queries
1565 owen-d: Feature/per tenant splitby
1562 sandeepsukhani: limit for concurrent tail requests
1558 owen-d: ingester.max-chunk-age
1484 pstibrany: loki: use new runtimeconfig package from Cortex

Promtail

1840 slim-bean: promtail: Retry 429 rate limit errors from Loki, increase default retry limits
1775 slim-bean: promtail: remove the read lines counter when the log file stops being tailed
1770 adityacs: Fix single job with multiple service discovery elements
1765 adityacs: Fix error in templating when extracted key has nil value
1743 dtennander: Promtail: Ignore dropped entries in subsequent metric-stages in pipelines.
1687 adityacs: Fix panic in labels debug message
1683 slim-bean: promtail: auto-prune stale metrics
1678 slim-bean: promtail: metrics pipeline count all log lines
1666 adityacs: Support entire extracted value map in template pipeline stage
1664 adityacs: Support custom prefix name in metrics stage
1660 rfratto: pkg/promtail/positions: handle empty positions file
1652 cyriltovena: --dry-run Promtail.
1649 cyriltovena: Pipe data to Promtail
1602 slim-bean: Improve promtail configuration docs

Helm

1731 billimek: [promtail helm chart] - Expand promtail syslog svc to support values
1688 fredgate: Loki stack helm chart can deploy datasources without Grafana
1632 lukipro: Added support for imagePullSecrets in Loki Helm chart
1620 rsteneteg: [promtail helm chart] option to set fs.inotify.max_user_instances with init container
1617 billimek: [promtail helm chart] Enable support for syslog service
1590 polar3130: Helm/loki-stack: refresh default grafana.image.tag to 6.6.0
1587 polar3130: Helm/loki-stack: add template for the service name to connect to loki
1585 monotek: [loki helm chart] added ingress
1553 got-root: helm: Allow setting 'loadBalancerSourceRanges' for the loki service
1529 tourea: Promtail Helm Chart: Add support for passing environment variables

Jsonnet

1776 Eraac: fix typo: Not a binary operator: =
1767 joe-elliott: Dashboard Cleanup
1766 joe-elliott: Move dashboards out into their own json files
1757 slim-bean: promtail-mixin: Allow choosing promtail name
1756 sh0rez: fix(ksonnet): named parameters for containerPort
1749 slim-bean: Increasing the threshold for a file lag and reducing the severity to warning
1748 slim-bean: jsonnet: Breakout promtail mixin.
1739 cyriltovena: Fixes frontend args in libsonnet.
1735 cyriltovena: Allow to configure global limits via the jsonnet deployment.
1705 cyriltovena: Add overrides file for our jsonnet library.
1699 pracucci: Increased production distributors memory request and limit
1689 shokada: Add headers for WebSocket
1665 cyriltovena: Query frontend service should be headless.
1613 cyriltovena: Fixes config change in the result cache

Fluent Bit

1791 cyriltovena: Improve fluentbit logfmt.
1717 adityacs: Fluent-bit: Fix panic error when AutoKubernetesLabels is true

Fluentd

1811 JamesJJ: Error handling: Show data stream at "debug" level, not "warn"
1728 irake99: docs: fix outdated link to fluentd
1703 Skeen: fluent-plugin-grafana-loki: Update fluentd base image to current images (edge)
1656 takanabe: Convert second(Integer class) to nanosecond precision
1646 takanabe: Fix rubocop violation for fluentd/fluent-plugin-loki
1603 tarokkk: fluentd-plugin: add URI validation

Docs

1781 candlerb: Docs: Recommended schema is now v11
1771 rfratto: change slack url to slack.grafana.com and use https
1738 jgehrcke: docs: observability.md: clarify lines vs. entries
1707 dangoodman: Fix regex in pipeline-example.yml
1697 oke-py: fix promtail/templates/NOTES.txt to show correctly port-forward command
1675 owen-d: maintainer links & usernames
1673 cyriltovena: Add Owen to the maintainer team.
1671 shokada: Update tanka.md so that promtail.yml is the correct format
1648 ShotaKitazawa: loki-canary: fix indent of DaemonSet manifest written in .md file
1642 slim-bean: Improve systemd field docs
1641 pastatopf: Correct syntax of rate example
1634 takanabe: Unite docs for fluentd plugin
1619 shaikatz: PeriodConfig documentation fix dynamodb -> aws-dynamo
1611 owen-d: loki frontend docs additions
1609 Lusitaniae: Fix wget syntax in documentation
1608 PabloCastellano: Documentation: Recommend using the latest schema version (v11)
1601 rfratto: Clarify regex escaping rules
1598 cyriltovena: Update tanka.md doc.
1586 MrSaints: Fix typo in changelog for 1.3.0
1504 hsraju: Updated configuration.md

Logcli

1808 slim-bean: logcli: log the full stats and send to stderr instead of stdout
1682 adityacs: BugFix: Fix logcli --quiet parameter parsing issue
1644 cyriltovena: This improves the log output for statistics in the logcli.
1638 owen-d: adds query stats and org id options in logcli
1573 cyriltovena: Improve logql query statistics collection.

Loki Canary

1653 slim-bean: Canary needs its logo
1581 slim-bean: Add sleep to canary reconnect on error

Build

1780 slim-bean: build: Update the CD deploy task name
1762 dgzlopes: Bump testify to 1.5.1
1742 slim-bean: build: fix deploy on tagged build
1741 slim-bean: add darwin and freebsd binaries to release output
1740 rfratto: Fix 32-bit Promtail ARM docker builds from Drone
1710 adityacs: Add goimport local-prefixes configuration to .golangci.yml
1647 mattmendick: Attempting to add informational only feedback for codecov
1640 rfratto: ci: print error messages when an API request fails
1639 rfratto: ci: prune docker tags prefixed with "master-" older than 90 days
1637 rfratto: ci: pin plugins/manifest image tag
1633 rfratto: ci: make manifest publishing run in serial
1629 slim-bean: Ignore markdown files in codecoverage
1628 rfratto: Exempt proposals from stale bot
1614 mattmendick: Codecov: Update config to add informational flag
1600 mattmendick: Codecov circleci test [WIP]

Tooling

1577 pstibrany: Move chunks-inspect tool to Loki repo

1.3.0 (2020-01-16)

What's New??

With 1.3.0 we are excited to announce several improvements focusing on performance!

First and most significant is the Query Frontend:

1442 cyriltovena: Loki Query Frontend

The query frontend allows for sharding queries by time and dispatching them in parallel to multiple queriers, giving true horizontal scaling ability for queries. Take a look at the jsonnet changes to see how we are deploying this in our production setup. Keep an eye out for a blog post with more information on how the frontend works and more information on this exciting new feature.

In our quest to improve query performance, we discovered that gzip, while good for compression ratio, is not the best for speed. So we introduced the ability to select from several different compression algorithms:

1411 cyriltovena: Adds configurable compression algorithms for chunks

We are currently testing out LZ4 and snappy, LZ4 seemed like a good fit however we found that it didn't always compress the same data to the same output which was causing some troubles for another important improvement:

1438 pstibrany: pkg/ingester: added sync period flags

Extending on the work done by @bboreham on Cortex, @pstibrany added a few new flags and code to synchronize chunks between ingesters, which reduces the number of chunks persisted to object stores and therefore also reduces the number of chunks loaded on queries and the amount of de-duplication work which needs to be done.

As mentioned above, LZ4 was in some cases compressing the same data with a different result which was interfering with this change, we are still investigating the cause of this issue (It may be in how we implemented something, or may be in the compression code itself). For now we have switched to snappy which has seen a reduction in data written to the object store from almost 3x the source data (with a replication factor of 3) to about 1.5x, saving a lot of duplicated log storage!

Another valuable change related to chunks:

1406 slim-bean: allow configuring a target chunk size in compressed bytes

With this change you can set a chunk_target_size and Loki will attempt to fill a chunk to approx that size before flushing (previously a chunk size was a hard coded 10 blocks where the default block size is 262144 bytes). Larger chunks are beneficial for a few reasons, mainly on reducing API calls to your object store when performing queries, but also in reducing overhead in a few places, especially when processing very high volume log streams.

Another big improvement is the introduction of accurate rate limiting when running microservices:

1486 pracucci: Add ingestion rate global limit support

Previously the rate limit was applied at each distributor, however with traffic split over many distributors the limit would need to be adjusted accordingly. This meant that scaling up distributors required changing the limit. Now this information is communicated between distributors such that the limit should be applied accurately regardless of the number of distributors.

And last but not least on the notable changes list is a new feature for Promtail:

1275 bastjan: pkg/promtail: IETF Syslog (RFC5424) Support

With this change Promtail can receive syslogs via TCP! Thanks to @bastjan for all the hard work on this submission!

Important things to note:

1519 Changes a core behavior in Loki regarding logs with duplicate content AND duplicate timestamps, previously Loki would store logs with duplicate timestamps and content, moving forward logs with duplicate content AND timestamps will be silently ignored. Mainly this change is to prevent duplicates that appear when a batch is retried (the first entry in the list would be inserted again, now it will be ignored). Logs with the same timestamp and different content will still be accepted.
1486 Deprecated -distributor.limiter-reload-period flag / distributor's limiter_reload_period config option.

All Changes

Once again we can't thank our community and contributors enough for the significant work that everyone is adding to Loki, the entire list of changes is long!!

Loki

1526 codesome: Support for aggregation
1522 cyriltovena: Adds support for the old query string regexp in the frontend.
1519 rfratto: pkg/chunkenc: ignore duplicate lines pushed to a stream
1511 sandlis: querier: fix panic in tailer when max tail duration exceeds
1499 slim-bean: Fix a panic in chunk prefetch
1495 slim-bean: Prefetch chunks while processing
1496 cyriltovena: Add duplicates info and remove timing informations.
1490 owen-d: Fix/deadlock frontend queue
1489 owen-d: unifies reverse iterators
1488 cyriltovena: Fixes response json encoding and add regression tests.
1486 pracucci: Add ingestion rate global limit support* 1493 pracucci: Added max streams per user global limit
1480 cyriltovena: Close iterator properly and check nil before releasing buffers.
1473 rfratto: pkg/querier: don't query all ingesters
1470 cyriltovena: Validates limit parameter.
1448 cyriltovena: Improving storage benchmark
1445 cyriltovena: Add decompression tracing instrumentation.
1442 cyriltovena: Loki Query Frontend
1438 pstibrany: pkg/ingester: added sync period flags
1433 zendern: Using strict parsing for yaml configs
1425 pstibrany: pkg/ingester: Added possibility to disable transfers.
1423 pstibrany: pkg/chunkenc: Fix BenchmarkRead to focus on reading chunks, not converting bytes to string
1421 pstibrany: pkg/chunkenc: change default LZ4 buffer size to 64k.
1420 cyriltovena: Sets the chunk encoding correctly when creating chunk from bytes.
1419 owen-d: Enables Series API in loki
1413 pstibrany: RangeQuery benchmark optimizations
1411 cyriltovena: Adds configurable compression algorithms for chunks
1409 slim-bean: change the chunk size histogram to allow for bigger buckets
1408 slim-bean: forgot to register the new metric for counting blocks per chunk
1406 slim-bean: allow configuring a target chunk size in compressed bytes
1405 pstibrany: Convert string to bytes once only when doing string filtering.
1396 pstibrany: pkg/cfg: print help only when requested, and print it on stdout
1383 beornf: Read websocket close in tail handler
1071 rfratto: pkg/ingester: limit total number of errors a stream can return on push
1545 joe-elliott: Critical n => m conversions
1541 owen-d: legacy endpoint 400s metric queries

Promtail

1515 slim-bean: Promtail: Improve position and size metrics
1485 p37ruh4: Fileglob parsing fixes
1472 owen-d: positions.ignore-corruptions
1453 chancez: pkg/promtail: Initialize counters to 0 when creating client
1436 rfratto: promtail: add support for passing through journal entries as JSON
1426 wphan: Support microsecond timestamp format
1416 pstibrany: pkg/promtail/client: missing URL in client returns error
1275 bastjan: pkg/promtail: IETF Syslog (RFC5424) Support

Fluent Bit

1455 JensErat: fluent-bit-plugin: re-enable failing JSON marshaller tests; pass error instead of logging and ignoring
1294 JensErat: fluent-bit: multi-instance support
1514 shane-axiom: fluent-plugin-grafana-loki: Add fluentd_thread label when flush_thread_count > 1

Fluentd

1500 cyriltovena: Bump fluentd plugin to 1.2.6.
1475 Horkyze: fluentd-plugin: call gsub for strings only

Docker Driver

1414 cyriltovena: Adds tenant-id for docker driver.

Logcli

1492 sandlis: logcli: replaced GRAFANA_* with LOKI_* in logcli env vars, set default server url for logcli to localhost

Helm

1534 olivierboudet: helm : fix fluent-bit parser configuration syntax
1506 terjesannum: helm: add podsecuritypolicy for fluent-bit
1431 eugene100: Helm: fix issue with config.clients
1430 olivierboudet: helm : allow to define custom parsers to use with fluentbit.io/parser annotation
1418 evalsocket: Helm chart url added in helm.md
1336 terjesannum: helm: support adding init containers to the loki pod
1530 WeiBanjo: Allow extra command line args for external labels like hostname

Jsonnet

1518 benjaminhuo: Fix error 'Field does not exist: jaeger_mixin' in tk show
1501 anarcher: jsonnet: fix common/defaultPorts parameters
1497 cyriltovena: Update Loki mixin to include frontend QPS and latency.
1478 cyriltovena: Fixes the typo in the result cache config of the Loki ksonnet lib.
1543 sh0rez: fix(ksonnet): use apps/v1

Docs

1531 fitzoh: Documentation: Add note on using Loki with Amazon ECS
1521 rfratto: docs: Document timestamp ordering rules
1516 rfratto: Link to release docs in README.md, not master docs
1508 cyriltovena: Fixes bad json in Loki API documentation.
1505 sandlis: doc: fix sample yaml in docs for installing promtail to k8s
1481 terjesannum: docs: fix broken promtail link
1474 Eraac: : information about max_look_back_period
1471 cyriltovena: Update README.md
1466 Eraac: : Update IAM requirement
1441 vtereso: : README spelling fix
1437 daixiang0: fix all misspell
1432 joe-elliott: Removed unsupported encodings from docs
1399 vishesh92: Docs: Add configuration docs for redis
1394 chancez: Documentation: Fix example AWS storage configuration
1227 daixiang0: Add docker install doc
1560 robshep: Promtail Docs: Update output.md
1546 mattmendick: Removing third-party link
1539 j18e: docs: fix syntax error in pipeline example

Build

1494 pracucci: Fixed TOUCH_PROTOS in all DroneCI pipelines
1479 owen-d: TOUCH_PROTOS build arg for dockerfile
1476 owen-d: initiates docker daemon for circle windows builds
1469 rfratto: Makefile: re-enable journal scraping on ARM

New Members!

1415 cyriltovena: Add Joe as member of the team.

1.2.0 (2019-12-09)

One week has passed since the last Loki release, and it's time for a new one!

Notable Changes

We have continued our work making our API Prometheus-compatible. The key changes centered around API compatibility are:

1370 slim-bean: Change /loki/api/v1/label to loki/api/v1/labels
1381 owen-d: application/x-www-form-urlencoded support

Meanwhile, @pstibrany has done great work ensuring that Loki handles hash collisions properly:

1247 pstibrany: pkg/ingester: handle labels mapping to the same fast fingerprint.

Other Changes

❤️ All PR's are important to us, thanks everyone for continuing to help support and improve Loki! ❤️

Features

1372 cyriltovena: Let Loki start when using the debug image.
1300 pstibrany: pkg/ingester: check that ingester is in LEAVING state when transferring chunks and claiming tokens. Required when using memberlist client.

Bug Fixes/Improvements

1376 jstaffans: Fluentd: guard against nil values when sanitizing labels
1371 cyriltovena: Logql benchmark and performance improvement.
1363 cyriltovena: Fixes fluentd new push path API.
1353 pstibrany: docs: Fix grpc_listen_host and http_listen_host.
1350 Eraac: documentation: iam requirement for autoscaling

1.1.0 (2019-12-04)

It's been a busy 2 weeks since the 1.0.0 release and quite a few important PR's have been merged to Loki.

The most significant:

1322 rfratto: Fix v1 label API to be Prometheus-compatible

Some might call this a breaking change, we are instead calling it a bug fix as our goal was to be prometheus compatible and we were not 😃

But please be aware if you are using the /loki/api/v1/label or /loki/api/v1/label/<name>/values the JSON result will be different in 1.1.0

Old result:

{
  "values": [
    "label1",
    "label2",
    "labeln"
  ]
}

New result:

{
  "status": "success",
  "data": [
    "label1",
    "label2",
    "labeln"
  ]
}

ALSO IMPORTANT

1160 daixiang0: replace gzip with zip

Binaries will now be zipped instead of gzipped as many people voiced their opinion that zip is likely to be installed on more systems by default.

If you had existing automation to download and install binaries this will have to be updated to use zip instead of gzip

Notable Fixes and Improvements

Broken version info in startup log message:

1095 pstibrany: Makefile changes to allow easy builds with or without vendoring. Also fixes version bug for both cases.
The hashing algorithm used to calculate the hash for a stream was creating hash collisions in some instances. Please Note this is just one part of the fix and is only in Promtail, the second part for Loki can be tracked in PR1247 which didn't quite make the cut for 1.1.0 and will be in 1.2.0:

1254 pstibrany: pkg/promtail/client: Handle fingerprint hash collisions
Thank you @putrasattvika for finding and fixing an important bug where logs were some logs were missed in a query shortly after a flush!

1299 putrasattvika: storage: fix missing logs with batched chunk iterator
Thank you @danieldabate for helping to again improve our API to be more Prometheus compatible:

1355 danieldabate: HTTP API: Support duration and float formats for step parameter
LogQL will support duration formats that are not typically handled by Go like [1d] or [1w]

1357 cyriltovena: Supports same duration format in LogQL as Prometheus

Everything Else

❤️ All PR's are important to us, thanks everyone for continuing to help support and improve Loki! ❤️

1349 Eraac: documentation: using parsable value in example
1343 dgzlopes: doc(configuration): Fix duration format.
1342 whothey: Makefile: add debug symbols to loki and promtail debug builds
1341 adamjohnson01: Update loki helm chart to support service account annotations
1340 adamjohnson01: Pull in cortex changes to support IAM roles for EKS
1339 cyriltovena: Update gem version.
1333 daixiang0: fix broken link
1328 cyriltovena: Fixes linter warning from the yacc file.
1326 dawidmalina: Wrong api endpoint in fluent-plugin-grafana-loki
1320 roidelapluie: Metrics: use Namespace everywhere when declaring metrics
1318 roidelapluie: Use tenant as label name for discarded_samples metrics
1317 roidelapluie: Expose discarded bytes metric
1316 slim-bean: Removing old file needed for dep (no longer needed)
1312 ekeih: Docs: Add missing ) in LogQL example
1311 pstibrany: Include positions filename in the error when YAML unmarshal fails.
1310 JensErat: fluent-bit: sorted JSON and properly convert []byte to string
1304 pstibrany: promtail: write positions to new file first, move to target location afterwards
1303 zhangjianweibj: grafana#1302
1298 rfratto: pkg/promtail: remove journal target forced path
1279 rfratto: Fix loki_discarded_samples_total metric
1278 rfratto: docs: update limits_config to new structure from #948
1276 roidelapluie: Update fluentbit README.md based on my experience
1274 sh0rez: chore(ci): drone-cli
1273 JensErat: fluent-bit: tenant ID configuration
1266 polar3130: add description about tenant stage
1262 Eraac: documentation: iam requirement for autoscaling
1261 rfratto: Document systemd journal scraping
1249 cyriltovena: Move to jsoniter instead of default json package
1223 jgehrcke: authentication.md: replace "user" with "tenant"
1204 allanhung: fluent-bit-plugin: Auto add Kubernetes labels to Loki labels

1.0.0 (2019-11-19)

🎉 Nearly a year since Loki was announced at KubeCon in Seattle 2018 we are very excited to announce the 1.0.0 release of Loki! 🎉

A lot has happened since the announcement, the project just recently passed 1000 commits by 138 contributors over 700+ PR's accumulating over 7700 GitHub stars!

Internally at Grafana Labs we have been using Loki to monitor all of our infrastructure and ingest around 1.5TB/10 billion log lines a day. Since the v0.2.0 release we have found Loki to be reliable and stable in our environments.

We are comfortable with the state of the project in our production environments and think it's time to promote Loki to a non-beta release to communicate to everyone that they should feel comfortable using Loki in their production environments too.

API Stability

With the 1.0.0 release our intent is to try to follow Semver rules regarding stability with some aspects of Loki, focusing mainly on the operating experience of Loki as an application. That is to say we are not planning any major changes to the HTTP API, and anything breaking would likely be accompanied by a major release with backwards compatibility support.

We are currently NOT planning on maintaining Go API stability with this release, if you are importing Loki as a library you should be prepared for any kind of change, including breaking, even in minor or bugfix releases.

Loki is still a young and active project and there might be some breaking config changes in non-major releases, rest assured this will be clearly communicated and backwards or overlapping compatibility will be provided if possible.

Changes

There were not as many changes in this release as the last, mainly we wanted to make sure Loki was mostly stable before 1.0.0. The most notable change is the inclusion of the V11 schema in PR's 1201 and 1280. The V11 schema adds some more data to the index to improve label queries over large amounts of time and series. Currently we have not updated the Helm or Ksonnet to use the new schema, this will come soon with more details on how it works.

The full list of changes:

1280 owen-d: Fix duplicate labels (update cortex)
1260 rfratto: pkg/loki: unmarshal module name from YAML
1257 rfratto: helm: update default terminationGracePeriodSeconds to 4800
1251 obitech: docs: Fix promtail releases download link
1248 rfratto: docs: slightly modify language in community Loki packages section
1242 tarokkk: fluentd: Suppress unread configuration warning
1239 pracucci: Move ReservedLabelTenantID out from a dedicated file
1238 oke-py: helm: loki-stack supports k8s 1.16
1237 joe-elliott: Rollback google.golang.org/api to 0.8.0
1235 woodsaj: ci: update triggers to use new deployment_tools location
1234 rfratto: Standardize schema used in match stage
1233 wapmorgan: Update docker-driver Dockerfile: add tzdb
1232 rfratto: Fix drone deploy job
1231 joe-elliott: Removed references to Loki free tier
1226 clickyotomy: Update dependencies to use weaveworks/common upstream
1221 slim-bean: use regex label matcher to not alert on any tail route latencies
1219 MightySCollins: docs: Updated Kubernetes docs links in Helm charts
1218 slim-bean: update dashboards to include the new /loki/api/v1/* endpoints
1217 slim-bean: sum the bad words by name and level
1216 joe-elliott: Remove rules that reference no longer existing metrics
1215 Eraac: typo url
1214 takanabe: Correct wrong document paths about querying
1213 slim-bean: Fix docker latest and master tags
1212 joe-elliott: Update loki operational
1206 sandlis: ksonnet: fix replication always set to 3 in ksonnet
1203 joe-elliott: Chunk iterator performance improvement
1202 beorn7: Simplify regexp's
1201 cyriltovena: Update cortex to bring v11 schema
1189 putrasattvika: fluent-plugin: Add client certificate verification
1186 tarokkk: fluentd: Refactor label_keys and and add extract_kubernetes_labels configuration

0.4.0 (2019-10-24)

A huge thanks to the 36 contributors who submitted 148 PR's since 0.3.0!

Notable Changes

With PR 654 @cyriltovena added a really exciting new capability to Loki, a Prometheus compatible API with support for running metric style queries against your logs! Take a look at how to write metric queries for logs

PLEASE NOTE: To use metric style queries in the current Grafana release 6.4.x you will need to add Loki as a Prometheus datasource in addition to having it as a Log datasource and you will have to select the correct source for querying logs vs metrics, coming soon Grafana will support both logs and metric queries directly to the Loki datasource!
PR 1022 (and a few others) @joe-elliott added a new set of HTTP endpoints in conjunction with the work @cyriltovena to create a Prometheus compatible API as well as improve how labels/timestamps are handled

IMPORTANT: The new /api/v1/* endpoints contain breaking changes on the query paths (push path is unchanged) Eventually the /api/prom/* endpoints will be removed
PR 847 owes a big thanks to @cosmo0920 for contributing his Fluent Bit go plugin, now loki has Fluent Bit plugin support!!
PR 982 was a couple weeks of painstaking work by @rfratto for a much needed improvement to Loki's docs! Check them out!
PR 980 by @sh0rez improved how flags and config file's are loaded to honor a more traditional order of precedence:
1. Defaults
2. Config file
3. User-supplied flag values (command line arguments)
PLEASE NOTE: This is potentially a breaking change if you were passing command line arguments that also existed in a config file in which case the order they are given priority now has changed!
PR 1062 and 1089 have moved Loki from Dep to Go Modules and to Go 1.13

Loki

Features/Improvements/Changes

Loki 1171 cyriltovena: Moves request parsing into the loghttp package
Loki 1145 joe-elliott: Update /loki/api/v1/push to use the v1 json format
Loki 1128 sandlis: bigtable-backup: list backups just before starting deletion of wanted backups
Loki 1100 sandlis: logging: removed some noise in logs from live-tailing
Loki/build 1089 joe-elliott: Go 1.13
Loki 1088 pstibrany: Updated cortex to latest master.
Loki 1085 pracucci: Do not retry chunks transferring on shutdown in the local dev env
Loki 1084 pracucci: Skip ingester tailer filtering if no filter is set
Loki/build1062 joe-elliott: dep => go mod
Loki 1049 joe-elliott: Update loki push path
Loki 1044 joe-elliott: Fixed broken logql request filtering
Loki/tools 1043 sandlis: bigtable-backup: use latest bigtable backup docker image with fix for list backups
Loki 1030 polar3130: fix typo in error messages
Loki/tools 1028 sandlis: bigtable-backup: verify backups to work on latest list of backups
Loki 1022 joe-elliott: Loki HTTP/JSON Model Layer
Loki 1016 slim-bean: Revert "Updated stream json objects to be more parse friendly (#1010)"
Loki 1010 joe-elliott: Updated stream json objects to be more parse friendly
Loki 1009 cyriltovena: Make Loki HTTP API more compatible with Prometheus
Loki 1008 wardbekker: Improved Ingester out-of-order error for faster troubleshooting
Loki 1001 slim-bean: Update new API paths
Loki 998 sandlis: Change unit of duration params to hours to align it with duration config at other places in Loki
Loki 980 sh0rez: feat: configuration source precedence
Loki 948 sandlis: limits: limits implementation for loki
Loki 947 sandlis: added a variable for storing periodic table duration as an int to be …
Loki 938 sandlis: vendoring: update cortex to latest master
Loki/tools 930 sandlis: fix incrementing of bigtable_backup_job_backups_created metric
Loki/tools 920 sandlis: bigtable-backup tool fix
Loki/tools 895 sandlis: bigtable-backup-tool: Improvements
Loki 755 sandlis: Use grpc client config from cortex for Ingester to get more control
Loki 654 cyriltovena: LogQL: Vector and Range Vector Aggregation.

Bug Fixes

Loki 1114 rfratto: pkg/ingester: prevent shutdowns from processing during joining handoff
Loki 1097 joe-elliott: Reverted cloud.google.com/go to 0.44.1
Loki 986 pracucci: Fix panic in tailer due to race condition between send() and close()
Loki 975 sh0rez: fix(distributor): parseError BadRequest
Loki 944 rfratto: pkg/querier: fix concurrent access to querier tail clients

Promtail

Features/Improvements/Changes

Promtail/pipeline 1179 pracucci: promtail: fix handling of JMESPath expression returning nil while parsing JSON
Promtail/pipeline 1123 pracucci: promtail: added action_on_failure support to timestamp stage
Promtail/pipeline 1122 pracucci: promtail: initialize extracted map with initial labels
Promtail/pipeline 1112 cyriltovena: Add logql filter to match stages and drop capability
Promtail/journal 1109 rfratto: Clarify journal warning
Promtail 1083 pracucci: Increased promtail's backoff settings in prod and improved doc
Promtail 1026 erwinvaneyk: promtail: fix externalURL and path prefix issues
Promtail 976 slim-bean: Wrap debug log statements in conditionals to save allocations
Promtail 973 ctrox: tests: Set default value for BatchWait as ticker does not accept 0
Promtail 969 ctrox: promtail: Use ticker instead of timer for batch wait
Promtail 952 pracucci: promtail: add metrics on sent and dropped log entries
Promtail 934 pracucci: promtail: do not send the last batch - to ingester - if empty
Promtail 921 rfratto: promtail: add "max_age" field to configure cutoff for journal reading
Promtail 883 adityacs: Add pipeline unit testing to promtail

Bugfixes

Promtail 1194 slim-bean: Improve how we record file size metric to avoid a race in our file lagging alert
Promtail/journal 1072 rfratto: build: enable journal in promtail linux release build

Docs

Docs 1176 rfratto: docs: add example and documentation about using JMESPath literals
Docs 1139 joe-elliott: Moved client docs and add serilog example
Docs 1132 kailwallin: FixedTypo.Update README.md
Docs 1130 pracucci: docs: fix Promtail / Loki capitalization
Docs 1129 pracucci: docs: clarified the relation between retention period and table period
Docs 1124 geowa4: Client recommendations documentation tweaks
Docs 1106 cyriltovena: Add fluent-bit missing link in the main documentation page.
Docs 1099 pracucci: docs: improve table manager documentation
Docs 1094 rfratto: docs: update stages README with the docker and cri stages
Docs 1091 daixiang0: docs(stage): add docker and cri
Docs 1077 daixiang0: doc(fluent-bit): add missing namespace
Docs 1073 flouthoc: Re Fix Docs: PR grafana#1053 got erased due to force push.
Docs 1069 daixiang0: doc: unify GOPATH
Docs 1068 daixiang0: doc: skip jb init when using Tanka
Docs 1067 rfratto: Fix broken links to docs in README.md
Docs 1064 jonaskello: Fix spelling of HTTP header
Docs 1063 rfratto: docs: fix deprecated warning in api.md
Docs 1060 rfratto: Add Drone CI badge to README.md
Docs 1053 flouthoc: Fix Docs: Change Imagepull policy to IfNotpresent / Add loki-canary b…
Docs 1048 wassan128: Loki: Fix README link
Docs 1042 daixiang0: doc(ksonnet): include ksonnet-lib
Docs 1039 sh0rez: doc(production): replace ksonnet with Tanka
Docs 1036 sh0rez: feat: -version flag
Docs 1025 oddlittlebird: Update CONTRIBUTING.md
Docs 1024 oddlittlebird: Update README.md
Docs 1014 polar3130: Fix a link to correct doc and fix a typo
Docs 1006 slim-bean: fixing lots of broken links and a few typos
Docs 1005 SmilingNavern: Fix links to correct doc
Docs 1004 rfratto: docs: fix example with pulling systemd logs
Docs 1003 oddlittlebird: Loki: Update README.md
Docs 984 tomgs: Changing "Usage" link in main readme after docs change
Docs 983 daixiang0: update positions.yaml location reference
Docs 982 rfratto: Documentation Rewrite
Docs 961 worr: doc: Add permissions that IAM roles for Loki need
Docs 933 pracucci: doc: move promtail doc into dedicated subfolder
Docs 924 pracucci: doc: promtail known failure modes
Docs 910 slim-bean: docs(build): Update docs around releasing and fix bug in version updating script
Docs 850 sh0rez: docs: general documentation rework

Build

Build 1157 daixiang0: Update golint
Build 1133 daixiang0: bump up golangci to 1.20
Build 1121 pracucci: Publish loki-canary binaries on release
Build 1054 pstibrany: Fix dep check warnings by running dep ensure
Build/release 1018 slim-bean: updating the image version for loki-canary and adding the version increment to the release_prepare script
Build/CI 997 slim-bean: full circle
Build/CI 996 rfratto: ci/drone: fix deploy command by escaping double quotes in JSON body
Build/CI 995 slim-bean: use the loki-build-image for calling circle
Build/CI 994 slim-bean: Also need bash for the deploy step from drone
Build/CI 993 slim-bean: Add make to the alpine image used for calling the circle deploy task from drone.
Build/CI 992 sh0rez: chore(packaging): fix GOPATH being overwritten
Build/CI 991 sh0rez: chore(packaging): deploy from drone
Build/CI 990 sh0rez: chore(ci/cd): breaking the circle
Build 989 sh0rez: chore(packaging): simplify tagging
Build 981 sh0rez: chore(packaging): loki windows/amd64
Build 958 daixiang0: sync release pkgs name with release note
Build/CI 914 rfratto: ci: update apt-get before installing deps for rootless step
Build 911 daixiang0: optimize image tag script

Deployment

Ksonnet 1023 slim-bean: make promtail daemonset name configurable
Ksonnet 1021 rfratto: ksonnet: update memcached and memcached-exporter images
Ksonnet 1020 rfratto: ksonnet: use consistent hashing in memcached client configs
Ksonnet 1017 slim-bean: make promtail configmap name configurable
Ksonnet 946 rfratto: ksonnet: remove prefix from kvstore.consul settings in loki config
Ksonnet 926 slim-bean: feat(promtail): Make cluster role configurable

Helm 1174 rally25rs: loki-stack: Add release name to prometheus service name.
Helm 1152 nicr9: docs(helm): fix broken link to grafana datasource
Helm 1134 minhdanh: Helm chart: Allow additional scrape_configs to be added
Helm 1111 ekarlso: helm: Add support for passing arbitrary secrets
Helm 1110 marcosnils: Bump grafana image in loki helm chart
Helm 1104 marcosnils: : Deploy prometheus from helm chart
Helm 1058 polar3130: Helm: Remove default value of storageClassName in loki/loki helm chart
Helm 1056 polar3130: Helm: Fix the reference error of loki/loki helm chart
Helm 967 makocchi-git: helm chart: Add missing operator to promtail
Helm 937 minhdanh: helm chart: Add support for additional labels and scrapeTimeout for serviceMonitors
Helm 909 angelbarrera92: Feature: Add extra containers to loki helm chart
Helm 855 ikeeip: set helm chart appVersion while release
Helm 675 cyriltovena: Helm default ingester config

Loki Canary

Loki-canary 1137 slim-bean: Add some additional logging to the canary on queries
Loki-canary 1131 rfratto: pkg/canary: use default HTTP client when reading from Loki

Logcli

Logcli 1168 sh0rez: feat(cli): order flags by categories
Logcli 1115 pracucci: logcli: introduced QueryStringBuilder utility to clean up query string encoding
Logcli 1103 pracucci: logcli: added --step support to query command
Logcli 987 joe-elliott: Logcli: Add Support for New Query Path

Tooling

Dashboards 1188 joe-elliott: Adding Operational dashboards
Dashboards 1143 joe-elliott: Improved compression ratio histogram
Dashboards 1126 joe-elliott: Fix Loki Chunks Dashboard
Tools 1108 joe-elliott: Updated push path to current prod

Plugins

DockerDriver 972 cyriltovena: Add stream label to docker driver
DockerDriver 971 cyriltovena: Allow to pass max-size and max-file to the docker driver
DockerDriver 970 mindfl: docker-driver compose labels support

Fluentd 928 candlerb: fluent-plugin-grafana-loki: Escape double-quotes in labels, and suppress labels with value nil

Fluent Bit 1155 cyriltovena: rollback fluent-bit push path until we release 0.4
Fluent Bit 1096 JensErat: fluent-bit: edge case tests
Fluent Bit 847 cosmo0920: fluent-bit shared object go plugin

Misc

Loki is now using a Bot to help keep issues and PR's pruned based on age/relevancy. Please don't hesitate to comment on an issue or PR that you think was closed by the stale-bot which you think should remain open!!

Github 965 rfratto: Change label used to keep issues from being marked as stale to keepalive
Github 964 rfratto: Add probot-stale configuration to close stale issues.

0.3.0 (2019-08-16)

Features/Enhancements

Loki 877 pracucci: loki: Improve Tailer loop
Loki 870 sandlis: bigtable-backup: update docker image for bigtable-backup tool
Loki 862 sandlis: live-tailing: preload all the historic entries before query context is cancelled
Loki 858 pracucci: loki: removed unused TestGZIPCompression
Loki 854 adityacs: Readiness probe for querier
Loki 851 cyriltovena: Add readiness probe to distributor deployment.
Loki 894 rfratto: ksonnet: update ingester config to transfer chunks on rollout

Build 901 sh0rez: chore(packaging): set tag length to 7
Build 900 sh0rez: chore(ci/cd): fix grafanasaur credentials and CircleCI image build
Build 891 sh0rez: chore(ci/cd): build containers using drone.io
Build 888 rfratto: Makefile: disable building promtail with systemd support on non-amd64 platforms
Build 887 slim-bean: chore(packaging): Dockerfile make avoid containers
Build 886 sh0rez: chore(packaging): wrong executable format
Build 855 ikeeip: set helm chart appVersion while release

Promtail 856 martinbaillie: promtail: Add ServiceMonitor and headless Service
Promtail 809 rfratto: Makefile: build promtail with CGO_ENABLED if GOHOSTOS=GOOS=linux
Promtail 730 rfratto: promtail: Add systemd journal support

809, 730 NOTE: Systemd journal support is currently limited to amd64 images, arm support should come in the future when the transition to building the arm image and binaries is done natively via an arm container

Docs 896 dalance: docs: fix link format
Docs 876 BouchaaraAdil: update Docs: update Retention section on Operations doc file
Docs 864 temal-: docs: Replace old values in operations.md
Docs 853 cyriltovena: Add governance documentation

Deployment 874 slim-bean: make our ksonnet a little more modular by parameterizing the chunk and index stores
Deployment 857 slim-bean: Reorder relabeling rules to prevent pod label from overwriting config define labels

857 POSSIBLY BREAKING: If you relied on a custom pod label to overwrite one of the labels configured by the other sections of the scrape config: job, namespace, instance, container_name and/or __path__, this will no longer happen, the custom pod labels are now loaded first and will be overwritten by any of these listed labels.

Fixes

Loki 897 pracucci: Fix panic in tailer when an ingester is removed from the ring while tailing
Loki 880 cyriltovena: fix a bug where nil line buffer would be put back
Loki 859 pracucci: loki: Fixed out of order entries allowed in a chunk on edge case

Promtail 893 rfratto: pkg/promtail/positions: remove executable bit from positions file

Deployment 867 slim-bean: Update read dashboard to include only query and label query routes
Deployment 865 sandlis: fix broken jsonnet for querier

Canary 889 slim-bean: fix(canary): Fix Flaky Tests

Pipeline 869 jojohappy: Pipeline: Fixed labels process test with same objects

Logcli 863 adityacs: Fix Nolabels parse metrics

0.2.0 (2019-08-02)

There were over 100 PR's merged since 0.1.0 was released, here's a highlight:

Features / Enhancements

Loki: 521 Query label values and names are now fetched from the store.
Loki: 541 Improvements in live tailing of logs.
Loki: 713 Storage memory improvement.
Loki: 764 Tailing can fetch previous logs for context.
Loki: 782 Performance improvement: Query storage by iterating through chunks in batches.
Loki: 788 Querier timeouts.
Loki: 794 Support ingester chunk transfer on shutdown.
Loki: 729 Bigtable backup tool support.

Pipeline: 738 Added a template stage for manipulating label values.
Pipeline: 732 Support for Unix timestamps.
Pipeline: 760 Support timestamps without year.

Helm: 641 Helm integration testing.
Helm: 824 Add service monitor.
Helm: 830 Customize namespace.

Docker-Plugin: 663 Created a Docker logging driver plugin.

Fluent-Plugin: 669 Ability to specify keys to remove.
Fluent-Plugin: 709 Multi-worker support.
Fluent-Plugin: 792 Add prometheus for metrics and update gems.

Build: 668,762 Build multiple architecture containers.

Loki-Canary: 772 Moved into Loki project.

Bugfixes

There were many fixes, here are a few of the most important:

Promtail: 650 Build on windows.
Fluent-Plugin: 667 Rename fluent plugin.
Docker-Plugin: 813 Fix panic for newer docker version (18.09.7+).

0.1.0 (2019-06-03)

First (beta) Release!

Files

CHANGELOG.md

Latest commit

History

CHANGELOG.md

File metadata and controls

Main/Unreleased

All Changes

Loki

Enhancements

Fixes

Changes

Promtail

Enhancements

Fixes

LogCLI

Fixes

Mixins

Enhancements

Fixes

FluentD

Enhancements

Jsonnet

2.8.2 (2023-05-03)

Loki

Security

Promtail

Enhancements

2.8.1 (2023-04-24)

Loki

Fixes

Changes

Build

Promtail

Fixes

2.8.0 (2023-04-04)

Loki

Enhancements

Fixes

Changes

Build

Promtail

Enhancements

Fixes

Changes

LogCLI

Enhancement

Fluent Bit

Loki Canary

Enhancements

Jsonnet

Fixes

Build

Notes

Dependencies

2.7.5 (2023-03-28)

Loki

Fixes

2.7.4 (2023-02-24)

Loki

Fixes

Build

Promtail

Enhancements

Fixes

2.7.3 (2023-02-01)

Loki

Fixes

Build

2.7.2 (2023-01-25)

Loki

Fixes

Notes

Dependencies

2.7.1 (2022-12-09)

Loki

Enhancements

Fixes

Changes

Promtail