Engarde - Stay on guard with Envoy Access Logs

Parse default envoy access logs like a champ with engarde and jq

Motivation

Envoy access log messages are packed with a lot of useful information but in an unstructured log format. Without prior context, or even with context it can get cumbersome to visually inspect these log messages to extract useful information.

I was inspired by this tweet from Megan O'Keefe on twitter to create this tool for better readability of the envoy/istio-proxy access logs.

In addition, a special shout out to Richard Li from ambassador.io for this excellent article that provides more details on each of the subcomponents of the log message.

Installing

Homebrew

brew tap nitishm/homebrew-engarde
brew install engarde

Scoop

scoop bucket add engarde https://github.com/nitishm/scoop-bucket-engarde
scoop install engarde

Source

go get github.com/nitishm/engarde

This should install the compiled binary to your $GOBIN (or $GOPATH/bin).

Otherwise, clone the repository and build the binary manually using,

This package uses gomodules. Ensure that GO111MODULE=on is set if building outside $GOPATH in go version 1.11+

git clone https://github.com/nitishm/engarde.git
cd engarde/
go build -o engarde .
mv engarde /usr/local/bin/
export PATH=$PATH:/usr/local/bin/

Example (default format only)

Prerequisites

jq must be in your $PATH. If you do not have jq installed please download the binary here

Envoy

echo '[2016-04-15T20:17:00.310Z] "POST /api/v1/locations HTTP/2" 204 - 154 0 226 100 "10.0.35.28" "nsq2http" "cc21d9b0-cf5c-432b-8c7e-98aeb7988cd2" "locations" "tcp://10.0.2.1:80"' | engarde | jq

2019/09/03 22:31:33 Reading input from STDIN. Use the pipe "|" operator to redirect traffic to engarde
{
  "authority": "locations",
  "bytes_received": "154",
  "bytes_sent": "0",
  "duration": "226",
  "method": "POST",
  "protocol": "HTTP/2",
  "request_id": "cc21d9b0-cf5c-432b-8c7e-98aeb7988cd2",
  "response_flags": "-",
  "status_code": "204",
  "timestamp": "2016-04-15T20:17:00.310Z",
  "upstream_service": "tcp://10.0.2.1:80",
  "upstream_service_time": "100",
  "uri_path": "/api/v1/locations",
  "user_agent": "nsq2http",
  "original_message": "[2016-04-15T20:17:00.310Z] \"POST /api/v1/locations HTTP/2\" 204 - 154 0 226 100 \"10.0.35.28\" \"nsq2http\" \"cc21d9b0-cf5c-432b-8c7e-98aeb7988cd2\" \"locations\" \"tcp://10.0.2.1:80\""
}

Istio Proxy

kubectl logs -f foo-app-1 -c istio-proxy | engarde --use-istio | jq

{
  "authority": "hello-world",
  "bytes_received": "148",
  "bytes_sent": "171",
  "duration": "4",
  "method": "GET",
  "protocol": "HTTP/1.1",
  "request_id": "c0ce81db-4f5a-9134-8a5c-f8c076c91652",
  "response_flags": "-",
  "status_code": "200",
  "timestamp": "2019-09-03T05:37:41.341Z",
  "upstream_service": "192.168.89.50:9001",
  "upstream_service_time": "3",
  "upstream_cluster": "outbound|80||hello-world.default.svc.cluster.local",
  "upstream_local": "-",
  "downstream_local": "10.97.86.53:80",
  "downstream_remote": "192.168.167.113:39953",
  "uri_path": "/index",
  "user_agent": "-",
  "mixer_status": "-",
  "original_message": "[2019-09-03T05:37:41.341Z] \"GET /index HTTP/1.1\" 200 - \"-\" 148 171 4 3 \"-\" \"-\" \"c0ce81db-4f5a-9134-8a5c-f8c076c91652\" \"hello-world\" \"192.168.89.50:9001\" outbound|80||hello-world.default.svc.cluster.local - 10.97.86.53:80 192.168.167.113:39953 -"
}

Projects using Engarde

A UI powered by Engarde and Envoy docs for viewing envoy access logs on the browser : greghanson/engarde-viewer