Support evaluating metrics and switching states

[ripienaar]

To deliver on the hopes of reacting to metric values we'd need to initiate state changes.

I imagine you could add some queries to the metric watcher using expr and based on their boolean returns you'd do success/fail transitions.

Today we support storing the metrics to machine data (#2157) but I'd like to eventually improve on that by storing in a small localised tsdb, think hours of data not days, so that expr functions can do things lke average over recent metrics etc

But we can start with just the stored data for now

[ripienaar]

name: demo
type: expression
state_match:
  - X
  - Y
fail_transition: FAIL
success_transition: SUCCESS
interval: 1m
properties:
  fail_when: ... # expression thats true or false, on true it will trigger fail transition
  success_when: ... # expression thats true or false, on true it will trigger success transition

false from the expressions will just be ignored. We'd add a few functions to allow looking up data etc

The data layer should get a new hook where these watchers can register for updates so they will get notified if data changes and also when they enter the matching states. It will also check on interval to cater for external updates to data

We'd need some flip-flop or true-if-already-true induced repeated transitions. Typically I think such a watcher would be sat on state_match on both the success and fail target states so it would be possible that once true each true will trigger. So will need to cache previous state like many others watchers do and ignore if previous == current to avoid this.