[ADDED] Allow aggregating JS metrics and service observations from multiple accounts in one config

[piotrpio]

This PR adds the ability to have a single config (and single NATS connection) to aggregate metrics and observations from multiple accounts.

In order to achieve that, a valid set of imports/exports has to be present in configuration. This can be achieved either by stream exports from source accounts into aggregate account, or by service exports from aggregate account to source accounts. For examples, look at test/jetstream.conf (for advisories config) and test/services.conf (for service observation config)

• For ServiceObsConfig, new fields are added (ExternalAccountTokenPosition and ExternalServiceNamePosition) to inform surveyor where the account name and service name (optionally) will be found in subject. These have to match with the imports/exports mapping in server config

• For jetstream advisories, in order to use aggregated metrics, these subjects should be exported:

  • $JS.EVENT.METRIC.ACC.*, wildcard is the account name
  • $JS.EVENT.ADVISORY.ACC.*, wildcard is the account name

[caleblloyd]

collector_statz already places account-specific labels under account, not source_account. so it might be good to use account here to remain consistent with that

[caleblloyd]

I think we need to support re-mapping these subjects in the aggregate account. Could be something like:

• metric_subject - default to api.JSAdvisoryPrefix + ".>"
• metric_external_account_token_position - default to 0
• advisory_subject - default to api.JSMetricPrefix + ".>"
• advisory_external_account_token_position - default to 0

As it stands, if JetStream is enabled in the aggregate account its own JS Metrics / Advisories will overlap with the ones we are forcing it to import

[piotrpio]

Well, they would not exactly overlap, as there is the ACC suffix. Although if using wildcards, then then becomes a problem. So I agree, we should move the account position to be either a prefix (as you sugested) or tucked inside the advisory subject, e.g. $JS.API.ACC.<account_name>.METRICS. But if we do so, why do you feel we need to allow re-mapping this subject? It should be unique + it does not collide with any other $JS.API subject.

I'm worried about adding complexity to an already complex feature. For service observations we had to make the subject configurable, there's no way around it. But here, adding 4 more configurable fields when you would probably use the default anyway, seems excessive. Unless I'm missing a use case of course, in which case this has to be added :)

[caleblloyd]

If we're going to leave it hardcoded, we should move it to something completely different such as $surveyor.js.event.acc.{accountID}

It would be nice to be able export a single subject from each account if desired - $JS.EVENT.> instead of 2 for $JS.EVENT.METRIC.> and $JS.EVENT.ADVISORY.>

There is no use case being missed, I just thought it was odd from an API standpoint to offer flexibility on the Service Observations and no flexibility on the JS Advisories.

[piotrpio]

It is possible to export a single subject, take a look at how it's done in test/jetstream.comf:

# export in aggregate account
exports [
    { service: '$JS.EVENT.*.ACC.*.>', account_token_position: 5 }
]
#import in source account
imports [
   { service: { account: aggregate_service, subject: '$JS.EVENT.*.ACC.a.>' }, to: '$JS.EVENT.*.>' }
]

It is important to set the to subject in import to have both the * wildcard (which is normally METRICS or ADVISORIES) as well as > to match the rest of the subjects. If you just do $JS.EVENT.>, server complains that the mapping is invalid.

[piotrpio]

And you're right, this is inconsistent, I will add the ability to configure the advisories subjects

[piotrpio]

I added configuring subjects for advisories + improved validation for both advisories and observations if external config is used.

[caleblloyd]

This looks good, my final thought is that we may want to get rid of the JSON Tags for Account Token position so that it is only exposed in library mode initially. Then once we are happy with Library usage we can add the JSON tags back to expose to CLI mode

[piotrpio]

@caleblloyd I reverted the ability to set external_account_config from file.