Increase replication factor for some blocks on store-gateways

[56quarters]

Each block owned by store-gateways is replicated to three store-gateways. When there are many queries that touch a particular block this can result in unbalanced CPU usage between store-gateways leading to higher costs. By far, recent blocks are queried more than older blocks.

From an internal cluster, we see that most queries only touch the most recent data:

• ~92% of Select() calls that hit store-gateways touch data from the last 25h
• ~50% of Select() calls that hit store-gateways touch data from the last 73h
• Less than 1% of Select() calls that hit store-gateways touch data older than 28d ago
• Less than 0.1% of Select() calls that hit store-gateways touch data older than 30d ago

In order to support spreading load for more recent blocks to more store-gateways, we should introduce the ability to override the configured replication factor (three by default) to something higher. The mechanism for picking overridden replication factor may be configurable or may be based on a variety of factors.

This issue proposed to add the ability to override the replication factor and default behavior or doing this based on the age or duration of blocks and iterating on the exact behavior in further PRs.

copied from an internal issue and discussion