Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce configurable parameters for ES client node draining #407

Merged
merged 13 commits into from
May 16, 2024
Merged

Introduce configurable parameters for ES client node draining #407

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
53d434c
Add draining specification to eds.
Apr 29, 2024
121a871
Inject draining specification to ESClient.
Apr 29, 2024
9cffaa7
Add tests for fetching elasticsearch draining specification.
Apr 29, 2024
da7f656
Use draining config in elasticsearch client.
Apr 29, 2024
4dbe951
Remove unused constants to control draining behaviour in elasticsearch.
Apr 29, 2024
8e73bea
Add draining configuration to tests to they will pass.
Apr 29, 2024
860b76a
Reformat code for es_client.
Apr 29, 2024
c209e44
Refactor: Rename "systemUnderTest" with "client" to prevent shadowing.
Apr 30, 2024
911e84a
Test: Add `TestDrainRetriesUntilMax`
Apr 30, 2024
b98db2c
Test: Add `TestDrainRetriesUntilSuccess`
Apr 30, 2024
ceff6d2
Update Readme to reflect changes in draining configurations.
Apr 30, 2024
9b3a058
Put draining configuration under experimental property to be able to …
May 14, 2024
b2e5752
Resolve errors in tests.
May 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 31 additions & 24 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,34 +61,41 @@ spec:
scaleDownThresholdDurationSeconds: 1800
scaleDownCooldownSeconds: 3600
diskUsagePercentScaledownWatermark: 80
experimental:
draining:
maxRetries: 999
maximumWaitTimeDurationSeconds: 30
minimumWaitTimeDurationSeconds: 10
```

### Custom resource properties


| Key | Description | Type |
|----------|---------------|---------|
| spec.replicas | Initial size of the StatefulSet. If auto-scaling is disabled, this is your desired cluster size. | Int |
| spec.excludeSystemIndices | Enable or disable inclusion of system indices like '.kibana' when calculating shard-per-node ratio and scaling index replica counts. Those are usually managed by Elasticsearch internally. Default is false for backwards compatibility | Boolean |
| spec.skipDraining | Allows the ES Operator to terminate an Elasticsearch node without re-allocating its data. This is useful for persistent disk setups, like EBS volumes. Beware that the ES Operator does not verify that you have more than one copy of your indices and therefore wouldn't protect you from potential data loss. (default=false) | Boolean |
| spec.scaling.enabled | Enable or disable auto-scaling. May be necessary to enforce manual scaling. | Boolean |
| spec.scaling.minReplicas | Minimum Pod replicas. Lower bound (inclusive) when scaling down. | Int |
| spec.scaling.maxReplicas | Maximum Pod replicas. Upper bound (inclusive) when scaling up. | Int |
| spec.scaling.minIndexReplicas | Minimum index replicas. Lower bound (inclusive) when reducing index copies. (reminder: total copies is replicas+1 in Elasticsearch) | Int |
| spec.scaling.maxIndexReplicas | Maximum index replicas. Upper bound (inclusive) when increasing index copies. | Int |
| spec.scaling.minShardsPerNode | Minimum shard per node ratio. When reached, scaling up also requires adding more index replicas. | Int |
| spec.scaling.maxShardsPerNode | Maximum shard per node ratio. Boundary for scaling down. | Int |
| spec.scaling.scaleUpCPUBoundary | (Median) CPU consumption/request ratio to consistently exceed in order to trigger scale up. | Int |
| spec.scaling.scaleUpThresholdDurationSeconds | Duration in seconds required to meet the scale-up criteria before scaling. | Int |
| spec.scaling.scaleUpCooldownSeconds | Minimum duration in seconds between two scale up operations. | Int |
| spec.scaling.scaleDownCPUBoundary | (Median) CPU consumption/request ratio to consistently fall below in order to trigger scale down. | Int |
| spec.scaling.scaleDownThresholdDurationSeconds | Duration in seconds required to meet the scale-down criteria before scaling. | Int |
| spec.scaling.scaleDownCooldownSeconds | Minimum duration in seconds between two scale-down operations. | Int |
| spec.scaling.diskUsagePercentScaledownWatermark | If disk usage on one of the nodes exceeds this threshold, scaling down will be prevented. | Float |
| status.lastScaleUpStarted | Timestamp of start of last scale-up activity | Timestamp |
| status.lastScaleUpEnded | Timestamp of end of last scale-up activity | Timestamp |
| status.lastScaleDownStarted | Timestamp of start of last scale-down activity | Timestamp |
| status.lastScaleDownEnded | Timestamp of end of last scale-down activity | Timestamp |
| Key | Description | Type |
|-----------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|
| spec.replicas | Initial size of the StatefulSet. If auto-scaling is disabled, this is your desired cluster size. | Int |
| spec.excludeSystemIndices | Enable or disable inclusion of system indices like '.kibana' when calculating shard-per-node ratio and scaling index replica counts. Those are usually managed by Elasticsearch internally. Default is false for backwards compatibility | Boolean |
| spec.skipDraining | Allows the ES Operator to terminate an Elasticsearch node without re-allocating its data. This is useful for persistent disk setups, like EBS volumes. Beware that the ES Operator does not verify that you have more than one copy of your indices and therefore wouldn't protect you from potential data loss. (default=false) | Boolean |
| spec.scaling.enabled | Enable or disable auto-scaling. May be necessary to enforce manual scaling. | Boolean |
| spec.scaling.minReplicas | Minimum Pod replicas. Lower bound (inclusive) when scaling down. | Int |
| spec.scaling.maxReplicas | Maximum Pod replicas. Upper bound (inclusive) when scaling up. | Int |
| spec.scaling.minIndexReplicas | Minimum index replicas. Lower bound (inclusive) when reducing index copies. (reminder: total copies is replicas+1 in Elasticsearch) | Int |
| spec.scaling.maxIndexReplicas | Maximum index replicas. Upper bound (inclusive) when increasing index copies. | Int |
| spec.scaling.minShardsPerNode | Minimum shard per node ratio. When reached, scaling up also requires adding more index replicas. | Int |
| spec.scaling.maxShardsPerNode | Maximum shard per node ratio. Boundary for scaling down. | Int |
| spec.scaling.scaleUpCPUBoundary | (Median) CPU consumption/request ratio to consistently exceed in order to trigger scale up. | Int |
| spec.scaling.scaleUpThresholdDurationSeconds | Duration in seconds required to meet the scale-up criteria before scaling. | Int |
| spec.scaling.scaleUpCooldownSeconds | Minimum duration in seconds between two scale up operations. | Int |
| spec.scaling.scaleDownCPUBoundary | (Median) CPU consumption/request ratio to consistently fall below in order to trigger scale down. | Int |
| spec.scaling.scaleDownThresholdDurationSeconds | Duration in seconds required to meet the scale-down criteria before scaling. | Int |
| spec.scaling.scaleDownCooldownSeconds | Minimum duration in seconds between two scale-down operations. | Int |
| spec.scaling.diskUsagePercentScaledownWatermark | If disk usage on one of the nodes exceeds this threshold, scaling down will be prevented. | Float |
| spec.experimental.draining.maxRetries | MaxRetries specifies the maximum number of attempts to drain a node. | Int |
| spec.experimental.draining.maximumWaitTimeDurationSeconds | MaximumWaitTimeDurationSeconds specifies the maximum wait time in seconds between retry attempts after a failed node drain. | Int |
| spec.experimental.draining.minimumWaitTimeDurationSeconds | MMinimumWaitTimeDurationSeconds specifies the minimum wait time in seconds between retry attempts after a failed node drain. | Int |
| status.lastScaleUpStarted | Timestamp of start of last scale-up activity | Timestamp |
| status.lastScaleUpEnded | Timestamp of end of last scale-up activity | Timestamp |
| status.lastScaleDownStarted |  Timestamp of start of last scale-down activity | Timestamp |
| status.lastScaleDownEnded |  Timestamp of end of last scale-down activity | Timestamp |


## How it scales
Expand Down
8 changes: 7 additions & 1 deletion cmd/e2e/test_environment.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"fmt"
"net/url"
"os"
"time"

"github.com/sirupsen/logrus"
"github.com/zalando-incubator/es-operator/operator"
Expand Down Expand Up @@ -90,5 +91,10 @@ func setupESClient(defaultServiceEndpoint, version string) (*operator.ESClient,
if err != nil {
return nil, err
}
return &operator.ESClient{Endpoint: endpoint}, nil
config := &operator.DrainingConfig{
MaxRetries: 999,
MinimumWaitTime: 10 * time.Second,
MaximumWaitTime: 30 * time.Second,
}
return &operator.ESClient{Endpoint: endpoint, DrainingConfig: config}, nil
}
38 changes: 38 additions & 0 deletions docs/zalando.org_elasticsearchdatasets.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,44 @@ spec:
description: Exclude management of System Indices on this Data Set.
Defaults to false
type: boolean
experimental:
description: Experimental represents configurations marked as experimental
that may change in future releases. Currently, manages the draining
behavior.
properties:
draining:
description: Draining controls behaviour of the EDS while draining
nodes
properties:
maxRetries:
default: 999
description: MaxRetries specifies the maximum number of attempts
to drain a node. The default value is 999.
format: int32
minimum: 0
type: integer
maximumWaitTimeDurationSeconds:
default: 30
description: MaximumWaitTimeDurationSeconds specifies the
maximum wait time in seconds between retry attempts after
a failed node drain. The default value is 30 seconds.
format: int64
minimum: 0
type: integer
minimumWaitTimeDurationSeconds:
default: 10
description: MinimumWaitTimeDurationSeconds specifies the
minimum wait time in seconds between retry attempts after
a failed node drain. The default value is 10 seconds.
format: int64
minimum: 0
type: integer
required:
- maxRetries
- maximumWaitTimeDurationSeconds
- minimumWaitTimeDurationSeconds
type: object
type: object
replicas:
description: |-
Number of desired pods. This is a pointer to distinguish between explicit
Expand Down
29 changes: 27 additions & 2 deletions operator/elasticsearch.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,13 @@ type operatingEntry struct {
logger *log.Entry
}

// DrainingConfig specifies the configuration settings for the behavior of draining Elasticsearch nodes.
type DrainingConfig struct {
MaxRetries int
MinimumWaitTime time.Duration
MaximumWaitTime time.Duration
}

// NewElasticsearchOperator initializes a new ElasticsearchDataSet operator instance.
func NewElasticsearchOperator(
client *clientset.Clientset,
Expand Down Expand Up @@ -241,10 +248,10 @@ func (o *ElasticsearchOperator) runAutoscaler(ctx context.Context) {
for _, es := range resources {
if es.ElasticsearchDataSet.Spec.Scaling != nil && es.ElasticsearchDataSet.Spec.Scaling.Enabled {
endpoint := o.getElasticsearchEndpoint(es.ElasticsearchDataSet)

client := &ESClient{
Endpoint: endpoint,
excludeSystemIndices: es.ElasticsearchDataSet.Spec.ExcludeSystemIndices,
DrainingConfig: o.getDrainingConfig(es.ElasticsearchDataSet),
}

err := o.scaleEDS(ctx, es.ElasticsearchDataSet, es, client)
Expand Down Expand Up @@ -676,7 +683,8 @@ func (o *ElasticsearchOperator) operateEDS(eds *zv1.ElasticsearchDataSet, delete

// TODO: abstract this
client := &ESClient{
Endpoint: endpoint,
Endpoint: endpoint,
DrainingConfig: o.getDrainingConfig(eds),
}

operator := &Operator{
Expand Down Expand Up @@ -731,6 +739,23 @@ func (o *ElasticsearchOperator) getElasticsearchEndpoint(eds *zv1.ElasticsearchD
}
}

// DrainingConfig returns the draining specification which control how should we handle draining nodes.
func (o *ElasticsearchOperator) getDrainingConfig(eds *zv1.ElasticsearchDataSet) *DrainingConfig {
// Fallback to default configurations if draining configuration is not specified.
if eds.Spec.Experimental == nil || eds.Spec.Experimental.Draining == nil {
return &DrainingConfig{
MaxRetries: 999,
MinimumWaitTime: 10 * time.Second,
MaximumWaitTime: 30 * time.Second,
}
}
return &DrainingConfig{
MaxRetries: int(eds.Spec.Experimental.Draining.MaxRetries),
MinimumWaitTime: time.Duration(eds.Spec.Experimental.Draining.MinimumWaitTimeDurationSeconds) * time.Second,
MaximumWaitTime: time.Duration(eds.Spec.Experimental.Draining.MaximumWaitTimeDurationSeconds) * time.Second,
}
}

type ESResource struct {
ElasticsearchDataSet *zv1.ElasticsearchDataSet
StatefulSet *appsv1.StatefulSet
Expand Down
Loading
Loading