Load Scheduling for Elasticsearch

Introduction

By default, this policy detects when the Elasticsearch service is overloaded using the search thread pool queue size metric. The policy is based on the adaptive load scheduling component.

All the Elasticsearch related metrics are collected by the Elasticsearch OpenTelemetry Collector so if the system under observation requires using different metrics for the overload confirmation, the list of available metrics can be used to configure the policy. The following PromQL query (with appropriate filters) is used as SIGNAL for the load scheduler:

avg(elasticsearch_node_thread_pool_tasks_queued)

Configuration

Blueprint name: load-scheduling/elasticsearch

Parameters

policy

Parameter	policy.components
Description	List of additional circuit components.
Type	Array of Object (aperture.spec.v1.Component)
Default Value	Expand []

Parameter	policy.policy_name
Description	Name of the policy.
Type	string
Default Value	__REQUIRED_FIELD__

Parameter	policy.resources
Description	Additional resources.
Type	Object (aperture.spec.v1.Resources)
Default Value	Expand flow_control: classifiers: []

Parameter	policy.elasticsearch
Description	Configuration for Elasticsearch OpenTelemetry receiver. Refer https://docs.fluxninja.com/integrations/metrics/elasticsearch for more information.
Type	Object (elasticsearch)
Default Value	Expand agent_group: default endpoint: __REQUIRED_FIELD__ password: __REQUIRED_FIELD__ username: __REQUIRED_FIELD__

Parameter	policy.search_queue_threshold
Description	Threshold of search thread pool tasks queued at which point to start throttling traffic.
Type	Number (double)
Default Value	__REQUIRED_FIELD__

policy.load_scheduling_core

Parameter	policy.load_scheduling_core.dry_run
Description	Default configuration for setting dry run mode on Load Scheduler. In dry run mode, the Load Scheduler acts as a passthrough and does not throttle flows. This config can be updated at runtime without restarting the policy.
Type	Boolean
Default Value	false

Parameter	policy.load_scheduling_core.kubelet_overload_confirmations
Description	Overload confirmation signals from kubelet.
Type	Object (kubelet_overload_confirmations)
Default Value	Expand {}

Parameter	policy.load_scheduling_core.overload_confirmations
Description	List of overload confirmation criteria. Load scheduler can throttle flows when all of the specified overload confirmation criteria are met.
Type	Array of Object (overload_confirmation)
Default Value	Expand []

Parameter	policy.load_scheduling_core.aiad_load_scheduler
Description	Parameters for AIMD throttling strategy.
Type	Object (aperture.spec.v1.AIADLoadSchedulerParameters)
Default Value	Expand alerter: alert_name: AIAD Load Throttling Event load_multiplier_linear_decrement: 0.05 load_multiplier_linear_increment: 0.025 load_scheduler: selectors: - control_point: __REQUIRED_FIELD__ max_load_multiplier: 2 min_load_multiplier: 0

---

Schemas

driver_criteria

Parameter	enabled
Description	Enables the driver.
Type	Boolean
Default Value	__REQUIRED_FIELD__

Parameter	threshold
Description	Threshold for the driver.
Type	Number (double)
Default Value	__REQUIRED_FIELD__

---

overload_confirmation_driver

Parameter	pod_cpu
Description	The driver for using CPU usage as overload confirmation.
Type	Object (driver_criteria)
Default Value	Expand {}

Parameter	pod_memory
Description	The driver for using memory usage as overload confirmation.
Type	Object (driver_criteria)
Default Value	Expand {}

---

kubelet_overload_confirmations

Parameter	criteria
Description	Criteria for overload confirmation.
Type	Object (overload_confirmation_driver)
Default Value	__REQUIRED_FIELD__

Parameter	infra_context
Description	Kubernetes selector for scraping metrics.
Type	Object (aperture.spec.v1.KubernetesObjectSelector)
Default Value	__REQUIRED_FIELD__

---

overload_confirmation

Parameter	operator
Description	The operator for the overload confirmation criteria. oneof: `gt | lt | gte | lte | eq | neq`
Type	string
Default Value

Parameter	query_string
Description	The Prometheus query to be run. Must return a scalar or a vector with a single element.
Type	string
Default Value

Parameter	threshold
Description	The threshold for the overload confirmation criteria.
Type	Number (double)
Default Value

---

elasticsearch

Parameter	agent_group
Description	Name of the Aperture Agent group.
Type	string
Default Value	default

Parameter	endpoint
Description	Endpoint of the Elasticsearch.
Type	string
Default Value	__REQUIRED_FIELD__

Parameter	indices
Description	Index filters that define which indices are scraped for index-level metrics.
Type	Array of string
Default Value

Parameter	initial_delay
Description	Defines how long this receiver waits before starting.
Type	string
Default Value

Parameter	nodes
Description	Node filters that define which nodes are scraped for node-level and cluster-level metrics.
Type	Array of string
Default Value

Parameter	password
Description	Password of the Elasticsearch.
Type	string
Default Value	__REQUIRED_FIELD__

Parameter	skip_cluster_metrics
Description	If true, cluster-level metrics will not be scraped.
Type	Boolean
Default Value

Parameter	username
Description	Username of the Elasticsearch.
Type	string
Default Value	__REQUIRED_FIELD__

---

Dynamic Configuration

note

The following configuration parameters can be dynamically configured at runtime, without reloading the policy.

Parameters

Parameter	dry_run
Description	Dynamic configuration for setting dry run mode at runtime without restarting this policy. In dry run mode the scheduler acts as pass through to all flow and does not queue flows. It is useful for observing the behavior of load scheduler without disrupting any real traffic.
Type	Boolean
Default Value	__REQUIRED_FIELD__

---