Flux Meter

See also

Flux Meter reference

The Flux Meter is a type of policy resource that provides a way to instrument requests, matching a Selector, into a Prometheus histogram. The histogram created by Flux Meter measures the workload latency by default.

Example:

flux_meters:
  cart-service1-app:
    selectors:
      - control_point: cart-service2-app
        label_matcher:
          match_labels:
            http.target: /service3
        service: kong-server.demoapp.svc.cluster.local

Naming Convention

Each Flux Meter is identified by its unique name. The generated histogram gets labeled with this name. It is therefore recommended to assign globally unique names to Flux Meters.

The Journey of a Request

This diagram details the journey of a request, as it traverses through the selectors which forward the request to the matching Flux Meters.

In addition to the Flux Meters shown in the diagram, the request might get processed by other matching flow control components before a decision is returned to the service. The service executes its logic based on the decision, and an OpenTelemetry span representing the flow gets generated and forwarded to the Agent. The OTel pipeline in the Agent processes the span(s) and reports the generated metrics to Prometheus.

Metric

The default metric tracked by Flux Meter is the flow's workload duration in milliseconds. The Flux Meter might be configured to track any arbitrary metric from OpenTelemetry attributes on the span or log streams. For example, any of the metrics defined in the Envoy access log specification might be used by Flux Meter.

Buckets

The buckets used by histogram metric can be provided as configuration to Flux Meter. Buckets effect the accuracy of quantile calculations at Prometheus through PromQL. Prometheus recommends defining buckets close to the actual distribution of metric values to achieve good accuracy.

note

Buckets are needed only for quantile queries, for example, getting the 95th percentile of duration across pods in a service. The buckets do not matter if you are only interested in the average duration or throughput metrics from a Flux Meter.

Usage

PromQL components can refer to the metrics generated by Flux Meter. The histogram metric generated by Flux Meter is named flux_meter and has the following labels:

1. flux_meter_name: Name of the Flux Meter metric
2. decision_type: Flow control decision from Agent
3. http_status_code: HTTP status code of the flow. Relevant only for traffic-based Control Points.
4. flow_status: Protocol independent status for the flow.
5. Other common labels available at all Agents, such as instance.

info

For more details, refer to metrics and labels collected by Flux Meters.

Query to get average duration (assuming default Flux Meter metric):

sum(increase(flux_meter_sum{flux_meter_name=\"<name>\"}[10s]))/sum(increase(flux_meter_count{flux_meter_name=\"<name>\"}[10s]))

Query to get requests per second:

sum(rate(flux_meter_count{flux_meter_name=\"<name>\"}[10s]))

Query to get 95th percentile of duration:

histogram_quantile(0.95, sum(rate(flux_meter_bucket{flux_meter_name=\"<name>\"}[5m])) by (le))

PromQL Components in a Circuit can use the Flux Meter metric in a PromQL query as described above. The PromQL Component generates a signal representing the results of the PromQL query.

Use cases

• Control: A control loop can use a duration signal generated using the Flux Meter metric as a signal to throttle the request rate at a service or to stop a feature rollout.
• Observability: Flux Meters can measure and alert on SLOs of a service down to fine-grained API attributes such as endpoints, user types (subscriber user compared to guest user).