Push versus Pull Telemetry at Cloud Scale: Performance Benchmarking in Kubernetes-Based OpenTelemetry Environments

Abstract

In today's cloud-native environments, telemetry pipelines are becoming more and more vital for maintaining visibility, distributed traceability, automated monitoring, and intelligent incident response for large-scale containerized environments. Most observability ecosystems today are either push or pull based, with push-based telemetry architectures having agents and exporters actively publish metrics and traces to central collectors, or pull-based architectures having monitoring systems actively pollute observability endpoints exposed by services and infrastructure nodes. Although both paradigms have been widely embraced in industry, there are few empirical comparisons that can be reproduced to assess their performance characteristics under operational conditions in the cloud. This paper provides a comprehensive benchmark study to understand and analyze the performance difference between push-based vs. pull-based telemetry architectures, with the emphasis on evaluating push-based telemetry in the context of Kubernetes cluster environments, equipped with OpenTelemetry collectors, Prometheus-based monitoring system, and synthetic workload generators to simulate production-like workloads. The study assesses end to end latency distributions, telemetry throughput, operational overhead, cost per event, end to end system behavior when put under sustained workloads, and end to end system recovery behavior after induced failures. Experimental evidence shows that the push architecture can be more adaptable under burst-intensive loads due to its flexibility in buffering mechanisms and asynchronous delivery of events, whereas the pull architecture has better consistency of query results, easier visibility of the operation, and more predetermined semantics for monitoring under steady load. This paper presents a hybrid telemetry architecture especially designed for the context of AI/machine learning infrastructure environments where the burst-resilience and scalable query semantics are concurrently required. The proposed framework combines push side buffering, and ingestion flexibility with pull side observability and analytics. The study introduces a benchmark methodology that is reproducible and experimentally validated hybrid telemetry design that can enhance the observability performance of modern distributed cloud-native infrastructure.