How to Cite:
N. Karthiga Devi, "Cost-Aware Data Streaming Optimization for Elastic Cloud Infrastructure" IJERMES, Vol. 1, No. 1, pp. 8-13, 2026.

Abstract:
The rapid growth of Internet of Things (IoT) applications, financial platforms, intelligent transportation systems, online services, and cloud-native enterprise applications has increased the demand for continuous data-stream processing with low latency and high availability. Elastic cloud infrastructure provides the ability to dynamically scale computational and storage resources according to workload variations; however, unrestricted elasticity can introduce substantial operational costs, particularly when streaming workloads exhibit bursty or unpredictable behavior. Conventional stream-processing approaches primarily optimize throughput, latency, and fault tolerance while treating infrastructure cost as a secondary concern. This research proposes a cost-aware data streaming optimization framework that jointly considers workload intensity, processing latency, resource utilization, data-ingestion rates, and cloud resource prices when making scaling and workload-placement decisions. The proposed methodology combines real-time stream monitoring, workload forecasting, cost estimation, adaptive resource allocation, and policy-based scaling. A conceptual evaluation demonstrates that cost-aware optimization can reduce unnecessary resource provisioning while preserving service-level objectives (SLOs). The results indicate that predictive scaling and workload-aware resource allocation are particularly effective for highly variable streaming workloads, whereas reactive scaling remains useful for sudden workload bursts. The study contributes a cloud-native optimization model that integrates economic efficiency with streaming performance and provides a foundation for autonomous resource management in elastic data-processing environments.

Keywords: Cloud Computing, Data Streaming, Cost Optimization, Elastic Infrastructure, Stream Processing, Autoscaling, Workload Prediction, Resource Allocation, Cloud Economics.

References:
[1] Apache Software Foundation. (2024). Apache Kafka documentation. https://kafka.apache.org/documentation/
[2] Apache Software Foundation. (2024). Apache Spark: Structured Streaming programming guide. https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
[3] Buyya, R., Srirama, S. N., Casale, G., Calheiros, R. N., Simmhan, Y., Varghese, B., Gelenbe, E., Javadi, B., Vaquero, L. M., Netto, M. A. S., Toosi, A. N., Rodríguez, M. A., Llorente, I. M., & others. (2018). A manifesto for future generation cloud computing: Research directions for the next decade. ACM Computing Surveys, 51(5), 1–38. https://doi.org/10.1145/3241737
[4] Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., & Tzoumas, K. (2015). Apache Flink: Stream and batch processing in a single engine. IEEE Data Engineering Bulletin, 38(4), 28–38.
[5] Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A distributed messaging system for log processing. In Proceedings of the NetDB Workshop (pp. 1–7).
[6] Mao, M., & Humphrey, M. (2011). Auto-scaling to minimize cost and meet application deadlines in cloud workflows. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 1–11). ACM. https://doi.org/10.1145/2063384.2063449
[7] Shahrad, M., Balkind, J., & Wentzlaff, D. (2017). Architectural implications of variability in cloud applications. In Proceedings of the 2017 IEEE International Symposium on Workload Characterization (pp. 195–204). IEEE.
[8] Vavilapalli, V. K., Murthy, A. C., Douglas, C., Agarwal, S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H., Seth, S., Saha, B., Curino, C., O’Malley, O., & others. (2013). Apache Hadoop YARN: Yet another resource negotiator. In Proceedings of the 4th Annual Symposium on Cloud Computing (pp. 1–16). ACM. https://doi.org/10.1145/2523616.2523633
[9] Zaharia, M., Chen, A., Davidson, A., Ghodsi, A., Hong, S. A., Konwinski, A., Murching, S., Nykodym, T., Ogilvie, M., Parkhe, M., Sen, A., Sottile, R., Xin, R., & Zumar, M. (2016). Apache Spark: A unified engine for big data processing. Communications of the ACM, 59(11), 56–65. https://doi.org/10.1145/2934664