How to Cite:
Durga Marimuthu, "Real-Time Data Streaming Pipelines for Cloud-Based AI and Machine Learning Workloads" IJERMES, Vol. 1, No. 1, pp. 1-7, 2026.

Abstract:
The rapid growth of artificial intelligence (AI) and machine learning (ML) applications has created an increasing demand for real-time data processing architectures capable of continuously ingesting, transforming, analyzing, and delivering high-velocity data. Traditional batch-oriented data pipelines are often inadequate for AI and ML workloads that require low-latency access to continuously changing information, particularly in applications such as fraud detection, recommendation systems, predictive maintenance, intelligent cybersecurity, Internet of Things (IoT) analytics, and autonomous decision-making. This research examines the architecture and operational characteristics of real-time data streaming pipelines designed for cloud-based AI and ML workloads. The proposed conceptual framework integrates distributed event ingestion, stream processing, feature engineering, model inference, monitoring, and elastic cloud resource management into a unified pipeline. Technologies such as Apache Kafka, Apache Flink, Apache Spark Structured Streaming, cloud object storage, feature stores, containerized model-serving platforms, and Kubernetes are considered as representative components. The study evaluates the pipeline according to latency, throughput, scalability, fault tolerance, data freshness, resource utilization, and model-serving efficiency. The analysis indicates that event-driven streaming architectures can significantly improve data freshness and inference responsiveness when compared with conventional batch pipelines. However, challenges remain in state management, exactly-once processing, feature consistency, infrastructure elasticity, model versioning, and operational governance. The research identifies an important gap between general-purpose cloud streaming systems and AI-specific streaming pipelines, particularly concerning coordinated optimization of data processing and model inference. The proposed architecture provides a foundation for developing adaptive, scalable, and cost-aware streaming infrastructures for next-generation cloud AI applications.

Keywords: Real-Time Data Streaming, Cloud Computing, Machine Learning, Artificial Intelligence, Apache Kafka, Apache Flink, Stream Processing, ML Inference, Feature Engineering, Cloud-Native Architecture

References:
[1] Armbrust, M., Das, T., Davidson, A., Ghodsi, A., Or, A., Rosen, J., Stoica, I., Xin, R., & Zaharia, M. (2018). Structured streaming: A declarative API for real-time applications in Apache Spark. Proceedings of the 2018 International Conference on Management of Data, 601–613. https://doi.org/10.1145/3183713.3190664
[2] Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. (2016). Borg, Omega, and Kubernetes. ACM Queue, 14(1), 70–93. https://doi.org/10.1145/2898442.2898444
[3] Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., & Tzoumas, K. (2015). Apache Flink: Stream and batch processing in a single engine. IEEE Data Engineering Bulletin, 38(4), 28–38.
[4] Dean, J., & Ghemawat, S. (2008). MapReduce: Simplified data processing on large clusters. Communications of the ACM, 51(1), 107–113. https://doi.org/10.1145/1327452.1327492
[5] Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A distributed messaging system for log processing. Proceedings of the NetDB Workshop, 1–7.
[6] Kreps, J. (2014). Questioning the Lambda Architecture. O'Reilly Radar. https://www.oreilly.com/radar/questioning-the-lambda-architecture/
[7] Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., & Dennison, D. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems, 28, 2503–2511.
[8] Stonebraker, M., Çetintemel, U., & Zdonik, S. (2005). The 8 requirements of real-time stream processing. ACM SIGMOD Record, 34(4), 42–47. https://doi.org/10.1145/1107499.1107504
[9] Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., & Stoica, I. (2013). Discretized streams: Fault-tolerant streaming computation at scale. Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, 423–438. https://doi.org/10.1145/2517349.2522737
[10] Zaharia, M., Xin, R. S., Wendell, P., Das, T., Armbrust, M., Dave, A., Meng, X., Rosen, J., Venkataraman, S., Franklin, M. J., Ghodsi, A., Gonzalez, J., Shenker, S., & Stoica, I. (2016). Apache Spark: A unified engine for big data processing. Communications of the ACM, 59(11), 56–65. https://doi.org/10.1145/2934664