Managing uncertainties of ecologically sustainable real-time multimedia event processing over the cloud and edge

This thesis presents a framework for managing uncertainties in ecologically sustainable real-time Multimedia Event Processing (MEP) systems across Cloud and Edge environments. As the demand for Big Data applications and Internet of Things (IoT) devices grows, so does the energy consumption required to support data centres and Edge devices. This has led to an urgent need for energy-efficient solutions, particularly in the context of the Internet of Multimedia Things (IoMT), where MEP systems play a vital role in applications such as smart cities, health monitoring, and autonomous driving. These systems rely on computationally intensive Deep Neural Networks (DNNs) for tasks like computer vision (CV), which can drive up both energy consumption and computational demands. Therefore, this work focuses on the optimisation of real-time MEP systems, seeking to balance the trade-offs between accuracy, speed, and energy consumption while addressing the uncertainties introduced by real-world deployment scenarios. The main contribution of this thesis are the 32 open-source resources that combined provide a real-time sustainable MEP framework that adapts to varying Quality of Service (QoS) requirements by effectively managing the accuracy-speed-energy trade-offs and the uncertainties from real-world deployments. The framework refines an existing MEP system by introducing a query-specific optimisation approach for balancing these trade-offs. It integrates a new event-sourcing-based architecture that improves the flexibility and adaptability of the system. This architectural shift enables 87% faster adaptation times compared to the previous version, which was crucial to reduce a major source of uncertainty in the adaptation goals. A WebRTC-based video streaming service was also incorporated to ensure sub-second latency in delivering processed video streams to end users, satisfying real-time requirements for mission-critical scenarios. In addition to the core framework improvements, this work introduces a novel real-time early filtering pipeline for high-definition video analytics on commodity Edge devices, such as Raspberry Pis, which are resource-constrained and lack GPUs. The pipeline uses an Edge-optimised hierarchical image pre-processing method that accelerates the performance of the CV models, enabling the processing of video at speeds exceeding 80 frames per second (FPS) at 1080p resolution. This approach leverages context-aware machine learning models that are fine-tuned using a data augmentation technique, which requires just 10 seconds of original footage from the target camera. The synthetic data augmentation, using Generative AI and Cut-and-Paste methods, trains the model to the specific background and objects of interest in the video stream, allowing the filter to operate in real-time, even in resource-constrained environments. This early filtering pipeline outperforms the state-of-the-art Nano-YoloV5 model and other state-of-the-art Edge video analytics enabling methods, achieving 48.8 times higher speed and maintaining competitive accuracy. Furthermore, this thesis addresses the challenge of managing uncertainties in MEP systems, which can arise from imprecision in hardware profiling, variability in user QoS requirements, and delays in system monitoring. By transforming the adaptation process into a multi-objective optimisation problem, the framework incorporates uncertainty-aware mechanisms that handle different types of uncertainty. A fuzzy-based Multi-Criteria Decision Making (MCDM) method was developed to rank service workers while managing imprecision in hardware measurements and user criteria. Experimental results demonstrate that the uncertainty-aware approach yields improvements in energy efficiency and latency, saving up to 1.2 kilowatt-hours of energy and reducing latency by 213 seconds, with only a minimal accuracy loss of 0.29%. In conclusion, this work contributes to the development of real-time sustainable MEP systems that are capable of operating efficiently and effectively in dynamic and uncertain real-world environments. The open-source framework, designed for both Cloud and Edge devices, provides a robust and extensible solution for real-time multimedia analytics, with the potential for wide application in IoT and smart-city scenarios.

Funder

European Regional Development Fund
Science Foundation Ireland

Publisher

University of Galway

Rights

Attribution-NonCommercial-NoDerivatives 4.0 International

cbn

Collections

University of Galway Theses (PhD Theses)