Abstract
Operating buildings at a hierarchical level enhances grid flexibility by leveraging the diverse behavior of individual buildings. While Model Predictive Control (MPC) excels at local control of buildings using physics-based models, Reinforcement Learning (RL) offers data-driven adaptability for coordinating systems where detailed modeling is infeasible. This study proposes and evaluates a hierarchical RL-MPC framework given the advantages of each approach for managing energy systems across a cluster of buildings. This framework proposes a learning-based MPC at the decentralized building level to optimize HVAC operations (i.e., regulating indoor air temperature setpoints) based on a local system identification process via a Long-Short-Term Memory (LSTM) model. At the centralized district level, RL control (RLC) is designed to coordinate shared energy resources using thermal and electrical storage systems. The overall hierarchical control framework is implemented in the CityLearn environment, an open-source Gymnasium framework. To meet use-case needs, CityLearn is extended by incorporating additional energy system models, including a Fresnel solar thermal collector, thermal buffer storage, and an absorption chiller. The proposed hierarchical control framework is validated by real-world data from a cluster of buildings in Cyprus and evaluated under both typical and extreme weather conditions. The extreme weather condition is selected according to the MeteoSwiss definition, representing a heatwave period in Cyprus in July 2024. Performance is assessed using key performance indicators (KPIs), including solar self-consumption, ramping, daily peak demand, energy cost, and thermal discomfort. Results show that the proposed hierarchical RL-MPC approach consistently outperforms standalone RLC, particularly in stressed scenarios. Specifically, the hierarchical RL-MPC has achieved improvements in solar self-consumption of 9%, energy cost reduction of 7%, and enhanced thermal discomfort of 20% under typical weather conditions. Under extreme weather conditions, the KPIs improvements are 18%, 17%, and 5%, respectively. These findings highlight the benefits of integrating model-based and data-based control strategies within a hierarchical architecture to improve energy flexibility, efficiency, and resilience in smart energy communities.