Recently, the team of Zhang Kai has made new progress in the field of reservoir production optimization, and the related research has been published in the SPE Journal. The paper is titled " Dual- Reward Reinforcement Learning with Intrinsic Exploration Mechanisms for Real-Time Reservoir Management".
Innovation: Although existing reinforcement learning–based reservoir production optimization methods have shown strong potential in high-dimensional dynamic decision-making problems, they still suffer from insufficient global exploration capability and weak representation of interwell relationships in complex reservoir environments. To address these challenges, this study proposes a dual-reward coupled reinforcement learning production optimization method from a global perspective. By constructing a collaborative mechanism between extrinsic rewards and intrinsic exploration rewards, the proposed method guides the agent to conduct more comprehensive global exploration in complex reservoir environments. Meanwhile, a graph neural network is introduced to model well-pattern topology and interwell connectivity, thereby enhancing the model’s ability to characterize dynamic reservoir features. Specifically, the reservoir production optimization problem is formulated as a Markov decision process, and a random network distillation method is employed to construct the intrinsic reward mechanism. Combined with deep reinforcement learning, the proposed framework enables real-time injection-production optimization. Results demonstrate that the proposed method effectively improves global search capability in complex reservoirs, achieving higher net present value and superior waterflooding performance.
Abstract: In the process of oil and gas reservoir development, the natural heterogeneity of the reservoir makes the development performance extremely sensitive to the matching of the injection-production well control scheme. Reasonable production optimization is the key to achieving balanced displacement and efficient development. Due to the complexity of oil and gas reservoirs and the large number of variables, traditional production optimization methods often face problems, such as complex and time-consuming calculations, hindering their ability to obtain global optimal solutions. Reinforcement learning (RL), with its intrinsic strengths in sequential decision-making and high-dimensional problem solving, offers a promising alternative for production optimization. Nevertheless, existing RL approaches exhibit limited exploration capabilities in complex reservoir environments, often becoming trapped in local optima. Additionally, they typically fail to incorporate well pattern information effectively, overlooking the location information and the connectivity between injection and production wells. To address these challenges, we propose a dual-reward coupled RL production optimization algorithm based on a global perspective. Specifically, the production optimization problem will be modeled as a Markov decision process. Through the intrinsic reward mechanism, the agent is encouraged to conduct deeper exploration to overcome the optimization limitations in complex environments, and the well network production information is integrated into the model in combination with the graph neural network (GNN) to enhance the modeling ability of the dynamic characteristics of the reservoir. In this way, the agent can effectively avoid relying only on local objectives for optimization and then achieve a more balanced injection and production strategy. The proposed method adaptively learns by engaging in intrinsic-extrinsic directional interactions with the uncertain reservoir environment, effectively leveraging accumulated well control experience in a manner that is similar to real-world field operating mode. Simulation results derived from two reservoir models demonstrate that, in comparison with other optimization methods, the proposed approach achieves higher net present value (NPV) and exhibits outstanding performance in oil displacement.
The new expanded SPE Journal covers the fundamental research and practical aspects of subsurface energy resources for sustainable oil and gas exploration and production, while broadening its focus on circular carbon economy, energy transition, and alternative/renewable energy sources. General topics include but are not limited to drilling and completion, production and operations, reservoir engineering, formation evaluation, petrophysics, rock physics, geology, geophysics, geochemistry, geomechanics, numerical simulation and modeling, injection/producing facilities, oilfield chemistry, water management and treatment, carbon capture, utilization, and storage (CCUS), hydrogen transport and storage, geothermal energy, digitalization, artificial intelligence, data analytics, economics, health, safety, environment, sustainability, and other special topics of current interest. The journal currently has an impact factor of 3.0. This journal currently appears in the JCR Q2 or the Chinese Academy of Sciences ranking 3.
Paper link:
https://doi.org/10.2118/228320-PA
Citation:
Xin Guojing, Zhang Kai, Wang Haochen, Sun Zifeng, Zhang Liming, Liu Piyang, Wang Yang, Yao Jun. Dual-Reward Reinforcement Learning with Intrinsic Exploration Mechanisms for Real-Time Reservoir Management[J]. SPE Journal, 2025, 30(08): 4501-4517.
