Recently, Zhang Kai's team has made significant progress in the field of automatic history matching in reservoirs. Their research results have been published in the journal Geoenergy Science and Engineering, in a paper titled "A latent space method with maximum entropy deep reinforcement learning for data assimilation"。
Innovation：We propose a latent space method based on maximum entropy deep reinforcement learning to address high-dimensional problems in data assimilation. This method constructs a latent space by integrating dimensionality reduction techniques with the state, action, and reward settings for the agent, allowing deep reinforcement learning to extend to complex systems with a large number of parameters and perform data assimilation in a reduced space. Additionally, we employed the maximum entropy deep reinforcement learning algorithm Soft Actor-Critic (SAC) to efficiently explore the parameter space. This approach overcomes the limitations of previous deep reinforcement learning algorithms, which were only suitable for small-scale problems, enabling them to handle medium-scale and even large-scale data assimilation tasks. The effectiveness of this method in data assimilation was demonstrated through comparisons with traditional data assimilation algorithms and other deep reinforcement learning algorithms on 2D synthetic and 3D reservoir cases.
Abstract：Data assimilation aims to calibrate the uncertain state or parameter vectors of a system by matching simulation results with observations, which is crucial for uncertainty quantification and optimization. Although traditional methods of data assimilation have shown promising results in practical applications, they need well-designed iteration rules (i.e., gradient, covariance, search strategies). Deep reinforcement learning (DRL) can solve data assimilation problems by trial and error, which does not require convoluted iteration rules. However, previous DRL methods cannot tackle high-dimensional data assimilation problems efficiently. In this work, we propose a latent space method with maximum entropy DRL for data assimilation to extend DRL to complicated systems with lots of parameters. Solutions can be found in a reduced space, through the interaction between the agent represented by artificial neural networks and the environment of numerical simulation. The proposed method contains two key points. First, to make the method applicable to high-dimensional problems, we construct a latent space method by integrating a dimensionality reduction method with the state, action, and reward settings for the agent. Second, a maximum entropy DRL algorithm, Soft Actor-Critic (SAC), was employed to efficiently explore the parameter space. The proposed method overcomes the limitation that previous DRL algorithms are only suitable for small-scale cases, and enables DRL to deal with medium-scale or even large-scale data assimilation problems. The performance of the proposed method was validated by comparison with a deep reinforcement learning algorithm and traditional data assimilation algorithms on 2D synthetic and 3D reservoir cases.
Geoenergy Science and Engineering covers a broad range of topics in earth energy and sustainable hydrocarbon production, aiming to publish articles with a particular focus on energy transition and achieving net-zero emission targets. Formerly known as the Journal of Petroleum Science and Engineering, its 2022 impact factor is 4.4, CiteScore is 8.8, and it is categorized as Q1 in the 2022 JCR rankings and in the second tier of the Chinese Academy of Sciences' engineering and technology category.
Paper link:
https://doi.org/10.1016/j.geoen.2024.213275
Citation:
Zhang Jinding, Zhang Kai, Wang Zhongzheng, Zhou Wensheng, Liu Chen, Zhang Liming, Ma Xiaopeng, Liu Piyang, Bian Ziwei, Kang Jinzheng, Yang Yongfei, Yao Jun. A latent space method with maximum entropy deep reinforcement learning for data assimilation[J]. Geoenergy Science and Engineering, 2024: 213275. DOI:10.1016/j.geoen.2024.213275.
