On Data-Driven Equation Discovery | by George Miloshevich

Photo by ThisisEngineering RAEng on Unsplash

Using analytical expressions verified through experiments has been a key aspect of scientific success, especially in physics. From the fundamental law of gravitation to quantum mechanics and beyond, the combination of observations and physical principles has played a crucial role in advancing our understanding of the natural world. However, as we face challenges like climate change, fusion, and computational biology, there is a growing need for concise yet robust reduced models that maintain physical consistency at a lower cost. This is where scientific machine learning comes in.

Scientific machine learning is an emerging field that aims to provide solutions by combining the power of machine learning with scientific principles. In this article, we will review recent data-driven equation discovery methods that target scientists and engineers with a basic understanding of machine learning or statistics.

Rather than simply fitting the data well, these methods focus on extracting equations from the data. This approach ensures that the models are more interpretable and less prone to overfitting. An example of the limitations of data-driven models can be seen in Ptolemy’s model of geocentrism, which was observationally accurate until Kepler’s heliocentric model surpassed it by incorporating fundamental physical principles.

In fields like physics, biology, and social sciences, where first principle approaches may not be as effective, statistical system identification already plays a significant role. Machine learning offers various methods for predicting the evolution of a system directly from data. Deep neural networks, in particular, have made significant advances in weather forecasting, as demonstrated by Google’s DeepMind and others. However, the use of fully data-driven models may not generalize well when the conditions under which the data is generated change, such as in the case of climate change. Therefore, extracting equations from data offers a more reliable and interpretable approach.

One method that has shown promise in extracting equations from data is Genetic Programming (GP). This approach involves generating candidate symbolic functions, deriving partial derivatives, and comparing them with numerically estimated derivatives from the data. The process is repeated until a sufficient level of accuracy is achieved. GP has been successfully applied in various systems, including identifying equations from Feynman lectures on physics and discovering ocean parameterizations in climate models. However, GP can be inefficient and requires human intervention to ensure the accuracy of the results.

Another method that has gained popularity in recent years is Sparse Identification of Nonlinear Dynamics (SINDy). This method aims to reconstruct first-order Ordinary Differential Equations (ODEs) from trajectory data. It involves choosing a library of monomials as basis functions and finding the sparse coefficient vector that represents the optimal choice for the right-hand side of the ODE. The goal is to promote sparsity, meaning that most values of the coefficient vector are set to zero. This helps prevent overfitting and ensures a more parsimonious model.

To achieve sparsity, SINDy uses Sparse Regression, such as the Sequential Threshold Least Squares (STLS) method. This method solves for the coefficients using standard least squares and then sequentially eliminates small coefficients while applying least squares each time. The tolerance for the smallness of coefficients is controlled by a hyperparameter, which can be determined through Pareto analysis or other methods.

In conclusion, data-driven equation discovery methods offer a promising approach to address the challenges in scientific research and modeling. By combining the power of machine learning with fundamental physical principles, these methods provide concise yet robust reduced models that maintain physical consistency at a lower cost. Whether it’s extracting equations through Genetic Programming or using Sparse Identification of Nonlinear Dynamics, these approaches offer interpretable and reliable solutions for scientists and engineers.

Source link