Derivatives as part of the computation
A simulator calculates how a physical system behaves for given parameters and operating conditions. We often also need to know how the results change when these inputs vary. Such derivatives are central both to solving the model equations and to fitting the model to observations or optimising its controls.
Automatic differentiation (AD) systematically applies the chain rule through the operations in a computer program. For differentiable computations, it produces derivatives with accuracy limited by floating-point arithmetic, without the step-size error associated with finite differences. It also reduces the need to derive and maintain derivative code manually.
How we can help
- Develop simulators with automatic differentiation: Structure model equations so that residuals and derivatives can be calculated from the same code.
- Calculate Jacobians and sensitivities: Adapt differentiation to nonlinear solvers, coupled models and analysis of parameter dependence.
- Enable calibration and optimisation: Calculate gradients of objective functions with respect to model parameters and controls.
- Select and integrate AD technology: Assess methods and libraries against software architecture, memory requirements and computational cost.
- Verify derivatives: Check implementations and investigate how branching, non-smooth functions and solver tolerances affect the results.
Automatic differentiation in simulator development
Implicit simulators commonly solve large, coupled nonlinear systems using Newton's method. This requires derivatives of the residuals with respect to the unknowns, assembled in a Jacobian matrix. Automatic differentiation allows these derivatives to be calculated from the same implementation that defines the model equations.
This is particularly useful when adding physical processes or modifying existing models. Derivative calculations follow changes in the equation code, reducing manual work and the risk of inconsistencies between residuals and Jacobians. Efficient implementation also requires exploiting the problem's sparsity, locality and block structure.
From model equations to gradients of simulation outputs
Derivatives of the model equations are also building blocks for calculating how an entire simulation depends on its inputs. We combine automatic differentiation with sensitivity calculations and adjoint methods for applications including model calibration, history matching and optimisation.
For implicit models, we can differentiate the discrete equations and use them to calculate sensitivities, rather than differentiating through every solver iteration. Adjoint methods are particularly useful when gradients of one or a few objective functions are needed with respect to many parameters.
Choosing the right differentiation approach
Automatic differentiation can operate in forward or reverse mode. Forward mode propagates derivatives from inputs to outputs and is often suitable for a small number of differentiation directions or local calculations with known structure. Reverse mode propagates information backwards from outputs and is often suitable when a few outputs depend on many inputs.
Implementations may use operator overloading, source-code transformation or compiler techniques. We work with both in-house and open-source AD libraries in C++, MATLAB, Julia and Python. The choice depends on the derivatives required, the organisation of the software, and requirements for performance and maintainability.
Numerical understanding remains essential
Automatic differentiation follows the computations that are actually implemented. It does not make a non-smooth model differentiable. Phase transitions, minimum and maximum operations, changes in boundary conditions and adaptive algorithms may require special treatment. For iterative solvers, the relationship between solver tolerances and sensitivity accuracy must also be considered.
We therefore combine AD with an understanding of the model's physics, discretisation and solution algorithms. Derivative calculations are verified using appropriate consistency tests and comparisons so that they can be used reliably in subsequent analyses.
Experience across models and applications
We use automatic differentiation in computational electrochemistry, flow in porous media and other models based on partial differential equations. Our experience also includes construction-site optimisation and models for biochar production. These applications share a need to connect model development with efficient and reliable derivative calculations.
Work with us
Do you need derivatives from an existing model, or want to build a simulator that supports sensitivity analysis and optimisation from the outset? Contact our Applied Computational Science research group to discuss method selection, implementation and simulator development.