Use the left and right arrow keys to navigate the presentation forward and backward respectively. You can also use the arrows at the bottom right of the screen to navigate with a mouse.
FAIR USE ACT DISCLAIMER: This site is for educational purposes only. This website may contain copyrighted material, the use of which has not been specifically authorized by the copyright holders. The material is made available on this website as a way to advance teaching, and copyright-protected materials are used to the extent necessary to make this class function in a distance learning environment. The Fair Use Copyright Disclaimer is under section 107 of the Copyright Act of 1976, allowance is made for “fair use” for purposes such as criticism, comment, news reporting, teaching, scholarship, education and research.
Courtesy of Pbroks13, CC BY-SA 3.0, via Wikimedia Commons
More generally, the tangent line approximation is one kind of general Taylor approximation.
Suppose we have a point \( x_0 \) fixed, and define \( x_1 \) as a small perturbation \[ x_1 = x_0+\delta_{x_1}, \]
If a function \( f \) has \( k \) continuous derivatives we can write \[ f(x_1) = f(x_0) + f'(x_0)\delta_{x_1} + \frac{f''(x_0)}{2!}\delta_{x_1}^2 + \cdots + \frac{f^{(k)}(x_0)}{k!} \delta_{x_1}^k + \mathcal{O}\left(\delta_{x_1}^{k+1}\right) \]
The \( \mathcal{O}\left(\delta_{x_1}^{k+1}\right) \) refers to terms in the remainder, that grows or shrinks like the size of the perturbation to the power \( k+1 \).
Another important practical example of using this Taylor approximation, when the function \( f \) has two continuous derivatives, is \[ f(x_0 + \delta_{x_1}) \approx f(x_0) + f'(x_0)\delta_{x_1} + f''(x_0) \frac{\delta_{x_1}^2}{2} \] which will be used shortly for obtaining solutions to several kinds of equations.
Particularly, this is strongly related to our second derivative test from univariate calculus.
At the moment, we consider how Taylor's expansion can be used at first order again to approximate the derivative.
Recall, we write
\[ \begin{align} f(x_1) &= f(x_0) + f'(x_0) \delta_{x_1} + \mathcal{O}\left( \delta_{x_1}^2\right) \\ \Leftrightarrow \frac{f(x_1) - f(x_0)}{ \delta_{x_1}} &= f'(x_0) + \mathcal{O}\left( \delta_{x_1}\right) \end{align} \]
This says that for a small value of \( \delta_{x_1} \), we can obtain the numerical approximation of \( f'(x_0) \) proportional to the accuracy of the largest decimal place of \( \delta_{x_1} \) by the difference on the left hand side.
This gives a forward finite difference equation approximation to the derivative.
We can similarly define a backward finite difference equation with \( \pmb{x}_1 := \pmb{x}_0 -\pmb{\delta}_{\pmb{x}_1} \).
In each case, we use the perturbation to parameterize the tangent-line approximation.
We have seen earlier the basic linear inverse problem,
\[ \begin{align} \mathbf{A}\pmb{x} = \pmb{b} \end{align} \] where \( \pmb{b} \) is an observed quantity and \( \pmb{x} \) are the unknown variables related to \( \pmb{b} \) by the relationships in \( \mathbf{A} \).
A similar problem exists when the relationship between \( \pmb{x} \) and \( \pmb{b} \) is non-linear, but we still wish to find some such \( \pmb{x} \).
Nonlinear inverse problem (scalar case)
Suppose we know the nonlinear, scalar function \( f \) that gives a relationship \[ \begin{align} f(x^\ast) = b \end{align} \] for an observed \( b \) but an unknown \( x^\ast \). Finding a value of \( x^\ast \) that satisfies \( f(x^\ast)=b \) is known as a nonlinear inverse problem.
Define a function \[ \begin{align} \tilde{f}(x) = f(x)-b. \end{align} \]
Thus solving the nonlinear inverse problem in one variable is equivalent to finding the appropriate \( x^\ast \) for which \[ \begin{align} \tilde{f}(x^\ast)= 0 . \end{align} \]
Finding a zero of a function, or root finding, is thus equivalent to a nonlinear inverse problem.
The Newton-Raphson method is one classical approach which has inspired many modern techniques.
Courtesy of Ralf Pfeifer, CC BY-SA 3.0, via Wikimedia Commons
As a quick example, let's consider the Newton algorithm built-in to Scipy.
Specifically, we will import the built-in newton
function from the optimize
sub-module of scipy
.
from scipy.optimize import newton
In the following, we define the cubic function \( f(x):=x^3 \), but we are interested in the value \( x^\ast \) for which \( f\left(x^\ast\right)=1 \)
def f(x): return (x**3 - 1)
The newton
function can be supplied an analytical derivative, if this can be computed, to improve the accuracy versus, e.g., a finite-differences approximation.
newton
:root = newton(f, 1.5, fprime=lambda x: 3 * x**2)
root
1.0
To expand our discussion to multiple variables, we will review some fundamental concepts of vector calculus.
Suppose we have a vector valued function, with a single argument:
\[ \begin{align} \pmb{x}:&\mathbb{R} \rightarrow \mathbb{R}^{N};\\ \pmb{x}(t) :=& \begin{pmatrix} x_1(t) & \cdots & x_{N}(t)\end{pmatrix}^\top; \end{align} \]
Tangent vector
Suppose \( \pmb{x}(t) \) is defined as above and that each of the component functions \( x_i(t) \) are differentiable. The tangent vector to the state trajectory \( \pmb{x} \) is defined as \[ \vec{x}:= \frac{\mathrm{d}}{\mathrm{d}t} \pmb{x}:= \begin{pmatrix}\frac{\mathrm{d}}{\mathrm{d}t} x_1(t) & \cdots & \frac{\mathrm{d}}{\mathrm{d}t} x_{N}(t)\end{pmatrix}^\top \]
In the above, the interpretation of the derivative defining a tangent line is extended into multiple variables;
Tangent spaces
Let \( \pmb{x}\in\mathbb{R}^{N} \) and \( \gamma(t) \) be an arbitrary differentiable curve \( \pmb{\gamma}:\mathbb{R}\rightarrow \mathbb{R}^{N} \) such that \( \pmb{\gamma}(0)= \pmb{x} \) with a tangent vector defined as \( \vec{\gamma}(0):= \frac{\mathrm{d}}{\mathrm{d}t}|_0 \pmb{\gamma} \). The tangent space at \( T_{\pmb{x}} \) is defined by the linear span of all tangent vectors as such through \( \pmb{x} \).