Review-09-Initial-Value-Problems-for-ODE.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Review-09-Initial-Value-Problems-for-ODE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answers to review questions from Chapter 9: Initial Value Problems for Ordinary Differential Equations <cite data-cite=\"heath2018scientific\">(Heath, 2018)</cite>.\n",
    "\n",
    "---\n",
    "Questions marked with $\\bigtriangledown$ are considered more difficult."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.1. True or false: An ODE solution that is un- bounded as time increases is necessarily unstable.\n",
    "\n",
    "False. An ODE solution whose **global error** is unbounded is unstable.  In constrast, the ODE $y' = k$ where $k$ is some constant has an unbounded solution given by $y(t) = kt + y_0$, but this ODE is stable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.2. True or false: In approximating a solution of an ODE numerically, the global error grows only if the solution sought is unstable.\n",
    "\n",
    "True. For unstable solutions, the effects of the local error accumulate and the growth of the global error is unbounded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.3. True or false: In solving an ODE numeri- cally, the roundoff error and the truncation error are independent of each other.\n",
    "\n",
    "False.  The truncation error can be reduced by using a smaller step-size, but this increases the roundoff error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.4. True or false: In solving an IVP for an ODE numerically, the global error is always at least as large as the sum of the local errors.\n",
    "\n",
    "False.  For asympotically stable ODE, the sum of the local errors will be greater than the global error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.5. True or false: For approximating a sta- ble solution of an ODE numerically, an implicit method is always stable.\n",
    "\n",
    "False.  In general, implicit methods have larger stability regions than explicit methods, but the allowable step-size for **all** implicit methods is not unlimited."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.6. True or false: In numerically approximating a stable solution of an ODE, one can take arbitrar- ily large time steps using an unconditionally stable method and still achieve any required accuracy.\n",
    "\n",
    "False.  The unconditional stability does not change the relationship between the choice of step size and the local error of the solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.7. True or false: Stiff ODEs are always diffi- cult and expensive to solve.\n",
    "\n",
    "False.  For stiff ODE some components in the solution vector $y(t)$ vary more rapidly with $t$ than others.  If the interval of integration does not intersect one of these fast-changing intervals, then solving the ODE is no different than it would be for non-stiff ODE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.8. (a) In general, does a differential equation, by itself, determine a unique solution?\n",
    "(b) If so, why, and if not, what additional infor- mation must be specified to determine a solution uniquely?\n",
    "\n",
    "(a) The ODE $y' = f(t, y)$ describes the slope of the solution curve at $t$, but does not determine a unique solution $y(t)$.  Some initial conditions are required in order to obtain a unique solution among the infinite family of solution curves which satisfy the ODE.\n",
    "\n",
    "(b) A differential equation has a unique solution only when some side conditions are given.  \n",
    "\n",
    "* If the side conditions $y_i(t) = y_i$ are all given at time $t=0$, then we have an initial value problem.\n",
    "* If the side conditions are specified at different values of $t$, then we have a boundary value problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.9. (a) What is meant by a first-order ODE?\n",
    "(b) Why are higher-order ODEs usually trans- formed into equivalent first-order ODEs before solving them numerically?\n",
    "\n",
    "(a) The order of an ODE is the highest order derivative.  The highest order derivative in a first-order ODE is the first derivative.\n",
    "\n",
    "(b) Because it is easier to solve first-order ODEs. The solution of a higher ODE can be transformed into solving a system of ODE.  The first equation in the system is $y'(t)$ and each subsequent equation is the first derivative of the prior equation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.10. (a) Describe in words the distinction be- tween a stable and an unstable solution of an ODE.\n",
    "(b) State a mathematical criterion for determining the stability of a solution of an ODE.\n",
    "(c) Can the stability or instability of an ODE so- lution change with time?\n",
    "\n",
    "(a) In an unstable solution the local error accumulates and as a result the growth in the global error is unbounded.  In a stable solution the local error does not accumulate or in an asymptotically stable solution the local error converges so that the growth in the global error is bounded.\n",
    "\n",
    "(b) Given the ODE $y' = \\lambda y$, the solution is stable when $\\lambda \\leq 0$ and otherwise is unstable.\n",
    "\n",
    "(c) No."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.11. Which of the following types of first-order ODEs have stable solutions?\n",
    "(a) An ODE whose solutions converge toward each other\n",
    "(b) An ODE whose Jacobian matrix has only eigenvalues with negative real parts\n",
    "(c) A stiff ODE\n",
    "(d) An ODE with exponentially decaying solu- tions\n",
    "\n",
    "(a) Asymptotically stable\n",
    "\n",
    "(b) Asymptotically stable\n",
    "\n",
    "(c) A stiff ODE will have large disparity in positive eigenvalues and hence unstable\n",
    "\n",
    "(a) Asymptotically stable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.12. Classify each of the following ODEs as hav- ing unstable, stable, or asymptotically stable so- lutions.\n",
    "(a)y′=y+t.\n",
    "(b)y′=y−t.\n",
    "(c)y′=t−y.\n",
    "(d)y′=1.\n",
    "\n",
    "(a) Unstable\n",
    "\n",
    "(b) Unstable\n",
    "\n",
    "(c) Asymptotically stable\n",
    "\n",
    "(d) Stable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.13. How does a typical numerical solution of an ODE differ from an analytical solution?\n",
    "\n",
    "An analytical solution to the ODE $y' = f(t, y)$ is an equation $y(t)$ that gives the exact value of the solution whereas a numerical solution is a table of approximate values of the solution $y(t)$ at a discrete set of points $t$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.14. (a) What is Euler’s method for solving an ODE?\n",
    "(b) Show at least one way it can be derived.\n",
    "\n",
    "(a) Euler's method computes the solution at time $t_{k+1}$ by extrapolating the solution at the previous time step $y_k$ using the product of the step size $h$ and slope $f(t_k, y_k)$.\n",
    "$$\n",
    "y_{k+1} = y_k + h f(t_k, y_k)\n",
    "$$\n",
    "\n",
    "(b) Euler's method can be derived from the Taylor Series expansion about $t$ by dropping the derivatives of second-order and higher.\n",
    "$$\n",
    "y(t + h) = y(t) + h y'(t) + \\frac{h^2}{2} y''(t) + \\cdots\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.15. In solving an ODE numerically, which is usually more significant, rounding error or trun- cation error?\n",
    "\n",
    "Truncation error used to approximate the solution is the dominant source of error.  The rounding error usually only becomes significant when the step size is made very small."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.16. Describe in words the difference between the local error and the global error in solving an IVP for an ODE numerically.\n",
    "\n",
    "Truncation error consists of local and global error.  The local error is the difference between the current solution and the solution obtained from the solution curve passing through the previous point.  The global error is the difference between the current solution and exact solution at the current point for the initial conditions provided in the problem. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.17. Under what condition is the global error in solving an IVP for an ODE likely to be smaller than the sum of the local errors at each step?\n",
    "\n",
    "The global error will be smaller than the sum of the local error for an asymptotically stable solution. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.18. (a) Define in words the error amplifica- tion or growth factor for one step of a numerical method for solving an IVP for an ODE.\n",
    "(b) Does the amplification factor depend only on the equation, only on the method of solution, or on both?\n",
    "(c) What is the value of the amplification factor for one step of Euler’s method?\n",
    "(d) What stability region does this imply for Eu- ler’s method?\n",
    "\n",
    "(a) The amplification or growth factor refers to the growth in the global error at each time step.\n",
    "\n",
    "(b) The amplification factor depends on the equation, method, and time step.\n",
    "\n",
    "(c) Euler's method is first-order accurate with growth factor $1 + h\\lambda$.\n",
    "\n",
    "(d) Euler's method is stable when $h \\leq 2/\\lambda$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.19. (a) What is the basic difference between an explicit method and an implicit method for solv- ing an ODE numerically?\n",
    "(b) Comparing these two types of methods, list one relative advantage for each.\n",
    "(c) Name a specific example of a method (or fam- ily of methods) of each type.\n",
    "\n",
    "(a) An explicit method uses the value of the solution at previous time steps to evaluate the solution at the current time step whereas an implicit method evaluates the solution $f(t_{k+1}, y_{k+1})$ before the solution at current time step is known (typically using some root-finding method).\n",
    "\n",
    "(b) Explicit methods are computationally cheaper than implicit methods.  Implicit methods usually have larger stability region than explicit methods and both backward Euler and trapezoid method are unconditionally stable.\n",
    "\n",
    "(c) The Euler method is explicit.  The backward Euler and trapezoid methods are implicit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.20. The use of an implicit method for solving a nonlinear ODE requires the iterative solution of a nonlinear equation. How can one get a good starting guess for this iteration?\n",
    "\n",
    "The solution at the previous time step can be used as a good starting guess for $y_{k+1}$ to solve $0 = y_k + h_k * f(t_{k+1}, y_{k+1}) - y_{k+1}$ for $y_{k+1}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.21. Is it possible for a numerical solution method to be unstable when applied to a stable ODE?\n",
    "\n",
    "A numerical solution can be unstable for any of the 3 factors listed below, not just the stability of the ODE (1).\n",
    "1. ODE to solve.\n",
    "2. Step size.\n",
    "3. Numerical method used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.22. What does it mean for the accuracy of a numerical method for solving ODEs to be of or- der p?\n",
    "\n",
    "Accuracy of order $p$ means that the global error is a power of the step size $O(h^p)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.23. (a) For solving ODEs, what is the highest- order accuracy that a linear multistep method can have and still be unconditionally stable?\n",
    "(b) Give an example of a method having these properties (by name or by formula).\n",
    "\n",
    "(a) Implicit multistep Adams methods of order 2 are unconditionally stable (see Table 9.1).\n",
    "\n",
    "(b) Trapezoid method.\n",
    "$$\n",
    "y_{k+1} = \\sum_{i=1}^2 \\frac{1}{2} y_{k+1-i} + h \\sum_{i=0}^2 \\frac{1}{2} f(t_{k+1-1}, y_{k+1-i})\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.24. Compare the stability regions (i.e., the sta- bility constraints on the step size) for the Euler and backward Euler methods for solving a scalar ODE.\n",
    "\n",
    "The Euler method for a sclar ODE $y' = \\lambda y$ is stable when $h \\leq 2/\\lambda$ whereas the backward Euler method is unconditionally stable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.25. For the backward Euler method, which fac- tor places a stronger restriction on the choice of step size: stability or accuracy?\n",
    "\n",
    "Since backward Euler is unconditionally stable, the accuracy places a stronger constraint on the choice of step size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.26. Which of the following numerical methods for solving a stable ODE numerically are uncon- ditionally stable?\n",
    "(a) Euler’s method\n",
    "(b) Backward Euler method\n",
    "(c) Trapezoid method\n",
    "\n",
    "(a) Euler's method is **not**\n",
    "\n",
    "(b) Backward Euler method is unconditionally stable and first-order accurate\n",
    "\n",
    "(c) Trapezoid method is unconditionally stable and second-order accurate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.27. (a) What is meant by a stiff ODE?\n",
    "(b) Why may a stiff ODE be difficult to solve nu- merically?\n",
    "(c) What type of method is appropriate for solving stiff ODEs?\n",
    "\n",
    "(a) For stiff ODE some components in the solution vector $y(t)$ vary more rapidly with $t$ than others.  The physical interpretation of this is that some components of a system have different time scales.  For example, a chemical reaction with one or more periods of very rapid transitions in concentration of constituent components.\n",
    "\n",
    "(b) A stiff ODE is more difficult to solve numerically because the step size might need to be more severely restricted in the region of rapid change in order to achieve the same level of accuracy.\n",
    "\n",
    "(c) Implicit methods increase the region of stability and are thus good for solving stiff ODE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\bigtriangledown$\n",
    "\n",
    "> 9.28. Suppose one is using the backward Euler method to solve a nonlinear ODE numerically. The resulting nonlinear algebraic equation at each step must be solved iteratively. If a fixed number of iterations are performed at each step, is the re- sulting method unconditionally stable? Why?\n",
    "\n",
    "Implicitness alone is not enough to guarantee stability.  Performing a fixed number of iterations in the root-finding subroutine as opposed to iterating until an error tolerance is reached might increase the local error of the solution.  Doing this for multiple steps will cause the local error to accumulate and result in growth in the global error.  Unbounded growth in the global error is the definition of unstable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.29. Explain why implicit methods are better than explicit methods for solving stiff ODEs nu- merically.\n",
    "\n",
    "A stiff ODE is difficult to solve numerically because some components of the solution vector $y(t)$ vary more rapidly with $t$ than others.  As a result, this generally means selecting a smaller step size than would otherwise be required.  Implicit methods increase the region of stability and thus loosen the restriction on step size placed by stiff ODE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.30. What is the simplest numerical method that is stable for solving a stiff ODE?\n",
    "\n",
    "Implicit single step methods such as backward Euler or trapezoid are the simplest methods for solving stiff ODE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\bigtriangledown$\n",
    "\n",
    "> 9.31. For solving ODEs numerically, why is it usually impractical to generate methods of very high accuracy by using many terms in a Taylor series expansion?\n",
    "\n",
    "Increasing the number of terms in the Taylor series generally means computing higher order derivatives which can be complicated.  The availability of automatic differentiation (AD) systems has loosened these restrictions and made use of higher order terms more practical."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.32. In solving an ODE numerically, with which type of method, Runge-Kutta or multistep, is it easier to supply values for the numerical solution at arbitrary output points within each step?\n",
    "\n",
    "Changing the step size is easy for Runge-Kutta methods whereas multistep methods are based on equally spaced intervals and the step size must remain fixed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.33. (a) What is the basic difference between a single-step method and a multistep method for solving an ODE numerically?\n",
    "(b) Comparing these two types of methods, list one relative advantage for each.\n",
    "(c) Name a specific example of a method (or fam- ily of methods) of each type.\n",
    "\n",
    "(a) Single step methods use information from a single previous point on the solution curve to compute the next point.  Multistep methods use information from more than one previous point to estimate the solution at the next point.\n",
    "\n",
    "(b) Advantages\n",
    "* Single step: changing the step size is easy.  \n",
    "* Multistep: obtain good estimates of local error from predictor and corrector pair.\n",
    "\n",
    "(c) Representative Methods\n",
    "* Single step: Runge-Kutta.  \n",
    "* Multistep: Adams predictor-corrector.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.34. List two advantages and two disadvan- tages of multistep methods compared with clas- sical Runge-Kutta methods for solving ODEs nu- merically.\n",
    "\n",
    "Advantages of multistep:\n",
    "* Implicit multistep methods are very effective for stiff ODE.\n",
    "* Obtain good estimates of local error from predictor and corrector pair.\n",
    "\n",
    "Disadvantages of multistep:\n",
    "* Step size must remain fixed.\n",
    "* Not self-starting eg requires history of solution values.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.35. What is the principal drawback of a Taylor series methods compared with Runge-Kutta meth- ods for solving ODEs numerically?\n",
    "\n",
    "Taylor series methods require higher order derivatives whch are generally complicated and must be explicitly programmed so that they can be passed to the ODE solver.  In contrast, Runge-Kutta replaces explicit higher order derivatives with finite difference approximations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.36. (a) What is the principal advantage of ex- trapolation methods for solving ODEs numeri- cally?\n",
    "(b) What are the disadvantages of such methods?\n",
    "\n",
    "(a) Extrapolation methods can achieve very high accuracy.\n",
    "\n",
    "(b) Extrapolation methods are less efficient and flexible than other methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.37. In using a multistep method to solve an ODE numerically, why might one still need to have a single-step method available?\n",
    "\n",
    "Multistep methods are not self-starting.  As a result, a single-step method might be used to compute the solution of the first time steps until there is sufficient number of prior points to use the multistep method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\bigtriangledown$\n",
    "\n",
    "> 9.38. Why are multistep methods for solv- ing ODEs numerically often used in predictor- corrector pairs?\n",
    "\n",
    "Multistep methods have an estimate of the local error from the difference between the predictor and corrector.  This estimate can be used to iteratively improve the solution obtained at the current step. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 9.41. For each of the following properties, state which type of ODE method, multistep or classical Runge-Kutta, more accurately fits the description: (a) Self starting\n",
    "(b) More efficient in attaining high accuracy (c) Can be efficient for stiff problems\n",
    "(d ) Easier to program\n",
    "(e ) Easier to change step size\n",
    "(f ) Easier to obtain a local error estimate\n",
    "(g) Easier to produce output at arbitrary interme- diate points within each step\n",
    "\n",
    "(a) Runge-Kutta\n",
    "\n",
    "(b) multistep\n",
    "\n",
    "(c) multistep\n",
    "\n",
    "(d) Runge-Kutta\n",
    "\n",
    "(e) Runge-Kutta\n",
    "\n",
    "(f) multistep\n",
    "\n",
    "(g) Runge-Kutta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}