Quiz: The Levenberg-Marquardt Algorithm — Theory and Implementation

1. The LM damped normal equations are $(J^\top J + \lambda D)\delta_\lambda = J^\top r$. At $\lambda = 0$, this gives the Gauss-Newton step. As $\lambda \to \infty$, the step approaches:

2. The gain ratio $\rho = (\mathcal{L}(\theta) - \mathcal{L}(\theta+\delta)) / (m(0) - m(\delta))$. A gain ratio of $\rho = 0.95$ means:

3. Prove that the LM step $\delta_\lambda$ is always a descent direction. The key ingredient is:

4. LM converges quadratically when the residuals vanish at the solution ($r(\theta^*) = 0$) but only linearly when $r(\theta^*) \ne 0$. What does slow convergence near the solution indicate about the model?

5. A Heston calibration starts from a cold start and converges in 45 iterations. The next day, starting from the previous day's calibrated parameters, it converges in 7 iterations. The reason is:

6. The LM algorithm with $D = I$ is applied to a problem where one parameter $\kappa \approx 2.0$ and another $\xi \approx 0.3$. What problem arises, and how does Marquardt's scaling $D = \mathrm{diag}(J^\top J)$ fix it?

7. During a calibration run, $\lambda$ grows from $10^{-4}$ to $10^{10}$ without the gradient norm decreasing below tolerance. The most likely root cause is:

8. A calibration algorithm reports "converged" after 3 iterations, but the final residuals are large ($\text{RMSE} = 5\%$ in vol space). What likely happened?