IIT Madras · Engineering

Optimization Theory and Algorithms

Prof. Uday K Khankhoje · 2025 · 82 free video lectures

This course will introduce the student to the basics of unconstrained and constrained optimization that are commonly used in engineering problems. The focus of the course will be on contemporary algorithms in optimization. Sufficient the oretical grounding will be provided to help the student…

▶ Play on Open Lecture HallCourse page ↗

Lectures

  1. Introduction to the course - 1 - Prerequisites, key elements
  2. Introduction to the course - 2 - Types of problems
  3. Introduction to the course - 3 - an optimization example to live longer
  4. Summary of background material - Linear Algebra 1
  5. Summary of background material - Linear Algebra II
  6. Summary of background material - Analysis I
  7. Summary of background material - Analysis II
  8. Summary of background material - Analysis III
  9. Summary of background material - Calculus 1
  10. Summary of background material - Calculus 2
  11. Summary of background material - Calculus 3
  12. Example of Multivariate Differentiation
  13. Gradient of Quadratic form and product rule
  14. Directional derivative, hessian, and mean value theorem
  15. Unconstrained optimization - 1 - roadmap of the course and Taylor’s theorem
  16. Unconstrained optimization - 2 - Identifying a local minima - 1st and 2nd order conditions
  17. Unconstrained optimization - 3 - proof of 1st order condition (split 3)
  18. Unconstrained optimization - 4 - overview of algorithms and choosing a descent direction
  19. Unconstrained optimization - 5 - properties of descent directions - steepest descent direction
  20. Unconstrained optimization - 6 - properties of descent directions - newton direction
  21. Unconstrained optimization - 7 - trust region methods
  22. A MATLAB session
  23. Introduction to Line Search
  24. Wolfe conditions
  25. Strong Wolfe conditions
  26. Backtracking Line Search
  27. Line Search - Analysis
  28. Line Search - Convergence and Rate - 1
  29. Line Search - Convergence and Rate - 2
  30. Convergence analysis of a descent algorithm - 1
  31. Convergence analysis of a descent algorithm - 2
  32. Implementation of an optimization algorithm in MATLAB
  33. Conjugate Gradient Methods - Introduction and Proof
  34. Visualizing Quadratic Forms
  35. Orthogonality and Conjugacy
  36. Conjugate Directions Method - Introduction and Proof
  37. Discussion on doubts
  38. More on Conjugate Directions Method
  39. Ways of Generating Conjugate Directions
  40. Expanding Subspace Theorem
  41. Discussion on doubts
  42. Conjugate Gradient Method
  43. MATLAB implementation on CGM
  44. Discussion on doubts
  45. Preconditioned Conjugate Gradient - Part 1
  46. Preconditioned Conjugate Gradient - Part 2
  47. Preconditioned Conjugate Gradient - Part 3
  48. Non-Linear Conjugate Gradient method
  49. Intro to Newton methods
  50. Newton methods and convergence
  51. Checks for positive definite matrices
  52. Hessian Modification
  53. Quasi newton methods
  54. BFGS method
  55. Least squares problems
  56. Linear least squares - Part 1
  57. Linear least squares - Part 2
  58. Solving least squares using SVD
  59. Non-linear least squares
  60. Constrained optimisation
  61. Single equality constraint
  62. Single inequality constraint
  63. Single inequality constraint - part 2
  64. Two inequality constraints example
  65. Linearised feasible directions
  66. Feasible sequences and tangent cone
  67. LICQ conditions
  68. KKT conditions (First order necessary conditions)
  69. Proof sketch for KKT conditions (part 1)
  70. Proof sketch for KKT conditions (part 2)
  71. Introduction to Projected gradient descent
  72. Projected gradient descent and proof of convergence
  73. Proof of convergence (Part 2)
  74. Subgradients and subdifferential
  75. Projection onto l1-ball
  76. Soft-thresholding example
  77. Recap of Projection onto l1-ball
  78. KKT and duality introduction
  79. Intuition of duality and dual problem
  80. Proof of concavity of the dual problem
  81. Proof of concavity of the dual problem - Part 2
  82. Proof of concavity of the dual problem - Part 3

From NPTEL (IITs and IISc) · CC BY-NC-SA. More from IIT Madras.

Open Lecture Hall is free, with no ads and no sign-up. It is not run by or connected to any university; the lectures are the universities' own free recordings, played from where they host them. About & rights