Dr. Bogdan Savchynskyy, Prof. Dr. Carsten Rother, SoSe 2018
Summary
Machine learning techniques are tightly coupled with optimization methods. Many techniques become practical only if there exists a supporting optimization tool.
In the seminar we will consider a number of optimization problems established in machine learning and corresponding solution methods. The latter range from continuous to combinatorial optimization.
Topics
Papers for presentation and discussion are in general pre-selected and grouped by subtopics, although alternatives can also be proposed by students, see below. The short introduction and paper assignment will be made at the first seminar session on 18 April 2018. Drop us an email if you have already decided about your preferences and want to do the selection in advance.Neural Networks and Convexity
- (assigned to E. Eulig) Amos, Xu, Kolter. Input-Convex-Neural-Networks. In ICML 2017
- Yoshua Bengio, Nicolas Le Roux, Pascal Vincent, Olivier Delalleau, Patrice Marcotte Convex Neural Networks. In NIPS 2006
- (assigned to M. Runz) Amos, Brandon, and J. Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. arXiv preprint arXiv:1703.00443 (2017).
Faster learning: Stochastic gradient and its variants
- Kingma, Diederik P., and Jimmy Ba. "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980 (2014).
- (assigned to I. Dehner) Schmidt, Mark, Nicolas Le Roux, and Francis Bach. "Minimizing finite sums with the stochastic average gradient." Mathematical Programming 162.1-2 (2017): 83-112.
- Johnson, Rie, and Tong Zhang. "Accelerating stochastic gradient descent using predictive variance reduction." Advances in neural information processing systems. 2013.
- Reddi, Sashank J., et al. "Stochastic variance reduction for nonconvex optimization." International conference on machine learning. 2016.
Combinatorial optimization and relaxations
- (assigned to K. Schwarz) Khalil, Elias, et al. "Learning combinatorial optimization algorithms over graphs." Advances in Neural Information Processing Systems. 2017.
- Lang, Hunter, David Sontag, and Aravindan Vijayaraghavan. "Alpha-expansion is Exact on Stable Instances." arXiv preprint arXiv:1711.02195 (2017)
- Werner, Tomas. "On Coordinate Minimization of Convex Piecewise-Affine Functions." arXiv preprint arXiv:1709.04989 (2017).
Paper of your choice
You may also give a presentation based on a paper which you have selected yourself,
as long as it fits the general topic of the seminar.
This must be discussed in advance with the teacher, however.
- (assigned to K. Roth) Ke Li and Jitendra Malik. "Learning to Optimize Neural Nets" arXiv preprint arXiv:1703.00441 (2017)
- (assigned to L. Kades) Max Welling and Yee Whye Teh. "Bayesian Learning via Stochastic Gradient Langevin Dynamics" ICML (2011)
- (assigned to L. Biasi) James Martens. "Deep learning via Hessian-free optimization" ICML (2010) and James Martens and Ilya Sutskever. "Learning Recurrent Neural Networks with Hessian-Free Optimization" ICML (2010)
Schedule
Important: The seminar on 27 June will already take place at 2pm.
6 June 2018
16:00 - E. Eulig
17:00 - K. Roth
13 June 2018
16:00 - L. Biasi
20 June 2018
16:00 - L. Kades
17:00 - I. Dehner
27 June 2018
14:00 - K. Schwarz
15:00 - M. Runz
Information
- Seminar: Wed, 16:00 – 18:00, Raum: Mathematikon B (Berliner Str. 43), SR B128
Ring the door bell labelled "HCI am IWR" to be let in. The seminar room is in the 3rd floor. - Credits: 2 SWS / 2 or 4 CP depending on course of study
The presentations will be scheduled at the first meeting on 18 April 2018.
This information can also be found in the LSF.
Registration
Please register for the seminar in Müsli. If you have trouble registering there drop an email to lisa.kruse@iwr.uni-heidelberg.de.