CIM Seminar

Abstract: Current mathematical models for deep learning networks are based on approximation theory and harmonic analysis [Wiatowski,18, Daubechies,19]. Other approaches explore the relevance of tropical geometry [Maragos, 19] to describe networks with Rectified Linear Units (ReLUs) [Zhang18, Arora18].

The Matheron-Maragos-Banon-Barrera (MMBB) [Matheron75, Maragos89, Banon91] representation theorems provide an astonishing general formulation for any nonlinear operator between complete lattices, based on combinations of infimum of dilations and supremum of erosions. The theory is relevant when the basis (minimal kernel) of the operators can be learnt. In the case of non-increasing or non-translation-invariant operators the constructive decomposition of operators become more complex but would still be based on basic morphological dilation, erosion, anti-dilation and anti-erosion.  In this talk, I will first discuss the theoretical interest of the MMBB representation theory to model nonlinear layers and operators in deep learning networks and to highlight their interest to propose more general non-linearity activations and layers.

Any network architecture combining convolution, down/up-sampling, ReLUs, etc. could be seen at first sight as incompatible with a lattice theory formulation. In fact, as it was shown by Keshet [Keshet02, Keshet03], low-pass filters, decimation/interpolation, Gaussian/Laplacian pyramids and other typical image processing operators, admit an interpretation as erosions and adjunctions in the framework of (semi)-lattices. In addition, max-pooling and ReLUs are just dilation operators. The notion of deepness or recurrence in a network can be seen as the iteration of basic operators. In the second part of the talk, I will therefore discuss a complete theoretical formulation of deep learning networks in terms of morphological operators and point out some open questions on the convergence to idempotency and the study of order stability in the corresponding (semi-)lattices [Hejmans92].

The seminar will both be held physically and via Zoom. A link to the Zoom meeting will be posted at the morning of the event on the CIM mail list and when registering to the event. The 40 first to register to the event will also be given a free lunch.

Registration to the event.

References:
[Arora18] R. Arora, A. Basu, P. Mianjy, A. Mukherjee. Understanding Deep Neural Networks with Rectified Linear Units. arXiv. 1611.01491, 2018.

[Banon91] G.J.F. Banon, J. Barrera. Minimal representations for translation-invariant set mappings by mathematical morphology. SIAM Journal Applied Mathematics, 51(6): 1782-1798, 1991.

[Daubechies19] Daubechies and R. DeVore and S. Foucart and B. Hanin and G. Petrova. Nonlinear Approximation and (Deep) ReLU Networks. \emph{arXiv} 1905.02199, 2019

[Hejmans92] H.J.A.M. Hejmans, J. Serra. Convergence, continuity, and iteration in mathematical morphology. Journal of Visual Communication and Image Representation, 3(1): 84-102, 1992.

[Keshet02] R. Keshet. A Morphological View on Traditional Signal Processing. In Mathematical Morphology and its Applications to Image and Signal Processing. Computational Imaging and Vision, Vol 18. Springer, 2002.

[Keshet03] R. Keshet, H.J.A.M. Heijmans. Adjunctions in Pyramids, Curve Evolution and Scale-Spaces.
\emph{International Journal of Computer Vision,} 52: 139-151, 2003.

[Matheron75] G. Matheron. Random sets and integrad geometry. NewYork, Wiley, 1975.

[Maragos89] P. Maragos. A representation theory for morphological image and signal processing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2(6): 1989.

[Maragos19] P. Maragos and E. Theodosis. Tropical Geometry and Piecewise-Linear Approximation of Curves and Surfaces on Weighted Lattices. arXiv 1912.03891, 2019.

[Wiatowski18] T. Wiatowski and H. Bölcskei. A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction. IEEE Transactions on Information Theory, 64(3): 1845--1866, 2018.

[Zhang18] L. Zhang, G. Naitzat, L.-H. Lim. Tropical Geometry of Deep Neural Networks.  arXiv. 1805.07091, 2018.