Information Theoretic Measures Applied to Deep Learning Models

It is possible to improve existing deep learning architecture performance with the use of added information theoretic measures. In this work, two different deep learning architectures are modified and experiments are carried out over multiple data sets to determine their effectiveness. Varational Autoencoders can be combined with mutual information maximization to create disentangled representations that are more useful for downstream tasks like classification. In Contrastive losses, such as the one used by SimCLR, substituting and enhacing I with other mutual information NCE measurements shows potential for improving the limited mutual information estimation capabilities that batch size enforces on NCE.