Utilizing Effective Linearity in Deep Neural Networks for Model Compression

My supervisor, Dr. Hannah Pinson, had done research into inherent linearity. The question that arose for her, was whether the non-linearity of the activation functions in Convolutional Neural Networks were necessary for performance. Her initial research indicated that many of this non-linear capacity wasn't needed. Thus, the topic of my thesis was born. If these non-linear activation functions aren't always needed, and can be removed (as her research showed), could we perhaps compress these now linear parts? After all, if you remove the non-linear activation functions, all you're left with is convolutional and MLP layers. These are just linear transformations, and multiple linear transformations can be combined into one.

I started my research by diving into the literature. Only her paper and one other had previously considered that models might be behaving linearly and thus a target for compression. The other work, by Razzhigaev et al., instead focused on transformers. They showed that these models often perform largely linear transformations on the input. Additionally, they provided the basis for the most successful method of compression in my research: linear approximation. Razzhigaev et al. attempted to replace parts of a model that were inherently linear with a single linear layer. This layer was first trained to mimic the linear transformer block(s), and was then used to replace them. For those familiar with Knowledge Distillation, it is a familiar approach to compression. Another suggested approach, by Dror et al., was layer merging. This approach only worked on CNNs, but described how consecutive convolutional layers can be combined into one bigger layer. This merged layer would encompass the same transformation as the individual layers would've done. Finally, three ways of identifying linearity were found, one from Razzhigaev et al., one from Pinson et al., and one envisioned by my supervisor.

Experiments were done on both transformers and CNNs. The different combinations of linearity metric and compression method were all attempted and run multiple times. Hybridizations with existing compression methods were also attempted. Finally, the behavior of the linearity metrics was analysed and compared to more traditional compression metrics. Compression using layer merging didn't achieve any good results. Compression through linear approximation was very promising, and came close to benchmark compression methods in certain situations. The Procrustes-based linearity metric, proposed by Razzhigaev et al., provided the best and most stable compression results. Hybridization didn't manage to outperform the sum of its parts, but the setup and combinations were very limited and rudimentary. No correlation could be found between the linearity metrics and other compression metrics.

Overall, my research indicates a clear potential for compression based on inherent linearity. The linearity can be detected in multiple models, even of different architectures. The linearity can also be compressed through linear approximation, without catastrophic collapse of performance. I've learned a lot about compression and doing research, and my interest in inherent linearity has been piqued. I hope to further research this novel and emerging topic in the future. I've linked the paper and the repository, so please feel free to read my work, try out my code, and build upon my ideas!