3 years ago

Emergence and Relevance of Criticality in Deep Learning.

Matteo Marsili, Juyong Song, Junghyo Jo

Deep learning has been successfully applied to various tasks, but its underlying mechanism remains unclear. Neural networks map input data to hidden states in deep layers. As deeper layers have fewer degrees of freedom, subsets of data are transformed into identical states in the deep layers. In this sense, deep learning can be considered as a hierarchical data grouping process. In this Letter, we discover that deep learning forces the size distributions of the data clusters to follow power laws with a different power exponent within each layer. In particular, we identify a critical layer where the cluster size distribution obeys a reciprocal relationship between rank and frequency, also known as Zipf's law. Deep learning ensures balanced data grouping by extracting similarities and differences between data. Furthermore, we verify that the data structure in the critical layer is most informative to reliably generate patterns of training data. Therefore, the criticality can explain the operational excellence of deep learning and provide a useful concept for probing optimal network architectures.

Publisher URL: http://arxiv.org/abs/1710.11324

DOI: arXiv:1710.11324v1

You might also like
Discover & Discuss Important Research

Keeping up-to-date with research can feel impossible, with papers being published faster than you'll ever be able to read them. That's where Researcher comes in: we're simplifying discovery and making important discussions happen. With over 19,000 sources, including peer-reviewed journals, preprints, blogs, universities, podcasts and Live events across 10 research areas, you'll never miss what's important to you. It's like social media, but better. Oh, and we should mention - it's free.

  • Download from Google Play
  • Download from App Store
  • Download from AppInChina

Researcher displays publicly available abstracts and doesn’t host any full article content. If the content is open access, we will direct clicks from the abstracts to the publisher website and display the PDF copy on our platform. Clicks to view the full text will be directed to the publisher website, where only users with subscriptions or access through their institution are able to view the full article.