SPONTANEOUS CATEGORIZATION AND SELF-LEARNING WITH DEEP AUTOENCODER MODELS
DOI:
https://doi.org/10.18372/2306-1472.80.14274Keywords:
artificial intelligence, machine learning, neural networks, unsupervised learningAbstract
In this studythe author investigates information processing in deep autoencoder models. It is demonstrated that unsupervised training of autoencoders of certain class can result in emergence of compact and structured internal representations of the input data space that can be correlated with higher level categories. The authors propose and demonstrate the practical possibility to detect and measure this emergent information structure by applying unsupervised density clustering in the activation space of the focal hidden layer of the model. Based on the findings of the studya new approach to training neural network models is proposed that is based on the emergent in unsupervised training information landscape, that is iterative, driven by the environment, requires minimal supervision and with interesting similarities to learning of biologic systems. In conclusion, a discussion of theoretical foundations of spontaneous categorization in self-learning systemsis provided.
References
Silver D. Shrittwieser J. Simonyan K. Antonoglou I. Huang A. et al (2017), Mastering the game of Go without human knowledge, Nature, volume 550, 354-359. https://doi.org/10.1038/nature24270
Le Q.V. Ranzato M.A. Monga R. Devin M. Chen K. et al. (2012), Building high-level features using large scale unsupervised learning, arxXiv:1112.6209. https://doi.org/10.1109/ICASSP.2013.6639343
Hassabis D. Kumaran D. Summerfield C. Botvinick M. (2017), Neuroscience inspired Artificial Intelligence, Neuron 95, 245-258. https://doi.org/10.1016/j.neuron.2017.06.011
Bengio Y. (2009), Learning deep architectures for AI, Foundations and Trends in Machine Learning Vol. 2 no. 1, 1-127. https://doi.org/10.1561/2200000006
Spinosa E., de Carvalho A.C.P.L.F.., Gama J. (2007), OLINDDA: a cluster-based approach for detecting novelty and concept drift in data streams, 2007 ACM Symposium on Applied Computing (SAC), Seoul. https://doi.org/10.1145/1244002.1244107
Fanizzi N., d'Amato C., Esposito F. (2008) Conceptual clustering and its application to concept drift and novelty detection. The Semantic Web: Research and Applications, ESWC
Albertini M.K., de Mello R.F. (2007), A self-organizing neural network approach to novelty detection, ACM symposium on Applied computing, Seoul, 462-466. https://doi.org/10.1145/1244002.1244110
Cassisi C., Ferro A., Guigno R., Pigola G., Pulvirenti A. (2013), Enhancing density-based clustering: parameter reduction and outlier detection, Information Systems, Vol. 38 no.3, 317-330. https://doi.org/10.1016/j.is.2012.09.001
Japkowicz N., Myers C., Gluck M. (1995), A novelty detection approach to classification, 14th international joint conference on Artificial intelligence Montreal, Vol.1, 518-523.
Pimentel M, Clifton D, Clifton L, TarassenkoL(2014). A review of novelty detection, Signal Processing, Vol. 99, 215-249. https://doi.org/10.1016/j.sigpro.2013.12.026
Hornik K, Stinchcombe M., White H. (1989), Multilayer feedforward neural networks are universal approximators, Neural Networks, Vol.2 no 5, 359-366. https://doi.org/10.1016/0893-6080(89)90020-8
He K., Zhang X., Ren S., Sun J. (2015), Deep residual learning for image recognition, arXiv:1512.03385. https://doi.org/10.1109/CVPR.2016.90
H. Peterson, D. Gustaffson, D. Bergstrom (2016), Hyperspectral image analysis using deep learning, 6th Int. Conf. Image Proc. Theory, Tools and Appl. (IPTA) Oulu U.S.A. https://doi.org/10.1109/IPTA.2016.7820963
A. Banino, C. Barry, D. Kumaran (2018), Vector-based navigation using grid-like representations in artificial agents, Naturevol. 557 pp. 429-433. https://doi.org/10.1038/s41586-018-0102-6
S. Dolgikh (2018), Spontaneous concept learning with deep autoencoder, Int. J. Comp. Intel. Syst. vol. 12 no. 1 pp. 1-12. https://doi.org/10.2991/ijcis.2018.25905178
Getting P. A. (1989), Emerging principles governing the operation of neural networks, Annual Review of Neuroscience, Vol.12, 185-204. https://doi.org/10.1146/annurev.ne.12.030189.001153
Keras: Python deep learning library. Available at: https://keras.io/
Waikato Internet Traffic Storage (WITS) passive datasets.Available at: https://wand.net.nz/wits
Wright C., Monrose F., Masson G. M., "HMM Profiles for Network Traffic Classification", ACM DMSEC, pp. 9-15, 2004. https://doi.org/10.1145/1029208.1029211
Comaniciu D. and Meer P. (2002), Mean Shift: A robust approach toward feature space analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, 603-619. https://doi.org/10.1109/34.1000236
Matplotlib. Available at: https://matplotlib.org
Ng A. (2011), Sparse autoencoder CS294A lecture notes, 1-19.Available at: http://ailab.chonbuk.ac.kr/seminar_board/pds1_files/sparseAutoencoder.pdf.
A. Gogna, A. Majumdar (2019), Discriminative Autoencoder for Feature Extraction: Application to Character Recognition" Neural Processing Letters, vol. 49 no. 3 pp 1723-1735. https://doi.org/10.1007/s11063-018-9894-5
