TWO-LEVEL SYSTEM FOR TUNING PARAMETERS OF ARTIFICIAL NEURAL NETWORKS

O. I. Chumachenko, S. V. Shymkov, A. T. Kot

Abstract


This paper focuses on the process of adjusting weights and biases of feed-forward ANN during their training process. A new algorithm for tuning artificial neural networks parameters has been proposed to overcome some limitations of existing optimization algorithms and to improve the training process of neural networks. This proposed algorithm combines the benefits of genetic algorithm and gradient-based optimization algorithms to improve the speed of training artificial neural networks and to increase the prediction accuracy of resulting network. The results of artificial neural networks training for classification task using two-level algorithm are presented and compared in performance with various gradient-based optimization algorithms.

Keywords


Neural networks; parametric tuning; training; optimization; genetic algorithms

References


Kurt Hornik, Maxwell Stinchcombe, Halbert White, “Multilayer feedforward networks are universal approximators,” Neural Networks, vol. 2, Issue 5, pp. 359–366, 1989. Print. doi:10.1016/0893-6080(89)90020-8

Simon Haykin, Neural Networks: A Comprehensive Foundation, 2nd ed., Upper Saddle River, NJ: Prentice Hall PTR, 1998. Print; ISBN: 0132733501

Nick McClure, TensorFlow Machine Learning Cookbook: Over 60 recipes to build intelligent machine learning systems with the power of Python, 2nd ed., Birmingham, UK: Packt Publishing, 2018. Print; ISBN: 1789131685

David Kriesel, A Brief Introduction to Neural Networks [Online]. Available: http://www.dkriesel.com/en/science/neural_network

Yurii Nesterov, “A method for unconstrained convex minimization problem with the rate of convergence o(1/k2),” Soviet. Math. Docl., vol. 269, pp. 543–547, 1983. Print.

Kurt Hornik, Maxwell Stinchcombe, Halbert White, “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization,” The Journal of Machine Learning Research, vol. 12, pp. 2121–2159, 2011. Print.

Matthew D. Zeiler, ADADELTA: An Adaptive Learning Rate Method, 2012 [Online]. Available: https://arxiv.org/abs/1212.5701

Diederik P. Kingma and Jimmy Ba, “Adam: A Method for Stochastic Optimization,” Presented at at the 3rd International Conference for Learning Representations, San Diego, 2015 [Online]

Available: https://arxiv.org/abs/1412.6980

David E. Goldberg, Genetic Algorithms in Search, Optimization & Machine Learning, Boston, MA: Addison-Wesley Longman Publishing Co., 1989. Print; ISBN: 0201157675

Dan Simon, Evolutionary Optimization Algorithms, Hoboken, NJ: John Wiley & Sons, Inc., 2013. Print; ISBN: 0470937416

Eckart Zitzler, Marco Laumanns, and Lothar Thiele (May 2001), SPEA2: Improving the Strength Pareto Evolutionary Algorithm, Swiss Federal Institute of Technology, Zurich, Switzerland [Online] Available: https://www.research-collection.ethz.ch/bitstream/handle/20.500.11850/145755/eth-24689-01.pdf, doi: 10.3929/ethz-a-004284029


Full Text: PDF

Refbacks

  • There are currently no refbacks.


Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.