详情

原文传递 Linear Layers and Partial Weight Reinitialization for Accelerating Neural Network Convergence.

题名：	Linear Layers and Partial Weight Reinitialization for Accelerating Neural Network Convergence.
作者：	Hyatt, J. S.; Lee, M. S.
关键词：	Artificial neural networks, Machine learning, Data compression, Pruning, Network compression, Explainability, Image painting, Network training, Cnn(convolutional neural network)
摘要：	We present two new approaches for accelerating the training of a neural network: 1) self-pruning using collapsible linear layers, and 2) mid-training weight reinitialization. By following each nonlinear layer with linear layers, then folding these linear layers into subsequent nonlinear layers after training, we are able to reproduce the benefits of overparameterizing the network, then pruning individual elements after training. We also periodically reinitialize the weights of nonlinear elements that do not improve the networks performance during training, freezing retained weights for several epochs to force the reinitialized weights to accommodate information already learned. Both methods demonstrate substantial gains: the resulting models are simpler than those attained by standard pruning and initialization methods, require fewer computations to train, and are more accurate than networks trained with those methods.
报告类型：	科技报告