Publications by authors named "Florian L Kreyssig"

1 Publications

  • Page 1 of 1

A distributed optimisation framework combining natural gradient with Hessian-free for discriminative sequence training.

Neural Netw 2021 May 29;143:537-549. Epub 2021 May 29.

Cambridge University Engineering Department, Trumpington Street, Cambridge, CB2 1PZ, UK. Electronic address:

This paper presents a novel natural gradient and Hessian-free (NGHF) optimisation framework for neural network training that can operate efficiently in a distributed manner. It relies on the linear conjugate gradient (CG) algorithm to combine the natural gradient (NG) method with local curvature information from Hessian-free (HF). A solution to a numerical issue in CG allows effective parameter updates to be generated with far fewer CG iterations than usually used (e.g. 5-8 instead of 200). This work also presents a novel preconditioning approach to improve the progress made by individual CG iterations for models with shared parameters. Although applicable to other training losses and model structures, NGHF is investigated in this paper for lattice-based discriminative sequence training for hybrid hidden Markov model acoustic models using a standard recurrent neural network, long short-term memory, and time delay neural network models for output probability calculation. Automatic speech recognition experiments are reported on the multi-genre broadcast data set for a range of different acoustic model types. These experiments show that NGHF achieves larger word error rate reductions than standard stochastic gradient descent or Adam, while requiring orders of magnitude fewer parameter updates.
View Article and Find Full Text PDF

Download full-text PDF

Source Listing
May 2021