The effect of bottlenecks on generalisation in backpropagation neural networks

Xu Zang

doi:10.1007/978-3-642-17534-3_21

The effect of bottlenecks on generalisation in backpropagation neural networks

Xu Zang^*

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Many modifications have been proposed to improve back-propagation's convergence time and generalisation capabilities. Typical techniques involve pruning of hidden neurons, adding noise to hidden neurons which do not learn, and reducing dataset size. In this paper, we wanted to compare these modifications' performance in many situations, perhaps for which they were not designed. Seven famous UCI datasets were used. These datasets are different in dimension, size and number of outliers. After experiments, we find some modifications have excellent effect of decreasing network's convergence time and improving generalisation capability while some modifications perform much the same as unmodified back-propagation. We also seek to find a combine of modifications which outperforms any single selected modification.

Original language	English
Title of host publication	Neural Information Processing
Subtitle of host publication	Models and Applications - 17th International Conference, ICONIP 2010, Proceedings
Pages	168-176
Number of pages	9
Edition	PART 2
DOIs	https://doi.org/10.1007/978-3-642-17534-3_21
Publication status	Published - 2010
Event	17th International Conference on Neural Information Processing, ICONIP 2010 - Sydney, NSW, Australia Duration: 22 Nov 2010 → 25 Nov 2010

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Number	PART 2
Volume	6444 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	17th International Conference on Neural Information Processing, ICONIP 2010
Country/Territory	Australia
City	Sydney, NSW
Period	22/11/10 → 25/11/10

Access to Document

10.1007/978-3-642-17534-3_21

Cite this

Zang, X. (2010). The effect of bottlenecks on generalisation in backpropagation neural networks. In Neural Information Processing: Models and Applications - 17th International Conference, ICONIP 2010, Proceedings (PART 2 ed., pp. 168-176). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6444 LNCS, No. PART 2). https://doi.org/10.1007/978-3-642-17534-3_21

Zang, Xu. / The effect of bottlenecks on generalisation in backpropagation neural networks. Neural Information Processing: Models and Applications - 17th International Conference, ICONIP 2010, Proceedings. PART 2. ed. 2010. pp. 168-176 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 2).

@inproceedings{55db1b15ccbe44d1830c6fa4a1f96108,

title = "The effect of bottlenecks on generalisation in backpropagation neural networks",

abstract = "Many modifications have been proposed to improve back-propagation's convergence time and generalisation capabilities. Typical techniques involve pruning of hidden neurons, adding noise to hidden neurons which do not learn, and reducing dataset size. In this paper, we wanted to compare these modifications' performance in many situations, perhaps for which they were not designed. Seven famous UCI datasets were used. These datasets are different in dimension, size and number of outliers. After experiments, we find some modifications have excellent effect of decreasing network's convergence time and improving generalisation capability while some modifications perform much the same as unmodified back-propagation. We also seek to find a combine of modifications which outperforms any single selected modification.",

keywords = "backpropagation, bottleneck, neural network, noise, pruning",

author = "Xu Zang",

year = "2010",

doi = "10.1007/978-3-642-17534-3_21",

language = "English",

isbn = "3642175333",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

number = "PART 2",

pages = "168--176",

booktitle = "Neural Information Processing",

edition = "PART 2",

note = "17th International Conference on Neural Information Processing, ICONIP 2010 ; Conference date: 22-11-2010 Through 25-11-2010",

}

Zang, X 2010, The effect of bottlenecks on generalisation in backpropagation neural networks. in Neural Information Processing: Models and Applications - 17th International Conference, ICONIP 2010, Proceedings. PART 2 edn, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. PART 2, vol. 6444 LNCS, pp. 168-176, 17th International Conference on Neural Information Processing, ICONIP 2010, Sydney, NSW, Australia, 22/11/10. https://doi.org/10.1007/978-3-642-17534-3_21

The effect of bottlenecks on generalisation in backpropagation neural networks. / Zang, Xu.
Neural Information Processing: Models and Applications - 17th International Conference, ICONIP 2010, Proceedings. PART 2. ed. 2010. p. 168-176 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6444 LNCS, No. PART 2).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - The effect of bottlenecks on generalisation in backpropagation neural networks

AU - Zang, Xu

PY - 2010

Y1 - 2010

N2 - Many modifications have been proposed to improve back-propagation's convergence time and generalisation capabilities. Typical techniques involve pruning of hidden neurons, adding noise to hidden neurons which do not learn, and reducing dataset size. In this paper, we wanted to compare these modifications' performance in many situations, perhaps for which they were not designed. Seven famous UCI datasets were used. These datasets are different in dimension, size and number of outliers. After experiments, we find some modifications have excellent effect of decreasing network's convergence time and improving generalisation capability while some modifications perform much the same as unmodified back-propagation. We also seek to find a combine of modifications which outperforms any single selected modification.

AB - Many modifications have been proposed to improve back-propagation's convergence time and generalisation capabilities. Typical techniques involve pruning of hidden neurons, adding noise to hidden neurons which do not learn, and reducing dataset size. In this paper, we wanted to compare these modifications' performance in many situations, perhaps for which they were not designed. Seven famous UCI datasets were used. These datasets are different in dimension, size and number of outliers. After experiments, we find some modifications have excellent effect of decreasing network's convergence time and improving generalisation capability while some modifications perform much the same as unmodified back-propagation. We also seek to find a combine of modifications which outperforms any single selected modification.

KW - backpropagation

KW - bottleneck

KW - neural network

KW - noise

KW - pruning

UR - http://www.scopus.com/inward/record.url?scp=78650216235&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-17534-3_21

DO - 10.1007/978-3-642-17534-3_21

M3 - Conference contribution

SN - 3642175333

SN - 9783642175336

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 168

EP - 176

BT - Neural Information Processing

T2 - 17th International Conference on Neural Information Processing, ICONIP 2010

Y2 - 22 November 2010 through 25 November 2010

ER -

Zang X. The effect of bottlenecks on generalisation in backpropagation neural networks. In Neural Information Processing: Models and Applications - 17th International Conference, ICONIP 2010, Proceedings. PART 2 ed. 2010. p. 168-176. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 2). doi: 10.1007/978-3-642-17534-3_21

The effect of bottlenecks on generalisation in backpropagation neural networks

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this