OPT OpenIR  > 光学影像学习与分析中心
Randomly translational activation inspired by the input distributions of ReLU
Cao, Jiale1; Pang, Yanwei1; Li, Xuelong2,3; Liang, Jingkun4
Department光学影像学习与分析中心
2018-01-31
Source PublicationNEUROCOMPUTING
ISSN0925-2312
Volume275Pages:859-868
Contribution Rank2
Abstract

Deep convolutional neural networks have achieved great success on many visual tasks (e.g., image classification). Non-linear activation plays a very important role in deep convolutional neural networks (CNN). It is found that the input distribution of non-linear activation is like Gaussian distribution and the most of the inputs are concentrated near zero. It makes the learned CNN likely sensitive to the small jitter of the non-linear activation input. Meanwhile, CNN is easily prone to overfitting with deep architecture. To solve the above problems, we make full use of the input distributions of non-linear activation and propose the randomly translational non-linear activation for deep CNN. In the training stage, non-linear activation function is randomly translated by an offset sampled from Gaussian distribution. In the test stage, the non-linear activation with zero offset is used. Based on our proposed method, the input distribution of non-linear activation is relatively scattered. As the result, the learned CNN is robust to the small jitter of the non-linear activation input. Our proposed method can be also seen as the regularization of non-linear activation to reduce overfitting. Compared to the original non-linear activation, our proposed method can improve classification accuracy without increasing computation cost. Experimental results on CIFAR-10/CIFAR-100, SVHN, and ImageNet demonstrate the effectiveness of the proposed method. For example, the reductions of error rates with VGG architecture on CIFAR-10/CIFAR-100 are 0.55% and 1.61%, respectively. Even when the noise is added to the input image, our proposed method still has much better classification accuracy on CIFAR-10/CIFAR-100. (C) 2017 Elsevier B.V. All rights reserved.

 

KeywordCnn Non-linear Activation Relu The Input Distributions Of Relu Random Translation Rt-relu
DOI10.1016/j.neucom.2017.09.031
Indexed BySCI ; EI
Language英语
WOS IDWOS:000418370200081
EI Keywords20174204277681
Citation statistics
Cited Times:3[WOS]   [WOS Record]     [Related Records in WOS]
Document Type期刊论文
Identifierhttp://ir.opt.ac.cn/handle/181661/30823
Collection光学影像学习与分析中心
Affiliation1.Tianjin Univ, Sch Elect & Informat Engn, Tianjin 300072, Peoples R China;
2.Chinese Acad Sci, Xian Inst Opt & Precis Mech, Ctr OPT IMagery Anal & Learning OPTIMAL, State Key Lab Transient Opt & Photon, Xian 710119, Shaanxi, Peoples R China;
3.Univ Chinese Acad Sci, Beijing 100049, Peoples R China;
4.Hainan Trop Ocean Univ, Coll Ocean Informat Engn, Sanya 572022, Peoples R China
Recommended Citation
GB/T 7714
Cao, Jiale,Pang, Yanwei,Li, Xuelong,et al. Randomly translational activation inspired by the input distributions of ReLU[J]. NEUROCOMPUTING,2018,275:859-868.
APA Cao, Jiale,Pang, Yanwei,Li, Xuelong,&Liang, Jingkun.(2018).Randomly translational activation inspired by the input distributions of ReLU.NEUROCOMPUTING,275,859-868.
MLA Cao, Jiale,et al."Randomly translational activation inspired by the input distributions of ReLU".NEUROCOMPUTING 275(2018):859-868.
Files in This Item:
File Name/Size DocType Version Access License
Randomly translation(853KB)期刊论文出版稿限制开放CC BY-NC-SAApplication Full Text
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Cao, Jiale]'s Articles
[Pang, Yanwei]'s Articles
[Li, Xuelong]'s Articles
Baidu academic
Similar articles in Baidu academic
[Cao, Jiale]'s Articles
[Pang, Yanwei]'s Articles
[Li, Xuelong]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Cao, Jiale]'s Articles
[Pang, Yanwei]'s Articles
[Li, Xuelong]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.