摘要
针对宽度学习系统稀疏过程忽视不同权重重要性的变化,易出现误剪枝的问题,该文提出了基于动态稀疏训练的宽度学习系统。在标准宽度学习系统的目标函数中引入正则化项约束输出权重阈值,通过对输出权重和输出权重阈值的联合训练寻找出最优网络参数和稀疏网络结构。针对每一个输出权重引入输出权重阈值,根据输出权重重要性的改变,生成控制模型结构的输出权重掩码。通过动态训练,寻找网络结构和网络精度之间最优的平衡,提升模型整体性能。为了验证所提方法的有效性,在UCI公共数据集上选择多个数据集进行仿真实验。实验结果表明所提方法可以在不降低模型性能的同时,利用动态稀疏的方式稀疏模型。
[Objective]In sparse broad learning systems,the changing importance of output weights is overlooked.Some weights are unimportant in the early stages of model training but become important after being trimmed,making their recovery challenging.Inspired by dynamic sparse training in neural networks,this paper proposes a width learning system utilizing dynamic sparse training to compensate for pruning errors during model training and improve overall model performance while maintaining model sparsity.[Methods]This system introduces a regularization term to constrain the output weight threshold in the objective function of a standard-width learning system.It seeks optimal network parameters and a sparse network structure through joint training of output weights and their thresholds.Introduce an output weight threshold for each output weight,and generate an output weight mask for the control model structure based on changes in the importance of the output weight.The mask is jointly generated using weights and their thresholds and dynamically adjusts weight threshold during training to prune and restore output weights based on changes in weight importance.This system can indirectly sparsify models using the mask while retaining output weights,achieving an optimal balance between network structure and accuracy through dynamic training,and improving the overall model performance by minimizing the incorrect pruning of weights.There is the greatest improvement in accuracy on the dataset'BUTCSP',with an increase of approximately 30.12%.This article introduces exponential powers as regularization terms to constrain the weight threshold in the loss function of a standard-width learning system and adds a weight mask to the error term of the loss function.The alternating direction multiplier method is used to optimize and solve the objective function.[Results]To verify the effectiveness of the broad learning system based on dynamic sparse training(BLSDST),this paper uses six UCI public datasets for simulation.The performance of the system was compared with those of the broad learning system(BLS)and lasso broad learning system(L1BLS).Results indicate that the BLSDST achieves a balance between model accuracy and sparsity by constraining the weight-threshold regularization term.Further,it can reduce model complexity without sacrificing model accuracy while compensating for the impact of model pruning on model performance.[Conclusions]Experimental results show that the proposed system can achieve model dynamic sparsity without reducing model performance and even improving it.
作者
李海港
孙娟
曹义湾
褚菲
余淼
张勇
LI Haigang;SUN Juan;CAO Yiwan;CHU Fei;YU Miao;ZHANG Yong(School of Information and Control Engineering,China University of Mining and Technology,Xuzhou 221116,China)
出处
《实验技术与管理》
CAS
北大核心
2024年第12期53-60,共8页
Experimental Technology and Management
基金
国家自然科学基金项目(62273348,61973304)
教育部产学合作协同育人项目(2022030014)
江苏省大学生创新训练项目(202410290143Y)
中国矿业大学教学研究项目(2021YB20,2022KCSZ03)。
关键词
宽度学习系统
增量学习
动态稀疏
权重阈值
broad learning system
incremental learning
dynamic sparsity
weight threshold
