Recently,the top journal in the field of artificial Intelligence and Patternrecognition, IEEE Transactions on Pattern Analysis and MachineIntelligence (TPAMI), acceptedthelatest research results of Professor Li Xinde's team from the Schoolof Automation, Southeast University. The study creatively combinesclassical control theory with cutting-edge meta-learning and proposesa new algorithm called "Layer Adaptive PID Optimizer (LA-PID)",providing an efficient and stable new solution to break through thedata scarcity bottleneck in few-shot learning. The previous versionof the work was presented at the top international conference onMachine Learning (ICML) in 2024, and this journal version publishedin TPAMI is a comprehensive theoretical deepening and experimentalexpansion. The study was led by doctoral students Zhang Pengfei andYu Le from Southeast University, and was supported by the NationalNatural Science Foundation of China and other projects.
Figure1 Methodological heuristic framework: PID control loop +Meta-learning
Inreal-world scenarios such as medical diagnosis and adaptation ofunmanned systems, obtaining large amounts of labeled data is oftencostly or extremely difficult. Few-shot learning techniques aredesigned to enable models to master new tasks with only a fewexamples, but their training process often faces challenges ofinstability and slow convergence due to insufficient data. Most ofthe existing methods rely on fixed gradient update rules, making itdifficult to fine-tune and adaptively according to thecharacteristics of different tasks and different levels of thenetwork. To address this challenge, Professor Li's team drew keyinspiration from the fundamentals of control engineering. Theresearchers compared the training process of neural networks to theregulation problem of a dynamic system and for the first time deeplyintegrated the principle of the PID(proportional-integral-derivative) controller, which is widely usedin industrial control, into the model optimization framework. Thisinnovative LA-PID optimizer intelligently coordinates the currentgradient, the cumulative historical gradient, and the trend ofgradient changes, acting like an "intelligent regulator"that accelerates training convergence while effectively suppressingfluctuations and enhancing learning stability. The outstandingcontribution of this work lies in the realization of "one layer,one policy" dynamic fine-tuning. Unlike traditional methods thatapply uniform rules to the entire network, LA-PID uses a lightweightauxiliary network to analyze the statistical characteristics of eachlayer of network parameters in real time and independently generatethe most appropriate PID control gain for each layer. This enablesboth the shallow layers responsible for extracting edge textures andthe deep layers responsible for understanding semantic concepts to beoptimized in the most suitable way for their own characteristics,greatly enhancing the model's adaptability and expression efficiency.
Experimentalresults show that the algorithm has achieved leading performance inseveral internationally recognized few-shot learning benchmarks.LA-PID outperformed previous mainstream methods in classificationtasks on general-use datasets such as mini-ImageNet and CIFAR-FS, aswell as in rigorous cross-domain tests from natural images to fineclassification images of birds. Visualization analysis furtherconfirmed that the network trained by it was able to focus on morecritical and discriminative regions in the images. This study is asuccessful practice of the intersection of control science andartificial intelligence. It not only provides a powerful new tool forfew-shot learning, but more importantly, it opens up new paths foroptimizing the entire training process of deep learning models. Thisnew research paradigm transforms the learning process of intelligentsystems into an observable and controllable dynamic system, providingkey ideas for building learning systems with higher stability,efficiency and controllability in the future.
Figure2 Visual comparison of CAM heat maps of key network layers
Figure3 Visualization of model loss guided by dynamic PID parameters ondifferent datasets
