Poh, Soon Chang and Chan, Chee Seng and Lim, Chee Kau (2024) Efficient label-free pruning and retraining for Text-VQA Transformers. Pattern Recognition Letters, 183. pp. 1-8. ISSN 0167-8655, DOI https://doi.org/10.1016/j.patrec.2024.04.024.
Full text not available from this repository.Abstract
Recent advancements in Scene Text Visual Question Answering (Text-VQA) employ autoregressive Transformers, showing improved performance with larger models and pre -training datasets. Although various pruning frameworks exist to simplify Transformers, many are integrated into the time-consuming training process. Researchers have recently explored post -training pruning techniques, which separate pruning from training and reduce time consumption. Some methods use gradient -based importance scores that rely on labeled data, while others offer retraining -free algorithms that quickly enhance pruned model accuracy. This paper proposes a novel gradient -based importance score that only necessitates raw, unlabeled data for post -training structured autoregressive Transformer pruning. Additionally, we introduce a Retraining Strategy (ReSt) for efficient performance restoration of pruned models of arbitrary sizes. We evaluate our approach on TextVQA and ST-VQA datasets using TAP, TAP dagger dagger and SaL double dagger- Base where all utilize autoregressive Transformers. On TAP and TAP dagger dagger , our pruning approach achieves up to 60% reduction in size with less than a 2.4% accuracy drop and the proposed ReSt retraining approach takes only 3 to 34 min, comparable to existing retraining -free techniques. On SaL double dagger- Base, the proposed method achieves up to 50% parameter reduction with less than 2.9% accuracy drop requiring only 1.19 h of retraining using the proposed ReSt approach. The code is publicly accessible at https://github.com/soonchangAI/LFPR.
| Item Type: | Article |
|---|---|
| Funders: | UNSPECIFIED |
| Uncontrolled Keywords: | Transformer; Pruning; Scene text visual question answering |
| Subjects: | Q Science > QA Mathematics > QA75 Electronic computers. Computer science |
| Divisions: | Faculty of Computer Science & Information Technology |
| Depositing User: | Ms. Juhaida Abd Rahim |
| Date Deposited: | 19 Sep 2024 01:40 |
| Last Modified: | 19 Sep 2024 01:40 |
| URI: | http://eprints.um.edu.my/id/eprint/45137 |
Actions (login required)
 |
View Item |
