Enhancing waste recognition with vision-language models: A prompt engineering approach for a scalable solution

Abstract

Conventional unimodal computer vision models, trained on limited bespoke waste datasets, face significant challenges in classifying waste images in material recovery facilities, where waste appears in diverse forms. Maintaining performance of these models requires frequent fine-tuning with extensive data augmentation, which is resource-intensive, time-consuming and ultimately unsustainable for scalable applications. This study implements multimodal image classification by using state-of-the-art Vision-Language Models (VLMs), showcasing their adaptability in data-scarce scenarios with zero-shot classification and their scalability through few-shot and fully supervised learning. A trade-off between accuracy and inference speed is sought as a criterion for selecting optimal model. It is demonstrated that targeted prompt engineering can significantly enhance VLM adaptability and scalability, as evidenced by the increase in zero-shot waste image classification accuracy of the optimal model from 82.71% to 90.48% without task specific finetuning. Additionally, experimenting with the optimal model resulted in an impressive fully supervised classification accuracy of 97.18%. This research contributes to the waste management literature by revealing the potential of VLMs as a scalable solution for waste image classification and by introducing a targeted prompt engineering method to enhance model performance as a final-stage optimization strategy.

Keywords: Computer vision; Deep learning; Prompt engineering; Sustainable waste management; Vision-language model; Zero-shot learning.