Benchmarking Top Vision Language Models (VLMs) for Image Classification

Introduction

In the rapidly evolving field of artificial intelligence, the ability to accurately interpret and analyze visual data is becoming increasingly crucial. From autonomous vehicles to medical imaging, the applications of image classification are vast and impactful. However, as the complexity of tasks grows, so does the need for models that can seamlessly integrate multiple modalities, such as vision and language, to achieve more robust and nuanced understanding.

This is where Vision Language Models (VLMs) come into play, offering a powerful approach to multimodal learning by combining image and text inputs to generate meaningful outputs. But with so many models available, how do we determine which one performs best for a given task? This is the problem we aim to address in this blog.

The primary goal of this blog is to benchmark Top Vision Language Models on an image classification task using a basic dataset and compare its performance to our general-image-recognition model. Additionally, we will demonstrate how to use the model-benchmark tool to evaluate these models, providing insights into their strengths and weaknesses. By doing so, we hope to shed light on the current state of VLMs and guide practitioners in selecting the most suitable model for their specific needs.

What are Vision Language Models (VLMs)

A Vision Language Model (VLM) is a type of multimodal generative model that can process both image and text inputs to generate text outputs. These models are highly versatile and can be applied to a variety of tasks, including but not limited to:

• Visual Document Question Answering (QA): Answering questions based on visual documents.
• Image Captioning: Generating descriptive text for images.
• Image Classification: Identifying and categorizing objects within images.
• Detection: Locating objects within an image.
  

The architecture of a typical VLM consists of two main components:

1. Image Feature Extractor: This is usually a pre-trained vision model like Vision Transformer (ViT) or CLIP, which extracts features from the input image.
2. Text Decoder: This is typically a Large Language Model (LLM) such as LLaMA or Qwen, which generates text based on the extracted image features

These two components are fused together using a modality fusion layer before being fed into the language decoder, which produces the final text output.

General architecture of vlm, image taken from hf blog.

There are many Vision Language Models available on the Clarifai Platform, including GPT-4o, Claude 3.5 Sonnet, Florence-2, Gemini, Qwen2-VL-7B, LLaVA, and MiniCPM-V. Try them out here!

Current State of VLMs

Recent rankings indicate that Qwen-VL-Max-0809 has outperformed GPT-4o in terms of average benchmark scores. This is significant because GPT-4o was previously considered the top multimodal model. The rise of large open-source models like Qwen2-VL-7B suggests that open-source models are beginning to surpass their closed-source counterparts, including GPT-4o. Notably, Qwen2-VL-7B, despite its smaller size, achieves results that are close to those of commercial models.

Experiment setup

Hardware

The experiments were conducted on Lambda Labs hardware with the following specifications:

Models of Interest

We focused on smaller models (less than 20B parameters) and included GPT-4o as a reference. The models evaluated include:

Results

Dataset: Caltech256

The Caltech256 dataset consists of 30,607 images across 256 classes, plus one background clutter class. Each class contains between 80 and 827 images, with image sizes ranging from 80 to 800 pixels. A subset of 21 classes (including background) was randomly selected for evaluation.