LAVE: Zero-shot VQA Evaluation on Docmatix with LLMs - Do We Still Need Fine-Tuning?

What changed

Although the generated answers semantically align with the reference answers, as illustrated in Figure 1, they still receive low scores. This raises these questions: Should we fine-tune the models to improve these metrics, or should we develop new metrics that better align with human perception? This shift is increasingly relevant with the rise of synthetic datasets such as Docmatix, SciGraphQA, SimVQA used to fine-tune Vision Language Models (VLMs).

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• Fine-tuning now available for GPT-4o
• Fine-tuning LLMs to 1.58bit: extreme quantization made easy
• Fine-tuning 20B LLMs with RLHF on a 24GB consumer GPU

• Fine-tuning now available for GPT-4o similarity 0.17
• Fine-tuning LLMs to 1.58bit: extreme quantization made easy similarity 0.17
• Fine-tuning 20B LLMs with RLHF on a 24GB consumer GPU similarity 0.17