FineVision:
Open Data Is All You Need

huggingface.co/datasets/HuggingFaceM4/FineVision

To use the dataset, simply load it with:

from datasets import load_dataset, get_dataset_config_names

# Get all subset names and load the first one
available_subsets = get_dataset_config_names('HuggingFaceM4/FineVision')
ds = load_dataset(
  'HuggingFaceM4/FineVision',
  name=available_subsets[0],
  split='train', streaming=True,
)

# Inspect the first sample
ds[0]

Why this dataset?

Even though open-weight Vision-Language Models are becoming ever more powerful, the accessibility of the training data used for these models is lagging behind. This data is often proprietary and inaccessible to the broader community. Projects like The Cauldron, LLaVa, and Cambrian aim to provide such datasets, but get quickly outpaced by the speed of the field and the emergence of novel applications for VLMs, like agentic tasks. For FineVision we set out to combine and unify existing available data sources to create a large and high-quality dataset. As a first step we need to collect and standardize the datasets.

How did we build FineVision?

FineVision was a giant act of data curation. We started by collecting publicly available datasets, and augmenting underrepresented categories. We then evaluated all datasets for duplicated data internally and benchmark contamination. This data is then cleaned and rated, before being added to the final mixture.

Data Collection

We manually collected over 200 image-text datasets from various publicly available sources and processed them to unify their formatting. On top of that, some datasets are not presented in chat form, so we converted them into question-answer pairs. In some cases, this goes as far as synthetically creating questions for all samples. Finally, we adressed underrepresented domains, such as GUI-oriented data. To fill this gap, we create and add a new dataset that was compiled from existing GUI datasets, after applying chat normalization and unifying the action space to convert their specific formats into a more general GUI action space.

Cleaning

After gathering all the sub-datasets, every turn is cleaned. We removed all individual turns whose combined question and answer length exceeds 8192 tokens. We resize big images to have a longest side of 2048 pixels while keeping the aspect ratio, and discard samples with corrupted images.

Rating

Finally, we rate every single turn in our dataset across 4 axes. For this, we used a LLM and VLM-as-a-judge pipeline (using Qwen3-32B and Qwen2.5VL-32B-Instruct), to rate every turn on a scale from 1-5 in these 4 categories:

• Text Formatting Quality: How is the quality of the answer both linguistically and structurally? (Question and Answer)
• Question-Answer Relevance: Does the answer properly respond to the question? (Question and Answer)
• Visual Dependency: How much does the question depend on visual information to be answered? (Question only)
• Image-Question Correspondence: How well does the image support answering the question? (Image and Question)

This is the distribution of scores across the different filters for FineVision:

FineVision Base Dataset

We classify FineVision’s subsets into 9 categories: Captioning & Knowledge, Chart & Table, General VQA, Grounding & Counting, Mathematics, Naive OCR, OCR QA, Science and Text-only (Fig. 1).

There are multiple ways to count the data in a multimodal dataset. The most common are the number of samples and the number of images. Additionally, a single sample can consist of multiple question/answer pairs in the form of a multi-turn conversation. Similarly to text-only datasets, the number of answer tokens is also interesting, since these are the tokens the model is actually trained on. We count all these characteristics for FineVision and arrive at 17.3M images, 24.3M samples, 88.9M turns, and 9.5B answer tokens. Based on these 4 distributions, multiple different mixtures are possible. In conjunction with the provided ratings, we encourage the community to create their own mixtures and experiment with the data. For example, large categories could be downsamples, while high-quality data could be upsampled. After collecting and processing the data, we run multiple experiments and ablations to provide practical recommendations on how to train small, data-centric VLMs.

Experimental Setup

To ensure a fair comparison between different configurations, we use the same setup and evaluations for all of our ablations. This enables us to compare FineVision to other publicly available datasets as well as experiment with different intra-dataset configurations.

Model Architecture: nanoVLM

For all ablations and experiments, we train a 460M parameter VLM, since it provides a good trade-off between training time and model performance. We utilize the lightweight nanoVLM training framework with SmolLM2-360M-Instruct as the text backbone, and SigLIP2-Base-512 as the vision encoder. We experimented with a classic 2-stage training schedule where the first stage is used to train mainly the Modality Projection to align the Language and Image Embeddings, and the second stage is used to train the whole model. Interestingly, we did not observe any significant benefits from this additional first stage compared to training the whole model directly at our size and training duration, so we settled on a single-stage training for most ablations.

Baseline Datasets

We use three similar open source alternatives as baselines to compare our dataset to: The Cauldron, LLaVA-OneVision and Cambrian-7M.

Evaluations

We utilize lmms-eval during training to evaluate our ablations in a reproducible manner. We evaluate on a diverse set of 11 benchmarks: AI2D, ChartQA, DocVQA, InfoVQA, MME, MMMU, MMStar, OCRBench, ScienceQA, TextVQA and SEED-Bench. Since these benchmarks cover different topics and produce results on different scales, e.g. AI2D returns the accuracy of the exact matches (0-1), but MME returns a continuous score (0-2800), aggregating them is not trivial. In our ablations the relative performance between the different configurations matters, so to provide a robuts summary metric we determine the rank of each model compared to the others in every benchmark at every training step and average it over all the benchmarks. This way, we can judge where different configurations rank among each other over the course of training. To keep a sense of how big the absolute difference between models is, we also provide an average over all metrics and incorporate MME by normalizing it between 0 and 1.

Training Configuration

Each of our ablations trains the 460M model with a maximal image size of 1536x1536 pixel (without resizing smaller images) and a maximal input token length of 4096. This results in a maximum batch size of 2 for a single H100, which we adapt with 8 steps of gradient accumulation on each of the 32 GPUs for an effective batch size of 512. In all single stage configurations we train for 20k Steps on 32 H100s for approximately 20h while evaluating all 11 benchmarks every 1k Steps. If not specified otherwise, the “Baseline” in our intra dataset ablations refers to a training run on the full unfiltered and unchanged dataset. In this configuration, a full epoch of the unfiltered FineVision dataset takes 12k steps.

Experiments

While many interesting questions could be investigated, we mainly focus on the aspects of the training that are influenced by the data. Before we dive into the internal details of FineVision, let’s have a look at our performance against the baselines.

How does FineVision compare to other open datasets?

Here we see the first interesting trend: VLMs still benefit from training on a larger, more diverse dataset than what was available until today. FineVision doesn’t lead the race in the first few thousand training steps, after all, it does include new tasks such as pointing and agentic browsing, so it shouldn’t be better at first. But after seeing enough varied data, FineVision clearly shows the best performance across a wide set of benchmarks, which can be seen in its average ranking (Fig. 2). One epoch of FineVision in our setup takes 12k training steps, so we train for close to 2 epochs in these ablations. Looking at the average benchmark, we can see how the models saturate around different points: 18k steps for Cambrian, 12k for LLaVa and 7k for the Cauldron. In particular, over 11 different benchmarks, FineVision achieves an average improvement of 40.7% over the Cauldron, 12.1% over Cambrian, and 46.3% over LLaVa, which increases to 51.3%, 18.6% and 58.0% when comparing the deduplicated versions of the datasets. Additionally, FineVision includes data for tasks such as agentic browsing, and counting and pointing, which are not part of the other baselines.

How much test data is in publicly available datasets?

We investigate data leakage by finding images from test sets that appear in the dataset. For this, we constructed an image deduplication pipeline. We used this pipeline to compare all images in FineVision to all images of 66 image-text benchmarks from the lmms-eval framework.

For the comparison, we embed the images using the SSCD descriptor, and compute the cosine similarity between a given image in FineVision and all images from the test-set embeddings. Whenever a sample has a similarity higher than a threshold of 0.95 it is assumed to be a duplicate.

While our tests with various thresholds show that this is still flagging more false-positives than false-negatives, given the scale of data we have, we preferred to err on the side of caution.

Below is an example of a correctly identified Duplicate (“Photo”), a false-positive with a similarity score above 0.95 (“Chart”) and a false-negative with a similarity score below 0.95 (“Drawing”) (Fig. 3).

We open-source the deduplication pipeline here as well as the precomputed test-set embedding’s here.

We repeated this deduplication procedure on all the baselines to analyse how contaminated they are. We found that all baselines contain between 2-3% images from test benchmarks, and removing them results in a performance drop of 2.4-2.8%. Interestingly, we find that for some benchmarks the difference is negligible, while other benchmarks suffer significantly. For example, after deduplicating, ScienceQA falls by 14.49% on average while OCRBench only drops by 1.08%. This deduplications also shows that FineVision contains the smallest relative amount of duplicated data at 1%, and also suffers the smallest performance drop over all benchmarks after deduplication at just 1.45%.

Additionally, we experimented with removing all found samples from all datasets to see if the outcome is different from Fig. 2, but we observe the same distribution (Fig. 4).

How diverse are the datasets?

Similarly to the size comparison, we also wanted to evaluate the datasets for diversity. Evaluating the diversity of a dataset is a field of study for itself, which we will not dive into here, rather we borrow techniques from computer vision and use the already computed SSCD embeddings as a proxy of visual diversity. To not rely on a subsample of the dataset in estimating the diversity, we analyse the covariance metric of the full embeddings. From this covariance matrix, we can calculate the eigenvalues for analysis. We get the effective rank of the covariance matrix, which measures how uniformly the variance is distributed across dimensions, as well as the participation ratio, which measures how many dimensions actively contribute to the overall variance. To obtain a single diversity score for the datasets, we normalize the effective rank and participation ratio with the embedding dimension and compute their geometric mean. We observe that FineVision is not only the biggest, but also the most diverse dataset. Additionally, you can also clearly see that more images do not necessarily result in more diversity, since LLaVa is substantially less diverse than the Cauldron, even with more images.

Should you merge multiple questions for the same image into a single multi turn conversation?

Since the training of a VLM already builds upon pretrained vision and language backbones, datasets are usually not completely unstructured, but follow an image+question and answer structure. Some works have shown that consolidating multiple questions for the same image into a multi-turn conversation where the image is shown only once improves model performance, reduces training budget, and reduces the datasets’ memory footprint. We therefore experiment with deduplicating every image in our dataset internally using the same SSCD descriptors, manually inspect the resulting clusters, and merge fitting samples into a multi-turn conversation.

When training with the same training budget, we find that both models perform very similarly (Fig. 5). Some benchmarks favor one image/several turns, while others favor one image/one turn. Given this, we decide to release the dataset without merging multiple questions for the same image, and open-source the pipeline in case users want to explore this further.

Should you train on multilingual data if your language backbone was not?

There are some multilingual datasets in our mixture, but since our Language Backbone is only trained on English data, we experimented with removing all the multilingual, mainly Chinese, subsets. Our results show that there is a slight advantage in leaving the multilingual data, even if it was not part of the Language Backbone’s initial training. We believe this reinforces our hypothesis that more diversity in the dataset is generally preferable for VLM training. In our training setup with this configuration, one epoch over the whole non-deduplicated dataset equals ~12k steps, so the benefit of unseen languages only materializes after the first full epoch (Fig. 5).

How can you assess the quality of the dataset?

The usual goal for every dataset, to collect samples with the highest quality possible, is quite an abstract endeavour in practice, especially for multimodal datasets. Additionally, different training stages usually have different qualitative and quantitative requirements. Finally, tuning the mixtures of different categories is also reliant on how much data with what quality is available. For image-text datasets, there are 3 different combinatorial ways to evaluate a sample: text-only, image-only, and image-text correspondence. The question persists, how do you actually measure the quality of a sample, especially if you have to do so in 3 different ways. We propose doing so by leveraging both a LLM and a VLM as a judge.

To try to quantify the quality of the training data and the effect it has on the model’s performance, we run extensive ablations on our generated ratings.

Interestingly, both when only training on turns that have any of the 4 ratings under a certain threshold, as well as when training on turns where only a single rating at a time is used, we observe the same behaviour. Simply training on the most diverse data, that one containing all samples, outperforms in benchmarks (Fig. 6) (Fig. 7). This could mean multiple things. Firstly, we can see almost the same distribution in the ranks across all filters: from best to worst with an increase in the rating threshold. For example the visual dependency and the image correspondence rating both result in exactly the same distribution of rankings, corresponding to the natural order of options, 1 through 5. This could indicate that with a sufficiently large dataset that you train on long enough, it hurts more to remove samples, even if they were judged to be of low quality, than to train on them.

Additionally, the notion of quality in VLM datasets is inherently nuanced. Unlike LLMs, where pre-training often relies on massive web crawls, training a VLM is closer to the supervised fine-tuning (SFT) stage. We do not train on crawls of internet data, instead we train on individual samples of Image-Question and Answer pairs, and these datapoints are usually ‘curated rather than collected’. We also do not train on trillions of tokens, but on billions. This built-in curation provides a baseline level of quality from the start. FineVision follows this pattern: it brings together widely used VLM datasets along with a few new ones in low-resource domains. We could therefore be trying to measure and quantify noisy nuances in the quality of Image-Question-Answer Pairs, instead of using the fact that they are already curated SFT datasets as the measure for quality.

Alternatively, while we used state-of-the-art open source models to judge our datapoints, we still had to find a compromise between model quality and cost due to the raw required effort to rate every single turn of FineVision. The chosen models could simply not be powerful enough to recognize and judge the quality of samples. Even though our first proposal to judge the quality of multimodal data on a per-turn basis did not yield any improvement in model performance, we believe that this is still an exciting and important direction of research and hope the release of FineVision encourages the community to develop techniques for this at large scale.

Should you train in multiple stages?

The standard training procedure of a VLM usually follows at least two stages. First, you train only the connecting module, potentially in addition the image encoder, and then you train the whole model in a second stage. Some work has even introduced an additional Stage 2.5 (141), where you train the full model on a smaller subset of higher quality data. To investigate this on small models, we experiment both with single, dual and triple stage training.

1 Stage vs 2 Stages

To evaluate if pre-training the Modality Projection and the Vision Encoder provides any benefits to the final model performance, we conduct this experiment at a higher image resolution of 2048px and train substantially longer. We can see that even for training longer, the general difference in model performance is quite small. Individual benchmarks, do show differences (ScienceQA drops by 5% but OCRBench improves by 5% in the two-stage setup) (Fig. 8), so the better setup is individual to the desired model capabilities. This also shows that evaluation (and through this also correctly training) a VLM is not straightforward tasks, since availible benchmarks are limited proxies for the underlying model performance.

2 Stages vs 2.5 Stages

We also experiment if splitting the second stage results in any performance improvements.

We take the baseline, and continue training for another 20k steps, both with the unfiltered (>= 1) as well as filtered subsets of FineVision according to our ratings.

As in the previous results, we observe that the best outcome is simply achieved by training on as much and as diverse data as possible (Fig. 9). Like before, this could also be due to the way we filter the data, and a different quality measure might yield different results.

Conclusion

We introduce FineVision, a new state of the art open dataset to train VLMs, that is both bigger and more diverse than previous open source datasets. We provide extensive analysis regarding size, diversity, contamination and data-centric model training, and hope we can empower both further research and the community with this.