Vision-language model, from the ground up

The goal

Build a small model that looks at an image and generates a caption for it — and actually understand, at the tensor level, why every piece of the pipeline is there. Not an attempt to beat a benchmark or invent a new architecture. The architecture itself (LLaVA-style adapters) is well established; the point was to build it by hand and be able to explain every shape, every masked position, and every gradient path without hand-waving.

Architecture

Two large pretrained models stay completely frozen. A small projector in between is the only thing that ever trains:

image
  │
  ▼
SigLIP2 (frozen)              — 768-dim patch embeddings, 196 patches per image
  │
  ▼
Projector (trainable, ~1.5M params)
  Linear(768 → 896) → GELU → Linear(896 → 896)
  │
  ▼
Qwen2.5-0.5B-Instruct (frozen) — 896-dim embedding space
  │
  ▼
caption (next-token prediction)

SigLIP2 turns an image into 196 patch embeddings. The projector's only job is translating those embeddings into Qwen's 896-dimensional token-embedding space — nothing more. Both foundation models stay exactly as pretrained; only those ~1.5M projector parameters ever receive a gradient update.

How a forward pass actually works

The projected patch embeddings get concatenated with the tokenized caption's embeddings into one sequence, which goes through the frozen LLM in a single forward pass. The loss is standard next-token cross-entropy, but with the image-patch positions and any padding masked out (label -100, which PyTorch's cross-entropy loss ignores by convention). That masking matters: without it, the model would be penalized for "mispredicting" tokens at positions that were never meant to be predicted, and gradients would leak into places they shouldn't. Because the vision encoder and the LLM are both frozen, every gradient that does flow has exactly one place to go — back into the projector.

Data pipeline

Trained on COCO Captions, Karpathy split (~20K images × 5 captions each, ~100K training examples). Fetching that at scale over a home connection meant writing a small threaded downloader — 16 workers, a JSONL index for O(1) resumable appends, and graceful interrupt handling so a killed process didn't corrupt the index. The resulting dataset was zipped, pushed to Google Drive, then pulled down and unzipped to local disk on whatever GPU box was training that day — Drive-mount latency during training was a non-starter.

The infra journey

Getting a stable training loop running was its own project. Colab's free TPU credits ran out mid-experiment and the runtime kept disconnecting. Kaggle's P100 turned out to be incompatible with the PyTorch version needed, so that meant switching to a T4 — which then hit an out-of-memory error at batch size 16, fixed by dropping to 8. The setup that actually stuck was a RunPod L4 instance, training inside tmux so a dropped SSH session didn't kill the run, checkpointing every 200 steps, with emergency crash-safe saves so an OOM or a spot-instance preemption never meant starting over.

Proving it learned something, not just that Qwen is good

A 0.5B-parameter instruction-tuned LLM already has a strong prior for producing fluent, plausible-sounding English — which makes it easy to accidentally build something that outputs coherent nonsense with no real connection to the image. So the real test wasn't "does the caption read well," it was "does the caption actually depend on the pixels."

Two checks:

Trained vs. randomly-initialized projector, same images. With a randomly-initialized projector, the "captions" were incoherent garbage — stray code fragments and CJK characters, nothing resembling a description. With the trained projector, captions were specific and mostly correct: "a double decker bus with people on board." Not perfect — one ski-slope photo got a caption that caught the gist of the scene but missed the dog in it entirely, a "right gist, misses secondary detail" failure mode that's informative in its own right.

Held-out COCO val2017 images, never seen in training. Captions stayed coherent and varied across genuinely new images — "red double-decker bus," "purple bus," "bear in grass" — which is the actual evidence against memorization. A model that had just memorized training captions would fall apart the moment the input distribution shifted even slightly; this one didn't.

Tracing one example through every stage

The deeper goal was understanding, not just working code, so I wrote a probe script that runs a single real image through the entire pipeline and prints the actual tensor values at every stage: pixel normalization, patch embeddings, the projector's weight matrices, tokenization, the -100 label masking, the logits, the loss, and the gradients on the way back. That traced through to the actual "why" behind the design:

• Why the same projector weights are reused across all 196 patches (weight sharing) instead of learning a separate transform per patch position.
• Why a two-layer projector with a GELU in between has more representational capacity than a single linear map, and what that buys in practice.
• Why masking with -100 is the correct way to exclude positions from a loss, rather than zeroing them out or filtering them post-hoc.
• What backpropagation is actually doing, mechanically, when only a small subgraph of a much larger frozen network is trainable.

Where it stands

Trained, checkpointed, and validated against both a random-projector baseline and a held-out test set. The architecture, the failure modes, and the reasons behind both are all understood end to end — which was the actual goal from the start.