Image and caption, generated together

The core idea

Give the system one random number — a seed — and it produces both a stylized engraving image and a caption describing that exact image. Neither output is derived from the other: the image doesn't get captioned after the fact, and the caption doesn't get illustrated after the fact. Both come from the same shared internal representation.

Data preparation

Started from a Kaggle art dataset and isolated just the "engraving" style — 272 images after cleaning out corrupt files, resized to 512×512. Captions were generated once, offline, by running BLIP2 over the cleaned set ("an engraving of a woman and child in a bed"), producing a fixed captions.jsonl used as training data. That BLIP2 pass has nothing to do with how the trained model behaves at inference — it only exists to bootstrap ground-truth captions for training.

Architecture

seed (random number)
   │
   ▼
SeedEncoder (small MLP)
   │
   ├──► z_pooled ──────► CaptionDecoder (GPT-2 + learned prefix) ──► caption
   │
   └──► z_sequence ────► DiffusionBranch (Stable Diffusion 1.5 + LoRA) ──► image

The SeedEncoder maps the raw random seed into a semantic latent, z — the shared representation both branches read from and neither branch's output feeds back into. The DiffusionBranch is standard Stable Diffusion, except its cross-attention conditioning slot is fed z directly instead of a text prompt, with a LoRA trained on top so the base model picked up the engraving style specifically. The CaptionDecoder is GPT-2, but instead of reading pixels it reads a soft prefix derived from that same z — it literally cannot see the generated image, only what z encodes.

Keeping the two branches honest

Sharing z alone doesn't guarantee the image and caption actually agree — both branches could independently learn to look plausible while quietly drifting apart from each other. So every 10 training steps, the model generates a real image and a real caption from the same z, runs both through CLIP, and applies a contrastive loss that directly penalizes any semantic mismatch between them. That's the actual coherence guarantee — not an assumption that shared conditioning would be enough on its own.

Training

Three losses combined: the diffusion loss (does the image look like an engraving), the caption cross-entropy loss (does the caption read like real English), and the coherence loss above. Trained for roughly 2,500 steps — 18 epochs over the cleaned image set — on an L4 GPU. Losses came down cleanly: caption loss dropped from about 4.0 to 0.5, diffusion loss reached near-zero on many steps, and the coherence loss stabilized alongside both.

Inference and deployment

Given any seed, a MultimodalGenerator runs both branches and returns a real image plus a real caption. A small Gradio app wraps that: type a seed, hit generate, see both outputs. On the deployment side, a FastAPI layer adds per-key auth and rate limiting, deployable either directly on a GPU pod or — better for real scale — via Runpod Serverless, so GPU workers autoscale with traffic instead of running around the clock.

What actually broke

None of this shipped without real debugging. Four bugs, each one non-cosmetic — any of them left unfixed would have either crashed training or quietly produced a broken model:

• A PyYAML quirk silently turned configured learning rates into strings instead of floats.
• A transformers version upgrade changed CLIP's API return type mid-project, breaking the coherence loss.
• The worst one: Runpod's network-mounted filesystem was silently corrupting large file writes — checkpoints, even source files, came back as 0 bytes. The fix was moving everything onto local pod disk instead of the network mount.
• An out-of-memory retry loop had no exit condition and looped over 700 times before it got capped.

Where it stands

A trained, checkpoint-verified model (latest.pt, step 2496) runs end to end locally. Of the original stretch goals, prompt+seed conditioning, latent interpolation, and style-aware captions are done; a shareable public demo is the open item, prioritized whenever the pod spins back up.