---
base_model: google/gemma-3-27b-it
library_name: peft
pipeline_tag: image-text-to-text
license: gemma
tags:
- gemma3
- vlm
- lora
- peft
- screenshot-to-code
- react
- code-generation
language:
- en
---

# gemma-3-27b-it-vlm-react-screenshot-to-code

A LoRA adapter for **`google/gemma-3-27b-it`** (Gemma 3 27B IT, vision-language) that turns a
**screenshot of a UI into React component code**.

Give it an image of an interface; it returns a single self-contained React component that
reproduces the layout.

This is the 27B sibling of
[`gemma-3-4b-it-vlm-react-screenshot-to-code`](https://huggingface.co/Reubencf/gemma-3-4b-it-vlm-react-screenshot-to-code).

👉 **Live demo:** [Reubencf/react-screenshot-to-code](https://huggingface.co/spaces/Reubencf/react-screenshot-to-code)

## Model details

| | |
|---|---|
| Base model | `google/gemma-3-27b-it` (`Gemma3ForConditionalGeneration`) |
| Adapter type | LoRA (PEFT `0.15.1`) |
| Rank / alpha | `r=4`, `lora_alpha=8`, `lora_dropout=0.0` |
| Target modules | `q_proj`, `v_proj` |
| Adapted layers | Language model self-attention only, all 62 decoder layers (248 tensors) |
| Frozen | Full SigLIP vision tower and multimodal projector (listed in `exclude_modules`) |
| Adapter size | ~16 MB (float32) |
| Base precision | bfloat16 |

Only the language model's query and value projections are adapted — the vision encoder is
untouched, so the adapter changes *how the model writes code*, not how it sees images.

The rank is lower than the 4B adapter's (`r=4` against `r=16`) and it touches two projections
rather than four, so it is a lighter intervention on a much larger base.

> **Note on the base model.** The training run recorded its base as
> `togethercomputer/gemma-3-27b-it-VLM`, which is **not a resolvable Hub repo** (404). The
> bundled `config.json` matches `google/gemma-3-27b-it` exactly (hidden 5376, 62 text layers,
> 32 query heads / 16 KV heads, SigLIP-27 @ 896px, vocab 262208), and the adapter's own tensor
> shapes agree (`q_proj` 5376→4096, `v_proj` 5376→2048), so `base_model_name_or_path` has been
> corrected to point there. `unsloth/gemma-3-27b-it` is an ungated mirror of the same model if
> you would rather not request access.

## Training

Short LoRA run, recorded in `trainer_state.json`:

| | |
|---|---|
| Epochs | 1 |
| Steps | 26 (`max_steps=26`) |
| Train batch size | 1 |
| Logging / eval every | 1 / 5 steps |
| Total FLOPs | 1.03e18 |
| Start train loss | 0.624 |
| Final eval loss | **0.590** |

Eval loss fell at every checkpoint (0.631 → 0.617 → 0.603 → 0.593 → 0.590), so the run was
still improving when it stopped.

This is a **light adaptation, not a full fine-tune** — 26 steps at batch size 1. Expect it to
nudge output format and style toward React rather than to teach new visual reasoning. Treat
generations as a starting scaffold to edit, not production code.

## Usage

Requires `transformers >= 5.10.1` — the adapter keys use the refactored
`model.language_model.layers.*` module layout and will not match older checkpoints.

The base model is ~54 GB in bfloat16. On a single 24 GB card, load it 4-bit and skip the
merge (see below).

```python
import torch
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from peft import PeftModel
from PIL import Image

BASE = "google/gemma-3-27b-it"          # or "unsloth/gemma-3-27b-it" (ungated mirror)
ADAPTER = "Reubencf/gemma-3-27b-it-vlm-react-screenshot-to-code"

processor = AutoProcessor.from_pretrained(BASE)
model = Gemma3ForConditionalGeneration.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER).merge_and_unload().eval()

image = Image.open("screenshot.png").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text", "text":
         "Convert this screenshot into a single self-contained React component. "
         "Use Tailwind CSS classes for styling. Return only code."},
    ],
}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device, dtype=torch.bfloat16)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=1536, do_sample=True,
                         temperature=0.7, top_p=0.95)

print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
```

### Fitting on one 24 GB GPU

Load the base in NF4 and leave the adapter unmerged — `merge_and_unload()` folds the LoRA
deltas back into the base weights at their original precision, which a quantised base cannot
represent.

```python
from transformers import BitsAndBytesConfig

quant = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = Gemma3ForConditionalGeneration.from_pretrained(
    BASE, quantization_config=quant, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER).eval()   # note: no merge
```

## Prompt format

Standard Gemma 3 chat template (`chat_template.jinja` is included). Images are inserted as
`<start_of_image>` and expand to 256 soft tokens each at 896×896.

```
<bos><start_of_turn>user
<start_of_image>Convert this screenshot into a React component.<end_of_turn>
<start_of_turn>model
```

## Limitations

- **26 training steps.** Output quality is closer to the base model than to a task-specialised one.
- Single-screenshot, single-component. No routing, state management, or multi-page flows.
- Colours, spacing, and font sizes are approximated from the image — expect to tune them.
- Text in the screenshot is transcribed by the vision encoder and can be misread at small sizes.
- Generated code is unverified. Review it before running, as you would any model output.
- Inherits all limitations and the licence of the Gemma 3 base model.

## Licence

Governed by the [Gemma Terms of Use](https://ai.google.dev/gemma/terms). The adapter weights are
a derivative of Gemma 3 and are subject to the same terms.

### Framework versions

- PEFT 0.15.1
- Transformers 5.10.1