LNTTushar/tryn-mini-7m

TrynMini v1 — a sentence transformer built from scratch (research prototype)

A compact transformer sentence encoder implemented from scratch — embeddings, sinusoidal positional encoding, multi-head self-attention, feed-forward blocks, and masked mean pooling, all hand-written (no transformers library) and trained on STS-B plus synthetic similarity data.

This is an early research prototype, not a production encoder.

v1 shipped with a tiny auto-generated vocabulary (~164 tokens) learned from limited, templated training text. As a result it handles the phrases it saw in training but does not generalize to arbitrary English — most real-world words fall back to [UNK]. It's published as a learning milestone and for provenance.

For a model you can actually use, see LNTTushar/trynmini-v2-static-7m-v2 — a properly trained static embedder (30k vocab, 0.715 STSB-dev, 7.8 MB, CPU-only, 5-line quickstart). A re-trained transformer v2 with a real 30k vocabulary is also in progress.

Architecture

• Custom Transformer encoder: 4 layers · 6 attention heads · 384-dim · mean pooling
• Parameters: 3,299,584 (~3.3M)
• Model size: ~12.6 MB
• Vocabulary: ~164 tokens (auto-generated — the main limitation; fixed in v2's 30k vocab)
• Max sequence length: 128 tokens

Results

On its own STS-B evaluation (within the limits of the 164-token vocab):

Metric	Score
Spearman	0.677
Pearson	0.672

These numbers reflect the small-vocab setup and should not be compared directly to encoders trained on a full vocabulary.

What this project is

TrynMini is a from-scratch study of how sentence embedding models work, built day by day: tokenizer → transformer → distillation → static compression → calibration. v1 (this repo) is the first working transformer stage. The line continues in the static v2 models linked above, which are the ones intended for real use.

License

Apache-2.0.