Anyone up for making a FP8 for 4B embedding and reranker?
#25
by huggyfaceenjoyer - opened
Anyone up for making a FP8 for 4B embedding and reranker?
I'd love to know how to do it properly, I have been able to do it for the embedding, and it "seems" to work, but I am not 100% sure I did it correctly.
For what it's worth, I went with NVFP4 instead of FP8 (finer-grained, native FP4 tensor cores on Blackwell) and benchmarked both a weight-only and a full weights+activations variant against the BF16 original with the official MTEB evaluators, at 768/1024/1536/2048 dims:
- NVFP4A16 (weights only, activations stay BF16): ~99% of BF16 average MTEB score, ~2.7GB vs ~8GB BF16.
- NVFP4 (weights + activations): ~97-98% of BF16, same size, faster FP4 matmul.
Full per-task/per-dim tables and raw scores are in both model cards, in case it saves you the calibration work.