SpikingLM

SpikingLM is a BERT-base style masked-language model with spiking attention blocks.

This checkpoint uses:

• temporal Spiking BERT with T=4
• learnable Q/K/V scaling parameters initialized from 7
• LIF nodes for projection, Q, K, V, attention output, and MLP blocks
• FP16OptimizedExp2Softmax through self.learnmax(attention_scores) for attention normalization

Files

config.json
model.safetensors
tokenizer.json
tokenizer_config.json
special_tokens_map.json
vocab.txt
spiking_bert/modeling_spiking_bert.py
scripts/finetune_glue.py
requirements.txt

Usage

from safetensors.torch import load_file
from transformers import AutoConfig, AutoTokenizer
from spiking_bert import BertForMaskedLM

repo_or_path = "YOUR_USERNAME/SpikingLM"

config = AutoConfig.from_pretrained(repo_or_path)
config.T = 4
config._attn_implementation = "eager"

tokenizer = AutoTokenizer.from_pretrained(repo_or_path)
model = BertForMaskedLM(config)

state = load_file("model.safetensors")
model.load_state_dict(state)

If you clone the repository locally, replace repo_or_path with the local clone path and load model.safetensors from that directory.

Results

Masked-language-model evaluation stored with the checkpoint:

{"perplexity": 57.81198691730261}

Notes

Large binary weights are stored as model.safetensors. The model code requires torch, transformers, safetensors, and spikingjelly.