Aobangaming/lightning-60m

Model Card for Model ID

Model Details

Lightning is a small, autoregressive transformer which utilizes FlashAttention and MHA. This model is trained on a variety of books from a dataset(300 MB).

Model Description

Lightning utilizes FlashAttention and AdamW for performance and capability. Lightning is designed to provide quick, coherent outputs, with the downside of limited embedding.

• Developed by: AobanZ
• Model type: Transformer
• Language(s) (NLP): English
• License: MIT
• Finetuned from model [optional]: Aoban-2.7-L

Model Sources [optional]

• Repository: https://huggingface.co/Aobangaming/lightning-60m

Uses

Lightning is intended to be used for research, analysis and fine-tuning, stories and other. It is not intended to be used for professional advice, real writing or any kind of heavy work as generated outputs may be incorrect.

Direct Use

Lightning can be used directly for text generation, experimentation, and conversational interactions. Users can provide text prompts and generate responses using the model's built-in language modeling capabilities.

Direct use is primarily intended for research and experimentation. Outputs may be incomplete, inaccurate, repetitive, or unrelated to the input, and should be evaluated before being used for other purposes.

Downstream Use

Lightning may be fined-tuned for an AI Story makers, Research, and small continuation models. However, please note that generated outputs may be corrupted and/or incorrect.

Out-of-Scope Use

Heavy Work may overload the model, which will cause corrupted outputs and/or misinformation if implemented into a larger-app/ecosystem.

Bias, Risks, and Limitations

Lightning is designed to process english and conversational text ONLY and cannot be fined-tuned for any other uses(eg. Robotics)

Recommendations

We recommend users of Lightning to finetune the model on new text, and add necessary guardrails and precautions to prevent misuse.

How to Get Started with the Model

Use the code below to get started with the model.

import torch
from transformers import AutoModelForCausalLM
from tokenizers import Tokenizer

model_id = "Aobangaming/lightning-60m"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True
)

tokenizer = Tokenizer.from_pretrained(model_id)

prompt = "The"

encoded = tokenizer.encode(prompt, add_special_tokens=False)
input_ids = torch.tensor([encoded.ids])

outputs = model.generate(
    input_ids,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.8,
    top_k=40,
    top_p=0.6,
)
print(tokenizer.decode(outputs[0].tolist()))

Training Details

Training Data

Lightning was trained on the full Booksum dataset.

Training Procedure

Lightning was trained on an RTX 3050 GPU, using FlashAttention and MHA. The model was trained on a large dataset. It was not trained on fine-tuning datasets since memory issues.

Training Hyperparameters

Hyperparameter	Value	Comment
Precision	FP32
Optimizer	AdamW	Better weight decay
Learning rate	5e-4
Batch size	32	Adapted for larger dataset

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: RTX 3050 6GB
• Hours used: 1.5
• Cloud Provider: My Computer
• Compute Region: Asia
• Carbon Emitted: ~0.17 kg CO₂e

Technical Specifications

Model Architecture and Objective

Lightning is a casual decoder model which is autoregressive.

Hyperparameter	Value	Comment
Layers	8
D_MODEL	384	Optimized for 64dim/head
Attention Heads	6	Improved from lightning-30m
Vocabulary	~65830	w/ 210 Sequence length

Compute Infrastructure

Hardware

RTX 3050 6GB

Software

Windows 11, Intel i5-10400