Diamegs/PIT-4B-202112

PIT-4B — Point-In-Time GPT (Pre-trained, 2021-12)

Point-In-Time (PIT) is a family of GPT-style language models trained on chronologically-ordered monthly snapshots of FineWeb. Each checkpoint captures the state of knowledge available up to a specific month, making them suitable for temporal reasoning and point-in-time analysis tasks.

This is the base (pre-trained only) variant. For instruction-following, use the corresponding PIT-4B-FT checkpoint.

Model details

Property	Value
Snapshot month	2021-12
Training step	42325
Architecture	Decoder-only Transformer (GPT)
Layers	20
Hidden dim	4096
Attention heads	32
Vocab size	50304
Tokenizer	GPT-2 BPE
Position encoding	RoPE
Normalization	RMSNorm on Q/K + pre-norm
Activation	Squared ReLU
Weight tying	Yes (input emb ↔ lm_head)

Requirements

pip install transformers torch safetensors

Quick start

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "Diamegs/PIT-4B-202112"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,   # required for custom architecture
    torch_dtype=torch.bfloat16,
)
model = model.cuda()
model.eval()

Text generation

prompt = "In 2021, the global economy"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=200,
    do_sample=True,
    temperature=0.8,
    top_p=0.95,
    repetition_penalty=1.1,
    pad_token_id=tokenizer.eos_token_id,
)
n_prompt = inputs["input_ids"].shape[1]
print(tokenizer.decode(output[0][n_prompt:], skip_special_tokens=True))

Temporal reasoning example

Because this model was trained on data up to 2021-12, it reflects the world as it was known at that point. You can use this for point-in-time analysis:

# What does the model "know" about events before its cutoff?
prompt = "The most important AI developments in early 2021 were"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=150,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    pad_token_id=tokenizer.eos_token_id,
)
n_prompt = inputs["input_ids"].shape[1]
print(tokenizer.decode(output[0][n_prompt:], skip_special_tokens=True))

Weights format

Weights are stored in safetensors format (model.safetensors) — memory-mapped, fast to load, and safe (no arbitrary code execution).

Limitations

• Knowledge is limited to web text available up to 2021-12.
• No RLHF or safety fine-tuning has been applied (base model).
• The model may reproduce biases present in FineWeb training data.
• Not suitable for safety-critical applications without further alignment.

Citation

@misc{pit_llm,
  title     = {Point-In-Time LLM},
  author    = {Diamegs},
  year      = {2024},
  publisher = {HuggingFace},
  url       = {https://huggingface.co/Diamegs}
}