Glossary · Architecture

Rotary Positional Embedding (RoPE)

A technique for encoding token position information into a transformer model using rotation matrices, widely adopted for its ability to generalize well to longer sequences than seen during training.