heron1i-flash-lite: preview of Heron Language Models
A first look at the Heron language models: 9.9 million parameters, trained from scratch.
heron1i-flash-lite is the first public Heron language model: a 9.9M-parameter base model for English and code, trained from scratch. The weights and inference code are on Hugging Face, and a further-trained version, heron1i-flash-lite-v2, raises its Intelligence Index from 10.31 to 11.11.
Architecture
It is a 14-layer decoder, 240 wide, with grouped-query attention (6 query heads, 3 key-value heads). The design came out of a series of controlled training runs. Three changes beat the baseline and are built in:
- A short causal convolution (kernel 3) on the query, key and value projections, so each position mixes in the two tokens before it before attention runs.
- Rotary position embeddings on half of each attention head. The other half carries no position information.
- A soft cap on the output logits.
Ten of the fourteen layers attend to a 256-token window, and the rest see the full 1024-token context.
Results
Zero-shot, scored the way the Open SLM Leaderboard scores every model.
| Benchmark | Score |
|---|---|
| HellaSwag | 28.36 |
| ARC-Easy | 36.20 |
| ARC-Challenge | 24.15 |
| PIQA | 58.54 |
| ArithMark-3 | 35.60 |
| Intelligence Index | 10.31 |