heron1i-flash-lite: preview of Heron Language Models

A first look at the Heron language models: 9.9 million parameters, trained from scratch.

Article

heron1i-flash-lite is the first public Heron language model: a 9.9M-parameter base model for English and code, trained from scratch. The weights and inference code are on Hugging Face, and a further-trained version, heron1i-flash-lite-v2, raises its Intelligence Index from 10.31 to 11.11.

Architecture

It is a 14-layer decoder, 240 wide, with grouped-query attention (6 query heads, 3 key-value heads). The design came out of a series of controlled training runs. Three changes beat the baseline and are built in:

Ten of the fourteen layers attend to a 256-token window, and the rest see the full 1024-token context.

Results

Zero-shot, scored the way the Open SLM Leaderboard scores every model.

BenchmarkScore
HellaSwag28.36
ARC-Easy36.20
ARC-Challenge24.15
PIQA58.54
ArithMark-335.60
Intelligence Index10.31