News Weekly
ENहिन्दी
LV 10 XP
0% read
Your progress · 0/5 chapters
About 8 min total
Models & EconomicsISSUE #4 · STORY 36 OF 40Sep 28, 2026COMPANY CLAIM

A 309B open-weights model with no full-attention layers anywhere in it

NaiveAI released Naive-N0.5-Flash on 28 September 2026 under MIT: 309B total parameters, 15.5B active, a native 1M-token context built from 39 sliding-window and 9 sparse-attention layers, and weights published as 49 FP8 shards. Every performance figure is the vendor's own, with no independent reproduction.

A vast hall with a continuous row of identical booths whose openings reach only their immediate neighbour, sealed by a solid frieze above where a single wide window would be.

Read it your way

CHAPTER 1Picked for Explorer

The 60-second version

NaiveAI published Naive-N0.5-Flash on 28 September 2026: an MIT-licensed mixture-of-experts model with 309B total parameters, 15.5B active and a native 1M-token context window. The attention stack is the point. There is no full-attention layer anywhere in the 48.

309B total, 15.5B activeRoughly one twentieth of the parameters run per token, and the whole set still has to fit in memory.
Thirty-nine local, nine sparseA standard module is five sliding-window layers then one sparse layer, with the very first layer also sparse.
Weights you can actually getForty-nine safetensors shards of roughly 315GB are on Hugging Face, alongside the inference runtime.
Every speed number is the vendor'sThe 2,122 tokens-per-second figure comes from NaiveAI's own blog case study, on eight GPUs.
Finish this chapter for +15 XP
An archival table entirely covered in identical closed folders, with one reader passing its whole length, touching each folder once and carrying away only a fixed handful.
Move the volume, keep the rates

What NaiveAI's announced rates cost per month

DRAG THE SLIDER
Input at announced rates
$10
$0.10 per 1M input tokens
Output at announced rates
$40
$0.40 per 1M output tokens
Output premium at equal counts
$30
every month

Rates are NaiveAI's own published plan for a future API, not an observed tariff anyone has been billed against, and they were announced alongside the weights on 28 September 2026. Cached input reads are listed at $0.01 per 1M tokens, one tenth of the uncached input rate. Long context is where these numbers get interesting: a single request can carry a million tokens, so the input line can dominate a conversation before the model has written anything back.

Your next move · Explorer

Read the architecture, then the claims

1Count the layers yourself: 48 total, split 39 sliding-window and 9 sparse, with no full attention.
2Check what the sparse path keeps, since the indexer scans full history and the cache is retained.
3Separate published specifications from performance, because only the first kind is checkable.

Switch your reading mode at the top to see a different next move.

Tap to open

Things to keep an eye on

Pop quiz · unlock the No full attention badge

Did it stick?

0/3
What is unusual about this model's attention stack?+20 XP
What does the sparse-attention layer actually skip?+20 XP
Which figure here is a company claim rather than an observation?+20 XP
Your call · +5 XP

You are sizing a long-context serving budget. What matters most here?

Research dossierThe full research, labeled and sourcedCOMPANY CLAIM6 sources · 8 min read · 25 sections
Story identity
  • Story ID: S36
  • Title: NaiveAI open-weights Naive-N0.5-Flash: 309B MoE, 15.5B active, 1M native context, MIT licensed
  • Organization: NaiveAI
  • Category: Open-Weight Models / Model Architecture
  • Event date (actual event): 2026-09-28 (announcement date: 2026-09-28)
  • Evidence status: COMPANY CLAIM (confidence Medium)

Evidence labels used in this research

Source limitations (from discovery): Model card specifications are vendor-authored; no independent benchmark reproduction of the 1M-context behaviour or quality was available at discovery time.

✓

What happened?

🎓 For Explorer

On 2026-09-27/2026-09-28 (public announcements dated Sept 27 on company blog, files committed to HF on Sept 27), NaiveAI publicly released Naive-N0.5-Flash, an open-weight mixture-of-experts language model with:

  • 309B total parameters, 15.5B active parameters (MoE)
  • 48 transformer layers
  • Native 1M token context window
  • Hybrid attention stack: Sliding-Window Attention (SWA) + DeepSeek Sparse Attention (DSA), with no full-attention layers
  • MIT license for weights and inference code
  • Model weights published on Hugging Face (NaiveAI/Naive-N0.5-Flash) as FP8 shards (49 files totaling ~315GB)

The release is explicitly positioned as "built with AI" (AI-centered R&D) for coding and AI R&D workloads.

Δ

What changed?

Before: Typical frontier/open-weight long-context models often rely on full/self-attention variants (or hybrid designs that still include global/full-attention layers). Long-context serving costs scale with attention over full context; global attention layers at large context (1M) become significant decoding overhead.

Change: NaiveAI released a 309B MoE (15.5B active, ~5% active ratio) with native 1M context using a hybrid that replaces global/full-attention with DSA in selected layers - explicitly no full-attention layers anywhere in the network. The model ships under MIT license with inference code (NaiveRT) and weights.

After: A concrete open-weight architecture example exists that attempts to achieve 1M context without any full-attention layers, potentially changing cost structure assumptions for long-context inference if behavior holds at scale. This is a COMPANY CLAIM until independently reproduced.

An archival table entirely covered in identical closed folders, with one reader passing its whole length, touching each folder once and carrying away only a fixed handful.
↔

Before ??? Change ??? After

🎓 For Explorer
  • Before: Long-context (1M) model architectures typically incorporate some form of global/full attention (or rely on ROPE extrapolation with full attention) creating quadratic/expensive attention patterns at very long contexts; cost/latency grows with context length for those layers. Many comparable open models use different attention designs.
  • Change: Released Naive-N0.5-Flash with 39 SWA + 9 DSA layers, no full-attention layers, native 1M context. MIT-licensed weights + inference code (NaiveRT).
  • After (claimed): Native 1M context with per-token decoding cost that does not grow with context length in SWA-dominant design; sparse attention reduces computation/memory access in DSA layers. Claimed AI-optimized inference up to 2000 tokens/s (per blog) with NaiveRT. These are COMPANY CLAIMS.
⚙

How it works

5.1 Base architecture

  • Derived from open-weight MiMo-V2.5 base (per acknowledgments). Uses MoE with 309B total, 15.5B active.
  • 48 layers total, grouped into eight 6-layer modules.
  • Hybrid attention composition: 39 Sliding-Window Attention (SWA) layers + 9 DeepSeek Sparse Attention (DSA) layers.
  • Layer layout (company specification): "A standard module contains five SWA layers followed by one DSA layer, with the first layer of the first module also replaced by DSA." So modules 1-8: (DSA, SWA, SWA, SWA, SWA, SWA?) wait - "first layer of the first module also replaced by DSA" means module 1 structure becomes DSA, SWA, SWA, SWA, SWA, SWA? (5 more SWA after DSA in that module? Or standard is 5 SWA + 1 DSA = 6; replacing first layer with DSA in first module gives DSA + 4 SWA + 1 DSA in first module?). Total DSA layers = 1 (first module first layer) + 8 modules × 1 DSA (last of each module) = 9. SWA = 48-9=39. Yes matches.

5.2 Attention mechanisms

  • SWA: window size 128 tokens. Per-token decoding cost does not grow with context length (local attention). Sink bias incorporated.
  • DSA (DeepSeek Sparse Attention): lightweight indexer scores full history; backbone computes attention only over selected subset (top-K). Full KV cache retained; indexer still scans full history. Selects top 2,048 tokens for backbone attention in DSA layers.
  • Hybrid: predominantly 5:1 SWA–DSA layout. Entire network remains local or sparse; no full-attention layers.
  • GQA4 (4 KV groups). DSA implementation uses GQA (replacing MLA variant). Indexer has 16 query heads.

5.3 Training

Multi-stage with native 1M context (3.25T tokens total):

  1. Indexer Warmup (50B): only DSA indexer trained, others frozen. Layers converting to DSA keep full attention during forward pass at 1M; SWA unchanged. KL divergence aligns indexer and backbone attention distributions.
  2. Sparse Attention Training (3T): switch to sparse attention, CPT with LM loss at fixed LR, native 1M context. Adapts to sparse selection/aggregation.
  3. Learning Rate Decay (200B): SFT-style with LR decay, still 1M context, sparse path.

5.4 Inference (NaiveRT)

Company claims AI-optimized runtime (NaiveRT): mega-kernel fusion, Programmatic Dependent Launch (PDL), speculative decoding. Claimed performance: 50 tokens/s per user (Standard), up to 2000 tokens/s (Ultrafast). Single-stream decode peak 2122 tokens/s on 8 GPUs (from blog case study). FP8 mixed-precision. Deterministic sampling (bitwise up to seed). Released as open-source (NaiveRT GitHub) per blog.

!

Why it matters

🎓 For Explorer
  • Architecture proof-of-concept (claimed): Demonstrates 1M native context without any full-attention layers in a 309B model. If the sparse+local hybrid preserves quality at 1M while avoiding full-attention cost growth, this is a meaningful architectural direction.
  • Cost structure: SWA keeps per-token decode cost independent of context length; DSA reduces attention compute/memory access even if indexer scans full history. Full KV retained. Potentially better long-context serving economics - this is INTERPRETATION pending independent validation.
  • Permissive licensing: MIT license for weights and inference code lowers barriers for research, customization, and deployment (FACT as stated).
  • Active parameter ratio: 15.5B active out of 309B (~4.85%) - could offer efficiency tradeoffs depending on memory/compute vs quality.
  • AI-centered R&D narrative: Released alongside case studies (NaiveRT optimization, AutoWM experiment) emphasizing AI models doing R&D work.
✦

What became possible?

🎓 For Explorer

If the claimed behavior holds (COMPANY CLAIM):

  • Serving very long contexts (1M) with more predictable per-token latency characteristics (dominated less by growing full-attention cost).
  • Exploration of sparse attention at scale in production-style models without full-attention fallback.
  • Reproducibility/experimentation due to MIT license + published architecture details + inference code availability.

Reality: These remain possibilities claimed by vendor until independently verified with quality/latency benchmarks at 1M context.

◎

Implications

Technical

  • Attention design space: Hybrid SWA+DSA with zero full-attention layers is a concrete datapoint. The 5:1 SWA:DSA ratio (39:9 layers) and specific DSA params (top-2048, GQA4, 16 indexer heads) are implementation choices.
  • Long-context fidelity: Sparse attention (top-2048 selected from full history per DSA layer) trades completeness for efficiency. Whether this preserves recall for "needles in haystack" across full 1M and complex multi-hop retrieval is a key verification question (UNVERIFIED).
  • Memory: Full KV cache retained despite sparse attention - so memory footprint still scales with context (KV storage), even if attention compute reduced. This is important distinction.
  • Inference engineering: NaiveRT emphasizes mega-kernel fusion + PDL to make fused sparse path fast; shows attention layer count reduced from 29 kernels to 1 in the DSA layer case study (COMPANY CLAIM from blog).
  • Training complexity: Multi-stage training (indexer warmup with KL to full-attention reference) needed to adapt from base architecture to sparse attention - non-trivial.
  • Model scale: 309B MoE with FP8 shards (~315GB weights) requires substantial GPU memory for inference.

Developer

  • Access: MIT-licensed weights on Hugging Face (NaiveAI/Naive-N0.5-Flash). Inference code available (NaiveRT). API planned with published pricing ($0.10/$0.40/$0.01 per M tokens input/output/cache read).
  • Local deployment: FP8-capable NVIDIA GPUs required; weights ~315GB + overhead. Not trivial for consumer hardware.
  • Tooling: Transformers example provided (trust_remote_code=True). Chat template included.
  • Context handling: Native 1M context enables very long documents/tool outputs - but quality at 1M with sparse attention needs validation.
  • Experimentation: Permissive license enables fine-tuning, distillation, architecture research.

Enterprise

  • Cost model (claimed): API pricing competitive relative to frontier models; internal deployment economics depend on utilization and hardware. Claimed efficiency via sparse attention - unverified independently.
  • Use cases: Positioned for coding and AI R&D. Enterprise applicability depends on quality parity with alternatives at relevant tasks.
  • IP/licensing: MIT license is business-friendly (no copyleft complications).
  • Operational complexity: Large model (309B) requires serious infra; sparse attention may need specific runtime support for optimal performance.

Strategic

  • Architecture trend: Another data point validating sparse+local hybrid attention for long context at scale. Challenges dominant assumption that full attention is required for quality long-context.
  • Open weights: Significant open-weight release (309B, MIT) in same class as other large open models - increases competitive pressure.
  • AI-driven development: Narrative of "building frontier AI with AI" (recursive self-improvement direction) is strategically interesting if substantiated.
  • Geographic/market: NaiveAI is a relatively new player; execution claims need independent validation.
⚠

Risks & limitations

Risks
  • Verification gap: No independent reproduction of 1M-context quality, retrieval fidelity, or latency claims. All core technical claims are COMPANY CLAIMS.
  • Benchmark cherry-picking: Evaluation figures in blog compare against specific models with specific setups (Claude Code harness, etc.). Methodology details partially specified but independent replication missing.
  • Sparse attention tradeoffs: Top-2048 token selection per DSA layer may miss critical long-range dependencies in adversarial cases; "needles in haystack" at 1M unverified independently.
  • Claims inflation: "up to 2000 tokens/s" is a peak/Ultrafast mode claim - realistic throughput varies by setup.
  • Compute requirements: Large model size limits accessibility; FP8 GPU requirements substantial.
Limitations
  • Source independence: Primary evidence is vendor-controlled (HF model card, company blog). Aggregator aiweekly.co mentioned in discovery as inadequate alone - we found better primary sources (HF + naive.ai). Still vendor-authored.
  • English coverage: Limited independent technical analysis found in English at time of research; corroboration sparse.
  • Temporal freshness: Released ~2026-09-27; ecosystem/testing still developing.
  • Context behavior: We cannot independently verify 1M native context performance from web sources alone.
?

Open questions

  1. Does Naive-N0.5-Flash maintain quality parity with full-attention models on long-context tasks (multi-hop retrieval, long document QA, codebase reasoning) at 1M tokens? Unanswered (no independent evidence).
  2. What is actual end-to-end latency/throughput at 1M context in realistic workloads (not just peak)? Unanswered.
  3. How does sparse attention (top-2048) behave on tasks requiring precise retrieval from arbitrary positions deep in context? Unanswered.
  4. Are there failure modes where DSA indexer misses relevant tokens? Unanswered.
  5. How does NaiveRT perform across different hardware configurations? Partially addressed in blog, not independently verified.
·

Related stories / ecosystem

  • Builds on MiMo-V2.5 (Xiaomi) - acknowledged. Uses DeepSeek Sparse Attention (DSA) concepts - acknowledged.
  • Part of trend toward sparse attention at scale (following DeepSeek's work on MLA/DSA ideas).
·

Sources used

See. All URLs are absolute HTTPS.

·

Lab work / verification notes

See. Verification focused on HF repo presence, file artifacts, and model card claims. No reproduction attempted.

·

Evidence classification summary

  • Confirmed (FACT): Model exists on HF (https://huggingface.co/NaiveAI/Naive-N0.5-Flash), license MIT, file structure (49 safetensors shards), architecture numbers as stated in model card (309B MoE, 15.5B active, 48 layers, 39 SWA+9 DSA, 1M context). These are directly observable from HF page.
  • Company claims requiring verification: Performance benchmarks, inference speeds (2000 tok/s), "no full-attention layers" quality preservation, 1M context behavior. Confidence remains Medium.
  • Independent verification: None found.
·

Date and time of research

Research conducted: 2026-10-04

·

Researcher / agent

Story researcher (deep research subagent), following project rules (sequential, evidence discipline, absolute URLs).

·

Final answer / status

Status: COMPLETE (artifact ready for validation). Validator python scripts/validate_story.py S36 must pass.

→

What should you do with this?

Circle 1
Circle 1

Factual core: NaiveAI released Naive-N0.5-Flash, an open-weight 309B MoE (15.5B active) with native 1M context, hybrid SWA+DSA attention, and no full-attention layers, under MIT license (weights + inference code). All observable specs confirmed from HF/model card.

Circle 2
Circle 2

Interpretation: This is a concrete architectural proof-of-concept claiming long-context capability without full-attention. Economic implications depend on sparse attention preserving quality - currently uncorroborated independently.

Circle 3
Circle 3

Strategic implications: Demonstrates feasibility of sparse+local hybrid at scale in an open-weight 309B model. If validated independently, it could challenge assumptions about full-attention necessity for 1M context. Narrative of AI-driven R&D adds strategic interest.

↗

What happens next

🎓 For Explorer

Likely: Independent researchers will test 1M-context retrieval, quality, and latency. Ecosystem (inference runtimes) will experiment with NaiveRT. Success depends on whether sparse top-2048 selection preserves long-range dependencies in practice.

·

Editorial

S36 is technically interesting as an architecture experiment - the "no full-attention layers" claim is bold and specific. However, performance/quality at 1M context remains vendor-claimed with no independent corroboration. We maintain evidence status as COMPANY CLAIM (Medium confidence) per project rules - no upgrade without independent evidence.

Research sources

Primary Sources (3)
Primary
NaiveAI Research blog post - "Naive-N0.5-Flash: Building Frontier AI with AI"- Evidence: Detailed architecture, training (3.25T tokens, 3 stages), hybrid SWA+DSA with no full-attention, NaiveRT case study (2122 tok/s peak), evaluation figures and methodology notes, citations. Published Sept 27, 2026 (per post). - Type: Primary (vendor) - Date accessed: 2026-10-04
Visit source ↗
Primary
NaiveAI Hugging Face repo files (tree)- Evidence: File list confirms weights (model-*.safetensors), config.json, configuration_naive_n05_flash.py, LICENSE, README, chat_template.jinja. Total ~618GB (per tree view), 49 weight shards. - Type: Primary (vendor) - Date accessed: 2026-10-04
Visit source ↗
Primary
NaiveAI Hugging Face model card - Naive-N0.5-Flash- Evidence: Model specs (309B MoE, 15.5B active, 48 layers, 39 SWA + 9 DSA layers, no full-attention layers, 1M native context), license (MIT), files (FP8 shards), quick start, acknowledgments. - Type: Primary (vendor) - Date accessed: 2026-10-04
Visit source ↗
Secondary Sources (3)
Secondary
MiMo-V2.5 (base model) - acknowledgedBase model acknowledged.
Visit source ↗
Secondary
DeepSeek Sparse Attention (DSA) - concept reference (acknowledged)Referenced conceptually in blog; not used as direct evidence for this specific model's performance.
Visit source ↗
Secondary
aiweekly.co alert (mentioned in discovery, inadequate as sole source)- Evidence: Alert coverage of release; aggregator summary. - Type: Secondary — Discovery flagged as inadequate alone; replaced with primary sources above.
Visit source ↗
Up next · story 37 of 40
A superhuman Stratego system, reported at a few thousand dollars of compute
All storiesNext story