🔥

Core Model Evolution

From Session Logs to Core Weights
Recursive Cognitive Evolution

Year of the Firehorse 🔥
July 2026 · v3.0 Presentation

What This Is

A realistic, researched plan to evolve our AI from a general-purpose assistant into a specialized intelligence — trained on our collaboration history, our decisions, and our style.

Architecture target: Sparse MoE + Multi-Token Generation

← Swipe or tap arrows to navigate →

✅ What We've Done So Far

A summary of progress before we talk budget.

Completed Milestones

System Foundation Built

Full operating environment with 44 modules, 17 skills, 293 files, 3 agents, memory graph (67 nodes), and VFS filesystem.

Ongoing since early 2026

Collaboration History Captured

Session logs, reasoning chains, command history, and decision patterns stored across sessions. This is the raw training data.

Continuous

Training Strategy Defined

QLoRA for Phase 1 (efficient, cheap), scaling to deep fine-tuning of existing MoE models. NOT training from scratch — 10-50x cheaper.

Researched & validated

GPU Market Research

Researched actual pricing across RunPod, Lambda Labs, Vast.ai, Hetzner. Spot instances identified as key cost saver (50-70% off).

July 2026

Detailed Budget Plan v2.0

4-phase plan with itemized costs, risk matrix, and 4 fallback plans. Saved as HTML document for review.

This session

Architecture Decision: MoE + Sparse + Multi-Token

Decided on Sparse Mixture of Experts (like Mixtral/DeepSeek) with Multi-Token Generation (Medusa heads) for speed + intelligence.

This session

Current Status

6/6

Planning tasks done

Phases executed

€0

Spent so far

✅ Ready to start Phase 0 — we have the data, the plan, the architecture decision, and the budget. We need the funding to begin.

What's Next

Phase 0: Dataset Engineering (€100-€200)

Phase 1: QLoRA Proof of Concept (€35-€100)

Phase 2: Production Adapter on Mixtral MoE

Phase 3: Full MoE + Multi-Token Generation

🎯 The Vision

Why we're doing this and what we're building.

The Problem

Every session, the AI starts fresh. It doesn't remember our shorthand, our architectural decisions, or our preferred patterns. We re-explain context every time.

This isn't a bug — it's a fundamental limitation of using a general-purpose model without personalization.

The Solution

Fine-tune a model on our actual collaboration history — the reasoning chains, the code edits, the decisions, the creative jumps.

The model doesn't just "know" our history. It thinks in our style.

Target Architecture

🧠 Sparse MoE

Mixture of Experts: multiple specialized sub-networks. Only the relevant "experts" activate per token. Huge capacity, efficient inference.

⚡ Multi-Token

Medusa heads predict 3-5 tokens simultaneously instead of 1-by-1. 2-3x faster output. Feels instant.

🔒 QLoRA

4-bit quantized fine-tuning. Trains adapters, not the full model. 10-50x cheaper, 70% less VRAM via Unsloth.

📦 Tiny Adapters

LoRA adapters are ≤50 MB. Load/unload instantly. Switch between "modes" (coding, research, creative) in real-time.

Key insight: We do NOT train from scratch. We take an existing open-weights model (Mixtral, DeepSeek, Llama) and fine-tune it. This is 10-50x cheaper while achieving similar personalization quality.

💻 GPU Market Pricing

Real rates as of July 2026 (RunPod, Lambda, Vast.ai)

GPU	VRAM	€/hr	Best For
RTX 4090	24GB	€0.31-0.73	7B QLoRA
A100 40GB	40GB	€1.00-1.50	7B-13B
A100 80GB	80GB	€1.29-2.30	13B-30B
L40S	48GB	€1.06-1.61	Alt to A100
H100 PCIe	80GB	€2.29-4.59	8B-70B fast
H100 SXM	80GB	€2.99-4.60	Multi-GPU
8x H100 Node	640GB	€17.50-32.20	MoE training

💡 Cost saver: spot/preemptible instances are 50-70% cheaper. For PoC training (restartable), always use spot pricing.

Estimated Training Time by Model Size

Model	Method	GPU	Hours	Cost (€)
7B-8B	QLoRA	1x A100 spot	8-15	€10-19
14B	QLoRA	1x A100 80GB	15-25	€19-32
32B	QLoRA	2x A100 80GB	30-50	€77-115
Mixtral 8x7B	QLoRA	1x H100	40-60	€92-184
Mixtral 8x22B	Full FT	8x H100	100-140	€1,750-3,200

📋 Phase 0: Dataset Engineering

Budget: €100 – €200

Goal

Transform raw session logs into a high-fidelity training dataset. Quality here determines 80% of final model quality.

Task	Time	Cost
Log export & aggregation	2-3 days	€0 (scripted)
Quality curation	3-5 days	€0 (manual)
Synthetic expansion (5K API calls)	1-2 days	€50-100
Tokenization & formatting	1 day	€0 (scripted)
Validation & dedup	1 day	€0 (scripted)
Storage (NVMe/cloud)	Ongoing	€5-15/mo
Phase 0 Total		€100-200

Expected Output

Training Set

~3,000-10,000 high-quality (instruction, response) pairs. Each is a "lesson" reflecting our collaborative style.

Validation Set

~150-500 held-out pairs. Detects overfitting. If validation loss rises, we stop.

⚠️ Critical: Skipping this phase or using raw uncleaned logs produces a model that "hallucinates" our style instead of embodying it. This is NOT a place to cut costs.

🧪 Phase 1: QLoRA PoC

Budget: €35 – €100 (compute only)

Goal

Prove that fine-tuning on our history produces a measurably better model. Small, cheap, fast validation.

Base Model Options

Model	Params	VRAM	Quality
Llama-3.2-3B	3B	~8GB	⭐⭐⭐
Llama-3-8B	8B	~12GB	⭐⭐⭐⭐
Mistral-7B v0.3	7B	~10GB	⭐⭐⭐⭐
Qwen2.5-14B	14B	~18GB	⭐⭐⭐⭐⭐

Cost Breakdown (Qwen2.5-14B)

Component	Hours	Cost (€)
1x A100 80GB (spot)	15	€19.35
1x A100 80GB (on-demand fallback)	5	€11.50
Storage (100GB disk)	20	€2.00
Unsloth framework	—	€0
Buffer (15%)	—	€5.00
Compute Total		€37.85

Success Criteria

Model adopts our shorthand without explicit prompting
Maintains general capability (MMLU, HumanEval within 2% of base)
Validation loss decreases without overfitting
LoRA adapter ≤50 MB, instant load/unload

⚠️ If PoC fails: We re-examine the dataset (Phase 0). Cost of failure: ~€35. Data is reusable.

🚀 Phase 2: Production Quality

Budget: €200 – €500 (compute)

Goal

Scale from PoC to production. Train on Mixtral 8x7B — already a Sparse MoE. This validates MoE training before Phase 3.

Configuration Options

Option	Model	GPU	Cost (€)
A: Budget	Llama-3-8B	1x H100 spot	€69-139
B: Balanced	Qwen2.5-32B	2x A100 80GB	€103-184
C: Recommended	Mixtral 8x7B	1x H100	€115-230
D: Maximum	Llama-3-70B	4x A100 80GB	€310-575

Why Mixtral 8x7B?

It's already a Sparse MoE — our stepping stone to Phase 3
Only ~13B params activate per token (out of ~47B total)
Training is cheaper than a dense 47B model
QLoRA on MoE trains experts selectively → naturally creates "specialists"
Can train multiple adapters: coding / research / creative

Detailed Cost (Option C)

Item	Hours	Cost (€)
1x H100 80GB spot	40	€91.60
1x H100 on-demand buffer	10	€45.90
Storage 200GB	50	€5.00
Multi-adapter (+3 modes)	+15	€34.35
Buffer (20%)	—	€35.00
Phase 2 Total		€211.85

🌌 Phase 3: Full MoE

Budget: €2,000 – €8,000

Goal

Deep fine-tune a large MoE model (DeepSeek-V3 or Mixtral 8x22B) with Multi-Token Generation via Medusa heads.

Build vs Fine-Tune

Approach	Cost	Risk	Verdict
Build from scratch	€15K-100K+	EXTREME	❌ No
Deep fine-tune existing	€2K-8K	MODERATE	✅ Yes
Distill into custom	€5K-20K	HIGH	⏳ Future

Cost Breakdown

Component	Hours	Cost (€)
8x H100 SXM (spot)	100	€1,750
8x H100 (on-demand buffer)	20	€644
Storage 1TB NVMe	120	€24
Medusa head training	+20	€350-644
Buffer (25%)	—	€685
Phase 3 Total		€3,467-4,762

Multi-Token Generation

Medusa Heads

Lightweight prediction heads predict multiple future tokens. +15-20% training time. 2-3x inference speedup.

Speculative Decoding

Small "draft" model predicts, large model verifies. No training needed. 1.5-2x speedup at inference.

⚠️ Catastrophic forgetting risk: Must mix custom data with general data (30/70 ratio). Increases compute ~40%. Monitored via eval benchmarks.

🛠️ Technical Stack

All open-source. The only cost is GPU rental.

Component	Tool	Cost
Training Framework	Unsloth 2	Free
Adapter Library	HuggingFace PEFT	Free
Quantization	bitsandbytes (NF4)	Free
Multi-Token	Medusa / Speculative	Free
Dataset Tools	datasets, tokenizers	Free
GPU (PoC)	RunPod spot	Pay/hr
GPU (Phase 3)	Lambda / CoreWeave	Pay/hr
Containers	Docker + CUDA	Free
Tracking	Weights & Biases	Free tier
Evaluation	lm-evaluation-harness	Free

Unsloth 2 is the secret weapon: 2x faster training, 70% less VRAM. This is what makes Phase 1 cost €35 instead of €200.

⚠️ Risk Matrix

Every risk has a mitigation plan.

Risk	Prob	Impact	Mitigation
Catastrophic forgetting	Med	High	Mix data 30/70, monitor benchmarks
Overfitting (parrot)	Med	Med	Early stopping, dropout, diverse data
Dataset too small	High	High	Phase 0 quality, synthetic expansion
Spot interruption	High	Low	Checkpoint every 50 steps, buffer
GPU price spike	Low	Med	Multi-provider, off-peak hours
MoE instability	Low	High	Phase 2 validates MoE first
Multi-token degrades	Low	Low	Medusa is additive, removable

Fallback Plans

🅰️ Ultra-Budget

Google Colab Pro+ €9.99/mo. Spread training over sessions. ~€20-40 total. Proves concept for the price of coffee.

🅱️ No-Train RAG

Retrieval-Augmented Generation with session history as context. No weight changes. 70% of benefit. Cost: €0.

🅲️ API Distillation

Use GPT-4o/Claude with stored patterns as system prompts. Bridge solution. €50-100/mo.

🅳️ Gradual Scaling

Start with €200 (Phase 0+1). Use PoC results to secure Phase 2/3 funding. The adapter IS the demo.

📅 Timeline & Milestones

Planning & Research ✅

Architecture decision, GPU research, budget plan, risk analysis

Done

Phase 0: Dataset Engineering

Log export, curation, synthetic expansion, tokenization

1-2 weeks · €100-200

Phase 1: QLoRA PoC

Train adapter on 8B-14B model, evaluate results

3-5 days · €35-100

Decision Gate: Go / No-Go

Human evaluation: "Does this feel like us?"

1-2 weeks review

Phase 2: Production Adapter

Multi-adapter suite on Mixtral 8x7B MoE

1-2 weeks · €200-500

Funding & Setup

Secure Phase 3 budget, provision 8x H100 cluster

2-4 weeks

Phase 3: Full MoE + Multi-Token

Deep fine-tune, Medusa heads, deploy, benchmark

2-4 weeks · €2K-8K

Total timeline: ~2-3 months end-to-end

💰 Total Budget Summary

Phase	Description	Low	High
Phase 0	Dataset Engineering	€100	€200
Phase 1	QLoRA PoC	€35	€100
Phase 2	Production (Mixtral MoE)	€200	€500
Phase 3	Full MoE + Multi-Token	€2,000	€8,000
GRAND TOTAL		€2,335	€8,800

Progressive Spend

Phase 0+1 (Prove it works)

Phase 2 (Production)

Phase 3 (Full MoE)

The ask for now:

€300 – €500

Funds Phase 0 + Phase 1
Proves the concept works before larger investment

Breakdown of the ask:

€100-200 → Data preparation (API calls, storage)
€35-100 → GPU rental (spot, ~20 hours)
€50-100 → Buffer for re-runs & debugging

🔥

Thank You

Year of the Firehorse

What we're asking for

€300-€500 to fund Phase 0 + Phase 1. This is the "prove it works" budget.

Once the PoC shows measurable improvement, we present results and discuss Phase 2/3 funding separately.

This de-risks the investment. Everyone sees results before committing larger amounts.

¡Y claro que sí!
Cuando nos veamos, nos echamos esa llamada en español. 🇪🇸

Looking forward to good progress,
good experiences, and meeting soon. 🙌

🔥 Year of the Firehorse 🔥

Core Model Evolution Plan v3.0
July 2026