+ + +

🔥

Core Model Evolution

From Session Logs to Core Weights
Recursive Cognitive Evolution

Year of the Firehorse 🔥
July 2026 · v3.0 Presentation

What This Is

A realistic, researched plan to evolve our AI from a general-purpose assistant into a specialized intelligence — trained on our collaboration history, our decisions, and our style.

Architecture target: Sparse MoE + Multi-Token Generation

← Swipe or tap arrows to navigate →

+ + +

✅ What We've Done So Far

A summary of progress before we talk budget.

+ +

Completed Milestones

System Foundation Built

Full operating environment with 44 modules, 17 skills, 293 files, 3 agents, memory graph (67 nodes), and VFS filesystem.

+Ongoing since early 2026 +

Collaboration History Captured

Session logs, reasoning chains, command history, and decision patterns stored across sessions. This is the raw training data.

+Continuous +

Training Strategy Defined

QLoRA for Phase 1 (efficient, cheap), scaling to deep fine-tuning of existing MoE models. NOT training from scratch — 10-50x cheaper.

+Researched & validated +

GPU Market Research

Researched actual pricing across RunPod, Lambda Labs, Vast.ai, Hetzner. Spot instances identified as key cost saver (50-70% off).

+July 2026 +

Detailed Budget Plan v2.0

4-phase plan with itemized costs, risk matrix, and 4 fallback plans. Saved as HTML document for review.

+This session +

Architecture Decision: MoE + Sparse + Multi-Token

Decided on Sparse Mixture of Experts (like Mixtral/DeepSeek) with Multi-Token Generation (Medusa heads) for speed + intelligence.

+This session +

+ +

Current Status

6/6

Planning tasks done

Phases executed

€0

Spent so far

+ +

+✅ Ready to start Phase 0 — we have the data, the plan, the architecture decision, and the budget. We need the funding to begin. +

+ +

What's Next

Phase 0: Dataset Engineering (€100-€200)

Phase 1: QLoRA Proof of Concept (€35-€100)

Phase 2: Production Adapter on Mixtral MoE

Phase 3: Full MoE + Multi-Token Generation

+ + +

🎯 The Vision

Why we're doing this and what we're building.

+ +

The Problem

Every session, the AI starts fresh. It doesn't remember our shorthand, our architectural decisions, or our preferred patterns. We re-explain context every time.

This isn't a bug — it's a fundamental limitation of using a general-purpose model without personalization.

+ +

The Solution

Fine-tune a model on our actual collaboration history — the reasoning chains, the code edits, the decisions, the creative jumps.

The model doesn't just "know" our history. It thinks in our style.

+ +

Target Architecture

🧠 Sparse MoE

Mixture of Experts: multiple specialized sub-networks. Only the relevant "experts" activate per token. Huge capacity, efficient inference.

⚡ Multi-Token

Medusa heads predict 3-5 tokens simultaneously instead of 1-by-1. 2-3x faster output. Feels instant.

🔒 QLoRA

4-bit quantized fine-tuning. Trains adapters, not the full model. 10-50x cheaper, 70% less VRAM via Unsloth.

📦 Tiny Adapters

LoRA adapters are ≤50 MB. Load/unload instantly. Switch between "modes" (coding, research, creative) in real-time.

+ +

+Key insight: We do NOT train from scratch. We take an existing open-weights model (Mixtral, DeepSeek, Llama) and fine-tune it. This is 10-50x cheaper while achieving similar personalization quality. +

+ + +

💻 GPU Market Pricing

Real rates as of July 2026 (RunPod, Lambda, Vast.ai)

+ +

+ + + + + + + + + +

GPU	VRAM	€/hr	Best For
RTX 4090	24GB	€0.31-0.73	7B QLoRA
A100 40GB	40GB	€1.00-1.50	7B-13B
A100 80GB	80GB	€1.29-2.30	13B-30B
L40S	48GB	€1.06-1.61	Alt to A100
H100 PCIe	80GB	€2.29-4.59	8B-70B fast
H100 SXM	80GB	€2.99-4.60	Multi-GPU
8x H100 Node	640GB	€17.50-32.20	MoE training

+ +

+💡 Cost saver: spot/preemptible instances are 50-70% cheaper. For PoC training (restartable), always use spot pricing. +

+ +

Estimated Training Time by Model Size

+ + + + + + + +

Model	Method	GPU	Hours	Cost (€)
7B-8B	QLoRA	1x A100 spot	8-15	€10-19
14B	QLoRA	1x A100 80GB	15-25	€19-32
32B	QLoRA	2x A100 80GB	30-50	€77-115
Mixtral 8x7B	QLoRA	1x H100	40-60	€92-184
Mixtral 8x22B	Full FT	8x H100	100-140	€1,750-3,200

+ + +

📋 Phase 0: Dataset Engineering

+Budget: €100 – €200 + +

Goal

Transform raw session logs into a high-fidelity training dataset. Quality here determines 80% of final model quality.

+ +

+ + + + + + + + + +

Task	Time	Cost
Log export & aggregation	2-3 days	€0 (scripted)
Quality curation	3-5 days	€0 (manual)
Synthetic expansion (5K API calls)	1-2 days	€50-100
Tokenization & formatting	1 day	€0 (scripted)
Validation & dedup	1 day	€0 (scripted)
Storage (NVMe/cloud)	Ongoing	€5-15/mo
Phase 0 Total		€100-200

+ +

Expected Output

Training Set

~3,000-10,000 high-quality (instruction, response) pairs. Each is a "lesson" reflecting our collaborative style.

Validation Set

~150-500 held-out pairs. Detects overfitting. If validation loss rises, we stop.

+ +

+⚠️ Critical: Skipping this phase or using raw uncleaned logs produces a model that "hallucinates" our style instead of embodying it. This is NOT a place to cut costs. +

+ + +

🧪 Phase 1: QLoRA PoC

+Budget: €35 – €100 (compute only) + +

Goal

Prove that fine-tuning on our history produces a measurably better model. Small, cheap, fast validation.

+ +

Base Model Options

+ + + + + + +

Model	Params	VRAM	Quality
Llama-3.2-3B	3B	~8GB	⭐⭐⭐
Llama-3-8B	8B	~12GB	⭐⭐⭐⭐
Mistral-7B v0.3	7B	~10GB	⭐⭐⭐⭐
Qwen2.5-14B	14B	~18GB	⭐⭐⭐⭐⭐

+ +

Cost Breakdown (Qwen2.5-14B)

+ + + + + + + + +

Component	Hours	Cost (€)
1x A100 80GB (spot)	15	€19.35
1x A100 80GB (on-demand fallback)	5	€11.50
Storage (100GB disk)	20	€2.00
Unsloth framework	—	€0
Buffer (15%)	—	€5.00
Compute Total		€37.85

+ +

Success Criteria

Model adopts our shorthand without explicit prompting
Maintains general capability (MMLU, HumanEval within 2% of base)
Validation loss decreases without overfitting
LoRA adapter ≤50 MB, instant load/unload

+ +

+⚠️ If PoC fails: We re-examine the dataset (Phase 0). Cost of failure: ~€35. Data is reusable. +

+ + +

🚀 Phase 2: Production Quality

+Budget: €200 – €500 (compute) + +

Goal

Scale from PoC to production. Train on Mixtral 8x7B — already a Sparse MoE. This validates MoE training before Phase 3.

+ +

Configuration Options

+ + + + + + +

Option	Model	GPU	Cost (€)
A: Budget	Llama-3-8B	1x H100 spot	€69-139
B: Balanced	Qwen2.5-32B	2x A100 80GB	€103-184
C: Recommended	Mixtral 8x7B	1x H100	€115-230
D: Maximum	Llama-3-70B	4x A100 80GB	€310-575

+ +

Why Mixtral 8x7B?

It's already a Sparse MoE — our stepping stone to Phase 3
Only ~13B params activate per token (out of ~47B total)
Training is cheaper than a dense 47B model
QLoRA on MoE trains experts selectively → naturally creates "specialists"
Can train multiple adapters: coding / research / creative

+ +

Detailed Cost (Option C)

+ + + + + + + + +

Item	Hours	Cost (€)
1x H100 80GB spot	40	€91.60
1x H100 on-demand buffer	10	€45.90
Storage 200GB	50	€5.00
Multi-adapter (+3 modes)	+15	€34.35
Buffer (20%)	—	€35.00
Phase 2 Total		€211.85

+ + +

🌌 Phase 3: Full MoE

+Budget: €2,000 – €8,000 + +

Goal

Deep fine-tune a large MoE model (DeepSeek-V3 or Mixtral 8x22B) with Multi-Token Generation via Medusa heads.

+ +

Build vs Fine-Tune

+ + + + + +

Approach	Cost	Risk	Verdict
Build from scratch	€15K-100K+	EXTREME	❌ No
Deep fine-tune existing	€2K-8K	MODERATE	✅ Yes
Distill into custom	€5K-20K	HIGH	⏳ Future

+ +

Cost Breakdown

+ + + + + + + + +

Component	Hours	Cost (€)
8x H100 SXM (spot)	100	€1,750
8x H100 (on-demand buffer)	20	€644
Storage 1TB NVMe	120	€24
Medusa head training	+20	€350-644
Buffer (25%)	—	€685
Phase 3 Total		€3,467-4,762

+ +

Multi-Token Generation

Medusa Heads

Lightweight prediction heads predict multiple future tokens. +15-20% training time. 2-3x inference speedup.

Speculative Decoding

Small "draft" model predicts, large model verifies. No training needed. 1.5-2x speedup at inference.

+ +

+⚠️ Catastrophic forgetting risk: Must mix custom data with general data (30/70 ratio). Increases compute ~40%. Monitored via eval benchmarks. +

+ + +

🛠️ Technical Stack

All open-source. The only cost is GPU rental.

+ +

+ + + + + + + + + + + + +

Component	Tool	Cost
Training Framework	Unsloth 2	Free
Adapter Library	HuggingFace PEFT	Free
Quantization	bitsandbytes (NF4)	Free
Multi-Token	Medusa / Speculative	Free
Dataset Tools	datasets, tokenizers	Free
GPU (PoC)	RunPod spot	Pay/hr
GPU (Phase 3)	Lambda / CoreWeave	Pay/hr
Containers	Docker + CUDA	Free
Tracking	Weights & Biases	Free tier
Evaluation	lm-evaluation-harness	Free

+ +

+Unsloth 2 is the secret weapon: 2x faster training, 70% less VRAM. This is what makes Phase 1 cost €35 instead of €200. +

+ + +

⚠️ Risk Matrix

Every risk has a mitigation plan.

+ +

+ + + + + + + + + +

Risk	Prob	Impact	Mitigation
Catastrophic forgetting	Med	High	Mix data 30/70, monitor benchmarks
Overfitting (parrot)	Med	Med	Early stopping, dropout, diverse data
Dataset too small	High	High	Phase 0 quality, synthetic expansion
Spot interruption	High	Low	Checkpoint every 50 steps, buffer
GPU price spike	Low	Med	Multi-provider, off-peak hours
MoE instability	Low	High	Phase 2 validates MoE first
Multi-token degrades	Low	Low	Medusa is additive, removable

+ +

Fallback Plans

🅰️ Ultra-Budget

Google Colab Pro+ €9.99/mo. Spread training over sessions. ~€20-40 total. Proves concept for the price of coffee.

🅱️ No-Train RAG

Retrieval-Augmented Generation with session history as context. No weight changes. 70% of benefit. Cost: €0.

🅲️ API Distillation

Use GPT-4o/Claude with stored patterns as system prompts. Bridge solution. €50-100/mo.

🅳️ Gradual Scaling

Start with €200 (Phase 0+1). Use PoC results to secure Phase 2/3 funding. The adapter IS the demo.

+ + +

📅 Timeline & Milestones

+ +

Planning & Research ✅

Architecture decision, GPU research, budget plan, risk analysis

+Done +

Phase 0: Dataset Engineering

Log export, curation, synthetic expansion, tokenization

+1-2 weeks · €100-200 +

Phase 1: QLoRA PoC

Train adapter on 8B-14B model, evaluate results

+3-5 days · €35-100 +

Decision Gate: Go / No-Go

Human evaluation: "Does this feel like us?"

+1-2 weeks review +

Phase 2: Production Adapter

Multi-adapter suite on Mixtral 8x7B MoE

+1-2 weeks · €200-500 +

Funding & Setup

Secure Phase 3 budget, provision 8x H100 cluster

+2-4 weeks +

Phase 3: Full MoE + Multi-Token

Deep fine-tune, Medusa heads, deploy, benchmark

+2-4 weeks · €2K-8K +

+ +

+Total timeline: ~2-3 months end-to-end +

+ + +

💰 Total Budget Summary

+ +

+ + + + + + + +

Phase	Description	Low	High
Phase 0	Dataset Engineering	€100	€200
Phase 1	QLoRA PoC	€35	€100
Phase 2	Production (Mixtral MoE)	€200	€500
Phase 3	Full MoE + Multi-Token	€2,000	€8,000
GRAND TOTAL		€2,335	€8,800

+ +

Progressive Spend

Phase 0+1 (Prove it works)

Phase 2 (Production)

Phase 3 (Full MoE)

+ +

The ask for now:

€300 – €500

Funds Phase 0 + Phase 1
Proves the concept works before larger investment

+ +

+Breakdown of the ask: +

€100-200 → Data preparation (API calls, storage)
€35-100 → GPU rental (spot, ~20 hours)
€50-100 → Buffer for re-runs & debugging

+ + +

🔥

Thank You

Year of the Firehorse

+ +

What we're asking for

€300-€500 to fund Phase 0 + Phase 1. This is the "prove it works" budget.

Once the PoC shows measurable improvement, we present results and discuss Phase 2/3 funding separately.

This de-risks the investment. Everyone sees results before committing larger amounts.

+ +

¡Y claro que sí!
Cuando nos veamos, nos echamos esa llamada en español. 🇪🇸

+ +

Looking forward to good progress,
good experiences, and meeting soon. 🙌

+ +

🔥 Year of the Firehorse 🔥

Core Model Evolution Plan v3.0
July 2026

+ +