Overfit runs local LLMs (Qwen, Llama, Phi, Gemma, Mistral, Bielik), RAG and agents — plus speech-to-text and text-to-speech — directly inside your process. It also serves an OpenAI-compatible API as a tiny Native-AOT binary or a ~34 MB container. No Python, no Ollama, no model server, no data egress.
A Polish local-agent preset loads Bielik from GGUF, indexes Polish documents, answers with RAG, calls C# tools and returns well-formed JSON — all in one ASP.NET process.
The Bielik preset is the most practical Polish demo: local Bielik GGUF, Polish documents, RAG, C# tool calling, business-decision JSON and metrics from one ASP.NET service.
cd Demo/LocalAgentAspNetDemo
.\download-bielik.cmd # Bielik GGUF -> C:\bielik
.\download-embedder.cmd # MiniLM embedder -> C:\minilm
.\run-bielik.cmd # launch-profile bielik
# Try:
POST /rag/query
POST /agent
POST /decision/refund
GET /metricsThe Local Agent demo exposes chat, RAG, tool calling, guaranteed JSON and metrics over HTTP.
Overfit is not a wrapper around a server. The runtime, retrieval, constrained JSON, tool dispatch and inference hot paths live in managed .NET.
Qwen2.5/Qwen3, Llama, Mistral, Phi-3.5/4, Gemma 2, Mixtral & MoE, Bielik and GPT-style models from GGUF, safetensors or ONNX — 0 B/token decode.
Whisper speech-to-text and text-to-speech with voice cloning — the full voice loop in one .NET process, on the CPU.
/v1/chat/completions (SSE), /v1/embeddings, /v1/audio/speech — a dependency-free server, Native-AOT into a ~34 MB container that idles at ~0% CPU. Point any OpenAI client at it.
Embeddings, built-in vector store and cosine search without an external vector DB.
Constrained decoding forces a valid tool-call envelope and dispatches straight to your C# delegate.
Structured output is valid by construction. Invalid tokens cannot be sampled under the active constraint.
Predictable CPU inference, explicit memory ownership, Native AOT-compatible paths and zero-allocation hot paths.
GGUF, K-quants, safetensors, ONNX, GPT-2 .bin, embeddings, LoRA, OCR and small deep networks.
LoRA stages, autograd training, Adam/SGD, LR schedules, CNNs, gradient checkpointing and data-parallel CPU training (a full 60k-image MNIST CNN trains in ~2 s).
Linear topology plus DAG support, skip connections, ResNet-style models and output parity checks vs PyTorch.
Overfit optimizes for pure-managed, in-process .NET deployment, low allocation pressure and predictable CPU behavior. It is not trying to beat llama.cpp or native GPU stacks on every raw throughput metric.
| Workload | Headline result | Allocation |
|---|---|---|
| Single inference Linear(784→10) | ~7.6× faster than ONNX Runtime | 0 B |
| Qwen2.5‑3B Q4_K_M decode | ~19 tok/s on CPU, up to ~24 tok/s with the opt-in repacked-GEMV mode; RAM footprint in llama.cpp range | ~1 B/token |
| Bielik-4.5B Q4_K_M preset | ~17 tok/s on CPU, −36% working set vs same-file llama.cpp | ~1 B/token |
| GPT-2 KV-cache decode | ~6.5× faster than naive O(N²) | 0 B/token |
Raw tokens/sec is not the main axis. llama.cpp / LLamaSharp can be faster. Overfit optimizes for pure C#, in-process deployment, Native AOT-friendly paths, low allocation pressure and no data egress.
Local model, document indexing, RAG, C# tools, guaranteed JSON and ASP.NET endpoints in your infrastructure.
Polish local agent over policies, support docs, contracts, claims and internal knowledge bases.
Move inference from Python, Ollama, ONNX Runtime or a model server into your .NET process.
Best first message: current stack, target deployment, model size, data constraints, whether Python/native binaries are allowed, and what the first demo must prove.
Direct email: devonbike@gmail.com