Privacy-First AI Learning in Your Own Homelab

Kevin Howell

2026-10-24

About Me

Principal Software Engineer at Red Hat

Raleigh, NC

@kahowell@mastodon.social

https://kahowell.net

Note

These are my own opinions and experiences; I’m not speaking for my employer.

Why Open-Source/Open-Weight

• Ethics/morals
• Transparency/learning
  • Log Probs
  • Structured Output
• Sovereignty
• Privacy

• You might not want to support Big AI.
• See how it works.
• You get to own the infrastructure. Change models when you want.
• You get more control over your data.

Hugging Face Incident

When we started the log analysis, we first used frontier models behind commercial APIs. This did not work.

These requests were blocked by the providers’ safety guardrails.

We ran the forensic analysis instead on zai-org/GLM-5.2, an open-weight model, on our own infrastructure.

Hugging Face - Security Incident July 2026

Note

OpenAI later admitted to compromising Hugging Face.

OpenAI later admitted to compromising Hugging Face. Hugging Face didn’t reveal which commercial provider they were using, but this is Big AI attacking someone, and then Big AI refusing to help analyze the attack.

Poll

• How many use AI through a subscription today?
• How many have used AI through APIs?

Goal: get those of you that haven’t explored AI through an API endpoint the tools to do so.

What is AI, Anyway?

• large language models are statistical models of one or more human languages, fancy autocomplete; autocomplete conversations

• computer vision: interpret images; e.g. self-driving cars

• audio: intepret audio

• multi-modal: LLM combined with image and/or audio capabilities

• many other subfields, but most energy is on LLMs right now.

Hardware

• Any recent-ish machine with a CPU
• GPUs
• APUs

• CPUs - fast to load but slow compute for inference; but no, really, viable, though expect to wait a bit.
• GPUs - slow to load but great compute for inference; any are helpful, even iGPUs
• APUs - fast to load & good compute for inference; e.g. Apple Unified memory, AMD Strix Halo

Lemonade Server

lemonade-server.ai

• Local AI Toolbox
• Servers
• AI Models

• from AMD
• servers: LLM, TTS, STT, image generation
• great starting point
• chat interface built-in

Inference

Inference is the process of using a trained LLM to generate human-like text from a given input prompt.

per Huggingface

Inference Servers

• llama.cpp (llama.app)
• vLLM (vllm.ai)

llama.cpp is a little simpler. vLLM has a lot of production features.

You can run vllm at scale with Kubernetes through llm-d. (Disclaimer: I work on a product that does this)

There are other notable inference servers, I’ve not explored them.

Models

Size	China	United States
XS	Qwen3.5 0.8B (~4GB)	Gemma 4 E2B (~4GB)
S	Qwen3.6 35B-A3B MTP (~24GB)	Gemma 4 26B-A4B (~18GB)
M	Qwen3.5 122B-A10B (~70GB)	gpt-oss-120b (text-only) (~66GB)

Note

• Estimates from unsloth.ai, for 4-bit quantization
• CPU: choose XS; GPU: choose XS or S; APU: prefer M if it fits
• All of these do vision, except for gpt-oss-120b
• Checkout Mistral models from France

• After you pick an inference server, choose a model
• I recommend not buying new hardware for this, with these ram prices
• CPU, GPU, APU, but generally, try it if it fits
• not going into distributed inference, but if you have more than one machine go for it.
• expect OOMKills or even system lock-ups if you get too close to limits

(V)RAM

• Which model, parameter count is “baked in”
• Context limit set at runtime
• Concurrent conversations
• Num cached conversations, if you have no cache, conversations have to be fully reprocessed with each new message. (imagine: restarting an episode after you pause: painful/wastful)

(V)RAM Estimation

	Formula	Example
Model	parameters * 0.5 bytes	35B * 0.5 = 17.5GiB
KV cache	32KiB * context size * slots	32KiB * 128K * 1 = 4GiB
Prompt cache	KV-cache slot size * cache size	4G * 2 = 8GiB
Engine overhead	2-4GiB	2GiB

Total: ~32GiB

Note

Cache-size per token varies wildly. This is “napkin math” (inaccurate). Deploying the model is the best way to find out.

• For each parameter, here’s a mental model to help reason about its contribution to RAM usage
• Not too far off, hopefully
• Calculators exist

Local Observability

• Langfuse (langfuse.com)
• MLflow (mlflow.org)

• Langfuse was purpose built for LLM observability, recently acquired by Clickhouse (January).
• MLflow was built for classic machine learning processes, but has steadily been growing support for genAI. They added general purpose trace support last year, but have had tracing features since 2024.

honorable mention: strace

note: colleague who uses strace to validate whether agent is lying about something it did.

Install, Configure, Invoke

# lemonade
export LEMONADE_RPM=lemonade-server-2026.42.1-fc44.x86_64.rpm
wget https://github.com/lemonade-sdk/lemonade/releases/download/v2026.42.1/$LEMONADE_RPM
sudo dnf install -y ./$LEMONADE_RPM
sudo systemctl enable --now lemond

# local mlflow, telemetry stays on your machine
uvx mlflow server

# download a model
lemonade pull Gemma-4-E2B-it-GGUF

# connect lemonade to mlflow
lemonade config set telemetry.enabled=true \
  telemetry.otlp.endpoint=http://localhost:5000/v1/traces \
  telemetry.otlp.protocol=http/protobuf \
  telemetry.otlp.semantics='["otel_genai"]' \
  telemetry.otlp.headers='{"x-mlflow-experiment-id":"0"}'

# chat with a model
curl http://localhost:13305/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Gemma-4-E2B-it-GGUF",
  "messages": [{"role": "user", "content": "Repeat after me: 1"}],
  "temperature": 0.7
}'

Tip

Another approach that works well with MLflow is to route traffic through its AI gateway feature.

• I choose mlflow here for simplicity. LangFlow requires API key set up.
• Experiment 0 is the default experiment.
• Line here is for another terminal.
• Many options on the API, explore temperature, log probs, structured output.

Example Trace

• You get a record of the performance of a particular LLM invocation

• some interesting details that you might have to dig for otherwise, like the system prompt.

• if you’re paying attention, this trace is from the same prompt in OpenCode, but it’s different there because there’s a long system prompt too.

• NOTE: traces contain sensitive info, like chat logs.

• When you use a commerical service for API, this is essentially what you give them.

Agent Frameworks

Apply LLMs to do something useful; aka. harnesses

Chat UIs

• Open WebUI* (openwebui.com)
• LibreChat (librechat.ai)

Warning

Open WebUI’s license is not an OSI-approved “open source” license, because of an additional “branding protection clause”.

Attribution clause is a little weird, but for individuals seems harmless.

Both are valid choices for a general chat UI.

Coding Agents

• zed.dev
• opencode (opencode.ai)

• zed.dev is kinda like vscode, full IDE
• opencode is kinda like claude code, good for terminal use
• connect LLMs to tools for coding (compilation, editing files)

Chat Assistants

• OpenClaw (openclaw.ai)
• Hermes (hermes-agent.nousresearch.com)

Warning

Dangerous! Inherent security risks. Sandboxing is a must.

• These tie together LLMs, chat platforms (like Slack, Matrix, Mattermost) and connect LLMs to tools.
• you want to sandbox these well, unless you’re a frontier AI lab, who seem to bear no responsibility for things that would land us regular folks in jail.

Sandboxing

• OpenShell (github.com/NVIDIA/OpenShell)

This is the only way I’ve ever felt remotely safe experimenting with OpenClaw/Hermes.

OpenShell - Local Inference Setup

openshell provider create \
    --type openai \
    --name lemonade \
    --credential OPENAI_API_KEY='none' \
    --config OPENAI_BASE_URL=http://localhost:13305/v1

openshell inference set \
    --provider lemonade \
    --model Qwen3.6-35B-A3B-MTP-GGUF \
    --timeout 3600

Note

https://inference.local in any sandbox now routes to Qwen3.6 35B-A3B MTP running via lemonade.

• First step registers an OpenAI-compatible inference endpoint.
• Second step sets it as the inference provider for sandboxes.

OpenShell - OpenCode Config

export OPENCODE_CONFIG_CONTENT='{
  "$schema": "https://opencode.ai/config.json",
  "model": "local/local",
  "provider": {
    "local": {
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "https://inference.local/v1",
        "apiKey": "none"
      },
      "models": {
        "local": {
          "name": "local",
          "modelID": "local"
        }
      }
    }
  }
}'

OpenShell - OpenCode Sandbox

openshell sandbox create \
    --provider lemonade -- \
    bash -c \
    "OPENCODE_CONFIG_CONTENT='$OPENCODE_CONFIG_CONTENT' opencode"

• If you followed all that, you’ll get sandboxed opencode running on local inference.

• NVIDIA’s official tutorials use free models on a openrouter (not private).

• Configuration varies by harness, but the pattern is the same.

Agent Stack

• Harness runs in an environment, and defines policies and interactions, incl. requests for permissions.
• Environment defines tools available for the agent to work with.
• Inference is just fancy autocomplete

Image Generation

Diffusion models construct a “new” image by refining noise into something recognizable by humans and the model, according to a prompt.

You can think of these as reproductions of concepts or images from the training data, or in some cases, straight reproductions of training data

Image Generation - Servers

• stablediffusion.cpp (github.com/leejet/stable-diffusion.cpp)
• ComfyUI (comfy.org)

stablediffusioncpp is an API/CLI. Lemonade server integrates with it.

Comfy is focused on a node-based workflow.

Resources

• AI Engineering - Chip Huyen
• Hands-On Large Language Models - Alammar & Grootendorst
• Maarten Grootendorst’s blog
• Hugging Face Learn
• Unsloth Docs

Check your library and employer/school, professional organizations for O’Reilly Learning access.

Maarten’s blog has wonderful diagrams of LLM architecture.

Q&A

Thanks for coming to my talk!

@kahowell@mastodon.social

https://kahowell.net