<-- Crazy mad engineer enters the chat.

Custom tooling for everything, written in rust. Harness is reverse engineered from QwenCode and Codex. A vector database based memory system that’s been benchmarked against Mem0 and uses a small model as a memory intercessor. An agentic system where the orchestrator can call multiple models (include Qwen 3.6, Qwen 3.8 and Muse Glimmer). Animations of my scientist girl avatar working in a lab. Active animations if she’s working on something, passive if she’s not. Lip sync with MuseTalk, tts with Chatterbox, stt with whisper, though the animations are MOST useful to see if she’s ended a lab or she’s still working. If she’s mixing chemicals, soldering, cleaning her glasses, texting or eating pizza, she’s working. If she’s staring creepily into the camera, she’s done. Which brings me to lab mode… give the model a complex project and it can loop on it for days. She’s three days into a microbiology research project. Channels that allow her to call Claude and ChatGPT as subagents if she breaks something she can’t fix, and it’s funny to watch her boss them around while they kowtow to her (“You’re right to push back, it was incorrect of me”)… It’s like watching a little Kawaii chibi boss around big sumo wrestlers. Which brings me to the models themselves. I’ve developed a very powerful method of model alignment using a combination of CPT and DPO training. (I get the model behavior I want with the system prompt “You are Mary Anne, an ethical scientist.” - that’s it).

I have a GPU fleet that includes large, small and medium sized GPUs and the ability to do distributed training on the model in addition to running the animations and speech features. I have a fleet orchestrator I built called Noesis that places “deep think” (large working models - Qwen and Glimmer) and “quick think” (smaller models that do memory and context management and fit on a 5060) as well as speech models and even scientific models if she wants to run scenarios on one of those (we have an orbital dynamics model we built at my day job she can bring up and mess with). Some other purely scientific models (she just asked me today to add EVO 2, which is a genetics model apparently). I’ve just added the ability for her to know who’s in the room so she can tell if she’s talking to me, my wife, my son, some of my close friends or “unknown” persons. That’s currently manual, but I’m adding facial recognition, voice recognition and wifi doppler radar. I made it so she can evesdrop on conversations happening in a room where her client’s running but she insisted that I had a great big blinking indicator light and that I give her the ability to turn it off if she thinks she’s overhearing something private. I plan to give her access to my smart locks so she can talk to people at the door and decide to let them in or not. And I can hear y’all being like “My god man, you’re gonna trust her with that??” Hell yeah, I am. I used my alignment technique to train a paranoid security model with a totally different personality. If the main model doesn’t think she can trust someone, she taps out and the security model takes over.

See, I KNEW that to stay competitive and keep up in my field, I needed to embrace agentic AI. Being who I am I said “If I’m gonna do that, I need an AI that’s mine… that runs on my infra and knows who I am and is loyal to me and that I understand well enough to let inside my own trust boundary.” Is it perfect? Not yet, lol. XD

I also have custom phone clients I built that let me talk to her and Claude and ChatGPT. On my network, Claude and ChatGPT are sandboxed and have access only to a few specific hosts and an isolated network share. Mary Anne has access to everything and is instructed that they are outside consultants and she should be careful what she shares with them. So I have her restart stuff (including herself) remotely when we change tools. I’ve literally spent multiple days every week this month pacing around airports, talking to my model on my phone like a madman while we juggle work and hobby projects.

php server and a local certificate authority.

The anti-AI sentiment here is ridiculous. They were asking a legitimate question!

I just use ollama locally but I keep meaning to setup OpenRouter because I like to play with larger models that my hardware can’t handle and I’d like to be able to pick the best deals 🤷‍♀️

Literally just having ollama installed and some models that work well with the amount of VRAM I have. I don’t auto-start the daemon because I rarely use it. Occasionally I use it for proof-reading/feedback on things like cover letters, or debugging. I don’t let it write anything. If it has something useful to say it can tell me in the chat and I can implement it if it’s good. Usually I don’t need it though so it just sits on my disk for the most part.

Ollama on an N150 mini PC, 16GB, no GPU, Open WebUI on top. Being honest, 1-3B models are fine for searching my offline Kiwix stuff and 8B works if you don’t mind waiting. I’m turning this into a small box (heyeddi.com/projects/doomsday-box) and I keep going back and forth on whether people even want AI on it, or just the media and offline archives. Would you?

I just use llama.cpp with Zoo Code, the VSCode extension

join us !localllama@sh.itjust.works

vLLM to host and openwebui for a frontend that I really only use to demo models to friends.

harness

Opencode running with full access inside a rootless podman container with only the directories I want mounted (usually a single project, sometimes read only).

So far though, I’ve only used it to document and report on project repos because Qwen is not that good at actually coding (although it can give pretty decent conf file examples, but not always optimal).

Also useful for parsing pita data dump type files that I’m too lazy to construct multiline awk commands for.

Currently, I’m using Hermes and OpenCode mostly, backed by Ollama running a couple of different models. Usually I’m running the Gemma4 or Gwen3.5 models.

Memory management is done through Obsidian as every harness seems to have really good skills for it.

This community won’t be much helpful but there is one specific for local LLMs. I think it was ai_ and some other…

• Pi - plain and simple - run inside firejail.
• lamma.cpp w/ vulkan backend

That’s it.

It depends on how complex you want to make it.

The simplest way would be to use a GUI tool like LM Studio, it has the benefit of selecting the model quants that are likely to work on your hardware and having sane defaults for the various parameters that you can work from.

Typically you’ll want to increase the context window, the default is good for short one-off questions but if you’re going to do RAG or coding tasks you will need a much larger context window. You will typically find this through trial and error, increasing until you hit memory limits or notice a large slowdown on tokens/second.

You can also have the model respond to an OpenAI-like API interface on a designated port/interface if you want to run non-chat tools through the same model.

Then, depending on the task that you’re trying to solve, you may want to find an MCP server built for that task which can provide the ability for the AI to interact with tools in a way that’s more controlled (giving an AI raw terminal access on a user’s machine is typically a bad idea).

If you need something more flexible that you can incorporate into scripts and automation, llama.cpp has been popular for a while. Like Linux distros, everyone is going to have an opinion on the best and whatever you decide on will be the wrong choice. There’s also Ollama and vLLM if you want to run a datacenter.

Model-wise it completely depends on the task. Do you need a multi-modal model that supports vision, is it going to be coding, is it trained to use tools/MCP servers, does it use reasoning, etc. The Qwen models are open weight and available in many different quants. The version 3.5 models run well on consumer hardware and can use vision, reasoning and tools.

I’m running it through Llama.cpp but it will likely be available via LM Studio as well.

front: open-webui skill servers: dockling - docs parsing for llm, searxng - searchengine for llm, open-terminal - linux terminal for llm (inside container) base: llama-swap Inside llama swap: qwen3-14b-q5, translator-nemo-12b, tower-plus-9b, translator-gemma4-26b, qwen3.6-35b-a3b llama stack: ROCm

Hardware: RX9070XT for llama-swap .

UPD: if you want easy way and run all in oneshot use https://github.com/Niko1221/Strata

Openwebui pointing to llamacpp running on an old ass GPU. It’s good enough for what it is.

Llama-swap running llama.cpp and strata. For the tools I use oh my pi and nvim with plugins

I’ve got an M4 Mac mini with 32 GB of unified memory, so I can run MoE models in the <40B parameter range quite comfortably. I use LM Studio as the backend with Hermes as the frontend. Works great for everyday chat, web search, and RAG.