My AI Went Off the Rails — In the Best Possible Way

I went down an AI rabbit hole. What started as tinkering with a local AI assistant on my home server turned into something I genuinely use every day. Let me share where I’ve ended up.

What is Hermes?

Last time I mentioned Hermes and how I got it installed, this time it’s actually doing things. Hermes (by Nous Research) is an agentic AI assistant I run locally and interact with via Telegram. What makes it useful isn’t just the chat, it’s the integrations.

What I’ve Connected So Far

🏠 Smart Home (Home Assistant) — Hermes can see all my smart home devices. I ran a quick experiment: I asked it to check everything for low battery warnings and add the needed batteries to my shopping list. It found several devices, identified the correct battery types, and updated the note in Joplin, automatically. That alone was worth the setup.

📓 Joplin Notes — I’ve been using Joplin for a couple of years now. I have a solid personal and professional knowledge base built up in there. Now Hermes has read/write access to it, I can ask it to summarize notes, pull specific info, or add new entries. The shopping list use case alone saves me time: I just send Hermes a photo of a product and it asks if I want to add it to the list.

📡 Tiny Tiny RSS (Self-hosted news) — Hermes can now read all the articles in my TT-RSS reader and summarize them for me in a clear, concise way. No more doom-scrolling headlines, just a clean brief when I ask for it.

Scheduled Tasks (Cron Jobs)

Hermes supports cron jobs, so I’ve set it up to fetch the latest news on the Iran-UAE situation every morning at 8am and update a dedicated Joplin note with a running summary. It keeps the full history and gives me a clean timeline view. I’ve even used this data to build a styled news site with all the updates, might make it public at some point.

The Memory Layer: Honcho

I came across Honcho in a Network Chuck video, it’s designed to build a profile of you over time as you interact with your AI, essentially giving it long-term memory. I got it running in Docker. Honest assessment so far: I haven’t noticed a dramatic difference yet, but it’s supposed to compound over time. Will report back.

Finding the Right Model

My local Ollama setup (16GB VRAM) wasn’t cutting it for agentic tasks — too much hallucination, too many structured output failures.

So I bought $10 of credits on OpenAI and $10 on Anthropic to test.

• GPT-5.5: great results, helped me set up most integrations, but I hit rate limits constantly and burned through the credits in under 24 hours.
• Claude Opus 4.7: fewer limitations, but burns tokens even faster. Excellent for complex reasoning.
• OpenRouter (DeepSeek V4 Pro): similar quality which is good enough for my use cases at a significantly lower cost. I’ve since switched to the Flash variant, which is more than enough for most of what I’m doing day-to-day.

What I’ve Learned

• Agentic AI eats credits fast. Budget accordingly.
• Simple tasks fill context windows quickly. Even a “small” multi-step task can balloon your context.
• Local models aren’t there yet, at least not at 16GB VRAM. For anything agentic, the hosted models win by a clear margin.

What I’ve Built (So Far)

An AI assistant that can:

• ✅ Access my smart home: read device status, toggle switches
• ✅ Read, create, edit, and delete Joplin notes
• ✅ Summarize my self-hosted RSS news feed on demand
• ✅ Run scheduled tasks and maintain a live timeline of curated information

What’s Next

• Prompt optimization: I need to get better at keeping context windows lean. So far my main workaround is simply resetting the session whenever I switch to a different task, which helps avoid carrying over irrelevant context. But I’d love to get more systematic about it, anyone with tips, please drop them in the comments.
• News summary website: pulling from TT-RSS and presenting articles in a clean summarized format.
• Broadcast infrastructure: more work on PTP, possibly configuring a transparent clock, and building out VMs for SMPTE 2110.

This article was drafted with the assistance of AI. Originally published on LinkedIn.