Minoko Labs

Managing network devices via SSH typically involves remembering IP addresses, usernames, and sometimes non-standard ports. The SSH config file (~/.ssh/config) eliminates this overhead by defining named aliases with pre-configured connection parameters. The Problem Connecting to a MikroTik switch requires typing the full connection string each time: ssh [email protected] scp backup.rsc [email protected]:/ This becomes tedious with multiple network devices, each potentially having different usernames, ports, or key files. Solution Create an SSH config file with host aliases. ...

The kube-prometheus-stack Helm chart deploys Alertmanager with a default configuration that routes all alerts to a “null” receiver—effectively discarding them. This post documents configuring Alertmanager to send notifications to Slack. The Problem Default Alertmanager configuration: receivers: - name: "null" route: receiver: "null" # All alerts discarded Alerts fire, but nobody gets notified. Solution Architecture ┌─────────────────────┐ ┌──────────────────────┐ ┌─────────────┐ │ Prometheus │────▶│ Alertmanager │────▶│ Slack │ │ (fires alerts) │ │ (routes & groups) │ │ (#alerts) │ └─────────────────────┘ └──────────────────────┘ └─────────────┘ │ ▼ ┌──────────────────────┐ │ Routing Rules │ ├──────────────────────┤ │ critical → 1h repeat │ │ warning → 4h repeat │ │ Watchdog → silenced │ └──────────────────────┘ Directory Structure monitoring/alertmanager/ ├── .env.example # Webhook URL template ├── .env # Actual webhook (gitignored) ├── create-secret.sh # Creates Kubernetes secret └── README.md # Setup documentation Setup Step 1: Create Slack Webhook Go to https://api.slack.com/apps Click “Create New App” → “From scratch” Name: Alertmanager, select your workspace Go to “Incoming Webhooks” → Toggle “Activate” Click “Add New Webhook to Workspace” Select the channel for alerts (e.g., #alerts) Copy the webhook URL Step 2: Create Kubernetes Secret # monitoring/alertmanager/.env.example SLACK_WEBHOOK_URL=https://hooks.slack.com/services/XXX/YYY/ZZZ SLACK_CHANNEL=#alerts #!/bin/bash # monitoring/alertmanager/create-secret.sh set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" if [ -f "$SCRIPT_DIR/.env" ]; then source "$SCRIPT_DIR/.env" else echo "Error: .env file not found" exit 1 fi kubectl create secret generic alertmanager-slack-config \ --from-literal=slack-webhook-url="${SLACK_WEBHOOK_URL}" \ --from-literal=slack-channel="${SLACK_CHANNEL}" \ --namespace=monitoring \ --dry-run=client -o yaml | kubectl apply -f - Run the setup: ...

Linkerd provides automatic mTLS between all pods in the mesh. This encrypts traffic and provides identity verification. However, it does not restrict which pods can communicate with each other. Any pod in the mesh can connect to any other pod. Kubernetes NetworkPolicies add an additional layer of security by defining explicit allow rules at the network level. This provides defense-in-depth: if Linkerd’s proxy is somehow bypassed, NetworkPolicies still enforce access control. ...

This post covers deploying a GitLab Runner inside a Kubernetes cluster using the Kubernetes executor. Each CI job spawns as a pod, runs its tasks, and is automatically cleaned up. Docker builds use Kaniko (rootless, no privileged containers), and job artifacts/dependencies are cached in MinIO. Architecture ┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐ │ GitLab CI Job │────▶│ Runner Manager │────▶│ Job Pod │ │ (push to repo) │ │ (polycephala) │ │ (auto-created) │ └─────────────────┘ └──────────────────┘ └─────────────────┘ │ ┌──────────────────┐ │ │ MinIO Cache │◀─────────────┘ │ (shared deps) │ └──────────────────┘ The runner manager pod runs continuously and polls GitLab for jobs. When a job is picked up, it creates a new pod in the gitlab-runner namespace, executes the job, and deletes the pod when complete. ...

This post documents the implementation of a Retrieval-Augmented Generation (RAG) system for gold market intelligence, running entirely on a homelab Kubernetes cluster with GPU acceleration. The Goal Build a self-hosted AI system that: Ingests gold market data from multiple sources (FRED, GoldAPI, RSS feeds) Stores embeddings in a vector database Provides natural language query capabilities using a local LLM Runs on an NVIDIA RTX 5070 Ti GPU Architecture ┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐ │ Data Ingestion │───▶│ Embedding Service │───▶│ Qdrant │ │ (CronJobs) │ │ (nomic-embed-text) │ │ (Vector Store) │ └─────────────────────┘ └─────────────────────┘ └──────────┬──────────┘ │ ┌─────────────────────┐ ┌─────────────────────┐ │ │ Query Service │◀───│ Ollama │◀──────────────┘ │ (RAG API + UI) │ │ (Llama 3.1 8B) │ └─────────────────────┘ └─────────────────────┘ │ │ │ ┌──────┴──────┐ ▼ │ RTX 5070 Ti │ Web UI @ :80 │ (16GB) │ └─────────────┘ Components Component Purpose Image Ollama LLM inference (Llama 3.1 8B) + embeddings (nomic-embed-text) ollama/ollama Qdrant Vector database for storing embeddings qdrant/qdrant Data Ingestion CronJobs fetching from FRED, GoldAPI, RSS Custom Python/FastAPI Embedding Service Converts text to vectors, stores in Qdrant Custom Python/FastAPI Query Service RAG pipeline + web UI Custom Python/FastAPI Data Sources Source Data Schedule FRED Gold price history, CPI, Fed Funds Rate, 10Y Treasury, USD Index Every 6 hours GoldAPI.io Real-time XAU/USD spot price Hourly RSS Feeds Market news from Investing.com Every 4 hours Implementation Repository Structure gold-intelligence/ ├── .gitlab-ci.yml ├── services/ │ ├── data-ingestion/ │ │ ├── Dockerfile │ │ ├── requirements.txt │ │ └── src/ │ │ ├── main.py │ │ └── collectors/ │ │ ├── fred.py │ │ ├── gold_api.py │ │ └── news_rss.py │ ├── embedding-service/ │ │ ├── Dockerfile │ │ ├── requirements.txt │ │ └── src/ │ │ ├── main.py │ │ ├── embedder.py │ │ └── qdrant_client.py │ └── query-service/ │ ├── Dockerfile │ ├── requirements.txt │ └── src/ │ ├── main.py │ ├── rag_pipeline.py │ ├── ollama_client.py │ └── static/ # Web UI ├── helm/ │ ├── data-ingestion/ │ ├── embedding-service/ │ ├── query-service/ │ ├── ollama-values.yaml │ └── qdrant-values.yaml └── kubernetes/ └── argocd/ Ollama Configuration The key to GPU acceleration is the runtimeClassName: nvidia in the Helm values: ...

Wiki.js was running on a LoadBalancer IP with no TLS. This post covers migrating to HTTPS using a Let’s Encrypt wildcard certificate managed by OPNsense, with automatic synchronization to Kubernetes. The Problem The wiki was accessible at http://192.168.2.204 with: No TLS encryption Direct LoadBalancer service exposure No ingress controller The goal: HTTPS with a publicly trusted certificate, no browser warnings. Architecture ┌─────────────────────────────────────────────────────────────────────────┐ │ OPNsense Firewall │ │ ┌────────────────────────────────────────────────────────────────────┐ │ │ │ ACME Client │ │ │ │ - Let's Encrypt account │ │ │ │ - Cloudflare DNS-01 validation │ │ │ │ - Wildcard cert: *.minoko.life │ │ │ │ - Auto-renewal at 60 days │ │ │ └────────────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────┘ │ OPNsense API (daily sync) ▼ ┌─────────────────────────────────────────────────────────────────────────┐ │ Kubernetes Cluster │ │ ┌─────────────────────┐ ┌─────────────────────────────────────────┐ │ │ │ letsencrypt-sync │ │ ingress-nginx namespace │ │ │ │ CronJob (5 AM) │───▶│ letsencrypt-wildcard secret │ │ │ │ - Fetch from API │ └─────────────────────────────────────────┘ │ │ │ - Build cert chain │ ┌─────────────────────────────────────────┐ │ │ │ - Update secrets │───▶│ wikijs namespace │ │ │ └─────────────────────┘ │ letsencrypt-wildcard secret │ │ │ └─────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ ingress-nginx-controller │ │ │ │ LoadBalancer: 192.168.2.224 │ │ │ │ │ │ │ │ Ingress: wiki.minoko.life ──▶ wikijs:80 │ │ │ │ TLS: letsencrypt-wildcard secret │ │ │ └─────────────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────┘ Prerequisites OPNsense with ACME plugin configured for Let’s Encrypt Cloudflare (or other DNS provider) for DNS-01 validation Existing wildcard certificate for *.minoko.life Step 1: Install ingress-nginx Controller Create the ingress-nginx infrastructure: ...

I wanted to add comments to this blog without using a third-party service like Disqus. After evaluating options, I chose Comentario - a self-hosted comment system that uses PostgreSQL for storage. This post covers the full deployment on Kubernetes with HAProxy TLS termination and custom theming. Why Comentario When choosing a comment system, I had a few requirements: Self-hosted - No third-party data collection PostgreSQL backend - I already run a shared PostgreSQL instance, so no extra backup infrastructure needed GitHub OAuth - Most of my readers are developers Simple embed - Just a script tag and web component I considered Remark42 (uses BoltDB) and Commento (abandoned), but Comentario hit all the marks. It’s an actively maintained fork of Commento with PostgreSQL support. ...

This post covers exposing an NVIDIA RTX 5070 Ti (Blackwell architecture) as a schedulable Kubernetes resource with time-slicing support, allowing multiple pods to share the GPU. Hardware Node GPU Memory Compute Capability polycephala NVIDIA GeForce RTX 5070 Ti 16 GB 12.0 (Blackwell, sm_120) The RTX 5070 Ti uses the new Blackwell architecture (GB203 chip) with compute capability sm_120. This creates compatibility challenges with some software that hasn’t been updated yet. ...

OPNsense includes Suricata for intrusion detection, but the built-in alerts page provides limited visibility. This post covers forwarding IDS alerts to Loki via syslog and visualizing them in Grafana alongside firewall logs. Architecture ┌─────────────────┐ UDP/514 ┌──────────────────┐ │ OPNsense │ RFC5424 │ Promtail │ │ ┌───────────┐ │ ───────────────▶ │ (syslog recv) │ │ │ Suricata │ │ │ 192.168.2.221 │ │ │ filterlog │ │ └────────┬─────────┘ │ └───────────┘ │ │ └─────────────────┘ ▼ ┌──────────────────┐ │ Loki │ │ (log storage) │ └────────┬─────────┘ │ ▼ ┌──────────────────┐ │ Grafana │ │ (dashboards) │ └──────────────────┘ Prerequisites OPNsense firewall with Suricata IDS enabled Kubernetes cluster with Loki deployed MetalLB or NodePort for exposing the syslog receiver Step 1: Enable Suricata IDS on OPNsense Navigate to Services → Intrusion Detection → Administration. ...

Certificates expire. In a Kubernetes homelab with Linkerd service mesh, this means the identity issuer certificate needs renewal annually. Without automation, this becomes a manual task that’s easy to forget until mTLS breaks across the cluster. This post covers installing cert-manager on a bare-metal kubeadm cluster and configuring it to automatically rotate Linkerd’s identity issuer certificate. The Problem Linkerd uses a two-tier PKI: Certificate Purpose Default Lifetime Trust Anchor Root CA for the mesh 10 years Identity Issuer Signs proxy certificates 1 year The identity issuer expires annually. When it does, new proxy sidecars cannot obtain valid certificates, breaking mTLS. The trust anchor rarely needs rotation, but the identity issuer requires attention. ...