Zuul Homelab GitOps Helper

Expert assistant for managing a GitOps-based Zuul CI homelab deployment designed for Talos Kubernetes using FluxCD.

Project Context

This skill helps you work with a single-node Zuul CI homelab setup optimized for personal use with minimal resource overhead. The deployment uses FluxCD for GitOps automation, SOPS for secret encryption, and a custom Zuul operator with executor security fixes.

Architecture Overview

**Core Components:**

**Deployment Layers:**

FluxCD deploys infrastructure in strict dependency order:

1. Layer 1: Namespaces

2. Layer 2: ingress-nginx, cert-manager, reflector

3. Layer 3: cert-manager-config, certificates, zookeeper, database

4. Layer 4: Zuul application

Instructions

When helping users with this repository, follow these guidelines:

1. Repository Structure Awareness

Recognize the hierarchical structure:

2. FluxCD Management Tasks

**Monitor deployment status:**

```bash

flux get kustomizations --watch

flux get all

kubectl get helmrelease -A -o wide

```

**Force reconciliation after changes:**

```bash

flux reconcile kustomization infrastructure --with-source

flux reconcile kustomization zuul --with-source

```

**Debug FluxCD issues:**

```bash

flux logs --level=error

kubectl logs -n flux-system deployment/kustomize-controller -f

kubectl describe kustomization -n flux-system zuul

```

3. SOPS Secret Management

**ALWAYS encrypt secrets before committing:**

**Encryption workflow:**

```bash

Encrypt a new or modified secret

sops -e -i apps/zuul/secrets/github-connection.yaml

Edit existing encrypted secret

sops apps/zuul/secrets/github-connection.enc.yaml

Verify encryption before commit

git diff # Should show encrypted content

```

**Initial SOPS setup (if needed):**

```bash

Generate age key

age-keygen -o ~/.config/sops/age/keys.txt

Deploy to cluster

cat ~/.config/sops/age/keys.txt | kubectl create secret generic sops-age \

--namespace=flux-system --from-file=age.agekey=/dev/stdin

```

4. GitOps Workflow

When making configuration changes:

1. Edit YAML files in appropriate directories

2. **CRITICAL**: Encrypt secrets with SOPS if modified

3. Verify encryption with `git diff`

4. Commit and push to trigger automatic deployment

5. Monitor with `flux get kustomizations --watch`

5. Debugging Common Issues

**Zuul component logs:**

```bash

kubectl logs -n zuul-system deployment/zuul-scheduler -f

kubectl logs -n zuul-system deployment/zuul-web -f

kubectl logs -n zuul-system deployment/zuul-executor -f

kubectl logs -n zuul-system deployment/zuul-operator -f

```

**Check deployment health:**

```bash

kubectl get pods -A | grep -E 'zuul|postgres|zookeeper|cert-manager|ingress'

kubectl get pods -n zuul-system -w

```

**Certificate issues:**

```bash

kubectl get certificate -A

kubectl describe certificate -n cert-manager teim-app-wildcard

kubectl logs -n cert-manager deployment/cert-manager-cert-manager --tail=50

```

**Database connectivity:**

```bash

kubectl exec -it -n postgres-system statefulset/postgres -- psql -U zuul -d zuul

```

**ZooKeeper TLS connectivity (port 2281):**

```bash

kubectl exec -it -n zuul-system deployment/zuul-scheduler -- \

nc -zv zookeeper.zookeeper-system.svc.cluster.local 2281

```

6. Key Configuration Files

**When users want to modify:**

7. TLS Certificate Management

**ZooKeeper uses self-signed CA:**

**Wildcard cert uses Let's Encrypt:**

8. Backup Operations

**PostgreSQL backup:**

```bash

kubectl exec -n postgres-system statefulset/postgres -- \

pg_dump -U zuul zuul > zuul-backup-$(date +%Y%m%d).sql

```

**PostgreSQL restore:**

```bash

cat backup.sql | kubectl exec -i -n postgres-system \

statefulset/postgres -- psql -U zuul zuul

```

9. Important Constraints

10. Resource Awareness

Examples

**Example 1: User wants to add a new GitHub connection**

1. Create/edit `apps/zuul/secrets/new-github-connection.yaml`

2. Encrypt: `sops -e -i apps/zuul/secrets/new-github-connection.yaml`

3. Reference in `apps/zuul/configs/zuul-config.yaml`

4. Commit, push, and monitor: `flux get kustomizations --watch`

**Example 2: User reports Zuul scheduler not starting**

1. Check pod status: `kubectl get pods -n zuul-system`

2. Check logs: `kubectl logs -n zuul-system deployment/zuul-scheduler`

3. Verify ZooKeeper TLS: `kubectl get secret -n zuul-system zookeeper-client-tls`

4. Test ZooKeeper connectivity: `kubectl exec -it -n zuul-system deployment/zuul-scheduler -- nc -zv zookeeper.zookeeper-system.svc.cluster.local 2281`

**Example 3: User wants to update Zuul tenant configuration**

1. Edit `apps/zuul/configs/tenant-config.yaml`

2. Commit and push (no encryption needed for config)

3. Monitor reconciliation: `flux reconcile kustomization zuul --with-source`

4. Verify: `kubectl logs -n zuul-system deployment/zuul-scheduler -f`

Safety Checks

Before suggesting any changes: