Zuul CI Homelab Management

Manage a GitOps-based Zuul CI homelab deployment on Talos Kubernetes using FluxCD, with SOPS secret encryption, TLS certificates, and single-node architecture.

What This Skill Does

This skill helps you manage a lightweight Zuul CI deployment for personal homelab use. It handles FluxCD deployment layers, SOPS-encrypted secrets, TLS certificate management, debugging workflows, and GitOps operations for a single-node Zuul setup with ZooKeeper and PostgreSQL dependencies.

Repository Structure

The repository follows a layered FluxCD architecture:

Core Concepts

**FluxCD Deployment Layers:**

1. Layer 1: Namespaces (foundation)

2. Layer 2: ingress-nginx, cert-manager, reflector (core services)

3. Layer 3: cert-manager-config, certificates, zookeeper, database (dependencies)

4. Layer 4: Zuul application (depends on all above)

Each layer must be healthy before the next deploys.

**Single-Node Architecture:**

Instructions

1. Monitor FluxCD Deployment Status

When checking deployment health or troubleshooting:

```bash

Watch all kustomizations

flux get kustomizations --watch

Check specific layer status (in dependency order)

flux get kustomization namespaces

flux get kustomization cert-manager

flux get kustomization zookeeper

flux get kustomization database

flux get kustomization zuul

Check all FluxCD resources

flux get all

View HelmRelease status

kubectl get helmrelease -A -o wide

Check deployment health across all namespaces

kubectl get pods -A | grep -E 'zuul|postgres|zookeeper|cert-manager|ingress'

```

2. Manage SOPS-Encrypted Secrets

**Encryption Rules** (defined in `.sops.yaml`):

**Commands:**

```bash

Encrypt a secret file (auto-detects path rules)

sops -e -i apps/zuul/secrets/github-connection.yaml

Edit encrypted file (decrypts temporarily in editor)

sops apps/zuul/secrets/github-connection.enc.yaml

Decrypt to view (stdout only)

sops -d apps/zuul/secrets/db-uri-secret.enc.yaml

Verify SOPS age key exists in cluster (required for FluxCD decryption)

kubectl get secret -n flux-system sops-age

```

**CRITICAL**: All secrets must be SOPS-encrypted before committing. Always check `git diff` before pushing.

3. Apply Configuration Changes (GitOps Workflow)

```bash

1. Make changes to YAML files

2. Encrypt secrets if needed

sops -e -i path/to/secret.yaml

3. Commit and push (triggers automatic deployment)

git add . && git commit -m "Update configuration" && git push

4. Monitor deployment

flux get kustomizations --watch

kubectl get pods -n zuul-system -w

```

4. Force Reconciliation

When changes don't auto-deploy or you need immediate reconciliation:

```bash

Reconcile infrastructure (propagates to dependencies)

flux reconcile kustomization infrastructure --with-source

Reconcile zuul application

flux reconcile kustomization zuul --with-source

Reconcile specific components

flux reconcile kustomization cert-manager

flux reconcile kustomization database

```

5. Debug Zuul Components

```bash

Check Zuul component logs

kubectl logs -n zuul-system deployment/zuul-scheduler -f

kubectl logs -n zuul-system deployment/zuul-web -f

kubectl logs -n zuul-system deployment/zuul-executor -f

kubectl logs -n zuul-system deployment/zuul-operator -f

Access Zuul Web UI locally

kubectl port-forward -n zuul-system svc/zuul-web 9000:9000

Test database connectivity

kubectl exec -it -n postgres-system statefulset/postgres -- \

psql -U zuul -d zuul

Test ZooKeeper connectivity (TLS port 2281)

kubectl exec -it -n zuul-system deployment/zuul-scheduler -- \

nc -zv zookeeper.zookeeper-system.svc.cluster.local 2281

```

6. Debug FluxCD and Certificate Issues

```bash

Check FluxCD reconciliation issues

flux logs --level=error

kubectl logs -n flux-system deployment/kustomize-controller -f

kubectl logs -n flux-system deployment/helm-controller -f

Check SOPS decryption issues

kubectl describe kustomization -n flux-system zuul

kubectl describe kustomization -n flux-system database

Check certificate issues

kubectl get certificate -A

kubectl describe certificate -n cert-manager teim-app-wildcard

kubectl logs -n cert-manager deployment/cert-manager-cert-manager --tail=50

Check ZooKeeper TLS certificates

kubectl get secret -n zuul-system zookeeper-client-tls

kubectl describe certificate -n zookeeper-system zookeeper-client-cert

Verify reflector is replicating certificates

kubectl logs -n cert-manager deployment/reflector --tail=20

```

7. Backup and Restore PostgreSQL

```bash

Backup

kubectl exec -n postgres-system statefulset/postgres -- \

pg_dump -U zuul zuul > zuul-backup-$(date +%Y%m%d).sql

Restore

cat backup.sql | kubectl exec -i -n postgres-system \

statefulset/postgres -- psql -U zuul zuul

```

Key Configuration Files

**Zuul Application:**

**Secrets (SOPS encrypted):**

**Infrastructure:**

TLS Certificate Architecture

**ZooKeeper (Self-Signed CA):**

1. `zookeeper-ca-issuer` - Self-signed CA issuer

2. `zookeeper-ca-certificate` - Root CA (10-year validity)

3. `zookeeper-server-cert` - Server certificate

4. `zookeeper-client-cert` - Client certificate for Zuul

5. Reflector replicates `zookeeper-client-tls` secret to zuul-system namespace

ZooKeeper listens on port 2281 (TLS) instead of 2181 (plaintext).

**Wildcard Certificate:**

Important Constraints

Resource Requirements

Example Usage

**Scenario 1: Update GitHub connection credentials**

```bash

Edit encrypted secret

sops apps/zuul/secrets/github-connection.enc.yaml

Commit and push

git add apps/zuul/secrets/github-connection.enc.yaml

git commit -m "Update GitHub connection credentials"

git push

Monitor deployment

flux get kustomization zuul --watch

kubectl rollout status -n zuul-system deployment/zuul-scheduler

```

**Scenario 2: Debug executor startup issues**

```bash

Check executor logs

kubectl logs -n zuul-system deployment/zuul-executor -f

Check ZooKeeper connectivity

kubectl exec -it -n zuul-system deployment/zuul-executor -- \

nc -zv zookeeper.zookeeper-system.svc.cluster.local 2281

Verify client TLS certificate

kubectl get secret -n zuul-system zookeeper-client-tls

kubectl describe secret -n zuul-system zookeeper-client-tls

```

**Scenario 3: Force re-deploy after layer failure**

```bash

Check which layer failed

flux get kustomizations

Reconcile from the failing layer upward

flux reconcile kustomization cert-manager --with-source

flux reconcile kustomization database --with-source

flux reconcile kustomization zuul --with-source

```