k8s-airflow/infra/airflow/rabbitmq/08-rabbitmq-install-guide.md

Rabbitmq on k8s deployment

本文件說明如何在 k8s 上部署 RabbitMQ Cluster（使用官方 RabbitMQ Cluster Kubernetes Operator）

1. 安裝 RabbitMQ Cluster Operator

使用官方 manifest:

kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"

執行後可以看到 Operator 的 Deployment 跑起來：

kubectl get all -n rabbitmq-system

會看到：

deployment.apps/rabbitmq-cluster-operator  1/1 Running

2. 配置動態存儲（StorageClass + PVC）

使用 Helm 安裝官方的 NFS 動態 Provisioner：

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
helm repo update

執行安裝：

helm install nfs-rabbitmq-airflow \
  nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
  --namespace airflow \
  --set nfs.server=10.10.0.85 \
  --set nfs.path=/srv/nfs/airflow/rabbitmq \
  --set storageClass.name=nfs-airflow-rabbitmq \
  --set storageClass.reclaimPolicy=Retain

說明：

• nfs.server：NFS Server IP
• nfs.path：NFS export 的根目錄
• storageClass.name：建立的 StorageClass 名稱
• reclaimPolicy=Delete：
• PVC 刪除時，對應的 PV 與 NFS 子目錄會一併刪除
• 適合 RabbitMQ 這類可重建的服務
• reclaimPolicy=Retain：
• PVC 刪除時，對應的 PV 與 NFS 子目錄會保留
• 正式環境建議採用手動刪除

驗證 StorageClass 是否建立成功

kubectl get storageclass

應該看到類似結果：

NAME           PROVISIONER                                                     RECLAIMPOLICY   VOLUMEBINDINGMODE
nfs-airflow-rabbitmq   cluster.local/nfs-rabbitmq-airflow-nfs-subdir-external-provisioner   Retain          Immediate           true                   13s

代表：

• Kubernetes 已具備可用的 動態 NFS StorageClass
• 之後只要 PVC 指定 storageClassName: nfs-airflow-rabbitmq → 就會自動建立對應的 PV

3. 建立 RabbitMQ Cluster

sudo vi airflow/rabbitmq-cluster.yml

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq-cluster
  namespace: rabbitmq-system
spec:
  replicas: 3

  persistence:
    storageClassName: "nfs-airflow-rabbitmq"
    storage: 10Gi

  override:
    statefulSet:
      spec:
        template:
          spec:
            # 排到 master/control-plane
            nodeSelector:
              node-role.kubernetes.io/control-plane: ""

            tolerations:
              - key: "node-role.kubernetes.io/control-plane"
                operator: "Exists"
                effect: "NoSchedule"
              - key: "node-role.kubernetes.io/master"
                operator: "Exists"
                effect: "NoSchedule"
            # 3個 Pod 分散在不同節點
            affinity:
              podAntiAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  - labelSelector:
                      matchExpressions:
                        - key: app.kubernetes.io/name
                          operator: In
                          values:
                            - rabbitmq-cluster
                    topologyKey: kubernetes.io/hostname

            # CRD 需要 containers 一起出現
            containers:
              - name: rabbitmq
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-mgmt-nodeport
  namespace: rabbitmq-system
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: rabbitmq-cluster
    app.kubernetes.io/component: rabbitmq
  ports:
    - name: management
      port: 15672
      targetPort: 15672
      nodePort: 31672

kubectl apply -f airflow/rabbitmq-cluster.yml

kubectl get pods -n airflow

會看到：

my-rabbitmq-cluster-server-0   Running
my-rabbitmq-cluster-server-1   Running
my-rabbitmq-cluster-server-2   Running

4. 建立給 airflow 用的帳號

kubectl exec -n airflow airflow-rabbitmq-cluster-server-0 -c rabbitmq -- \
  rabbitmqctl add_user airflow airflow
  
kubectl exec -n airflow airflow-rabbitmq-cluster-server-0 -c rabbitmq -- \
  rabbitmqctl set_user_tags airflow management

kubectl exec -n airflow airflow-rabbitmq-cluster-server-0 -c rabbitmq -- \
  rabbitmqctl set_permissions -p / airflow ".*" ".*" ".*"