# Rabbitmq on k8s deployment

本文件說明如何在 k8s 上部署 RabbitMQ Cluster（使用官方 RabbitMQ Cluster Kubernetes Operator）



## 1. 安裝 RabbitMQ Cluster Operator

使用官方 manifest:
```shell
kubectl apply -f "https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml"
```
執行後可以看到 Operator 的 Deployment 跑起來：
```shell
kubectl get all -n rabbitmq-system
```
會看到：
```
deployment.apps/rabbitmq-cluster-operator  1/1 Running
```

## 2. 配置動態存儲（StorageClass + PVC）

[//]: # (使用 NFS Server 搭配 nfs-subdir-external-provisioner 來實作 動態 NFS StorageClass。)

[//]: # ()
[//]: # (編輯 /etc/exports)

[//]: # (```shell)

[//]: # ( sudo vi /etc/exports)

[//]: # (```)

[//]: # ()
[//]: # (添加以下字段)

[//]: # ()
[//]: # (```)

[//]: # (/srv/nfs/rabbitmq 10.10.0.0/16(rw,sync,no_subtree_check,no_root_squash))

[//]: # (```)

[//]: # ()
[//]: # (套用設定)

[//]: # ()
[//]: # (```shell)

[//]: # (sudo exportfs -ra)
[//]: # (```)

使用 Helm 安裝官方的 NFS 動態 Provisioner：

```shell
helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/
helm repo update
```
執行安裝：
```shell
helm install nfs-rabbitmq-airflow \
  nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
  --namespace airflow \
  --set nfs.server=10.10.0.85 \
  --set nfs.path=/srv/nfs/airflow/rabbitmq \
  --set storageClass.name=nfs-airflow-rabbitmq \
  --set storageClass.reclaimPolicy=Retain
```
說明：
-	nfs.server：NFS Server IP
-	nfs.path：NFS export 的根目錄
-	storageClass.name：建立的 StorageClass 名稱
-	reclaimPolicy=Delete： 
  -	PVC 刪除時，對應的 PV 與 NFS 子目錄會一併刪除
  - 適合 RabbitMQ 這類可重建的服務
-   reclaimPolicy=Retain：
  -	PVC 刪除時，對應的 PV 與 NFS 子目錄會保留
  - 正式環境建議採用手動刪除



驗證 StorageClass 是否建立成功

```shell
kubectl get storageclass
```

應該看到類似結果：
```
NAME           PROVISIONER                                                     RECLAIMPOLICY   VOLUMEBINDINGMODE
nfs-airflow-rabbitmq   cluster.local/nfs-rabbitmq-airflow-nfs-subdir-external-provisioner   Retain          Immediate           true                   13s
```

代表：
- Kubernetes 已具備可用的 動態 NFS StorageClass
-  之後只要 PVC 指定 storageClassName: nfs-airflow-rabbitmq → 就會自動建立對應的 PV


## 3. 建立 RabbitMQ Cluster


```shell
sudo vi airflow/rabbitmq-cluster.yml
```

```yaml
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq-cluster
  namespace: rabbitmq-system
spec:
  replicas: 3

  persistence:
    storageClassName: "nfs-airflow-rabbitmq"
    storage: 10Gi

  override:
    statefulSet:
      spec:
        template:
          spec:
            # 排到 master/control-plane
            nodeSelector:
              node-role.kubernetes.io/control-plane: ""

            tolerations:
              - key: "node-role.kubernetes.io/control-plane"
                operator: "Exists"
                effect: "NoSchedule"
              - key: "node-role.kubernetes.io/master"
                operator: "Exists"
                effect: "NoSchedule"
            # 3個 Pod 分散在不同節點
            affinity:
              podAntiAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  - labelSelector:
                      matchExpressions:
                        - key: app.kubernetes.io/name
                          operator: In
                          values:
                            - rabbitmq-cluster
                    topologyKey: kubernetes.io/hostname

            # CRD 需要 containers 一起出現
            containers:
              - name: rabbitmq
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-mgmt-nodeport
  namespace: rabbitmq-system
spec:
  type: NodePort
  selector:
    app.kubernetes.io/name: rabbitmq-cluster
    app.kubernetes.io/component: rabbitmq
  ports:
    - name: management
      port: 15672
      targetPort: 15672
      nodePort: 31672
```

```shell
kubectl apply -f airflow/rabbitmq-cluster.yml
```

```shell
kubectl get pods -n airflow
```
會看到：
```
my-rabbitmq-cluster-server-0   Running
my-rabbitmq-cluster-server-1   Running
my-rabbitmq-cluster-server-2   Running
```


## 4. 建立給 airflow 用的帳號
```shell
kubectl exec -n airflow airflow-rabbitmq-cluster-server-0 -c rabbitmq -- \
  rabbitmqctl add_user airflow airflow
  
kubectl exec -n airflow airflow-rabbitmq-cluster-server-0 -c rabbitmq -- \
  rabbitmqctl set_user_tags airflow management

kubectl exec -n airflow airflow-rabbitmq-cluster-server-0 -c rabbitmq -- \
  rabbitmqctl set_permissions -p / airflow ".*" ".*" ".*"
```