Docker 部署 Redis 集群（三主三从）

2023-10-22

MiddleWare › Redis

Docker, Redis

一、集群搭建

1. 环境准备

$ mkdir redis-cluster
$ cd redis-cluster

$ vi redis.conf.tmpl
$ vi start.sh
$ vi stop.sh

$ chmod +x start.sh stop.sh

# 配置Docker网络
$ docker network create redis-net

(1) Redis 配置模板（redis.conf.tmpl）

#端口
port ${PORT}
#非保护模式
protected-mode no
#启用集群模式
cluster-enabled yes
cluster-config-file nodes.conf
#超时时间
cluster-node-timeout 5000
#集群各节点IP地址
cluster-announce-ip 10.4.7.200
#集群节点映射端口
cluster-announce-port ${PORT}
#集群总线端口
cluster-announce-bus-port 1${PORT}
#开启aof持久化策略
appendonly yes
#后台运行
#daemonize yes
#进程号存储
pidfile /var/run/redis_${PORT}.pid
#集群加密
#masterauth 123456
#requirepass 123456

(2) 集群启动脚本（start.sh）

#!/bin/bash
pwd=$PWD

for port in `seq 7001 7006`;
do
  mkdir -p $pwd/${port}/{conf,data} && PORT=${port} envsubst < $pwd/redis.conf.tmpl > $pwd/${port}/conf/redis.conf
done

for port in `seq 7001 7006`;
do
  docker run -d -it -p ${port}:${port} -p 1${port}:1${port} -v $pwd/${port}/conf/redis.conf:/usr/local/etc/redis/redis.conf -v $pwd/${port}/data:/data --privileged=true --restart always --name redis-${port} --net redis-net --sysctl net.core.somaxconn=1024 redis:6.2.12 redis-server /usr/local/etc/redis/redis.conf;
done

for port in `seq 7001 7006`;
do
  echo -n "$(docker inspect --format '{{ (index .NetworkSettings.Networks "redis-net").IPAddress }}' "redis-${port}")":${port}" ";
done

echo -e "\n"
read -p "请输入要启动的docker容器名称，默认redis-7001:" DOCKER_NAME
if [ ! $DOCKER_NAME ];
    then DOCKER_NAME='redis-7001';
fi
docker exec -it redis-7001 /bin/bash

(3) 集群停止脚本（stop.sh）

#!/bin/bash
docker stop redis-7001 redis-7002 redis-7003 redis-7004 redis-7005 redis-7006
docker rm redis-7001 redis-7002 redis-7003 redis-7004 redis-7005 redis-7006
rm -rf 7001 7002 7003 7004 7005 7006

2. 启动集群

$ ./start.sh
95ced1b26a6e2857fbe690e1e041f23182bff49e4cc3db4bb4c5744a37408a7b
79372ab2a43c33603cd3cedd477bc76069656e63bae66f32f350fae965dd2cd1
dd79151246a79df28f66772935ab7daf3d6755fd5eb82c555569f9b2ace878cf
fefd5047e5ba3cea40a260fdf41c4c3e2e8d4a8819587eb2ac75f9eb6b2b59c5
d873fa7510faa455f557eb08f2b909ab5ea14d569eb9760b5e844e6f3f367dc3
d7dcbc97ce0e6da5a477d9b7648adb661059b36b54ad86751b72cf5c13bb699a
172.19.0.2:7001 172.19.0.3:7002 172.19.0.4:7003 172.19.0.5:7004 172.19.0.6:7005 172.19.0.7:7006

请输入要启动的docker容器名称，默认redis-7001:
root@95ced1b26a6e:/data# cd /usr/local/bin/
root@95ced1b26a6e:/usr/local/bin# ./redis-cli --cluster create 172.19.0.2:7001 172.19.0.3:7002 172.19.0.4:7003 172.19.0.5:7004 172.19.0.6:7005 172.19.0.7:7006 --cluster-replicas 1
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.19.0.6:7005 to 172.19.0.2:7001
Adding replica 172.19.0.7:7006 to 172.19.0.3:7002
Adding replica 172.19.0.5:7004 to 172.19.0.4:7003
M: 7adf5de9770bd4899e1a386253fb5186ec11f614 172.19.0.2:7001
   slots:[0-5460] (5461 slots) master
M: cd44145cfcc0e78c88028e01fbbabdeda7e68f6b 172.19.0.3:7002
   slots:[5461-10922] (5462 slots) master
M: 0fafaad14022f0bb428130f13f5b3b09b9c1bc03 172.19.0.4:7003
   slots:[10923-16383] (5461 slots) master
S: 8d8cb66ebd87a87f5c53945aa4a7fe81fca57eab 172.19.0.5:7004
   replicates 0fafaad14022f0bb428130f13f5b3b09b9c1bc03
S: 95049616e3dd6ba1de444ddeee82204a172f13f6 172.19.0.6:7005
   replicates 7adf5de9770bd4899e1a386253fb5186ec11f614
S: e733d6820fe1f0acd52a5cbe3b2f7c26aba28f48 172.19.0.7:7006
   replicates cd44145cfcc0e78c88028e01fbbabdeda7e68f6b
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join

>>> Performing Cluster Check (using node 172.19.0.2:7001)
M: 7adf5de9770bd4899e1a386253fb5186ec11f614 172.19.0.2:7001
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 95049616e3dd6ba1de444ddeee82204a172f13f6 10.4.7.200:7005
   slots: (0 slots) slave
   replicates 7adf5de9770bd4899e1a386253fb5186ec11f614
M: 0fafaad14022f0bb428130f13f5b3b09b9c1bc03 10.4.7.200:7003
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 8d8cb66ebd87a87f5c53945aa4a7fe81fca57eab 10.4.7.200:7004
   slots: (0 slots) slave
   replicates 0fafaad14022f0bb428130f13f5b3b09b9c1bc03
M: cd44145cfcc0e78c88028e01fbbabdeda7e68f6b 10.4.7.200:7002
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: e733d6820fe1f0acd52a5cbe3b2f7c26aba28f48 10.4.7.200:7006
   slots: (0 slots) slave
   replicates cd44145cfcc0e78c88028e01fbbabdeda7e68f6b
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
root@95ced1b26a6e:/usr/local/bin# ./redis-cli -p 7001 -c
127.0.0.1:7001> cluster nodes
95049616e3dd6ba1de444ddeee82204a172f13f6 10.4.7.200:7005@17005 slave 7adf5de9770bd4899e1a386253fb5186ec11f614 0 1698818415452 1 connected
0fafaad14022f0bb428130f13f5b3b09b9c1bc03 10.4.7.200:7003@17003 master - 0 1698818416459 3 connected 10923-16383
8d8cb66ebd87a87f5c53945aa4a7fe81fca57eab 10.4.7.200:7004@17004 slave 0fafaad14022f0bb428130f13f5b3b09b9c1bc03 0 1698818415000 3 connected
7adf5de9770bd4899e1a386253fb5186ec11f614 10.4.7.200:7001@17001 myself,master - 0 1698818416000 1 connected 0-5460
cd44145cfcc0e78c88028e01fbbabdeda7e68f6b 10.4.7.200:7002@17002 master - 0 1698818416560 2 connected 5461-10922
e733d6820fe1f0acd52a5cbe3b2f7c26aba28f48 10.4.7.200:7006@17006 slave cd44145cfcc0e78c88028e01fbbabdeda7e68f6b 0 1698818415552 2 connected

其他查看集群信息

# 查看docker给每个节点分配的IP信息
$ docker network inspect redis-net | grep -i -E "name|ipv4address"

# 查看主节点信息
$ redis-cli -p 7001 cluster nodes | grep master
# 查看从节点信息
$ redis-cli -p 7001 cluster nodes | grep slave
# 查看集群信息
$ redis-cli --cluster info 10.4.7.200:7002
$ redis-cli --cluster check 10.4.7.200:7004

二、集群扩容

1. 准备新节点

$ vi start-port.sh

$ chmod +x start-port.sh
$ ./start-port.sh  # 安装7007和7008节点

(1) 启动指定端口脚本（start-port.sh）

#!/bin/bash
pwd=$PWD
read -p "请输入容器端口: " DOCKER_PORT
port=$DOCKER_PORT;
echo -e "$port"

mkdir -p $pwd/${port}/{conf,data} && PORT=${port} envsubst < $pwd/redis.conf.tmpl > $pwd/${port}/conf/redis.conf

docker run -d -it -p ${port}:${port} -p 1${port}:1${port} -v $pwd/${port}/conf/redis.conf:/usr/local/etc/redis/redis.conf -v $pwd/${port}/data:/data --privileged=true --restart always --name redis-${port} --net redis-net --sysctl net.core.somaxconn=1024 redis:6.2.12 redis-server /usr/local/etc/redis/redis.conf;

echo -n "启动 $(docker inspect --format '{{ (index .NetworkSettings.Networks "redis-net").IPAddress }}' "redis-${port}")":${port}" 成功!";
echo -e "\n"

2. 集群扩容

(1) 添加集群主节点

$ docker exec -it redis-7001 /bin/bash
# 将7007节点添加到7001节点所在的集群中
root@95ced1b26a6e:/data# redis-cli --cluster add-node 10.4.7.200:7007 10.4.7.200:7001
>>> Adding node 10.4.7.200:7007 to cluster 10.4.7.200:7001
......
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.4.7.200:7007 to make it join the cluster.
[OK] New node added correctly.
# 查看集群状态
root@95ced1b26a6e:/data# redis-cli -p 7001 cluster nodes | grep master
ffc2302ad8489488086963328cd98086d07584ca 10.4.7.200:7007@17007 master - 0 1698821288591 0 connected
0fafaad14022f0bb428130f13f5b3b09b9c1bc03 10.4.7.200:7003@17003 master - 0 1698821288591 3 connected 10923-16383
7adf5de9770bd4899e1a386253fb5186ec11f614 10.4.7.200:7001@17001 myself,master - 0 1698821287000 1 connected 0-5460
cd44145cfcc0e78c88028e01fbbabdeda7e68f6b 10.4.7.200:7002@17002 master - 0 1698821287383 2 connected 5461-10922
# 重新分配哈希槽：将7001,7002中的2000个哈希槽分给7007
root@95ced1b26a6e:/data# redis-cli --cluster reshard 10.4.7.200:7001 --cluster-from 7adf5de9770bd4899e1a386253fb5186ec11f614,cd44145cfcc0e78c88028e01fbbabdeda7e68f6b --cluster-to ffc2302ad8489488086963328cd98086d07584ca --cluster-slots 2000
Ready to move 2000 slots.
  Source nodes:
    M: 7adf5de9770bd4899e1a386253fb5186ec11f614 10.4.7.200:7001
       slots:[0-5460] (5461 slots) master
       1 additional replica(s)
    M: cd44145cfcc0e78c88028e01fbbabdeda7e68f6b 10.4.7.200:7002
       slots:[5461-10922] (5462 slots) master
       1 additional replica(s)
  Destination node:
    M: ffc2302ad8489488086963328cd98086d07584ca 10.4.7.200:7007
       slots: (0 slots) master
  Resharding plan:
    Moving slot 5461 from cd44145cfcc0e78c88028e01fbbabdeda7e68f6b
    Moving slot 5462 from cd44145cfcc0e78c88028e01fbbabdeda7e68f6b
    ......
Do you want to proceed with the proposed reshard plan (yes/no)? yes
Moving slot 5461 from 10.4.7.200:7002 to 10.4.7.200:7007:
Moving slot 5462 from 10.4.7.200:7002 to 10.4.7.200:7007:
......
# 再次查看集群状态
root@95ced1b26a6e:/data# redis-cli -p 7001 cluster nodes | grep master
ffc2302ad8489488086963328cd98086d07584ca 10.4.7.200:7007@17007 master - 0 1698821478000 7 connected 0-998 5461-6461
0fafaad14022f0bb428130f13f5b3b09b9c1bc03 10.4.7.200:7003@17003 master - 0 1698821478707 3 connected 10923-16383
7adf5de9770bd4899e1a386253fb5186ec11f614 10.4.7.200:7001@17001 myself,master - 0 1698821479000 1 connected 999-5460
cd44145cfcc0e78c88028e01fbbabdeda7e68f6b 10.4.7.200:7002@17002 master - 0 1698821479714 2 connected 6462-10922

(2) 添加集群从节点

$ docker exec -it redis-7001 /bin/bash
# 给7007节点添加一个从节点7008
root@95ced1b26a6e:/data# redis-cli --cluster add-node 10.4.7.200:7008 10.4.7.200:7007 --cluster-slave --cluster-master-id ffc2302ad8489488086963328cd98086d07584ca
>>> Adding node 10.4.7.200:7008 to cluster 10.4.7.200:7007
......
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.4.7.200:7008 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of 10.4.7.200:7007.
[OK] New node added correctly.
# 查看集群状态
root@95ced1b26a6e:/data# redis-cli -p 7001 cluster nodes
ffc2302ad8489488086963328cd98086d07584ca 10.4.7.200:7007@17007 master - 0 1698821809019 7 connected 0-998 5461-6461
95049616e3dd6ba1de444ddeee82204a172f13f6 10.4.7.200:7005@17005 slave 7adf5de9770bd4899e1a386253fb5186ec11f614 0 1698821810027 1 connected
0fafaad14022f0bb428130f13f5b3b09b9c1bc03 10.4.7.200:7003@17003 master - 0 1698821809524 3 connected 10923-16383
8d8cb66ebd87a87f5c53945aa4a7fe81fca57eab 10.4.7.200:7004@17004 slave 0fafaad14022f0bb428130f13f5b3b09b9c1bc03 0 1698821808516 3 connected
7adf5de9770bd4899e1a386253fb5186ec11f614 10.4.7.200:7001@17001 myself,master - 0 1698821809000 1 connected 999-5460
cd56cba5b26c7fbb1e72e9d8ee848769c53c6358 10.4.7.200:7008@17008 slave ffc2302ad8489488086963328cd98086d07584ca 0 1698821809000 7 connected
cd44145cfcc0e78c88028e01fbbabdeda7e68f6b 10.4.7.200:7002@17002 master - 0 1698821808013 2 connected 6462-10922
e733d6820fe1f0acd52a5cbe3b2f7c26aba28f48 10.4.7.200:7006@17006 slave cd44145cfcc0e78c88028e01fbbabdeda7e68f6b 0 1698821809523 2 connected

三、集群缩容

$ docker exec -it redis-7001 /bin/bash
# 移除7007的从节点7008
root@95ced1b26a6e:/data# redis-cli --cluster del-node 10.4.7.200:7008 cd56cba5b26c7fbb1e72e9d8ee848769c53c6358
>>> Removing node cd56cba5b26c7fbb1e72e9d8ee848769c53c6358 from cluster 10.4.7.200:7008
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.
# 将7007节点的哈希槽迁移到7001,7002,7003节点上（直接迁移）
root@95ced1b26a6e:/data# redis-cli --cluster reshard 10.4.7.200:7007 --cluster-from ffc2302ad8489488086963328cd98086d07584ca --cluster-to 7adf5de9770bd4899e1a386253fb5186ec11f614 --cluster-slots 700 --cluster-yes
root@95ced1b26a6e:/data# redis-cli --cluster reshard 10.4.7.200:7007 --cluster-from ffc2302ad8489488086963328cd98086d07584ca --cluster-to cd44145cfcc0e78c88028e01fbbabdeda7e68f6b --cluster-slots 700 --cluster-yes
root@95ced1b26a6e:/data# redis-cli --cluster reshard 10.4.7.200:7007 --cluster-from ffc2302ad8489488086963328cd98086d07584ca --cluster-to 0fafaad14022f0bb428130f13f5b3b09b9c1bc03 --cluster-slots 600 --cluster-yes
# 查看集群状态
root@95ced1b26a6e:/data# redis-cli -p 7001 cluster nodes
ffc2302ad8489488086963328cd98086d07584ca 10.4.7.200:7007@17007 master - 0 1698823201433 7 connected
95049616e3dd6ba1de444ddeee82204a172f13f6 10.4.7.200:7005@17005 slave 7adf5de9770bd4899e1a386253fb5186ec11f614 0 1698823200426 8 connected
0fafaad14022f0bb428130f13f5b3b09b9c1bc03 10.4.7.200:7003@17003 master - 0 1698823200000 10 connected 5862-6461 10923-16383
8d8cb66ebd87a87f5c53945aa4a7fe81fca57eab 10.4.7.200:7004@17004 slave 0fafaad14022f0bb428130f13f5b3b09b9c1bc03 0 1698823199521 10 connected
7adf5de9770bd4899e1a386253fb5186ec11f614 10.4.7.200:7001@17001 myself,master - 0 1698823200000 8 connected 0-699 999-5460
cd44145cfcc0e78c88028e01fbbabdeda7e68f6b 10.4.7.200:7002@17002 master - 0 1698823199420 9 connected 700-998 5461-5861 6462-10922
e733d6820fe1f0acd52a5cbe3b2f7c26aba28f48 10.4.7.200:7006@17006 slave cd44145cfcc0e78c88028e01fbbabdeda7e68f6b 0 1698823200000 9 connected
# 移除主节点7007
root@95ced1b26a6e:/data# redis-cli --cluster del-node 10.4.7.200:7007 ffc2302ad8489488086963328cd98086d07584ca
>>> Removing node ffc2302ad8489488086963328cd98086d07584ca from cluster 10.4.7.200:7007
>>> Sending CLUSTER FORGET messages to the cluster...
>>> Sending CLUSTER RESET SOFT to the deleted node.

参考

https://www.cnblogs.com/dooor/p/docker-redis0227.html