Redis 3.2集群环境搭建和测试

Redis 3开始支持了Cluster模式，增强了Redis的水平扩展能力，Redis Cluster的节点分片通过hash slot实现，每个节点上的键都属于16384(0~16383)个slots中的一个，每个节点负责处理一部分slots。Redis Cluster采用无中心节点方式实现，无需proxy代理，客户端直接与redis集群的每个节点连接，根据同样的hash算法计算出key对应的slot，然后直接在slot对应的Redis上执行命令。

Redis安装 环境准备 CentOS 7.2 redis1 192.168.61.516379(master) 6380(slave) redis2 192.168.61.526379(master) 6380(slave) redis3 192.168.61.536379(master) 6380(slave)

修改内核参数 vm.overcommit_memory = 1

vi /etc/sysctl.conf vm.overcommit_memory = 1 sysctl -p 编译安装Redis wget http://download.redis.io/releases/redis-3.2.8.tar.gz tar -zxvf redis-3.2.8.tar.gz cd redis-3.2.8 make make install PREFIX=/usr/local/redis mkdir /usr/local/redis/etc cp redis.conf /usr/local/redis/etc/

测试启动：

/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis.conf 集群初始化 配置

redis_6379.conf

daemonize yes pidfile /var/run/redis_6379.pid dir /var/lib/redis/redis_6379 logfile /var/log/redis/redis_6379.log appendonly yes bind 192.168.61.51 # bind 192.168.61.52 # bind 192.168.61.53 port 6379 cluster-enabled yes cluster-config-file nodes-6379.conf cluster-node-timeout 15000 cluster-slave-validity-factor 10 cluster-migration-barrier 1 cluster-require-full-coverage yes

redis_6380.conf

daemonize yes pidfile /var/run/redis_6380.pid dir /var/lib/redis/redis_6380 logfile /var/log/redis/redis_6380.log appendonly yes bind 192.168.61.51 # bind 192.168.61.52 # bind 192.168.61.53 port 6380 cluster-enabled yes cluster-config-file nodes-6380.conf cluster-node-timeout 15000 cluster-slave-validity-factor 10 cluster-migration-barrier 1 cluster-require-full-coverage yes

集群相关的配置：

cluster-enabled 是否开启集群配置 cluster-config-file 集群节点集群信息配置文件,每个节点都有一个,由redis生成和更新,配置时避免名称冲突 cluster-node-timeout 集群节点互连超时的阀值，单位毫秒 cluster-slave-validity-factor 进行故障转移时,salve会 申请成为master。有时slave会和master失联很久导致数据较旧，这样的slave不应该成为master。这个配置用来判断slave是否和master失联时间过长。 groupadd redis useradd -g redis -d /var/lib/redis -s /sbin/nologin redis mkdir -p /var/log/redis mkdir -p /var/lib/redis/redis_6379 mkdir -p /var/lib/redis/redis_6380 chown -R redis:redis /var/log/redis /var/lib/redis

redis_6379.service

[Unit] Description=redis After=network.target [Service] Type=forking User=redis ExecStart=/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_6379.conf Restart=on-failure [Install] WantedBy=multi-user.target

redis_6380.service

[Unit] Description=redis After=network.target [Service] Type=forking User=redis ExecStart=/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis_6380.conf Restart=on-failure [Install] WantedBy=multi-user.target systemctl enable redis_6379 systemctl enable redis_6380 systemctl start redis_6379 systemctl start redis_6380 主节点

配置成功后，在各个主机上查看redis进程，各个redis节点都运行在cluster模式下：

ps -ef | grep redis redis 2656 1 0 15:09 ? 00:00:00 /usr/local/redis/bin/redis-server 192.168.61.51:6379 [cluster] redis 2666 1 0 15:09 ? 00:00:00 /usr/local/redis/bin/redis-server 192.168.61.51:6380 [cluster]

使用 cluster meet 添加主节点:

./redis-cli -h 192.168.61.51 -p 6379 -c cluster meet 192.168.61.52 6379 cluster meet 192.168.61.53 6379

查看集群状态：

cluster info cluster_state:fail #集群状态 cluster_slots_assigned:0 #分配的slot数 cluster_slots_ok:0 #正确的slot数 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:3 #当前的节点数 cluster_size:0 cluster_current_epoch:2 cluster_my_epoch:0 cluster_stats_messages_sent:171 cluster_stats_messages_received:171

上面显示cluster的状态是fail，是由于没有分配slot，需要把16384分配到3个Node上，编写addslots脚本：

addslots.sh

#!/bin/bash for ((i=$2;i<=$3;i++)) do /usr/local/redis/bin/redis-cli -h $1 -p 6379 CLUSTER ADDSLOTS $i done

分别在redis1,redis2,redis3三个主机上执行:

./addslots.sh redis1 0 5461 ./addslots.sh redis2 5462 10922 ./addslots.sh redis3 10923 16383

再次查看集群状态：

CLUSTER INFO cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:3 cluster_size:3 cluster_current_epoch:2 cluster_my_epoch:0 cluster_stats_messages_sent:3495 cluster_stats_messages_received:3495 从节点

先添加从节点到集群中：

cluster meet 192.168.61.51 6380 cluster meet 192.168.61.52 6380 cluster meet 192.168.61.53 6380 CLUSTER INFO cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 #所有节点数6个 cluster_size:3 #主节点数3个 cluster_current_epoch:5 cluster_my_epoch:0 cluster_stats_messages_sent:4388 cluster_stats_messages_received:4388

查看各节点id:

CLUSTER NODES 5daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68 192.168.61.53:6380 master - 0 1488959931731 5 connected 8f60e94a47da98286795b2fbbcae1b3aea75e123 192.168.61.51:6380 master - 0 1488959933751 3 connected bd503b887831e642ac850af3fdd6e27c27086db7 192.168.61.51:6379 myself,master - 0 0 0 connected 0-5461 956f9f95b96bb02842609cea5dcda0cf89bfb78e 192.168.61.53:6379 master - 0 1488959935772 2 connected 10923-16383 877a1fad6a6c344a69707a335d2dcf94afeecbbd 192.168.61.52:6380 master - 0 1488959934761 4 connected 7290136f19a26c877c31f565068ea02badf11339 192.168.61.52:6379 master - 0 1488959936786 1 connected 5462-10922

下面分配从节点，主从关系如下：

master 192.168.61.51 6379 -> slave 192.168.61.52 6380 master 192.168.61.52 6379 -> slave 192.168.61.53 6380 master 192.168.61.53 6379 -> slave 192.168.61.51 6380 ./redis-cli -h 192.168.61.51 -p 6380 -c CLUSTER REPLICATE 5daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68 ./redis-cli -h 192.168.61.52 -p 6380 -c CLUSTER REPLICATE bd503b887831e642ac850af3fdd6e27c27086db7 ./redis-cli -h 192.168.61.53 -p 6380 -c CLUSTER REPLICATE 7290136f19a26c877c31f565068ea02badf11339 CLUSTER NODES 877a1fad6a6c344a69707a335d2dcf94afeecbbd 192.168.61.52:6380 slave bd503b887831e642ac850af3fdd6e27c27086db7 0 1488960443865 4 connected 7290136f19a26c877c31f565068ea02badf11339 192.168.61.52:6379 master - 0 1488960447913 1 connected 5462-10922 956f9f95b96bb02842609cea5dcda0cf89bfb78e 192.168.61.53:6379 master - 0 1488960444893 2 connected 10923-16383 5daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68 192.168.61.53:6380 myself,slave 7290136f19a26c877c31f565068ea02badf11339 0 0 5 connected 8f60e94a47da98286795b2fbbcae1b3aea75e123 192.168.61.51:6380 slave 5daaec9170e2cb0e8e4dfa7e313aa4749c9a0b68 0 1488960446907 5 connected bd503b887831e642ac850af3fdd6e27c27086db7 192.168.61.51:6379 master - 0 1488960445904 0 connected 0-5461

到这一步Redis 3的高可用和分配集群搭建完毕。

基于ruby的集群管理工具 安装ruby

安装ruby：

wget https://cache.ruby-lang.org/pub/ruby/2.4/ruby-2.4.0.tar.gz tar -zxvf ruby-2.4.0.tar.gz cd ruby-2.4.0 ./configure prefix=/usr/local/ruby make make install echo 'export PATH="$PATH:/usr/local/ruby/bin"' >> /etc/profile source /etc/profile ruby -v ruby 2.4.0p0 (2016-12-24 revision 57164) [x86_64-linux]

安装rubygems:

wget https://rubygems.org/rubygems/rubygems-2.6.10.tgz tar -zxvf rubygems-2.6.10.tgz cd rubygems-2.6.10 ruby setup.rb gem -v 2.6.10

安装zlib：

wget http://www.zlib.net/zlib-1.2.11.tar.gz tar -zxvf zlib-1.2.11.tar.gz cd zlib-1.2.11 ./configure --prefix=/usr/local/zlib make make install

安装 ruby-zlib:

cd ruby-2.4.0/ext/zlib ruby ./extconf.rb --with-zlib-dir=/usr/local/zlib make make install

上面的命令在 make 时报 make: *** No rule to make target /include/ruby.h', needed by zlib.o'.Stop. 。修改Makefile文件：

ORIG_SRCS = /root/source/ruby-2.4.0/ext/zlib/zlib.c SRCS = $(ORIG_SRCS) OBJS = /root/source/ruby-2.4.0/ext/zlib/zlib.o

安装redis gem：

gem sources --add http://rubygems.org gem sources --remove https://rubygems.org/ gem sources list gem q -rn redis gem install redis

redis源码下的redis-trib.rb即为集群管理工具：

./redis-trib.rb Usage: redis-trib <command> <options> <arguments ...> create host1:port1 ... hostN:portN --replicas <arg> check host:port info host:port fix host:port --timeout <arg> reshard host:port --from <arg> --to <arg> --slots <arg> --yes --timeout <arg> --pipeline <arg> rebalance host:port --weight <arg> --auto-weights --use-empty-masters --timeout <arg> --simulate --pipeline <arg> --threshold <arg> add-node new_host:new_port existing_host:existing_port --slave --master-id <arg> del-node host:port node_id set-timeout host:port milliseconds call host:port command arg arg .. arg import host:port --from <arg> --copy --replace help (show this help) 参考 Redis cluster tutorial Redis Cluster Specification Redis Cluster Commands