RabbitMQ Cluster & CAP

最近看了一下RabbitMQ对于分布式部署的问题，整理了一下关于CAP的问题

C是说数据完全一致，任意时刻任意节点数据一致

A是指任意时刻任意节点都可以读写成功

P是指在网络出现分区的情况下，可以操作数据

CA是指任意时刻，任意节点读写完全一致切一定可用，但如果出现分区则不能保证C

CP是指如果出现分区，则更小规模一些的分区直接下线，不可访问，放弃了A

AP是指每个分区可以独立活动，恢复的时候再同步，最终一致性，放弃了C

在rabbitmq中的cluster模式，默认是CA模式，其中

• 元数据的C和A由erlang提供
• queue中message的C和A由mirrored queue提供

但是一旦出现分区，则以上的C无法提供

其中cluster的模式：

• ignore (默认): 提供CA模式，如果出现分区，则C无法保证，且分区中的节点会各自为政，分区结束后不得不放弃一部分数据(元数据及queue中的message)，且无法merge
• pause-minority: 提供CP模式，即如果出现分区，则部分节点不可用，但可以保证数据一致性

会比较糟心的一点是，如果你把运行rabbitmq的虚拟机suspend的话，这台rabbitmq服务器无法感知到自己已宕机，也不知道其他机器已把自己列为无法访问的机器，也不会判定其他机器已宕机。

真正的CA是不存在的，因为出现P就没有C了，除非所有节点部署在同一台机器上。

https://www.quora.com/Whats-the-difference-between-CA-and-CP-systems-in-the-context-of-CAP-Consistency-Availability-and-Partition-Tolerance

mSOHU failure recovery

14:00  reported site is down to another team
14:09  the other team can not solve the problem and reported by my team leader that website is down
I was interviewing a person, then received a call from leader that website is down.
Come back to desk, found website is unable to access, resulting 504 Gateway timeout.
First consideration is backup plan, switch to backup version.
SA reported that nginx -s reload takes very long time.
Tracked server log which reported a redis is unable to connect.
Someone reports that 10.10 section is unable to access & found that IDC down.
Started local test server for isolate affects.

Found 2 functions are affected from this issue.

14:16 decided to remove functions involved temporayly

14:24  issue reported by TECH-NO

故障名称：北京电信兆维IDC2-3机房7-12排电力中断
故障时间：2015年5月19日13:54—未恢复
影响范围：北京电信兆维IDC2-3机房7-12排服务器所有公私网访问全部中断
故障原因：现场查看是运营商电力中断

14:25  packaged website and begins to deploy
14:27  package deployed, but switch version (nginx reload) stucks.
14:30  nginx finally reloaded and website begins to recover, part of site is unavailable due another service is still down.
14:37  deployed second part of patch
14:40  begins to fix rest parts of the website.
--  re-configured internal data service to available redis.
--  another part of website is recovered
--  utsf is down, and timeline to be filled
--  some redis is recovered, so that we re-deployed old version online.
17:00  found that some part of website is not same. but every thing is just fine.
17:04  re-deployed and switched server and everything works
17:08  found another issue is same effect.
17:16  isolated that issue and found one nginx is new to website, and switch server operation is not affected that server.
17:25  updated deploy script that includes that server.

-- till now everything is okay.

so, here is the resolution.
1. allow service to be unavailable, if it happens, fill with empty data.
2. replica of important services, and marks those failed to be untouched
3. unit and cell -ize services between IDCs, which boosts speed and also provide HA for the mainline service.
4. reporting flow improvement. TBD.
5. server partitioning

initializing ubuntu 14.04

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

# basic components

sudo apt-get install network-manager-openconnect proxychains mesa-utils python-dev git subversion vim libyaml-dev python-pip libmysqlclient-dev libxml2-dev libxslt1-dev libcurl4-gnutls-dev libssl-dev smartmontools ifstat
sudo pip install pyyaml django tornado bottle gunicorn gevent virtualenv


# extras

sudo add-apt-repository ppa:ubuntu-wine/ppa

sudo add-apt-repository ppa:webupd8team/java

sudo apt-get update
sudo apt-get install wine1.7

sudo apt-get install oracle-java8-installer

# after language support is okay

sudo apt-get remove fonts-arphic-ukai fonts-arphic-uming

# qqintl
winetricks qqintl
proxychains wget http://winezeug.googlecode.com/svn/trunk/winetricks_files/winetest.cat

# cx_oracle
cd /opt
unzip ~/Downloads/instantclient-sdk-linux.x64-12.1.0.2.0.zip
unzip ~/Downloads/instantclient-basic-linux.x64-12.1.0.2.0.zip
cd /opt/instantclient_12_1/
ln -s libclntsh.so.12.1 libclntsh.so
sudo cat "export ORACLE_HOME=/opt/instantclient_12_1" >> ~/.bashrc
sudo cat "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME" >> ~/.bashrc
sudo -E pip install cx_oracle