*Update: Thanks to Daniel Huber for making some corrections. He found an error in the systemd script as well as my Java/JDK stuff
A while ago, late 2015, IBM announced “IP quorum” support with version 7.6.
I didn’t have the need or opportunity to play around with this, but after a colleague just implemented an installation with an IP quorum I wanted to try it for myself. The storage system part is fairly easy but they leave it up to you how to implement the Linux part.
A bit of technical background
A quorum device is needed in IBM SVC “streched” or StorWize “HyperSwap” installation and acts as a “tie breaker” placed at an independent location outside your two main data centers.
The traditional quorum device for StorWize/SVC is an “extended quorum” qualified fibre channel array. Usually either the FC connectivity or the storage array (cost, rackspace, heat dispersion) at the third site become a problem. So, here comes the IP quorums to save the day.
The IP quorum consists of a simple java application that needs to be executed on a Linux server. However, IBM has narrowed the choice for your deployment considerably:
I don’t want to go into possible reasons why they do this, but if you are in a production environment make sure you fulfill the requirements.
Note that there is a gotcha with using the IP quorum (taken from the IBM documentation center):
Unlike quorum disks, all IP quorum applications must be reconfigured and redeployed to hosts when certain aspects of the system configuration change.
Now this is serious stuff, the “good old” FC quorum is like a fire and forget solution which you don’t need to touch after the setup.
Reading further on you’ll find some more information about the network quality required, in a typical metro/campus installation this should be of no concern but make sure you check this:
The maximum round-trip delay must not exceed 80 milliseconds (ms), which means 40 ms each direction.
A minimum bandwidth of 2 megabytes per second is guaranteed for node-to-quorum traffic.
If you are worried about the availability:
The maximum number of IP quorum applications that can be deployed is five.
Be grateful for this. You want to patch your Linux on a regular base, don’t you? And an 1U rack server or workstation offers not the same level of redundancy a storage system does. On the downside, if you have more than one installation you’ll have to redeploy the quorum to multiple locations after a major configuration change (see above).
Personal comment time:
Using an IP quorum looks like a nice thing, there is no useless storage array sitting in a third location, less costs and so on. However, make sure you know the implications for your storage environment. This is a kind of paradigm shift since you give a key part of your availability solution out of your hands and rely on network and servers. Organisation wise this might include working with different teams, so change management becomes crucial.
Make sure you know how the network behaves in a site disaster so that the quorum stays available, the only time your ip quorum is really needed it should stay available. Test this on a regular base. Make sure the server team doesn’t patch all your quorum servers at once and test them after updating.
Time for some deployment:
For my tests I am using a IBM StorWize V7000 generation 1.
Since all models (including SVC) share the same code base this howto shoud be valid for all platforms.
As you can see I am using the latest and greatest code level, version 7.8.
Creating the quorum apllication
The latest version gives me some advantages, including the fact that you can deploy and monitor the IP quorum from the UI:
Since my installation uses IPv4 only (shame on me), the option for the IPv6 download is greyed out. And yes, I am using the “superuser” account for this demonstration (I guess that’s the second mark).
As you can see in the screenshot you can do this in the command line by issuing
The application jar file is created in the /dumps directory, since you are already using the CLI I assume you know how to retrieve it from there with scp.
On the UI your browser will offer you a download, save it and transfer it to your quorum server.
I am using a VM based on RHEL 7.3, for a real life deployment this might be an option if you have an ESXi host in your third location. Otherwise a simple 1U rack server or even some kind of workstation might be a good option.
Deploying the quorum application
At this point you should have created the ip quorum application and installed a server with a supported Linux OS. At the end you’ll find a short summary of all commands, but here is a walk-through to understand the process.
I am not going to run an application as root if I do not have to, so creating a service user might be a good idea. Mine is named “ip-quorum”
My quorum application will be deployed into /usr/local/bin/ip_quorum and therefore I need to create the directory and change permissions accordingly:
Time to install Java, but not any Java – you’ll need IBM Java.
As Daniel pointed out I mentioned IBM Java but drifted off and used OpenJDK. Sorry for the error. As you can see it works but it will not be officially supported by IBM. Get your IBM Java here.
As I haven’t time to update the post with new instructions please beware that the next two steps describe OpenJDK instead of IBM Java:
Since I do not have a working satellite installation I’ll get mine straight from the only repositories. Have a look at the RedHat KB for instructions regarding satellite or RHEL6.
For SLES and RHEL you’ll need a working subscription if you are going to do it this way.
After the installation you might want to check the status
and start the applications once by running
java -jar /usr/local/bin/ip_quorum/ip-quorum.jar
As you can see the quorum application initiates a connections to your storage array, not the other way around as a normal service would. The target port on the StorWize is 1260/tcp if you need to implement a firewall rule.
Go back to your storage system to check the status:
Looks good, we made sure our quorum can reach the storage system on the network and application layer. Now terminate the application with CTRL-C .
Setting up a service for the quorum application
Since we want our quorum to start and stop with the server we need some kind of auto start implementation. There are multiple ways to achieve this, I choose a systemd service definition which I call “ip-quorum”.
Create the file with in the system folder of systemd
and add your content:
[Unit] Description=IBM Storwize IBM Quorum Service [Service] # Typo corrected by Daniel, it is Type with a capital letter. Type=simple ExecStart=/usr/bin/java -jar /usr/local/bin/ip_quorum/ip_quorum.jar StandardOutput=null User=ip-quorum Group=ip-quorum WorkingDirectory=/usr/local/bin/ip_quorum/ [Install] WantedBy=multi-user.target
Make sure you tell systemd that unit files have changed
Enable and start the service, do a quick check with “netstat” and on the storage system UI to see if the stuff is running
Here are the lines required, please make sure to check these, fill in the needed values and do not paste them blindly into your system
# creating the user and files useradd -r ip-quorum mkdir -p /usr/local/bin/ip_quorum cp (SOURCE)/ip_quorum.jar /usr/local/bin/ip_quorum chown -R ip-quorum:ip-quorum /usr/local/bin/ip_quorum cd /usr/local/bin/ip_quorum chmod 774 ip_quorum.jar # install java subscription-manager repos --enable rhel-7-server-supplementary-rpms yum install -y java-1.8.0-openjdk-headless # create service definition cat <<'EOF' >> /lib/systemd/system/ip-quorum.service [Unit] Description=IBM Storwize IBM Quorum Service [Service] Type=simple ExecStart=/usr/bin/java -jar /usr/local/bin/ip_quorum/ip_quorum.jar StandardOutput=null User=ip-quorum Group=ip-quorum WorkingDirectory=/usr/local/bin/ip_quorum/ [Install] WantedBy=multi-user.target EOF # start and enable the service systemctl daemon-reload systemctl enable ip-quorum systemctl start ip-quorum