VCSA 6.5 U1: vAPI status “yellow” and content library not started (possible fix)

As this is an error that affected me now at multiple customer installations, it is time for a blog 🙂

After upgrading a site to 6.5 U1 I noticed several issues:

  • The vAPI Endpoint status changed to “yellow”
  • The Content Library service would not start

The only resolve I found within the VMware KB was “restart the services.

As I didn’t help, I searched along and found VMware KB 2151085 with the cause of the error

 

The ts-config.properties file is deployed with the noreplace option. With this option, the ts-config.properties will no longer be overwritten, instead it is saved with the extension .rpmnew.
and a nice hint
This is a known issue seen with several upgrade paths to vSphere 6.5 Update 1. Not all upgrade paths are affected, VMware is investigating affected paths this article will be updated once confirmed.
I hope this helps someone else as the KB entry isn’t obvious.

Adding a second syslog server to a VCSA 6.5 (Appliance MUI)

Beware: This is probably not supported

I was asked if I could add another syslog server to an existing VCSA deployment. With the nice blog post from William Lam in mind, adding the second server should be easy. Just edit the configuration and there you are.

Except:
The UI won’t allow this.

I guess it is CLI time then, luckily the blog post mentions this:

A minor change, but syslog-ng is no longer being used within the VCSA and has been replaced by rsyslog.

So we are just looking at a matter of finding the right config file.

In the main file

  • /etc/rsyslog.conf

you can an “include” statement point towards the file

  • /etc/vmware-syslog/syslog.conf

The only content is our first syslog server, configured as a remote syslog target.

At this point adding the second server is not a big deal, the file now looks like this:

Remember to reload the syslog service afterwards.

 

Another gotacha (besides the lack of support):

Changing the settings via VAMI/UI will overwrite your modifications

I/O acceleration at host level – Part I: Overview & PrimaryIO

In my last to posts I have been rambling about certifications, but it is time to put something useful on this page.

When PernixData released 2.x of FVP they really got the attention of the vCommunity. Lots of posts and happy people all around about two years ago.

For those of you who are not familiar with the topic:
The idea of FVP and similar products is to take the I/O handling as close to the source as possible, which is in case of VMs the ESXi host. You can use memory or flash storage devices as a read or (mirrored-)write cache for your most demanding VMs. (Cached) Requests are instantly answered and load is taken from your storage system. This is especially useful if you have an older model with very little or no flash and/or experience performance problems and you 6want to protect your investment (read: no chance to upgrade hardware any time soon). There are of course other use cases but I’ll try to keep this simple.

But after Nutanix acquired PernixData they essentially buried FVP as a product.
However Nutanix fulfills the support for customers with an existing contract but by now you should have a “plan B” if you need I/O acceleration at host level.
FVP lacks vSphere 6.5 support and I find it hard to believe that great efforts are being made to deliver it anytime soon. (Side note: More than FVP I’ll be missing the PernixData Architect which delivers quite a good view into the storage layer and is very easy to handle.) *UPDATE: I am very sorry, I didn’t give the folks at Pernix/Nutanix the credit they deserve, update is targeted for 04/17 according to support.

So, let’s have a look at a possible alternative:

Last year PrimaryIO offered PernixData FVP customers their Application Performance Accelerator (APA) for VMware at no charge except support costs.

APA uses vSphere APIs for I/O Filtering (VAIO) for their implementation, which  is quite nice in my opinion since this is standardized within the vSphere environment. Here I take the liberty to quote directly from VMware:

VAIO is a Framework that enables their parties (Partners) to develop filters that run in ESXi and can intercept any IO requests from a guest operating system to a virtual disk. An IO will not be issued or committed to disk without being processed by IO Filters created by 3rd parties (source)

Right now there are two supported use cases (caching and replication) and according to VMware

caching will significantly increase the IOPS available, reduce latency, and increase hardware utilization rates (quote from the source from above)

If someone is interested in an overview of currently supported VAIO solutions, you may find it here.

What does APA offer?
According to their technical brief they do not cache randomly or only the most frequented blocks but do this acutally application aware (hence the name, I guess):

Only the most important application components such as frequently accessed tables or indexes that speed up queries are optimized, while less critical elements such as log records, replicas, audit entries, or ad hoc user activity are de-prioritized.

I’m not sure how to track this in the future and verify the claim, so comments are welcome and if I find a way I’ll let you know 🙂

Like FVP they APA offers the possibility to mirror the write cache across hosts which is a must if you take this into production. I’ll need to check if they support fault domains (i.e. two data center).

For a more details on how it works have a look at the VMware blogs where Murali Nagaraj, CTO of PrimaryIO posted about this.

 

Thank you for your attention, in part 2 I will continue on with the APA appliance deployment

 

Note: My posts on this topic are in no way sponsored by PrimaryIO.

Personal blog, remember? 🙂