Beyond SQL on VMware Best Practices

Going beyond SQL in VMware Best Practices has involved settings and configurations that are never mentioned in the SQL Server On VMware Best Practices Guide. Working in a healthcare environment means working with a large EMR (Electronic Medical Record) provider like EPIC. In tuning one of their many large sql databases they had some recommendations. Some of these are throwbacks to my old days of tweaking performance out of a desktop with settings like TCP parameter KeepAliveTime. How many of you go beyond the recommendations in the SQL Best Practice Guide?

floating book in quaint library
Knowledge shared becomes wisdom attained (Photo by Jaredd Craig on Unsplash)

I’ve listed here some of the settings that have been suggested for our larger more accessed and just plain busy sql servers:

ESXi Settings

Adjust the Round Robin IOPS limit from the default 1000 to 1 on each database LUN. Refer to VMware KBA 2069356 for more information on setting this parameter. (We already utilize Round Robin but each lun was set to the default)

Why would you want to make this change?
“The default of 1000 input/output operations per second (IOPS) sends 1000 I/O down each path before switching. If the load is such that a portion of the 1000 IOPS can saturate the bandwidth of the path, the remaining I/O must wait even if the storage array could service the requests. The IOPS or bytes limit can be adjusted downward allowing the path to be switched at a more frequent rate. The adjustment allows the bandwidth of additional paths to be used while the other path is currently saturated. “

How to make this change:

In ESXi 5.x/6.x:
for i in esxcfg-scsidevs -c |awk '{print $1}' | grep naa.xxxx; do esxcli storage nmp psp roundrobin deviceconfig set –type=iops –iops=1 –device=$i; done

Where, .xxxx matches the first few characters of your naa IDs.
 
To verify if the changes are applied, run this command:

esxcli storage nmp device list
 
You see output similar to:
 
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=iops,iops=1,bytes=10485760,useANO=0;lastPathIndex=1: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba33:C1:T4:L0, vmhba33:C0:T4:L0

Registry Settings

Configure Windows TCP Parameters in the Registry

The default setting for the Windows TCP parameter KeepAliveTime is two hours. This setting controls how often TCP sends a keep-alive packet to verify that an idle connection is still intact. Reducing it from two hours to five minutes helps windows detect and clean up stale network connections faster.


How to make this change:

Use regedit to create the DWORD KeepAliveTime (if it does not currently exist) at HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\ . Then modify the value to 300000 (time in milliseconds).

The default setting for the Windows TCP parameter TCPTimedWaitDelay is four minutes. This setting determines the time that must elapse before TCP/IP can release a closed connection and reuse its resources. By reducing the value of this entry, TCP/IP can release closed connections faster and provide more resources for new connections.

How to make this change:

Using regedit to create the REG_DWORD TCPTimedWaitDelay (if it does not currently exist) at HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\ . Then modify the value to 30 .

What kind of ESXi settings, Windows registry settings, config file changes, etc. do you implement in your environment that goes beyond the SQL Server On VMware Best Practices Guide ? As always, I look forward to your comments and sharing of knowledge.

HPE DL560 Gen9 Will Soon Be on VMware HCG

HPE DL560 Gen9 servers will soon be on the VMware HCG. Like many of you we plan ESXi upgrades to new versions months into the future. As we approach our planned upgrades we hope that our HPE hardware will be on VMware’s HCG (Hardware Compatibility Guide) as approved for the latest ESXi 6.7 version. I can’t remember the last time that our hardware wasn’t on the HCG as approved.

My understanding is that most people that are using HPE server hardware are using the DL360 models. Typically those models get qualified faster. For philosophical reasons we like to use the DL560 models. We currently have quite a few of the DL560 gen9 models that we want to use. So far HPE has not bothered to have the DL560 gen9 model qualified as compatible with ESXi 6.7 but they did qualify the gen10 model. Our HPE guy has hinted that the DL560 is just not purchased nearly as much as the DL360s and that contributes to the slowness of getting the hardware on the HCG.

Well I have good new for those in the same boat as us. HPE has notified us that the qualification is being performed and will be completed around April 2nd. This may or may not come true but I thought I’d share the information.

I am very interested to hear what you are doing in your datacenter with HPE and VMware. Are HPE DL360s more your flavor or do you like the HPE DL560s like we do? Comments are always welcome! I look forward to hearing from all of you especially if you have HPE DL560 gen9 servers in your environment.

Datacenter with full racks of servers with flashing lights.  HPE DL560 gen9 servers are among the other servers

Nutanix vs VMware – Who is bullying whom?

Nutanix vs VMware – Who is bullying whom? Can’t we just all get along? Apparently not! VMware and Nutanix are at it again. Nutanix claims that VMware is bullying them by responding to Nutanix’s ‘You Decide’ marketing. When you mess with the bull you get the horns. Head on over to The Register to check out their always unique reporting. I have not used Nutanix in my environment so I can’t give a knowledgeable or unbiased response.

Nutanix vs VMware

Come back and let me know what you think. Comments are always welcome!

VMware Recertification Is Up To You

The pain is over for all that hated the every 2 years recertification policy for VMware certs.  You get to decide when you want to upgrade.  What a novel concept! We’ll see how this plays out as to whether this is a good idea or not.  In the meantime check out the comments in the VMware subreddit. As always, Reddit delivers with comments like ‘This looks like a hostage video’.

Here is the ‘hostage’ video:

via VMware Certification: Recertification Is Changing and What It Means to You – VMware Education Services

Scratch File Location for VMHosts PowerCLi Script

I’ve been wanting to create a PowerCli script to automatically configure a scratch file location if one did not exist. This script would come in handy when moving to a new lun for the scratch location. After some quick google fu I came across a great script that met my requirements plus a little more.

Check out the post over at http://miklm.com/2018/01/configure-scratch-location-with-powercli/

In addition to creating the needed scratch file location this script indicates if it is already created and if the vmhost needs a reboot to complete the process. A csv file is created with a list of all the hosts that need a reboot. I made two small changes to the script. I added the cluster name to the exported name of the csv file. A disconnect-viserver command was added to the end of the script.

PowerCLi Script
Coding
Scripting Time

Feel free to comment and give suggestions on how to improve this script or any others I have shared. While you’re here check out my Schannel Set-Crypto PowerCLi Script Feedback is always appreciated.

Schannel Set-Crypto PowerCLI Script

Are you in need of configuring crypto or schannel (IIS) settings on more than a few Windows virtual machines?  The Set-Crypto PowerCLI script might be of interest to use in your VMware environment. Recently I had to configure all the Windows servers in our VMware environment to use the correct crypto/schannel settings.  The gold standard for Windows/IIS administrators for configuring crypto is a program called IISCrypto . 

Schannel Security
Schannel Security

I have been using IISCrypto for a few years now and it is hands down the easiest way to configure your Windows vms using the schannel settings you choose. It also has built-in ‘best practices’ settings for general security as well as pci and fips requirements.  This is a great tool for setting one server at a time but I needed to do this en masse.  I thought wouldn’t it be great if they had a command-line tool.  Well it turns out that they do have a command-line tool. It allowed me to create a script to configure all my servers for crypto/schannel settings.

Set-Crypto Schannel

The schannel PowerCLI script  Set-Crypto utilizes IISCrypto command-line, batch files, and template profiles created ahead of time based on ‘best practices’ for 2012R2 Windows Server and 2016 Windows Server respectively. You can also go to my Pastebin Set-Crypto folder to get the needed batch files and template profile files. You will need to get IISCrypto from Nartac software.  It is free to use but their site does not explain their licensing model so I do not know the implications of using their software.  Please contact Nartac software for more information regarding licensing.  Please read the assumptions at the beginning of the script. Always be sure to read and understand any code you download from the internet before running in a production environment.


Feel free to comment and give suggestions on how to improve this script or any others I have shared.  If you like this script you may want to check out one of my previous scripts Powershell SSL Certificate Script .

 

 

Last Week’s Most Interesting VMware KBs

Last Week’s VMware KBs of Interest for week ending May 27th, 2018

These are the KBs I found of the most interest to me in my environment.  They might be of interest to you as well.

vCenter Server Appliance Management Interface might not display the vCenter Server 6.7.0a patch  

If you were one of the psychos that likes to upgrade to the bleeding edge as soon as it goes GA then this one may have bitten you. There is currently no resolution but there is a workaround. 

 

While migrating from a Windows vCenter Server to a 6.5 vCenter Appliance, one or more services fail to start

If you utilize hosts files instead of a valid forward and reverse dns entry then this one my rear its ugly head during your migration.  There is a workaround with two parts.  You really should use the first part of the workaround and skip the second part which continues the use of hosts files.  This is why you are in this mess in the first place.  Please check it out.

Upgrading VM Virtual Hardware through Update Manager 6.x fails on Linux Virtual Machines

This is a known issue and there is no resolution currently.  Please see KB for a list of the symptoms to see if you are truly affected.

Hopefully these are helpful and hopefully reading these will prevent some issues before they happen. Please read the full list by clicking the picture below or the source link.

Get in there!

 

VMware Support News, Alerts, and Announcements

Source: VMware Support Insider

Backup VDS PowerCLI Script

I started doing all my VDS (Virtual Distributed Switch) backups manually which isn’t too hard. But you can do better with a Backup VDS script.  Even if you don’t realize it now, you will want to automate as much as possible.  Here is how I automated VDS backups. Starting with a powercli script from VCDX56 I made sure that I understood how it worked and modified it to fit my environment. 

Backup VDS PowerCLI Script Modification

You can edit your Backup VDS PowerCLI script to fit your needs. My modification started with a few variables which are automatically incorporated into the naming of the backup files.  To keep several versions of the backup files I added the date into the file naming as well. This really helps keep the vds backup files organized by name and date.  You will need a vCenter connection in the script.  If you want to utilize mine called Get-vCenter (It is very handy in a multi-vCenter environment) please feel free.

Backup VDS Code

For complete automation create a static array for all your vCenters and loop them. Then you nest this script in another loop.  Utilize Task Scheduler on a Windows computer to call the script on a schedule.  Then you have automated backups of all your VDS switches.

Caution: Like all code you download from the internet, please understand and modify the code accordingly to prevent unforeseen production problems.  Also known as career-altering events. 

 For this script the main command is Export-VDSwitch. I like to familiarize myself with any PowerCLI commands I am running in my environment.  The quickest way to get help and examples is to run from a Powershell prompt the command ‘help verb-noun -showwindow’.  In this case it would be help Export-VDSwitch -showwindow.

vCenter Server 6.7.0a Resolved Issues

Resolved Issues vCenter 6.7.0a

It is a good time to start planning your next upgrade so here are the highlights of all the Resolved Issues in 6.7.0a

Resolved Installation and Upgrade Issues

  • Upgrade to vCenter Server 6.7 might fail with an error 
  • The GUI installer might display one and the same error message caused by firstboot failures during stage 2 of the vCenter Server Appliance upgrade
  • Installation or upgrade to vCenter Server 6.7 might fail with an error “Failed to register Auto Deploy” during firstboot

Resolved vSphere Web Client and vSphere Client Issues

  • The vSphere Web Client might stop responding when you try to log in
  • You might not be able to disable the deduplication or encryption features by using the Allow Reduced Redundancy option in the Configure vSAN wizard in the vSphere Client
  • In a deployment without Internet connection, the VMware vSAN health service might alert that the Hardware Compatibility List (HCL) database is out of date

For more information including patches and isos for 6.7.0a please click the source link or picture below.

Source: VMware vCenter Server 6.7.0a Release Notes

Planning

4 Months Till vSphere 5.5 Support Ends

As usual VMware magnanimously offers to provide extended support for a ‘small’ purchase price.

“In the event you are unable to upgrade before the End of General Support (EOGS) and are active on Support and Subscription, you have the option to purchase extended support in one year increments for up to two years beyond the EOGS date. Visit VMware Extended Support for more information.”

If support is just a best-case scenario and you can stomach going it alone (with Google searching) then Technical Guidance with be available for another year until September 19, 2020.

Google Support
Google Support

“Technical Guidance for vSphere 5.5 is available until September 19, 2020 primarily through the self-help portal. During the Technical Guidance phase, VMware does not offer new hardware support, server/client/guest OS updates, new security patches or bug fixes unless otherwise noted. For more information, visit VMware Lifecycle Support Phases.”

Technical Support is always a good option to have and for all of you knowledge seekers it is a great way to learn the nuances of VMware products.  See my previous post Tech Support is Not a Dirty Word for my take.