The Importance of VCSA Services


This has been cross posted from my own blog Go check it out.

This week I was trying to deploy an OVA on a vCenter and it was throwing strange error messages. I chalked it down to the Flash client being dumb, but then I could not add a new Role for some permissions.

Once I started I discovered the issue it reminded me the importance of checking the basics.

This was on a VCSA 6.5 Update 1 with an external PSC.


As I said above the first sign of an issue was on the deployment of an OVA. In the Flash client I kept getting this error when selecting Deploy OVF Template:

This version of vCenter Server does not support Deploy OVF Template using this version of the vSphere Web Client. To Deploy OVF Template, login with version of vSphere Web Client.

OVA Deployment Error

I tried then using the HTML5 client and it would just sit at Validating at step three in the wizard:

OVA Deployment Error

I had a look at the VMware Knowledgebase and found KB2151085 but this was originally a new VCSA and had not been upgraded. I ran out of time to investigate further so ended up using PowerCLI to deploy the OVA I needed.

The next day I needed to create a new Role in vCenter and encountered further issues. The first was when I went to the Roles screen in the Flash client. It said I did not have permissions. I then tried using the SSO admin and had the same:

Roles Error

Again I checked the HTML5 client and found an empty screen:

Roles Error

As a final check I went to an existing folder and tried to add an existing permission to the folder. Going to the Add Permission wizard in the Flash client I got the error:

An internal error has occurred – Error #1034

Roles Error

and the error stack showed:

Roles Error

Googling Error #1034 showed a couple of entries and they fixed it by rebooting the appliance.

The Fix

At this point I knew I had a serious problem. My first thought would be to reboot the appliance but this would have involved a Change ticket with approvals and an out of hours reboot which I wanted to avoid.

I spent time wondering if there was a permissions error but I was trying the SSO local administrator account which has access to everything. I spent some time looking at the SSO configuration and the Groups but could not find the issue. I could rule out permissions as the cause.

Next I spent time looking at KB2151085 from above. I performed the steps listed even though I knew is was a bigger issue but wanted to rule it out. Of course that didn’t help.

At this point it was lunch time. As many of you know there is nothing better to help you figure out a problem than walking away from it for a while. So I went for a walk! When I got back I decided to get back to basics and check on the health of the appliance.

I checked the following in the VAMI:

  1. Health Status (all green)
  2. NTP status
  3. DNS setup
  4. Database utilization (had issues when VCSA disk partitions fill up)

That all looked good.

Then I moved to the vCenter UI. I had a look at the System Configuration under Administration and noticed in the Services Health a Warning:

Roles Error

Clicking on the Objects tab I could see the VMware vCenter Serverservice was in Warning. Looking further down the list I noticed something. The service vAPI Endpoint was stopped:

Roles Error

I knew the vAPI Endpoint is a critical vCenter service. Checking the properties of the the service showed it should have been running as the Startup Type was Automatic:

Roles Error

I started up the service and went back to the Roles screen using my domain account and could see Roles now:

Roles Error

I then check a simple OVA deployment and it worked.

Wrap Up

This taught me a lesson. I spent too long on Google and searching for a complex error and fix. I need to start with the basics first.

I also took note in our wiki of the services that should be running on the vCenter and PSC so that others not as familiar will know what should and should not be running.

I also wanted to blog this as the specific error codes matched other issues and KB articles that were not relevant. If anyone else hits similar problems I hope they land on this post and it helps them.

VMware Fling – DRS Dump Insight H5 Plugin

This is a cross post from my own site:

I was recently reading through fellow vExpert Wouter Kurten‘s “The VMware Labs flings monthly for September 2018” post and one of the flings in the post piqued my curiosity.

If you’re not familiar with VMware Flings, then head over to and have a look around.

“Flings are apps and tools built by our engineers and community that are intended to be played with and explored.”

The fling in question is the “DRS Dump Insight H5 Plugin”  , so I decide to get it up and running in my Ravello Cloud Lab.


a.  Head over to

b. check out the requirements. Note this is fling is only compatible with VCSA 6.5/6.7 and not the HTML5 Client Fling. Continue reading

vSphere 6.5 Update 1 Security Configuration Guide Released

This has been cross posted from my own blog Go check it out.


On the 12th March 2018 VMware released the latest version of the vSphere Security Configuration Guide. This is an indispensable guide for securing your vSphere infrastructure which I highly recommend all VMware admins read.


I have been following the guide for a few iterations now. Back in the early versions there were a lot of settings that could mean the over zealous administrator could have gone in and potentially caused problems. For example in the v5.1 version of the guide there were 172 settings listed over multiple sheets. In the latest version there are 68. A couple of reason for this are the mitigation change has been eradicated due to code changes or the guidance is no longer required because the software is secure by default.

Also included are some common sense ‘best practices’. This goal of secure by default can be seen in the graphs in the blog post from VMware. In vSphere 6.5 there were 24 settings available to harden the deployment. In 6.5 Update 1 there are now 10 due to VMware coding the guidelines into the code. So for that 68 Guidelines 10 are Hardening settings with 58 Non-Hardening (Audit only + Site Specific). Great job VMware! Continue reading

VMware vSphere: Optimize and Scale [V6] – On Demand Review

This has been cross posted from my own blog Go check it out!

I recently was able to take the VMware vSphere: Optimize and Scale [V6] – On Demand course from VMware. Why On Demand and not in a Classroom format? Simple – travel time and costs. I was actually looking for the Design & Deploy Fast Track course but annoyingly it seem to be scheduled very infrequently and only in London. With family and work commitments taking a week out to attend was pretty impossible.

So I started looking at the On Demand option. I was scheduled to take the VCAP6-DCV Deploy exam so the O&S On Demand course seemed like a good fit. This was my first time trying an On Demand course instead of Instructor led in class training. The interface is based on the Hands on Labs so if you are familiar with that you will be comfortable using it. The modules covered were: Continue reading

Terraform with vSphere – Part 2

This has been cross posted from my own blog Go check it out!


In Part 1 of this series we went about installing Terraform, verifying it was working and setting up Visual Studio Code. In this part we will cover some Terraform basics.

Terraform Components

The three Terraform Constructs we are going to look at are:

  • Providers
  • Resources
  • Provisioners

Providers are the resources or infrastructure we can interact with in Terraform. These can include AWS, Azure, vSphere, DNS, etc. A full list is available on the Terraform website. As you can see it’s a very big list. In this series we will concentrate on the VMware vSphere Provider.


Resources are the things we are going to use in the provider. In the vSphere realm this can be a Virtual Machine, Networking, Storage, etc.


Terraform uses Provisioners to talk to the back end infrastructure or services like vSphere to create your Resources. They essentially are used to execute scripts to create or destroy resources.

Setup Terraform for vSphere

Open up Visual Studio Code and create a new file called main.tfin the folder C:\Terraform. If you have added C:\Terraform to your Path environment variable save anywhere you like, but of course the best place for all of your Terrform files is source control…

Continue reading

VCAP6-DCV – So good I thought I’d take it twice

I’d been thinking about taking the plunge on the VCAP for a while, truth be told, thinking about it was pretty much all i’d done. So at the start of August i booked my VMworld ticket and decided to add on a VCAP exam voucher. My thinking was follow the advice of so many before me, set the date and with the impending deadline that would be enough to get my ass into gear.

So i received my voucher, scheduled my exam date, then promptly quit my job….it all happened swiftly and I didn’t really see it coming. So rather than spending four weeks dedicating myself to my study i spent a frantic four weeks trying to tidy up a number of outstanding projects before beginning another chapter of my career. So as far as exam preparation goes it couldn’t have gone much worse.

I found time to watch some vBrownbag design sessions, i finished half of Foundation in the Art of Infrastructure Design, i read a number of blog articles about what to expect but only managed a fraction of the study that i’d have ideally done.

The day before the exam I spent most of the day trying some last minute cramming but truly felt by that point the damage had been done. I didn’t feel there was much more i could take in so late in the day and that night i was joking with people about how i was failing a VCAP exam the following day.

As i entered the exam i felt pretty lethargic, the prospect of 3.5 hours doing an exam that i didn’t think i had much chance of passing filled me with apathy but well i was there now with nothing better to do. First question was a drag and drop, and to be honest i felt completely at home i knew the topic and promptly rattle off an answer.

I read enough guides to know most people suggest a strategy, dependant on your strengths do all the questions first, leaving all the time for the designs etc. After question one, all my strategising went out the window, i started to answer the second Drag and Drop but it was more complicated and i thought this will take a bit more thought, so i flagged the answer and moved on.

I did this for the next 16 questions, answering any quick hitters and skim reading some of the designs. There was no rhyme and reason to my strategy i just wanted to know what i was up against. When i got to the end i went back through the questions in numerical order with a similar mindset, if it looked “easier” i’d tackle it otherwise i’d move on.

After a couple of passes i’d done 1/3 of the designs and most of the drag and drops. I was about 90 minutes into the allotted time and about two thirds of the way through the exam. Obviously i’d now picked off all the low hanging fruit and i was left with everything that looked either tough or terrifying. I worked my through the remaining Drag and Drops, some i found really ambiguous so was battling internally with the correct answers..

In fact this was probably my biggest issue with the exam as a whole, there were certain answers where i could quite clearly see two schools of argument. For some of them i really felt like i could argue the case for two correct answers, obviously the exam isn’t graded that way but that was what felt so tough. Two answers look right, which one is most right, or more importantly more right in VMware’s eyes?? A customer has enough physical 10GB interfaces for virtual interface requirements should they use physical or VLAN separation? It depends isn’t an acceptable answer

All that remained were the 3 most complicated designs. I battled through them as best i could. One design alone took me in excess of an hour, it was about a physical/vDS design, with port groups and a LAG. I found it incredibly tough, it was a complicated design and there was lots I wasn’t 100% about. Anyway it got to the point where i couldn’t look at it anymore and clicked submit. I submitted about the 3 hour mark so despite everyone’s warnings i didn’t really find time that much of a constraint. “Sorry you have not passed”.

Prior to the exam i was fully prepared to fail and was just expecting to take the experience as a learning opportunity, now despite my lack of prep, as i was about to press the submit button, i genuinely felt I’ve got a chance here.  And it turns out i had,  i was so close to passing that, rather than accepting the expected defeat i was absolutely gutted. 20 odd points, that may just have been 1-2 questions, as is typical with VMware exams all you’re left with is a pretty useless vague list of things to get better at before trying again.

I was pretty downbeat and went to the pub to meet up with some friends. This was when my luck changed, in the pub i bumped into Kyle Jenner (who has an outstanding VCAP study guide on his blog) who i’d only met for the first time the night before. He knew i was sitting the VCAP so we talked about it. Anyway it turned out a lot of my experience married his first attempt, we were able to talk through some of the designs and he helped me see where i’d made some of my mistakes.

By the end of the night i felt pretty good about it again, i took a shot with little prep and got bloody close. That meant i was on the right track, my actual real world experience had got me within touching distance of a pass. A bit more discipline, a bit more study (especially vSan as it came up a couple of times and i’ve never been hands on with it) and i’d be ready to take a second crack.

to be continued…