VMware vSphere 6 API Limitation when querying performance Data

Recent vCenter versions (vSphere 5.5 Update 2d) have implemented some limitation when is read performance data. You might experience it when you have large number of VMs (more than 64 VMs). It results in all VMs graphs of MEM, DISK, NET, SWAP and COMP are all blank in the UI.

You can see in LPAR2RRD error log logs/error.log:
Fault string: This operation is restricted by the administrator - 'vpxd.stats.maxQueryMetrics'. Contact your system administrator.
vCenter reports it in vpxd.log:
The query size of N metrics exceeded the config.vpxd.stats.maxQueryMetrics limit of 64 metrics. Dropping.
The amount of 64 is the default value, so it should be always the same as long as you did not change it.
You can find the vpx.log either in %ALLUSERSPROFILE%VMWarevCenterServerlogsvmware-vpxvpxd.log on your windows based vCenter Server or in /var/log/vmware/vmware-vpx/vpxd.log on your vCenter Server Appliance

What it causes?

It is all about a new "Feature" in VMwares new vSphere 6.0. To be more specific it is located in the vCenter and it is a setting that limits the maximum performance metric database queries to a threshold of 64. Which is fairly low when you are planning to collect performance data for your whole vSphere. As VMware says on their knowledgebase this features should protect your vCenter from getting spammed by those requests, because that kind of spam could slow/shut down the performance of the vCenter.
VMware KB description.

How to fix it?

This "Feature" is controlled via two settings on the VMware vCenter (Appliance).

1. Necessary changes to the vSphere 6 API limitation can be found within the advanced configuration setting called config.vpxd.stats.maxQueryMetrics
  • Login to your vCenter Server using the the Webclient
  • Go to the Advanced Settings of the vCenter
    (vCenter Inventory Lists -> vCenter Servers -> Select the vCenter -> Manage Tab -> Settings -> Advanced Settings)
    maxQueryMetrics example 1
  • Make sure that the key config.vpxd.stats.maxQueryMetrics does not exist (per default this key it not there and that means it is hidden set to 64 by default)
    maxQueryMetrics example 2
  • Click "Edit" to add the new key
  • Add the key config.vpxd.stats.maxQueryMetrics with the value of your choice. Set -1 to simply delete the limit (or at least number of your VMs with some reserve for future grow)
  • Click ok to reconfigure the vCenter
  • Check if the key is shown
    maxQueryMetrics example 3

2. To change the second parameter you have to edit the web.xml file. You can find it on the windows based vCenter Server at C:ProgramDataVMwarevCenterServerdataperfchartstc-instancewebappsstatsreportWEB-INFweb.xml or for the vCenter Server Appliance at /usr/lib/vmware-perfcharts/tc-instance/webapps/statsreport/WEB-INF/web.xml .

Description of the change on vCenter Server Appliance (changing the windows web.xml should be pretty self explaining)
  • Open a remote session to your vCenter in one of below ways:
    • Use ssh (if its enabled)
    • open straight a console session to it. To open VMware console just press Alt+F1 and enter your root-credentials (this is not the Administrator-account you use to administrate your vSphere, it is really root!)
  • Enable BASH-access by typing: shell.set -enabled True
  • Launch BASH by typing: shell
  • Head to the directory using: cd /usr/lib/vmware-perfcharts/tc-instance/webapps/statsreport/WEB-INF/
  • Open the file using vi: vi ./web.xml
  • Search for "maxQuery" by typing: /maxQuery
  • Head to the <param-value>64</param-value>
  • Hit "i" to edit, change to value to the value of your choice or 0 for unlimited (or at least number of your VMs with some reserve for future grow)
    maxQueryMetrics example 4
  • Press esc-key and ":wq" to write and quit.
  • It should show "written"
  • Logout and head to the Webclient to restart the performance chart service
  • In the WebClient go to the System Configuration
    maxQueryMetrics example 5
  • Click the services, locate the VMware Performance Charts Service and right click + Restart the service
    maxQueryMetrics example 6
Here we go. Your vSphere is now capable to provide you with as much performance metrics database requests as you might need!