Free Server Performance Monitoring


 XORUX

VMware installation

Skip Prerequisites and Web tabs in case of configuring Virtual Appliance

  • LPAR2RRD can be hosted on any Unix or Linux Operating system
  • Rough disk space requirements: 3GB for 100 VMs, 20 datastores, 10 ESXis
  • Perl: arbitrary version coming with the OS with some additional modules, more below
  • Apache or any other Web server
Skip it in case of configuring Virtual Appliance

User creation

  • create lpar2rrd user under root:
    # useradd -c "LPAR2RRD user" -m lpar2rrd
    
  • Increase memory limits for user lpar2rrd and the WEB server user (under root)
    • AIX
      # chuser  data=-1  stack=-1 rss=-1 lpar2rrd
      # chuser  data=-1  stack=-1 rss=-1 apache
      
    • Linux
      # vi /etc/security/limits.conf
      
      @lpar2rrd        hard    stack           -1
      @lpar2rrd        soft    stack           -1
      @lpar2rrd        hard    data            -1
      @lpar2rrd        soft    data            -1
      @apache          hard    stack           -1
      @apache          soft    stack           -1
      @apache          hard    data            -1
      @apache          soft    data            -1
      
If you have already running STOR2RRD on the same host then skip to Web tab.

Linux RedHat, CentOS

    # yum install perl rrdtool rrdtool-perl httpd
    # yum install epel-release ed bc sharutils
    # yum install perl-TimeDate perl-XML-Simple perl-XML-SAX perl-XML-LibXML perl-Env perl-CGI perl-Data-Dumper perl-LWP-Protocol-https perl-PDF-API2 perl-libwww-perl libxml
    
    Note that rrdtool-perl and epel-release (this is necessary only for PDF reporting) might not be in your RedHat base repository especially for RHEL 6.x and olders.
    Install CentOS package instead rrdtool-perl-1.4.8-9.el7.x86_64.rpm and epel-release-7-9.noarch.rpm

Linux Debian, Ubuntu

    # umask 0022
    # apt-get install apache2 perl rrdtool ed bc
    # apt-get install librrdp-perl libxml-sax-perl libxml-simple-perl libtimedate-perl libenv-sanctify-perl libcgi-pm-perl libdata-dumper-simple-perl libpdf-api2-perl libxml2-utils
    # apt-get install liblwp-protocol-https-perl=6.04-2 libcrypt-ssleay-perl libio-socket-ssl-perl libmozilla-ldap-perl libxml-parser-perl
    
    Other available versions like 6.07-2, 6.06-2 or 6.02-1 have a bug which prevents working it with vCenter 6.7+.
    Only Debian based Linux distributions are affected.
    If you do not have access to the internet then get it from liblwp-protocol-https-perl 6.04-2
    Prevent updating it in operating system
    # apt-mark hold liblwp-protocol-https-perl
      liblwp-protocol-https-perl set on hold.
    

AIX

    Install RRDTool 1.4.8 with all prerequisites for AIX
    If you use yum on AIX then do not use rrdtool-1.7.0-4, it has a bug, use rrdtool-1.7.0-3 instead

Solaris

    # /opt/csw/bin/pkgutil -y -i pm_xml_simple pm_libxml_perl pm_xml_sax rrdtool
    
Skip it in case of configuring Virtual Appliance

Apache download

  • Linux CentOS, RedHat
    # yum install httpd
    
  • Linux Debian, Ubuntu
    # apt-get install httpd
    

Apache configuration

Apache start / restart

    Command depends on Apache version, one of these should work:
    # apachectl restart
    # /opt/freeware/apache/sbin/apachectl restart
    # /etc/init.d/httpd restart
    # service apache2 restart
    

Test web pages

    Point your web browser to http://<your lpar2rrd host>/lpar2rrd/ as soon as you configure LPAR2RRD
    Do not forget to ensure that Apache starts after the OS reboot (there must be a starting script in /etc/rc.... )

Apache configuration per platform and version

  • Linux CentOS, RedHat


    Append at the end of /etc/httpd/conf/httpd.conf following
    # vi /etc/httpd/conf/httpd.conf
    
    AddHandler cgi-script .sh
    # DocumentRoot  "/home/lpar2rrd/lpar2rrd/www/"
    Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
    <Directory "/home/lpar2rrd/lpar2rrd/www/">
        AllowOverride AuthConfig
        Options Indexes FollowSymLinks
        Require all granted
    </Directory>
    # CGI-BIN
    ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
    <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
        AllowOverride AuthConfig
        SetHandler cgi-script
        Options ExecCGI FollowSymLinks
        Require all granted
    </Directory>
    
  • Linux Debian, Ubuntu : Apache2 (2.4+)


    Enable CGI-BIN
    # a2enmod cgi
    
    Append this at the end of /etc/apache2/apache2.conf
    # vi /etc/apache2/apache2.conf
    
    AddHandler cgi-script .sh
    Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
    <Directory "/home/lpar2rrd/lpar2rrd/www/">
        Options Indexes FollowSymLinks 
        Require all granted
    </Directory>
    # CGI-BIN
    ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
    <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
        AllowOverride None
        Options +ExecCGI +SymLinksIfOwnerMatch
        Require all granted
        SetHandler cgi-script
    </Directory>
    
  • Apache 2.4+ generally


    Configure httpd.conf, it might be in different locations (depends of distribution)
    • /etc/httpd/httpd.conf
    • /etc/httpd/conf/httpd.conf
    • /etc/opt/freeware/apache/httpd.conf (AIX)
    • /opt/freeware/etc/httpd/httpd.conf (AIX)
    • /opt/freeware/etc/httpd/conf/httpd.conf (AIX)

    1. Verify alias_module and cgi_module is loaded (uncommented it, the path can be different)
      LoadModule alias_module /opt/freeware/lib/httpd/modules/mod_alias.so
      LoadModule cgi_module /opt/freeware/lib/httpd/modules/mod_cgi.so
      
    2. cgi_module could depend on mpm_prefork_module like here.
      Allow it as well (uncomment it) and disable mpm_worker_module, mpm_event_module
      LoadModule mpm_prefork_module /opt/freeware/lib/httpd/modules/mod_mpm_prefork.so
      # LoadModule mpm_worker_module /opt/freeware/lib/httpd/modules/mod_mpm_worker.so
      # LoadModule mpm_event_module modules/mod_mpm_event.so
      <IfModule mpm_prefork_module>
           LoadModule cgi_module /opt/freeware/lib/httpd/modules/mod_cgi.so
      </IfModule>
      
    3. Append this at the end of httpd.conf
      AddHandler cgi-script .sh
      Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
      <Directory "/home/lpar2rrd/lpar2rrd/www/">
          Options Indexes FollowSymLinks 
          Require all granted
      </Directory>
      # CGI-BIN
      ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
      <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
          AllowOverride None
          SetHandler cgi-script
          Options ExecCGI FollowSymLinks
          Require all granted
      </Directory>
      
      Sometimes appears necessary to remove "Require all granted" lines in both directives.

  • Apache 1.3 - 2.3


    Configure httpd.conf, it might be in different locations (depends of distribution)
    • /etc/httpd/httpd.conf
    • /etc/httpd/conf/httpd.conf

    # vi /etc/opt/freeware/apache/httpd.conf
    
    Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
    <Directory "/home/lpar2rrd/lpar2rrd/www/">
        Options Indexes FollowSymLinks 
        Order allow,deny
        Allow from all
    </Directory>
    # CGI-BIN
    ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
    <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
        AllowOverride None
        Options ExecCGI FollowSymLinks
        Order allow,deny
        Allow from all
    </Directory>
    
  • SELinux changes


    You need to disable the SELinux service protection to be able to utilize basic web services.
    You must do it in case you get "(13)Permission denied:" in Apache error_log.
    You have 2 options:

    • Disable SELinux only for Apache
      Step 1. Query for the Boolean value you need to change:
      # getsebool -a | grep httpd_dis
        httpd_disable_trans --> off
      
      Step 2. Disable the SELinux protection:
      # setsebool -P httpd_disable_trans=1
      
      Step 3. Verify that the Boolean has changed:
      # getsebool -a | grep httpd_dis
        httpd_disable_trans --> on
      
    • Globally disable SELinux
      # setenforce Permissive
      # vi /etc/sysconfig/selinux
        SELINUX=disabled
      

VMware monitoring diagram

Prerequisites

  • Create a user on each vCenter with read-only role.

    Create an account, it must be the global one (like below lpar2rrd in xorux.com domain)
    VMware user rights

    Assign lpar2rrd@xorux.com user read-only role
    VMware user rights


  • Download Linux VMware Perl SDK in tar.gz format from VMware site.
    It requires free registration at VMware.
    Does not matter if you are on AIX or Linux or if 32bit or 64bit package
    Use Perl-SDK-6.7.0 only in case you have LPAR2RRD v6.00+ in place
    You must use SDK 6.7 when you have vCenter 6.7+
    VMware-vSphere-Perl-SDK-6.7.0
    VMware-vSphere-Perl-SDK-6.5.0

  • Open https (443) port on form communication from LPAR2RRD server to each monitored vCenter
    perl /home/lpar2rrd/lpar2rrd/bin/conntest.pl <vCenter-host> 443  
      Connection to <vCenter-host> on port "443" is ok
    

Installation of VMware SDK from the UI

    You can use this method only on the Virtual Appliance.
    Select previously downloaded Linux VMware Perl SDK, and upload it.
    VMware Perl SDK installation 1

Installation of VMware SDK manually

  • Install script, download VMware-vSphere-Perl-SDK-6.0.0-2503617.x86_64.tar.gz into /tmp
    # su - lpar2rrd
    cd /home/lpar2rrd/lpar2rrd
    ./bin/vmware_install.sh /tmp
      /tmp/VMware-vSphere-Perl-SDK-6.0.0-2503617.x86_64.tar.gz found
      Extracting selected package to /home/lpar2rrd/lpar2rrd/vmware-vsphere-cli-distrib ...
      Installing selected libraries and apps to /home/lpar2rrd/lpar2rrd/vmware-lib ...
      ...
      Continue by define VMware hosts and their credentials
      UI: menu --> VMware --> Configure --> Add credentials
    
  • Test of vCenter connection (the last cmd must print vCenter time):
    Use double quotes around username and password only in this test
    cd /home/lpar2rrd/lpar2rrd
    . etc/lpar2rrd.cfg
    $PERL vmware-lib/apps/connect.pl --version
      vSphere SDK for Perl version: 6.0.0
      Script 'connect.pl' version: 1.0
    $PERL vmware-lib/apps/connect.pl --server <vCenter host> --username "lpar2rrd@your_domain" --password "XXXXX"
      Connection Successful
      Server Time : 2016-02-25T16:28:44.086369Z
    
    If you get output like bellow then follow our this procedures
    Server version unavailable at 'https://<vCenter host>:443/sdk/vimService.wsdl' at /home/lpar2rrd/lpar2rrd/vmware-lib/apps/../VMware/VICommon.pm line 704.
    

Configuration

  • Continue by define VMware hosts (vCenter) in the UI:
    http://your_web_server/lpar2rrd/
    Add credentials for every vCenter if it is not managed by the vCenter
    Settings ➡ VMware ➡ New ➡ place credentials (username in format: lpar2rrd@your_domain)
    Test vCenter connection via "Test" button.

    VMware install

  • Run "Connection test" for each added vCenter to assure that it is ok

  • When you use Virtual Appliance then use "run data load" button and wait a bit.
    Your vcenter appears in a few minutes after Ctrl-F5 (browser refresh). That is all!

  • When you do not use Virtual Appliance then run ./load.sh which starts collecting of VMware data
    cd /home/lpar2rrd/lpar2rrd
    ./load.sh 
    
  • You should see VMware hosts in the UI when load.sh finishes (reload the UI in the browser via Ctrl-F5)

Troubleshooting

  • If you get output like bellow then follow our this procedures
    cd /home/lpar2rrd/lpar2rrd
    . etc/lpar2rrd.cfg
    $PERL vmware-lib/apps/connect.pl --server <server name> --username "lpar2rrd"
      Server version unavailable at 'https://vcenter01:443/sdk/vimService.wsdl' at /lpar2rrd/lpar2rrd/vmware-lib/apps/..//VMware/VICommon.pm line 734.
    
  • Data is not being collected with this error in error.log
    more /home/lpar2rrd/lpar2rrd/logs/error.log
    Thu Feb 18 10:49:01 2016: vmware name: vcenter01 has not array of hosts ?!? : 
    ...
    
    The reason is that user you have configured for retrieving data from the vCenter probably has not read access to whole vCenter and it cannot access all required perf counters.
    Consult user rights setup with your vCenter administrator.

  • Test vCenter connection from the command line:
    1. With entering password manually:
      cd /home/lpar2rrd/lpar2rrd
      . etc/lpar2rrd.cfg
      $PERL vmware-lib/apps/connect.pl --server <vCenter host> --username "lpar2rrd@your_domain" --password "XXXXX"
          Connection Successful
          Server Time : 2016-08-02T06:58:26.355767Z
      
    2. By using saved password already:
      $PERL vmware-lib/apps/connect.pl --server <vCenter host> --username "vCenter read only user" --credstore .vmware/credstore/vicredentials.xml
      
      You should get the same answer with no password request.
      Use the same hostname and username as defined in LPAR2RRD VMware Configuration.

  • In case of a problem check our forum or contact us via support@lpar2rrd.com



OS agent is add-on feature for monitoring from operating system level.
It is monitoring CPU, memory utilization, paging, LAN and SAN traffic on all adapters.
It requires the OS agent deployment to every monitored VM.
The agent is written in Perl and calls basic OS commands to obtain required statistics like vmstat, iostat.

OS agent architecture

Additional information about the OS agent:

Prerequisites

  • Perl
  • Opened TCP communication between each VM and LPAR2RRD server on port 8162.
    Connections are initiated from VM side.
  • Additional disk space on LPAR2RRD server (about 40MB per each monitored VM)
  • Create preferable dedicated user lpar2rrd on each VM with minimum rights
    # useradd -c "LPAR2RRD agent user" -m lpar2rrd
    

OS agent installation (client)

  • Get the latest OS agent from download page

  • Linux installation under root
    # rpm -Uvh lpar2rrd-agent-5.00-0.noarch.rpm
    # rpm -qa|grep lpar2rrd-agent
      lpar2rrd-agent-5.00-0
    
  • Solaris x86 installation under root:
    # gunzip lpar2rrd-agent-5.00-0.solaris-i86pc.tar.gz
    # tar xf lpar2rrd-agent-5.00-0.solaris-i86pc.tar
    # pkgadd -d .
      The following packages are available:
      1  lpar2rrd-agent     LPAR2RRD OS agent 5.00
                            (i86pc) 5.00
     ...
    
    Solaris upgrade under root:
    # pkgrm lpar2rrd-agent
    # pkgadd -d .
    
  • Schedule its run every minute from the crontab on every VM.
    This line must be placed into lpar2rrd crontab:
    # su - lpar2rrd
    crontab -e 
    * * * * * /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl <LPAR2RRD-SERVER> > /var/tmp/lpar2rrd-agent.out 2>&1
    
    Replace <LPAR2RRD-SERVER> by hostname of your LPAR2RRD server.

  • You might need to add lpar2rrd user into /var/adm/cron/cron.allow under root user if above "crontab -e" fails.
    # echo "lpar2rrd" >> /var/adm/cron/cron.allow
    

LPAR2RRD server (daemon)

  • Edit etc/lpar2rrd.cfg and set following (if it is not already set):
    vi /home/lpar2rrd/lpar2rrd/etc/lpar2rrd.cfg
    
    LPAR2RRD_AGENT_DAEMON=1
    
  • The daemon is started when load.sh starts
    ./load.sh
      Starting LPAR2RRD daemon on port:8162
      ...
    
  • Assure it is running and listening on port 8162:
    ps -ef|grep lpar2rrd-daemon
      lpar2rrd 10617010 1 0 Mar 16 - 0:00 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/lpar2rrd-daemon.pl
    netstat -an| grep 8162
      tcp4  0  0  *.8162   *.*   LISTEN
    
  • OS agent data graphs will appear in the UI, use Ctrl-F5 to refresh your web browser

Troubleshooting

  • Client (agent) side:
    • Test if communication through the LAN is allowed.
      telnet  <LPAR2RRD-SERVER> 8162
        Connected to 192.168.1.1   .
        Escape character is '^]'.
      
      This is ok, exit either Ctrl-C or ^].

    • Check following agent files:
      data store: /var/tmp/lpar2rrd-agent-*.txt
      error log: /var/tmp/lpar2rrd-agent-*.err
      output log: /var/tmp/lpar2rrd-agent.out

    • run the agent from cmd line:
      /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl -d <LPAR2RRD-SERVER>
        ...
        Agent send     : yes : forced by -d 
        Agent send slp: sending wait: 4
        OS/HMC agent working for server: <LPAR2RRD-SERVER>
        store file for sending is /var/tmp/lpar2rrd-agent-<LPAR2RRD-SERVER>-lpar2rrd.txt
      
      It means that data has been sent to the server, all is fine
      Here is example when the agent is not able to sent data :
      /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl -d <LPAR2RRD-SERVER>
        ...
        Agent send     : yes : forced by -d 
        Agent send slp: sending wait: 1
        OS/HMC agent working for server: <LPAR2RRD-SERVER>
        store file for sending is /var/tmp/lpar2rrd-agent-<LPAR2RRD-SERVER>-lpar2rrd.txt
        Agent timed out after : 50 seconds /opt/lpar2rrd-agent/lpar2rrd-agent.pl:265
      
      It means that the agent could not contact the server.
      Check communication, port, above telnet example, DNS resolution of the server etc.

  • Server side:
    • test if the daemon on LPAR2RRD server is running, and checking the logs
      ps -ef|grep lpar2rrd-daemon
        lpar2rrd 10617010 1 0 Mar 16 - 0:00 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/lpar2rrd-daemon.pl
      cd /home/lpar2rrd/lpar2rrd
      tail logs/error.log-daemon
      tail logs/daemon.out
        new server has been found and registered: Linux (lpar=linuxhost01)
        mkdir : /lpar2rrd/data/Linux/no_hmc/linuxhost01/
      
      It means that new OS agent has been registered from linuxhost01 (Linux stand-alone example)

    • Test if OS agent data is being stored on the LPAR2RRD server and have actual timestamp:
      cd /home/lpar2rrd/lpar2rrd
      ls -l data/<server name>/*/<VM name>/*mmm
        -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/cpu.mmm
        -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/lan-en1.mmm
        -rw-r--r-- 2 lpar2rrd staff 10790264 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/mem.mmm
        -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/pgs.mmm
        -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/san-vscsi0.mmm
        -rw-r--r-- 2 lpar2rrd staff  3597208 Mar 17 16:16 data/<server name>/no_hmc/<VM name>/san_resp-vscsi0.mmm
      find data -name mem.mmm -exec ls -l {} \;
        ...
      
  • In case of a problem check our forum or contact us via support@lpar2rrd.com.
    We would need this data for start of troubleshooting.

Notes

    LPAR2RRD agent is available since 3.56 release.
    You will not need to upgrade LPAR2RRD agents regularly with each LPAR2RRD upgrade.
    Read release notes if that is necessary.
    Check OS agent upgrade steps.

Skip it in case of configuring Virtual Appliance

Install LPAR2RRD server

  • Download the latest LPAR2RRD server
    Upgrade your already running LPAR2RRD instance.

  • Install it:
    # su - lpar2rrd
    tar xvf lpar2rrd-6.XX.tar
    cd lpar2rrd-6.XX
    ./install.sh
    cd /home/lpar2rrd/lpar2rrd
    
  • Schedule to run it from lpar2rrd crontab (it might already exist there)
    crontab -l | grep load.sh
     
    
    Add if it does not exist as above
    crontab -e
    
    # LPAR2RRD UI
    0 * * * * /home/lpar2rrd/lpar2rrd/load.sh > /home/lpar2rrd/lpar2rrd/load.out 2>&1 
    
    Assure there is just one such entry in crontab.

  • You might need to add lpar2rrd user into /var/adm/cron/cron.allow if crontab command fails
    Allow it for lpar2rrd user as root user.
    # echo "lpar2rrd" >> /var/adm/cron/cron.allow
    
  • Initial start from cmd line:
    cd /home/lpar2rrd/lpar2rrd
    ./load.sh
    
  • Go to the web UI: http://<your web server>/lpar2rrd/
    Use Ctrl-F5 to refresh the web browser cache.