Free performance monitoring for VMware™ and IBM Power Systems™

Installation

To install LPAR2RRD follow all tabs from the left to the right.
Follow Virtual Appliance installation in case of usage of Virtual Appliance

  • LPAR2RRD can be hosted on any Unix or Linux Operating system
  • Rough disk space requirements:
    • IBM Power edition: 6GB for 100 LPARs with OS agents
    • IBM i OS agent: 170MB per each IBM i OS agent
    • VMware edition: 3GB for 100 VMs, 20 datastores, 10 ESXis
  • Perl: arbitrary version coming with the OS with some additional modules, more below
  • Apache or any other Web server
  • RRDTool: use RRDTool v1.3.5+ to allow graph zooming feature
If you have already running STOR2RRD on the same host then skip to Web server configuration

Software prerequisites

  • Apache
  • RRDTool with Perl support (rrdtool & rrdtool-perl)
  • Perl TimeDate module
  • Perl XML::Simple module
  • Perl PDF::API2 for PDF reporting

Linux

Install RRDTool, Apache and Perl on Linux like below (under root):
  • Linux RedHat, CentOS
    # yum list | egrep "httpd|rrdtool|perl\.|perl-TimeDate|perl-XML-Simple|perl-XML-SAX|CGI\.|perl-Env\.|sharutils"
        httpd.i386                             2.2.15-15.el6
        perl.i386                              4:5.10.1-119.el6_1.1
        rrdtool.i386                           1.2.30-1.wrl
        rrdtool-perl.i386                      1.2.30-1.wrl
        rrdcached.i386                         1.2.30-1.wrl
        perl-TimeDate.noarch                   1:1.16-11.1.el6
        perl-XML-SAX.noarch                    0.96-7.el6
        perl-XML-Simple.noarch                 2.14-8.el6
        perl-CGI.i386                          3.51-119.el6_1.1
        perl-Env.noarch                        1.04-2.el7
    # umask 0022 
    # yum install rrdtool rrdtool-perl httpd
    # yum install epel-release
    # yum install perl perl-TimeDate perl-XML-Simple perl-XML-SAX perl-XML-LibXML perl-Env perl-CGI perl-Data-Dumper ed bc perl-PDF-API2 sharutils
    
    Note that rrdtool-perl package might not be in your latest RedHat base repository. Install CentOS package instead http://mirror.centos.org/centos/7/os/x86_64/Packages/rrdtool-perl-1.4.8-9.el7.x86_64.rpm

  • Linux Debian, Ubuntu
    # umask 0022 
    # apt-get install apache2 perl rrdtool librrdp-perl libxml-sax-perl libxml-simple-perl libtimedate-perl libenv-sanctify-perl libdata-dumper-simple-perl libcgi-pm-perl ed bc libpdf-api2-perl
    
    When you have done that then skip to Web server configuration
    When you do not use yum or apt-get then go for manual RPM installation how is described in AIX section

AIX

Skip it in case of VMware installation

Allow data collection on the HMC

    Allow utilization data collection (either via the HMC GUI or globally via following cmd as hscroot)
    $ ssh hmc1 -l hscroot 
    hscroot@hmc1:~> chlparutil -r config -s 60
    hscroot@hmc1:~> lslparutil -r config  -F name,sample_rate
    PWR6A-9117-MMA-SN103A4B0,60
    PWR6B-9117-MMA-SN103B5C0,60
    
    Use padmin account for IVM, or sysadmin for SDMC/FLEX
    Repeat that for all your HMCs/IVMs/SDMCs.

Allow password less access to the HMC

    Create ssh keys (type enter when you are prompted for passphrase) under lpar2rrd user on LPAR2RRD server
    $ who am I
      lpar2rrd    pts/0       Jan 10 10:13     (192.168.1.240)
    $ ssh-keygen -t rsa
      Generating public/private rsa key pair.
      Enter file in which to save the key (/home/lpar2rrd/.ssh/id_rsa):
      Enter passphrase (empty for no passphrase):
      Enter same passphrase again:
      Your identification has been saved in /home/lpar2rrd/.ssh/id_rsa.
      Your public key has been saved in /home/lpar2rrd/.ssh/id_rsa.pub.
      The key fingerprint is:
      c0:05:85:ff:9c:90:39:da:d7:48:33:35:4f:cf:1a:21 lpar2rrd@p550-2  
    
  • HMC


    create a HMC account with role hmcviewer on the HMC server
    $ ssh -l hscroot hmc1
    hscroot@hmc1:~> mkhmcusr -u lpar2rrd -a hmcviewer --passwd abc1234
    
    Use already created user ssh-key for access configuration to the HMC:
    $ cat /home/lpar2rrd/.ssh/id_rsa.pub 
    'ssh-dss AAAAB3NzaC1kc3MAAACBALvbupPLnqy6UfJjgMG5SRnnFDfD013OtBxFv8P7qoHfGKCG0Vu2
    IhNCYttpYMmsMR+BWADo4c9oT7r92raLfVDjNW8uO5C5fatK305+sNqazbT91HDDNDKQnbpoKUqybVRC2
    BQbPR8ESh+ws3uHdxXSQKwOSyrHO6Nwkkx8/h4TAAAAFQDZgT9MxrEUQg4uOwxhciwwdoOO8QAAAIAH37
    x06Ia4FiCdlk3U9vuOI0QEvCKMXL+ZPFNoRcgiqiCnR2WeiaG5qM+odYWk/F/owV0Y/DmkmrrjPrgDADj
    S1uKoSy+NMg803+4cS1B06EMtia+RmMbWkr+kQZ90WvB8C5cxofzcllnQjztQxVrZnZPmMI73/SQdS7QB
    U9a9WwAAAIA3VAdMvP+ZO3zs57FaznMySoszxgNWxuINlcP61TQuyuCp5Bq3mFXwxFwJWPpqpbW5yHj9N
    1+3pscSsFxEQb9YVvb1oL9c5QdPDnmXZK8BHHmNZMn+ftuzJKHgR2lynaxQ4hxTnz+xT0ywGwokeuuCnr
    4G3Bftc70Yn9vCNjuzGQ== lpar2rrd@p550-2'
    
    Paste the pub key just created to mkauthkeys command on the HMC
    (as hscroot, all in one line, without carriage returns):
    hscroot@hmc1:~> mkauthkeys -u lpar2rrd -a 'ssh-dss AAAAB3NzaC1kc3MAAACBALvbupPL
    nqy6UfJjgMG5SRnnFDfD013OtBxFv8P7qoHfGKCG0Vu2IhNCYttpYMmsMR+BWADo4c9oT7r92raLfVDjNW
    8uO5C5fatK305+sNqazbT91HDDNDKQnbpoKUqybVRC2BQbPR8ESh+ws3uHdxXSQKwOSyrHO6Nwkkx8/h4T
    AAAAFQDZgT9MxrEUQg4uOwxhciwwdoOO8QAAAIAH37x06Ia4FiCdlk3U9vuOI0QEvCKMXL+ZPFNoRcgiqi
    CnR2WeiaG5qM+odYWk/F/owV0Y/DmkmrrjPrgDADjS1uKoSy+NMg803+4cS1B06EMtia+RmMbWkr+kQZ90
    WvB8C5cxofzcllnQjztQxVrZnZPmMI73/SQdS7QBU9a9WwAAAIA3VAdMvP+ZO3zs57FaznMySoszxgNWxu
    INlcP61TQuyuCp5Bq3mFXwxFwJWPpqpbW5yHj9N1+3pscSsFxEQb9YVvb1oL9c5QdPDnmXZK8BHHmNZ{{M
    n+ftuzJKHgR2lynaxQ4hxTnz+xT0ywGwokeuuCnr4G3Bftc70Yn9vCNjuzGQ== lpar2rrd@p550-2'
    
  • SDMC/FLEX


    create lpar2rrd user under sysadmin account with role SMMonitor:
    $ ssh -l sysadmin smdc1
    sysadmin@sdmc:~> smcli mkuser -u lpar2rrd -g smmon -p abc1234
    sysadmin@sdmc:~> smcli chuser -u lpar2rrd -e expire 9999 # do not let it expire
    sysadmin@sdmc:~> mkauthkeys -u lpar2rrd -a 'ssh-dss AAAAB3NzaC1kc3MAAACBALvbupPL
    nqy6UfJjgMG5SRnnFDfD013OtBxFv8P7qoHfGKCG0Vu2IhNCYttpYMmsMR+BWADo4c9oT7r92raLfVDjNW
    8uO5C5fatK305+sNqazbT91HDDNDKQnbpoKUqybVRC2BQbPR8ESh+ws3uHdxXSQKwOSyrHO6Nwkkx8/h4T
    AAAAFQDZgT9MxrEUQg4uOwxhciwwdoOO8QAAAIAH37x06Ia4FiCdlk3U9vuOI0QEvCKMXL+ZPFNoRcgiqi
    CnR2WeiaG5qM+odYWk/F/owV0Y/DmkmrrjPrgDADjS1uKoSy+NMg803+4cS1B06EMtia+RmMbWkr+kQZ90
    WvB8C5cxofzcllnQjztQxVrZnZPmMI73/SQdS7QBU9a9WwAAAIA3VAdMvP+ZO3zs57FaznMySoszxgNWxu
    INlcP61TQuyuCp5Bq3mFXwxFwJWPpqpbW5yHj9N1+3pscSsFxEQb9YVvb1oL9c5QdPDnmXZK8BHHmNZ{{M
    n+ftuzJKHgR2lynaxQ4hxTnz+xT0ywGwokeuuCnr4G3Bftc70Yn9vCNjuzGQ== lpar2rrd@p550-2'
    
  • IVM


    Create lpar2rrd user under padmin account with role view:
    $ ssh -l padmin ivm1
    padmin@ivm1:~> mkuser -attr pgrp=view lpar2rrd
    padmin@ivm1:~> passwd lpar2rrd
      Changing password for "lpar2rrd"
      lpar2rrd's New password:
      Enter the new password again:
    padmin@ivm1:~> su - lpar2rrd    # it must be done under the user identification!
    lpar2rrd@ivm1:~> mkauthkeys -a 'ssh-dss AAAAB3NzaC1kc3MAAACBALvbupPLnqy6UfJjgMG5SRnn
    FDfD013OtBxFv8P7qoHfGKCG0Vu2IhNCYttpYMmsMR+BWADo4c9oT7r92raLfVDjNW8uO5C5fatK305+sNq
    azbT91HDDNDKQnbpoKUqybVRC2BQbPR8ESh+ws3uHdxXSQKwOSyrHO6Nwkkx8/h4TAAAAFQDZgT9MxrEUQg
    4uOwxhciwwdoOO8QAAAIAH37x06Ia4FiCdlk3U9vuOI0QEvCKMXL+ZPFNoRcgiqiCnR2WeiaG5qM+odYWk/
    F/owV0Y/DmkmrrjPrgDADjS1uKoSy+NMg803+4cS1B06EMtia+RmMbWkr+kQZ90WvB8C5cxofzcllnQjztQ
    xVrZnZPmMI73/SQdS7QBU9a9WwAAAIA3VAdMvP+ZO3zs57FaznMySoszxgNWxuINlcP61TQuyuCp5Bq3mFX
    wxFwJWPpqpbW5yHj9N1+3pscSsFxEQb9YVvb1oL9c5QdPDnmXZK8BHHmNZ{{Mn+ftuzJKHgR2lynaxQ4hxT
    nz+xT0ywGwokeuuCnr4G3Bftc70Yn9vCNjuzGQ== lpar2rrd@p550-2'
    

Connectivity test

  • test the connection between the LPAR2RRD hosted server and HMC
    (should not request password any more) as lpar2rrd user:
    $ ssh -l lpar2rrd hmc1
      The authenticity of host 'hmc1 (9.138.236.97)' can′t be established.
      RSA key fingerprint is 97:e1:a5:c2:58:3c:c6:0c:d1:6f:4d:6e:62:9c:68:4b.
      Are you sure you want to continue connecting (yes/no)? yes
    lpar2rrd@hmc1:~> 
    
  • if you use a commercial SSH distribution (like F-Secure ssh) which supports only SSH2 format (above example was for OpenSSH format), then you need to convert public keys onto OpenSSH format before uploading the keys into HMC. Convert SSH2 to OpenSSH format and after that use it in mkauthkeys:
    $ ssh-keygen -i -f your-ssh2.pub > your-openssh.pub
    
Skip it in case of configuring Virtual Appliance

Apache download

  • AIX
    Apache 2.4.18 from IBM AIX Toolbox for Linux Applications
    Apache 2.4.4 from M. Perlz's archive.

  • Linux CentOS, RedHat
    # yum install httpd
    
  • Linux Debian, Ubuntu
    # apt-get install httpd
    

Apache configuration

Apache start / restart

    Command depends on Apache version, one of these should work:
    # apachectl restart
    # /opt/freeware/apache/sbin/apachectl restart
    # /etc/init.d/httpd restart
    # service apache2 restart
    

Test web pages

    Point your web browser to http://<your lpar2rrd host>/lpar2rrd/ as soon as you configure LPAR2RRD
    Do not forget to ensure that Apache starts after the OS reboot (there must be a starting script in /etc/rc.... )

Linux SELinux issue

    Linux: disable SELinux if you get "(13)Permission denied:" in Apache error_log
    # setenforce Permissive
    # vi /etc/sysconfig/selinux
      SELINUX=disabled
    

Apache configuration per platform and version

  • Linux CentOS, RedHat


    Append at the end of /etc/httpd/conf/httpd.conf following
    # vi /etc/httpd/conf/httpd.conf
    
    AddHandler cgi-script .sh
    # DocumentRoot  "/home/lpar2rrd/lpar2rrd/www/"
    Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
    <Directory "/home/lpar2rrd/lpar2rrd/www/">
        AllowOverride AuthConfig
        Options Indexes FollowSymLinks Includes MultiViews
        Require all granted
    </Directory>
    # CGI-BIN
    ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
    <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
        AllowOverride AuthConfig
        SetHandler cgi-script
        Options ExecCGI Includes FollowSymLinks
        Require all granted
    </Directory>
    
  • Linux Debian, Ubuntu : Apache2 (2.4+)


    Enable CGI-BIN
    # a2enmod cgi
    
    Append this at the end of /etc/apache2/apache2.conf
    # vi /etc/apache2/apache2.conf
    
    AddHandler cgi-script .sh
    Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
    <Directory "/home/lpar2rrd/lpar2rrd/www/">
        Options Indexes FollowSymLinks Includes MultiViews
        Require all granted
    </Directory>
    # CGI-BIN
    ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
    <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
        AllowOverride None
        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
        Require all granted
        SetHandler cgi-script
    </Directory>
    
  • Apache 2.4+ generally


    Configure httpd.conf, it might be in different locations (depends of distribution)
    • /etc/httpd/httpd.conf
    • /etc/httpd/conf/httpd.conf
    • /etc/opt/freeware/apache/httpd.conf (AIX)
    • /opt/freeware/etc/httpd/httpd.conf (AIX)

    1. Verify alias_module and cgi_module is loaded (uncommented it, the path can be different)
      LoadModule alias_module /opt/freeware/lib/httpd/modules/mod_alias.so
      LoadModule cgi_module /opt/freeware/lib/httpd/modules/mod_cgi.so
      
    2. cgi_module could depend on mpm_prefork_module like here.
      Allow it as well (uncomment it) and disable mpm_worker_module, mpm_event_module
      LoadModule mpm_prefork_module /opt/freeware/lib/httpd/modules/mod_mpm_prefork.so
      # LoadModule mpm_worker_module /opt/freeware/lib/httpd/modules/mod_mpm_worker.so
      # LoadModule mpm_event_module modules/mod_mpm_event.so
      <IfModule mpm_prefork_module>
           LoadModule cgi_module /opt/freeware/lib/httpd/modules/mod_cgi.so
      </IfModule>
      
    3. Append this at the end of httpd.conf
      AddHandler cgi-script .sh
      Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
      <Directory "/home/lpar2rrd/lpar2rrd/www/">
          Options Indexes FollowSymLinks Includes MultiViews
          Require all granted
      </Directory>
      # CGI-BIN
      ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
      <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
          AllowOverride None
          SetHandler cgi-script
          Options ExecCGI Includes FollowSymLinks
          Require all granted
      </Directory>
      
      Sometimes appears necessary to remove "Require all granted" lines in both directives.

  • Apache 1.3 - 2.3


    Configure httpd.conf, it might be in different locations (depends of distribution)
    • /etc/httpd/httpd.conf
    • /etc/httpd/conf/httpd.conf
    • /etc/opt/freeware/apache/httpd.conf (AIX)
    • /opt/freeware/etc/httpd/httpd.conf (AIX)

    # vi /etc/opt/freeware/apache/httpd.conf
    
    Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
    <Directory "/home/lpar2rrd/lpar2rrd/www/">
        Options Indexes FollowSymLinks Includes MultiViews
        Order allow,deny
        Allow from all
    </Directory>
    # CGI-BIN
    ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
    <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
        AllowOverride None
        Options ExecCGI Includes
        Order allow,deny
        Allow from all
    </Directory>
    

User creation

  • create lpar2rrd user on a server which owns the tool and will have access to HMC/IVM or vCenter/ESXi
  • add a user on Unix under root account
    # useradd -c "LPAR2RRD user" -m lpar2rrd
    
  • Increase memory limits for user lpar2rrd and the WEB server user (under root)
    • AIX
      # chuser  data=-1  lpar2rrd
      # chuser  stack=-1 lpar2rrd
      # chuser  rss=-1   lpar2rrd
      # chuser  data=-1  apache
      # chuser  stack=-1 apache
      # chuser  rss=-1   apache
      
    • Linux
      # vi /etc/security/limits.conf
      
      @lpar2rrd        hard    stack           -1
      @lpar2rrd        soft    stack           -1
      @lpar2rrd        hard    data            -1
      @lpar2rrd        soft    data            -1
      @apache          hard    stack           -1
      @apache          soft    stack           -1
      @apache          hard    data            -1
      @apache          soft    data            -1
      

Install LPAR2RRD

    # su - lpar2rrd
    $ tar xvf lpar2rrd-4.XX.tar
    $ cd lpar2rrd-4.XX
    $ ./install.sh
    $ cd /home/lpar2rrd/lpar2rrd
    
  • Configure parameters in etc/lpar2rrd.cfg.
    Install process should do most of that automatically
    Here is the list of parameters which need to be reviewed:
    $ vi /home/lpar2rrd/lpar2rrd/etc/lpar2rrd.cfg
    
      HMC_USER=lpar2rrd  
      HMC_LIST="hmc1 sdmc1 ivm1"   # example only, update it!
      WEBDIR=/home/lpar2rrd/lpar2rrd/www
      RRDTOOL=/usr/bin/rrdtool
      PERL=/usr/bin/perl
      # AIX usual location is:
      #RRDTOOL=/opt/freeware/bin/rrdtool
      #PERL=/opt/freeware/bin/perl
      PERL5LIB=/opt/freeware/lib/perl/5.8.8:/opt/freeware/lib/perl/5.8.0:/usr/opt/perl5/lib/site_perl/5.8.2:/usr/lib/perl5/vendor_perl/5.8.5:/usr/share/perl5:/usr/lib/perl5:/usr/opt/perl5/lib/site_perl/5.8.8/aix-thread-multi:/opt/freeware/lib/perl5/vendor_perl/5.8.8/ppc-thread-multi:/usr/lib64/perl5/vendor_perl:/usr/lib/perl5/vendor_perl
    
  • Sample rate check of configured HMCs and their servers (all must be set to 60!)
    $ cd ./bin
    $ ./sample_rate.sh  
      Going to check HMC as user lpar2rrd
      hmc1:PWR6A-9117-MMA-SN103A4X1 sample rate : 60, OK  
      hmc1:PWR6B-9117-MMA-SN103B5X1 sample rate : 60, OK
    
  • Allow "real-time" refresh on demand, WEB user (apache, nobody) needs to have access to SSH keys
    • find out WEB user
      $ ps -ef|egrep "apache|httpd"|grep -v grep|awk '{print $1}'|grep -v "root"|head -1 
        nobody
      
    • under root user copy SSH keys and change ownership to the WEB user
      # cp /home/lpar2rrd/.ssh/id_rsa /home/lpar2rrd/.ssh/realt_rsa  
      # chown nobody /home/lpar2rrd/.ssh/realt_rsa  
      # chmod 600 /home/lpar2rrd/.ssh/realt_rsa
      
  • Assure you have enough of disk space on the fs where is LPAR2RRD installed
    Roughly you might count 5MB/lpar (10MB/lpar in dual HMC setup) == 1GB/100lpars (without the OS agent)
    $ df -g /home # AIX
    $ df -h /home # linux
    
  • Start the tool itself
    $ ./load.sh
    
  • check the GUI via the web browser
    http://<your lpar2rrd host>/lpar2rrd/
  • Schedule to run the tool from lpar2rrd crontab
    Once or twice an hour seems to be reasonable time, depends how long load.sh takes in your env
    $ crontab -e  
    
    0 * * * * /home/lpar2rrd/lpar2rrd/load.sh > /home/lpar2rrd/lpar2rrd/load.out 2>&1 
    
  • Schedule IVE (HEA) data download if you use this technology (Ethernet virtualization)
    $ crontab -e  
    
    0,5,10,15,20,25,30,35,40,45,50,55 * * * * /home/lpar2rrd/lpar2rrd/load_hea.sh > /home/lpar2rrd/lpar2rrd/load_hea.out 2>&1
    
  • You might need to add lpar2rrd user into /var/adm/cron/cron.allow if crontab command fails as root user.
    # echo "lpar2rrd" >> /var/adm/cron/cron.allow
    
  • If you have any problems with the GUI then check:
    (note that the path to Apache logs might be different, search them in /var)
    • /var/opt/freeware/apache/logs/error_log # Apache error log
    • /var/opt/freeware/apache/logs/access_log # Apache access log
    • /home/lpar2rrd/lpar2rrd/logs/error-cgi.log # LPAR2RRD CGI-BIN log
    • check if web configuration is ok, it should list environment
      http://<your lpar2rrd host>/lpar2rrd-cgi/test.sh

Configuration of additional features

You migt follow with them once you assure that LPAR2RRD is running fine.
Skip it in case of VMware installation.

The tools does not support OS agent on VMware VMs as it is not necessary (vCenter provides enough of information about VMs)

OS agent is add-on feature for monitoring of OS memory utilization and paging (LAN & SAN traffic since 3.64).
It requires the OS agent deployment to every monitored LPAR.
The agent is written in Perl and calls basic OS commands to obtain required statistics like vmstat, lparstat and svmon.
CPU monitoring is still done agent-less based on HMC utilization data.

OS agent architecture

Additional information about the OS agent: NMON support via the OS agent
HMC monitoring via the OS agent
IBM i (AS/400) OS agent
Solaris OS agent

Prerequisites

  • Perl interpreter (AIX usually contains it) and preferably a user account on each LPAR.
  • Opened TCP communication between each LPAR and LPAR2RRD server on port 8162.
    Connections are initiated from LPARs.
  • Additional disk space on LPAR2RRD server (about 40MB per each monitored LPAR)
  • Create preferable dedicated user lpar2rrd on each host with minimum rights
    # useradd -s /usr/bin/ksh  -c "LPAR2RRD agent user" -m lpar2rrd
    
  • In case of VIO server do not use padmin account and mkuser cmd, that would not work!
    Create the account under root in the same way as on usual AIX (like above useradd).

OS agent installation (client)

  • Get the latest OS agent from download page

  • AIX & Linux installation under root
    # rpm -Uvh lpar2rrd-agent-4.95-0.aix5.1.ppc.rpm
    # rpm -qa|grep lpar2rrd-agent
      lpar2rrd-agent-4.95-0
    
  • Solaris installation under root:
    # gunzip lpar2rrd-agent-4.95-0.solaris-i86pc.tar.gz
    # tar xf lpar2rrd-agent-4.95-0.solaris-i86pc.tar
    # pkgadd -d .
      The following packages are available:
      1  lpar2rrd-agent     LPAR2RRD OS agent 4.95
                            (i86pc) 4.95
     ...
    
    Solaris upgrade under root:
    # pkgrm lpar2rrd-agent
    # pkgadd -d .
    
  • Schedule its run every minute from the crontab on every LPAR.
    This line must be placed into lpar2rrd crontab:
    # su - lpar2rrd
    $ crontab -e 
    * * * * * /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl <LPAR2RRD-SERVER> > /var/tmp/lpar2rrd-agent.out 2>&1
    
    Replace <LPAR2RRD-SERVER> by hostname of your LPAR2RRD server.

  • You might need to add lpar2rrd user into /var/adm/cron/cron.allow under root user if above "crontab -e" fails.
    # echo "lpar2rrd" >> /var/adm/cron/cron.allow
    

LPAR2RRD server (daemon)

  • Edit etc/lpar2rrd.cfg and set following (if it is not already set):
    $ vi /home/lpar2rd/lpar2rrd/etc/lpar2rrd.cfg
    
    LPAR2RRD_AGENT_DAEMON=1
    
  • The daemon is started when load.sh starts
    $ ./load.sh
      Starting LPAR2RRD daemon on port:8162
      ...
    
  • Assure it is running and listening on port 8162:
    $ ps -ef|grep lpar2rrd-daemon
      lpar2rrd 10617010 1 0 Mar 16 - 0:00 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/lpar2rrd-daemon.pl
    $ netstat -an| grep 8162
      tcp4  0  0  *.8162   *.*   LISTEN
    
  • OS agent data graphs will appear in the GUI, use Ctrl-F5 to refresh your web browser

Troubleshooting

  • Client (agent) side:
    • Test if communication through the LAN is allowed.
      $ telnet  <LPAR2RRD-SERVER> 8162
        Connected to 192.168.1.1   .
        Escape character is '^]'.
      
      This is ok, exit either Ctrl-C or ^].

    • Check following agent files:
      data store: /var/tmp/lpar2rrd-agent-*.txt
      error log: /var/tmp/lpar2rrd-agent-*.err
      output log: /var/tmp/lpar2rrd-agent.out

    • run the agent from cmd line:
      $ /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl -d <LPAR2RRD-SERVER>
        ...
        Agent send     : yes : forced by -d 
        Agent send slp: sending wait: 4
        OS/HMC agent working for server: <LPAR2RRD-SERVER>
        store file for sending is /var/tmp/lpar2rrd-agent-<LPAR2RRD-SERVER>-lpar2rrd.txt
      
      It means that data has been sent to the server, all is fine
      Here is example when the agent is not able to sent data :
      $ /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl -d <LPAR2RRD-SERVER>
        ...
        Agent send     : yes : forced by -d 
        Agent send slp: sending wait: 1
        OS/HMC agent working for server: <LPAR2RRD-SERVER>
        store file for sending is /var/tmp/lpar2rrd-agent-<LPAR2RRD-SERVER>-lpar2rrd.txt
        Agent timed out after : 50 seconds /opt/lpar2rrd-agent/lpar2rrd-agent.pl:265
      
      It means that the agent could not contact the server.
      Check communication, port, above telnet example, DNS resolution of the server etc.

  • Server side:
    • test if the daemon on LPAR2RRD server is running, and checking the logs
      $ ps -ef|grep lpar2rrd-daemon
        lpar2rrd 10617010 1 0 Mar 16 - 0:00 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/lpar2rrd-daemon.pl
      $ cd /home/lpar2rrd/lpar2rrd
      $ tail logs/error.log-daemon
      $ tail logs/daemon.out
        new server has been found and registered: Linux (lpar=linuxhost01)
        mkdir : /lpar2rrd/data/Linux/no_hmc/linuxhost01/
      
      It means that new OS agent has been registered from linuxhost01 (Linux stand-alone example)

    • Test if OS agent data is being stored on the LPAR2RRD server and have actual timestamp:
      $ cd /home/lpar2rrd/lpar2rrd
      $ ls -l data/<server name>/*/<lpar name>/*mmm
        -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/<hmc name>/<lpar name>/cpu.mmm
        -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/<hmc name>/<lpar name>/lan-en1.mmm
        -rw-r--r-- 2 lpar2rrd staff 10790264 Mar 17 16:16 data/<server name>/<hmc name>/<lpar name>/mem.mmm
        -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/<hmc name>/<lpar name>/pgs.mmm
        -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/<hmc name>/<lpar name>/san-vscsi0.mmm
        -rw-r--r-- 2 lpar2rrd staff  3597208 Mar 17 16:16 data/<server name>/<hmc name>/<lpar name>/san_resp-vscsi0.mmm
      $ find data -name mem.mmm -exec ls -l {} \;
        ...
      
  • In case of a problem check our forum or contact us via support@lpar2rrd.com

Notes

    LPAR2RRD agent is available since 3.56 release.
    You will not need to upgrade LPAR2RRD agents regularly with each LPAR2RRD upgrade.
    Read release notes if that is necessary.
    Check OS agent upgrade steps.


How it works

VMware monitoring diagram

Prerequisites

  • Create lpar2rrd user on each vCenter (or ESXi not managed by vCenter) with read-only role
  • Download Linux VMware-vSphere-Perl-SDK from VMware site.
    It requires free registration at VMware.
    Does not matter if you are on AIX or Linux or if 32bit or 64bit package
    Download VMware-vSphere-Perl-SDK-6.0.0-<version>.tar.gz, any (*tar.gz) will work. We need just a few Perl scripts from that package.

  • Open https (443) port on your infrastructure for incoming communication from LPAR2RRD server on each monitored vCenter (or ESXi which are nit managed by vCenter)
    $ telnet vmware01 443
      Trying 192.168.1.123...
      Connected to vmware01.
      Escape character is '^]'.
    
    Ctrl-C
    

Installation of VMware SDK

  • Install script
    # su - lpar2rrd
    $ cd /home/lpar2rrd/lpar2rrd
    $ ./bin/vmware_install.sh <directory where is located VMware-vSphere-Perl-SDK-6.0.0-2503617.x86_64.tar.gz >
      VMware-vSphere-Perl-SDK-6.0.0-2503617.x86_64.tar.gz found
      Extracting selected package to /home/lpar2rrd/dev/vmware-vsphere-cli-distrib ...
      Installing selected libraries and apps to /home/lpar2rrd/dev/vmware-lib ...
      ...
    
  • Test of vCenter (ESXi) connection (the last cmd must print vCenter time):
    Use double quotes around username and password only in this test
    $ . /home/lpar2rrd/lpar2rrd/etc/lpar2rrd.cfg
    $ /usr/bin/perl /home/lpar2rrd/lpar2rrd/vmware-lib/apps/connect.pl --version
      vSphere SDK for Perl version: 6.0.0
      Script 'connect.pl' version: 1.0
    $ /usr/bin/perl /home/lpar2rrd/lpar2rrd/vmware-lib/apps/connect.pl --server <vCenter host> --username "vCenter read only user" --password "XXXXX"
      Connection Successful
      Server Time : 2016-02-25T16:28:44.086369Z
    
  • Potential issue
    If connect.pl is reporting a problem with Crypt::SSLeay then install Perl-Crypt-SSLeay (AIX download)
    AIX:   # rpm -Uvh perl-Crypt-SSLeay-0.57-2.aix6.1.ppc.rpm
    Linux: # yum install perl-Crypt-SSLeay
    

Configuration

  • Continue by define VMware hosts (vCenter/ESXi) in the GUI:
    http://your_web_server/lpar2rrd/
    Add credentials for every vCenter or ESXi if it is not managed by the vCenter
    menu ➡ VMware ➡ Configure ➡ Add credentials

  • Now you can run ./load.sh which starts collecting of VMware data
    $ cd /home/lpar2rrd/lpar2rrd
    $ ./load.sh 
    
  • You should see VMware hosts in the GUI when load.sh finishes (reload the GUI in the browser via Ctrl-F5)

  • schedule to run LPAR2RRD from lpar2rrd crontab if it is not already done
    $ crontab -e
    0,20,40 * * * * /home/lpar2rrd/lpar2rrd/load.sh > /home/lpar2rrd/lpar2rrd/load.out 2>&1
    
  • You might need to add lpar2rrd user into /var/adm/cron/cron.allow if crontab command fails as root user.
    # echo "lpar2rrd" >> /var/adm/cron/cron.allow
    
  • If you have any problems with the GUI then check:
    (note that the path to Apache logs might be different, search them in /var)
    • /var/opt/freeware/apache/logs/error_log # Apache error log
    • /var/opt/freeware/apache/logs/access_log # Apache access log
    • /home/lpar2rrd/lpar2rrd/logs/error-cgi.log # LPAR2RRD CGI-BIN log

Troubleshooting

  • In case you miss some performance data then
  • VMware vSphere 6 API Limitation when querying performance Data: VMs graphs of MEM, DISK, NET, SWAP and COMP are all blank in the GUI
    You can see in LPAR2RRD error log logs/error.log:
    Fault string: This operation is restricted by the administrator - 'vpxd.stats.maxQueryMetrics'. Contact your system administrator.
    
    Follow this document to fix it.

  • When you do not see all data (especially for datastores or memory) then the reason is that VMware does not provide all required performance data.
    $ tail /home/lpar2rrd/lpar2rrd/logs/counter-info.txt 
      Wed Dec  2 10:00:04 2015: (F1) no real counter for Disk:read:KBps: Host_System: 
      Wed Dec  2 10:00:04 2015: (F1) no real counter for Memory:active:KB: 
      Wed Dec  2 10:00:04 2015: (F1) no real counter for Memory:granted:KB: 
    $ tail /home/lpar2rrd/lpar2rrd/logs/error.logs
      Tue Feb 02 11:38:39 2016: (F10) not defined real_sampling_period time_stamps= no timestamps for XIV01./bin/vmw2rrd.pl:2491 :
      Tue Feb 02 11:38:39 2016: (F13) not defined real_sampling_period time_stamps= no timestamps for XIV02./bin/vmw2rrd.pl:2491 :
    ...
    
    Adjust VMware performance counter settings to fix it, follow this document:
    How to change VMware counters
  • Data is not being collected with this error in error.log
    $ more /home/lpar2rrd/lpar2rrd/logs/error.log
    Thu Feb 18 10:49:01 2016: vmware name: vcenter01 has not array of hosts ?!? : 
    ...
    
    The reason is that user you have configured for retrieving data from the vCenter probably has not read access to whole vCenter and it cannot access all required perf counters.
    Consult user rights setup with your vCenter administrator.

  • Test vCenter/ESXi connection from the command line:
    1. With entering password manually:
      $ . /home/lpar2rrd/lpar2rrd/etc/lpar2rrd.cfg
      $ $PERL /home/lpar2rrd/lpar2rrd/vmware-lib/apps/connect.pl --server <vCenter host> --username "vCenter read only user"
          Connection Successful
          Server Time : 2016-08-02T06:58:26.355767Z
      
    2. By using saved password already:
      $ $PERL /home/lpar2rrd/lpar2rrd/vmware-lib/apps/connect.pl --server <vCenter host> --username "vCenter read only user" --credstore /home/lpar2rrd/lpar2rrd/.vmware/credstore/vicredentials.xml
      
      You should get the same answer with no password request.
      Use the same hostname and username as defined in LPAR2RRD VMware Configuration.

  • In case of a problem check our forum or contact us via support@lpar2rrd.com