Install Windows 10 on ESXi 4.1

ESXi 4.1 – Running Windows 10 and Windows Server 2016 on ESXi 4.1


This article provides the steps required to create a Windows 10 or Windows Server 2016 virtual machine on VMWare ESXi 4.1.

The procedure to create a Windows 10 or Windows Server 2016 VM on ESXi 4.1 consists of the following steps :-

  • Creating The Virtual Machine
  • Modifying The BIOS File Used

Creating The Virtual Machine

The first step to create a Windows 10 or Windows Server 2016 VM on ESXi 4.1 is to create the Virtual Machine by performing the following steps :-

  • Connect to the ESXi host using the vSphere Client
  • Righ Click on the server name and select New Virtual Machine
  • Select Custom for the Configuration type and then click on Next
  • When prompted enter the Name you wish for the new Virtual Machine and then click on Next
  • When prompted select the Datastore you wish the Virtual Machine to reside on and then click on Next
  • Leave the Virtual Machine Version as 8 and then click on Next  (version 7 works as well)
  • When prompted for the Guest Operating System select Microsoft Windows 2008 R2 (64-bit)
  • When prompted select the amount of Virtual Sockets and Cores Per Socket you require and then click on Next
  • When prompted select the amount of Memory you require and then click on Next
  • When prompted configure the Network Cards you require and then click on Next
  • When prompted configure the SCSI Controller as LSI Logic SAS and then click on Next
  • When promtped configure the Disks you require and then click on Next
  • When prompted change to change the  Virtual Disk Node click on Next
  • At the Ready To Complete screen click on Finish to create the new Virtual Machine

Modifying The BIOS File Used

Once the Virtual Machine has been created the next step is to modify the BIOS file used by Virtual Machine by performing the following steps :-

  • Download the bios file  from the VMware community and upload it to a DataStore on your ESXi 4.1 Server
  • Copy the filename bios.440.rom to the new VM’s folder
  • Connect to the ESXi Server through SSH and logon as the root user
  • Navigate to the Virtual Machine folder on the ESXi Server :-

cd /vmfs/volumes/{Datastore Name}/{Virtual Machine Name}

E.G. cd /vmfs/volumes/DataStore1/Win-2016-VM

  • Edit the vmx file for the Virtual Machine using vi :-

vi {Virtual Machine Name}

E.G. vi Win-2016.vmx

  • Add the following lines to the bottom of the vmx file:

bios440.filename = “bios.440.rom

mce.enable = TRUE

cpuid.hypervisor.v0 = FALSE

vmGenCounter.enable = FALSE

  • Save and exit the file

Once the steps above have been performed you should then be able to power on the VM and install Windows 10 or Windows Server 2016.

Email spoofing: SPF, DKIM and DMARC to the rescue

It’s estimated that nearly half of all email being sent today is considered SPAM.  Some are harmless messages trying to get you to click through to their website, others are more malignant in nature (phishing, spear-phishing, credential harvesting, etc).


Simple Message Transport Protocol (SMTP) was originally defined in 1982 by RFC 821 and later in 2008, updated by RFC 5321 to allow for extended SMTP additions.  The latter is the version of SMTP that is widely used today to send and receive email on the internet.  The vast majority of details of the SMTP implementation is beyond the scope of this post, however you are more than welcome to peruse RFC 5321 and all of its related RFC’s for some light reading before bedtime.


I want to focus on some mechanisms that are meant to combat email exploits such as spam and phishing.  When an email is composed and sent, there are fields of data in a “header” that are used to tell the email servers on the internet how to route the email to its intended target.  The mail client you use is responsible for putting the correct information into the header so it will reach its destination.  Generally, all you need to know is the destination email address, the subject of the email and the actual body of the email itself.  The client will pretty much fill in the rest.  This header is typically not exposed by most modern email clients unless you ask it to.  See below for an example header:

From: Media Temple user (
Subject: article: How to Trace a Email
Date: January 25, 2011 3:30:58 PM PDT
Return-Path: <>
Delivery-Date: Tue, 25 Jan 2011 15:31:01 -0700
Received: from ([]:54907) by with esmtp (Exim 4.63) (envelope-from <>) id 1KDoNH-0000f0-RL for; Tue, 25 Jan 2011 15:31:01 -0700
Received: by with SMTP id y22so795146pof.4 for <>; Tue, 25 Jan 2011 15:30:58 -0700 (PDT)
Received: by with SMTP id t17mr3929916rvm.251.1214951458741; Tue, 25 Jan 2011 15:30:58 -0700 (PDT)
Received: by with HTTP; Tue, 25 Jan 2011 15:30:58 -0700 (PDT)
Dkim-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=+JqkmVt+sHDFIGX5jKp3oP18LQf10VQjAmZAKl1lspY=; b=F87jySDZnMayyitVxLdHcQNL073DytKRyrRh84GNsI24IRNakn0oOfrC2luliNvdea LGTk3adIrzt+N96GyMseWz8T9xE6O/sAI16db48q4Iqkd7uOiDvFsvS3CUQlNhybNw8m CH/o8eELTN0zbSbn5Trp0dkRYXhMX8FTAwrH0=
Domainkey-Signature: a=rsa-sha1; c=nofws;; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=wkbBj0M8NCUlboI6idKooejg0sL2ms7fDPe1tHUkR9Ht0qr5lAJX4q9PMVJeyjWalH 36n4qGLtC2euBJY070bVra8IBB9FeDEW9C35BC1vuPT5XyucCm0hulbE86+uiUTXCkaB 6ykquzQGCer7xPAcMJqVfXDkHo3H61HM9oCQM=
Message-Id: <>
Mime-Version: 1.0
Content-Type: multipart/alternative; boundary="----=_Part_3927_12044027.1214951458678"
X-Spam-Status: score=3.7 tests=DNS_FROM_RFC_POST, HTML_00_10, HTML_MESSAGE, HTML_SHORT_LENGTH version=3.1.7
X-Spam-Level: ***
Message Body: This is a KnowledgeBase article that provides information on how to find email headers and use the data to trace a email.

As you can see, the majority of the information here would pretty much be greek to most people- hence the reason it is usually hidden.  One important thing to be aware of is that every line in an email header can be forged.  Only the “Received:” lines which are added by each email server that your message is relayed through can be trusted.


There are two fields in particular that are a common target of email attackers to make people think an email originated from somewhere it didn’t.  The “From:” and “Return-path:” fields are intended to describe who the message is from and where to send replies if needed.  Recall I said just a second ago that these lines can be forged.  Take for example the email below:



By looking at this email, you would think that I got an email from the former President of the United States.  Alas, not true.  See the header below for the real story (note the lines in bold red):

Received: from (XXXXXXXXXXXXXX)
by with HTTPS via
XXXXXXXXXXXXXX.NAMPRD02.PROD.OUTLOOK.COM; Tue, 9 Apr 2019 18:26:29 +0000
Received: from
(XXXXXXXXXXXXXX) with Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1771.15; Tue, 9 Apr
2019 18:26:28 +0000
Received: from
(XXXXXXXXXXXXXX) with Microsoft SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.1792.14 via Frontend
Transport; Tue, 9 Apr 2019 18:26:28 +0000
Authentication-Results: spf=softfail (sender IP is XXXXXXXXXXXXXX);; dkim=none (message not signed)
header.d=none;; dmarc=fail action=oreject;compauth=none reason=451
Received-SPF: SoftFail ( domain of transitioning discourages use of XXXXXXXXXXXXXX as permitted sender)
Received: from (XXXXXXXXXXXXXX) by (XXXXXXXXXXXXXX) with Microsoft SMTP
Server id 15.20.1771.16 via Frontend Transport; Tue, 9 Apr 2019 18:26:28
Date: Tue, 9 Apr 2019 13:26:27 -0500
To: <>
From: <>
Subject: Totally legit email I promise
Message-ID: <>
X-Mailer: swaks v20170101.0
X-MS-Exchange-Organization-ExpirationStartTime: 09 Apr 2019 18:26:28.3981
X-MS-Exchange-Organization-ExpirationStartTimeReason: OriginalSubmit
X-MS-Exchange-Organization-ExpirationInterval: 2:00:00:00.0000000
X-MS-Exchange-Organization-ExpirationIntervalReason: OriginalSubmit
X-EOPAttributedMessage: 0
X-EOPTenantAttributedMessage: 8fcaf227-a749-47aa-9f8e-5f979cbada7c:0
X-MS-Exchange-Organization-MessageDirectionality: Incoming
X-MS-Exchange-Organization-SCL: -1
MIME-Version: 1.0
Content-Type: text/plain
X-MS-Exchange-Organization-AuthAs: Anonymous
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 72497a18-4430-4243-8ac5-08d6bd18e1c2
X-MS-TrafficTypeDiagnostic: XXXXXXXXXXXXXX:
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Apr 2019 18:26:28.1289
X-MS-Exchange-CrossTenant-Network-Message-Id: 72497a18-4430-4243-8ac5-08d6bd18e1c2
X-MS-Exchange-CrossTenant-Id: 8fcaf227-a749-47aa-9f8e-5f979cbada7c
X-MS-Exchange-CrossTenant-FromEntityHeader: Internet
X-MS-Exchange-Transport-CrossTenantHeadersStamped: XXXXXXXXXXXXXX
X-MS-Exchange-Transport-EndToEndLatency: 00:00:01.7254887
X-MS-Exchange-Processed-By-BccFoldering: 15.20.1771.000

Note that both the “From:” and “Return-Path:” fields show the email as coming from the domain, however the “Received: from” field shows as the actual host that sent the email (I’ve commented out the hostname to protect myself from attacks).  Also note that this email failed SPF checks (more on that shortly).  There are literally dozens of websites that offer either very cheap or free email spoofing services- it’s very easy for attackers to do this.


Now let’s talk about some technologies that can protect us from this sort of thing.


SPF – Sender Policy Framework

SPF is a mechanism that can verify that the SMTP server that an email originated from is actually one of the valid email servers for that domain.  This is important because it is possible to spoof an email to make it look like it came from a given domain like for example.  SPF works by the owner of a domain adding a record to their DNS server that contains information about the valid email servers for that domain.

The problem with SPF is that each domain has to be responsible for publishing their own SPF records.  If they don’t, SPF isn’t able to be enforced.  Most email services will check for SPF and if it fails, will notify the recipient of the failure.  But again- this requires the sending domain to implement SPF in their DNS.  You as a recipient don’t have any control over other domains and their lack of desire to protect themselves from being spoofed.

Here’s a visual example of how it works:




DKIM – DomainKeys Identified Mail

DKIM goes a little bit further by verifying that an email address coming from a given domain is actually an authorized email address.  It doesn’t just verify the domain’s email servers like in SPF.  It does this by using a signature/key combination.  The sender adds a digital signature (aka private key) to each outgoing email that is meant to be validated by the recipient by means of a public key.  Note that you as an email user don’t have to remember to attach this signature- it’s done automatically on your email server(s) once you configure them.  The public key is stored in the sending organizations DNS servers.

Here’s a quick visual example of DKIM in action:



DMARC – Domain Message Authentication Reporting and Conformance

DMARC implements a combination of SPF and DKIM.  It also uses DNS records to vouch for authenticity of an email source.  The addition of reporting includes feedback to the sender organization so they can be made aware of people trying to spoof their emails.  Additionally, DMARC can tell the recipient mail server how to react to these types of email- either reject or quarantine.


See below for a visual example of DMARC in action:



By utilizing any or all of these protection mechanisms, companies can dramatically reduce the volume of spoofed emails received.  We note that all three mechanisms require action on the part of the sender in order for it to work.


Remember- it only takes one click to allow an attack to succeed and put you at risk.  Consider the volume of emails you receive and think about that for a minute!


Create VM from scratch on Oracle VM for x86


I’ve been tasked with creating a very small development environment for a customer based on OVM.  As you know- OVM Manager is needed to do pretty much everything in the Oracle VM world.  Usually you install it on a physical machine or a VM in another virtual environment such as VMware.  This customer wanted a single box with internal storage so we’re faced with creating a VM before we have OVMM up and running.


Let me note right away that THIS IS NOT A SUPPORTED DEPLOYMENT METHOD.  You will be laughed at by Oracle support if you try to call in with a problem and they will tell you to install it on a physical machine. Oracle doesn’t even like to suggest running it in a VM but that deployment method is supported, just not running on the server you’re trying to manage.


Having made that important disclaimer, I wanted to put this out to the Public because I myself had a hell of a time trying to figure this out on my own.  What’s more- there is virtually NO information in the interweb that shows how to do something like this.  I’m sure those with lots of Xen knowledge could have cobbled this together pretty quickly, but what fun would that be?  So here are the things you’re gonna need:

  • TightVNC client
  • OVM Server install media (I used 3.4.6)
  • OVM Manager install media (I used 3.4.6)
  • Oracle Linux or RedHat install ISO (minimum version 5.5.  I used 7.6)
  • A physical server with
    • lots of memory
    • lots of cores
    • lots of internal storage
    • at least one NIC

You need to install OVM server.  Installing OVM Server is trivial so I won’t go into that here.  If you run into problems, check out and look at the getting started guide for OVM Server for x86.  Once you have your OVM server running and you can log in, here’s what I did step by step:

  1. Create a directory inside /OVS called ovmm
  2. Copy the Oracle linux ISO to /OVS/ovmm/
  3. Create a virtual disk image using dd.  This command will create a 20gb sparse file.  I would make it more like 100gb in case you want to use it for a patching proxy server to patch your VMs from.
    • dd if=/dev/zero of=/OVS/ovmm/system.img bs=1k seek=20480k count=1
  4. Create a file called vm.cfg and populate the contents below.  This creates a VM with 2gb memory and 4 vcpu’s.  You’ll probably want to increase the memory to 16gb and maybe bump up the number of vcpu’s.  It also assumes the network adapter is eth0.  If yours is different, change it to match.  You will need to refer to the actual ISO filename that you downloaded for Oracle Linux in the file below.  I used Oracle linux 7.6 to host my OVM Manager.
    • name = 'OVM_Manager'
      kernel = '/usr/lib/xen/boot/hvmloader'
      device_model = '/usr/lib/xen/bin/qemu-dm'
      builder = 'hvm'
      memory = '2048'
      vcpus = 4
      acpi = 1
      apic = 1
      pae = 1
      disk = ['file:/OVS/ovmm/system.img,xvda,w', 'file:/OVS/ovmm/V980739-01.iso,xvdc:cdrom,r']
      vif = [ 'bridge=eth0' ]
      vfb = [ 'type=vnc,vnclisten=,vncdisplay=10' ]
  5. Start the Xen networking.  This is critical to create the xen network bridge that your VM is going to use to talk on the network.  This is the piece that I stumbled on for hours until I figured it out.  Take note of the NIC you want to use- the script will automatically pick whatever adapter can talk via your default gateway.  If you want to specify a different one, pass that adapter name after the start keyword.  Also you’ll want to add this command to the /etc/rc3.d/S99xendomains script which is responsible for starting up all the xen pieces in OVM.  This will ensure the xen bridge will start on boot automatically.
    • /etc/xen/scripts/network-bridge start
  6. Verify the status of your bridge
    • # brctl show
      bridge name  bridge id          STP enabled  interfaces
      eth0         8000.00212826379a  no           peth0
  7. Start the VM
    • xm create /OVS/ovmm/vm.cfg
  8. Fire up TightVNC and connect to the IP address of your OVM server on port 5910
  9. Complete the install of Oracle Linux
  10. Click on reboot when the install is done
  11. When the VM comes back up, it’s going to try to run the install from the cd again.  We need to stop the VM and make a change to the config file.  From the OVM Server, run:
    • xm destroy OVM_Manager
  12. Edit the vm.cfg file and remove the cdrom entry from the disk configuration so it doesn’t try to boot from that again.  The line should look like this when you’re done:
    • disk = ['file:/OVS/ovmm/system.img,xvda,w']
  13. Start the VM once more and connect to the console via VNC the same way you did earlier
    • xm create /OVS/ovmm/vm.cfg
  14. Now you have a perfectly functional linux VM in which to install OVM Manager.  Simply copy the zip file into the VM, extract it and run the installer.



Setting up fail2ban on a linux server

download (1)Fail2ban is an open source cross platform tool that leverages your firewall to block persistent threats from actors that are trying to break into your server.  There are a number of services that typically run by default on most standard linux distros.  I typically work mostly with Oracle Linux which is a derivative of Red Hat Enterprise Linux.  I like it because it’s free to download and use, you only have to pay for support if you want it.

Setting it up is fairly straightforward, out of the box it comes with a bunch of rule definitions that you can borrow and adjust to fit your needs.  There are a few things you have to do after installation to make it functional and I’ll walk through a very basic set of steps that you can build on.

fail2ban is found in the Extra Packages for Enterprise Linux (EPEL) repository.  Make sure it’s enabled before trying to install or you won’t get very far.  If you only want to install just that package and not leave the repository enabled, use this command to install:

# yum enablerepo=epel/x86_64 install fail2ban

If you want to leave the repository enabled for additional software installs or updates to fail2ban, there are a couple extra steps (this applies to Oracle Linux 7.x):

  • change the enabled= flag to 1 for the EPEL repository in /etc/yum.repos.d/public-yum-ol7.repo
name=Oracle Linux $releasever Developement Packages ($basearch)
enabled=1  <<- make sure this is a 1
  • install the fail2ban software
# yum install fail2ban

Once the software successfully installs, you should have a new folder called /etc/fail2ban.  You need to create /etc/fail2ban/jail.local with the contents below:

enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
bantime = -1
banaction = iptables-allports
findtime = 900
maxretry = 3

This file enables the sshd jail. This does the following:

  • enables the sshd jail
  • identifies the path to the log file that sshd writes to when users try to log into the system
  • sets the time to ban to forever
  • determines how many times over how many seconds it should be looking for attempts
    • findtime sets the window for the number of seconds to assess access attempts
    • maxretry sets how many times is one person allowed to try to log in before they’re banned

I’ve found these numbers seem to work well for me, YMMV.

NOTE: If you’re using selinux for added security, you’ll need to update your selinux policy and change the file context of jail.local so selinux doesn’t block you from making changes to it.  Here’s a quick overview of what I had to do:

    • update the selinux-policy* packages
# yum update -y selinux-policy*
    • change the context of the jail.local file to fail2ban_exec_t
# chcon -t fail2ban_exec_t jail.local
    • restart fail2ban and verify there are no selinux errors
# systemctl restart fail2ban

If you still have problems, drop me a message or google is always your friend!

Now, let’s make some final additions to make the IP address bans persist across restarts and reboots.

  • Create a file called /etc/fail2ban/persistent.bans
  • Modify /etc/fail2ban/actions.d/iptables-multiport.conf as seen below.  Note the lines in red bold that are added.
# Fail2Ban configuration file
# Author: Cyril Jaquier
# Modified by Yaroslav Halchenko for multiport banning


before = iptables-common.conf


# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
actionstart =  -N f2b-
               -A f2b- -j 
               -I  -p  -m multiport --dports  -j f2b-
               cat /etc/fail2ban/persistent.bans | awk '/^fail2ban-/ {print $2}' \
               | while read IP; do iptables -I fail2ban- 1 -s $IP -j ; done

# Option:  actionstop
# Notes.:  command executed once at the end of Fail2Ban
# Values:  CMD
actionstop =  -D  -p  -m multiport --dports  -j f2b-
              -F f2b-
              -X f2b-

# Option:  actioncheck
# Notes.:  command executed once before each actionban command
# Values:  CMD
actioncheck =  -n -L  | grep -q 'f2b-[ \t]'

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
actionban =  -I f2b- 1 -s  -j 
echo "fail2ban- " >> /etc/fail2ban/persistent.bans

# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
actionunban =  -D f2b- -s  -j 

  • Restart fail2ban
# systemctl restart fail2ban
  • Check the log file to make sure everything looks ok and in the case of selinux, that it allowed the configuration changes.
# journalctl -xe
  • Look at the firewall contents to see if any IP addresses have already been banned. If you have a system that’s directly connected to the internet and has port 22 forwarded you may see some in a matter of minutes.
# iptables -L -n

Happy hunting!

Log into sftp server with filezilla using ssl key


  1. create account on sftp server (optionally in a chroot sftp only environment for safety)
  2. generate an rsa ssl key with puttygen
  3. save the key in both public and private key (.ppk) format
  4. copy the public key to the remote system you’re connecting to in the appropriate location (usually ~username/.ssh/authorized_keys)
  • copy the private .ppk key you created to the local system
  • open filezilla
  • click file -> Site Manager
  • click New Site
    • Host: {address of sftp server}
    • Port: 22
    • Protocol: sftp
    • Logon Type: key file
    • user: {username created on the sftp server}
    • Key file: {browse to location where you saved the .ppk file}
    • click connect
  • In the left pane, cd to the location of the file you want to upload
  • Drag the file you want to upload from the left pane to the right pane and wait for the upload to complete
  • Close the program



Log into UNIX servers with key via Putty

  • Create an rsa keypair with no passphrase using puttygen
  • Save the public key
  • Save the private key (.ppk format)
  • Configure PuTTy to use your private key
    • Connection -> SSH -> Auth -> Private key for authentication
  • Configure PuTTy to automatically log you in with your username
    • Connection -> Data -> Auto-login Username
  • Save the profile in PuTTy
  • Copy your public key to ~/.ssh/authorized_hosts on each server you want to connect to

Oracle X7-2 HA ODA fiber interface issues

I was working with a customer to deploy an X7-2HA ODA awhile back.  They opted to use 10gbE fiber (as most customers do) for their public network interface.  One problem I ran into quite early on is what turned out to be a bug in the driver for those onboard SFP ports.  They actually can negotiate up to 25gb in addition to 10gb.  This is what caused my problem- the fiber switch ports couldn’t do 25gb and that’s what the onboard adapters was trying to negotiate.

I applied the updated driver to the NIC and rebooted.  Nothing.  No link no packets.  I verified with ethtool that the NIC was only advertising 10gb as its max speed and not 25 like before.  After some troubleshooting, I disabled autonegotiation and forced each of the two adapters to 10gb.  A few seconds after this the link came up and I was able to ping my default gateway!

After reboot however, same thing- no link.



I wound up having to put the following lines at the end of /etc/rc.local on each compute node:

# btbond1 - force em2/em3 to 10 gig Ethernet speed
/sbin/ethtool -s em2 autoneg off speed 10000
/sbin/ethtool -s em3 autoned off speed 10000
sleep 10
ping -c 5 {default gateway}
sleep 10
ping -c 5 {default gateway}

This basically ensures the adapters get forced to 10gb and makes them ping to “wake up” the interface at the end of the system boot process.  Make sure you don’t forget to limit the ping to 5 packets or something reasonable, otherwise guess what you’re going to see on your console every second until eternity?

I’ve been assured this has been fixed in future driver releases- and I’m pretty confident that it will be or there are going to be quite a few pissed off Oracle customers out there!

Rescan virtual disk in a VMware linux VM without reboot

I’ve run into this situation a number of times.  I get a request from a user to resize a filesystem and add some space to it.  Thankfully, through the magic of virtualization I can change the size of the disk on the fly (assuming there are no snapshots in effect).


Resizing the disk in VMware goes fine, so I log into the VM.  Normally for a physical machine with SAN or even SCSI disks, I’d go to /sys/class/scsi_host/ and figure out which adapter the disk I want to resize is sitting on.  Usually a combination of fdisk and lsscsi will give me the info I need here.  Good to go!  So I cd into the hostX folder that represents the right controller.  Here’s where things go south.  I’ve had luck with sending an all HCTL scan and the disk recognizes the new size:

echo "- - -" > scan


By the way, when I mentioned sending an all HCTL scan, let me explain what that means. HCTL stands for scsi_Host, Channel, Target and Lun. When you’re looking at the output of lsscsi as you’ll see below shortly, you’ll see some numbers separated by colons like such:

[2:0:3:0] disk VMware Virtual disk 1.0 /dev/sdd

The four numbers here represent the following
2 = scsi_host
    This is the numeric iteration of the host bus adapter that the scsi device is connected to.  Think of a scsi card or fiber channel card here.  The first scsi_host is usually the internal ones built into most servers as they usually get enumerated first during POST.

0 = Channel
    This is the channel on the HBA that is being referred to.  Think of a dual channel SCSI card or a dual port Fiber Channel HBA.  

3 = Target
    This refers to the SCSI target of the device we're looking at.  In this case, a good description would be an internal SCSI card that has a tape drive, CDROM and a couple hard drives attached to it.  Each of those devices would have a separate "target" to address that device specifically.

0 = Logical Unit Number or LUN
    This is the representation of a sub unit of a target.  A good example would be an optical disk library where the drive itself gets a SCSI target and assumes LUN 0, then the optical disks themselves get assigned LUNs so they can be addressed.  This also more commonly comes into play when you have a SAN that is exporting multiple disks (a.k.a. LUNs).  Say you have an EMC SAN that is presenting 30 disks to a server.  Based on conventional SCSI limitations, most cards can only address up to 15 targets per HBA channel (some will do up to 24 but they are extremely rare).  In this scenario you would need a couple SCSI HBAs to see all those disks.  Now picture thousands of disks... You see where I'm going with this.

I’ve always had luck seeing newly presented disks by doing an all HCTL scan, even in VMware.  But I always wound up having to reboot the damn VM just to get it to recognize the new size of the disk.  Well I stumbled upon a slightly different process today that lets me do what I’ve been trying to do.  Here’s the breakdown:

  • Determine which disk you want to resize.  fdisk -l usually does the trick:
[root@iscsi ~]# fdisk -l

Disk /dev/sda: 32.2 GB, 32212254720 bytes
64 heads, 32 sectors/track, 30720 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0009e4a5

Device Boot Start End Blocks Id System
/dev/sda1 * 2 501 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 502 30720 30944256 8e Linux LVM
Partition 2 does not end on cylinder boundary.

Disk /dev/mapper/VolGroup-lv_root: 27.5 GB, 27455913984 bytes
255 heads, 63 sectors/track, 3337 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/VolGroup-lv_swap: 4227 MB, 4227858432 bytes
255 heads, 63 sectors/track, 514 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb: 1099.5 GB, 1099511627776 bytes
255 heads, 63 sectors/track, 133674 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x02020202

Disk /dev/sdc: 12.9 GB, 12884901888 bytes
64 heads, 32 sectors/track, 12288 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/sdd: 16.6 GB, 32212254720 bytes
256 heads, 63 sectors/track, 3900 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sdd1 1 2081 16777215+ ee GPT
[root@iscsi ~]#
  • Ok so /dev/sdd is the disk I want to resize.  I go resize it in VMware and re-run fdisk but still get the same size- no shock there.
  • Now let’s use the lsscsi command to show us some information about the scsi devices that the OS sees.  You may have to install this tool first.
[root@iscsi ~]# lsscsi -v
[1:0:0:0] cd/dvd NECVMWar VMware IDE CDR10 1.00 /dev/sr0
dir: /sys/bus/scsi/devices/1:0:0:0 [/sys/devices/pci0000:00/0000:00:07.1/host1/target1:0:0/1:0:0:0]
[2:0:0:0] disk VMware Virtual disk 1.0 /dev/sda
dir: /sys/bus/scsi/devices/2:0:0:0 [/sys/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host2/target2:0:0/2:0:0:0]
[2:0:1:0] disk Nimble Server 1.0 /dev/sdb
dir: /sys/bus/scsi/devices/2:0:1:0 [/sys/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host2/target2:0:1/2:0:1:0]
[2:0:2:0] disk Nimble Server 1.0 /dev/sdc
dir: /sys/bus/scsi/devices/2:0:2:0 [/sys/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host2/target2:0:2/2:0:2:0]
[2:0:3:0] disk VMware Virtual disk 1.0 /dev/sdd
dir: /sys/bus/scsi/devices/2:0:3:0 [/sys/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host2/target2:0:3/2:0:3:0]


  • I used the -v (verbose) flag to have it tell me more information about each scsi device.  This gives us a great shortcut to decoding where in /sys/class/scsi_device our target resides.
  • To tell the scsi subsystem to rescan the scsi device, we simply echo a 1 to the rescan file which is located inside the folder identified above.  In our case, the folder is /sys/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host2/target2:0:3/2:0:3:0.  If you cd into this folder, you see a bunch of entries.  BE CAREFUL here, these file entries represent in most cases a live view of what the system is seeing or doing.  If you do the wrong thing, you could tell the disk to power off, or maybe something even more destructive.  There’s no failsafe, the OS isn’t going to ask you if you’re sure you want to do this- we’re poking around in the live kernel here through the “back door” if you will.  Here’s a listing of what my systems shows:
[root@iscsi 2:0:3:0]# ls -la
total 0
drwxr-xr-x 8 root root 0 Mar 21 09:22 .
drwxr-xr-x 4 root root 0 Mar 21 09:22 ..
drwxr-xr-x 3 root root 0 Mar 21 09:22 block
drwxr-xr-x 3 root root 0 Mar 21 09:33 bsg
--w------- 1 root root 4096 Mar 21 09:33 delete
-r--r--r-- 1 root root 4096 Mar 21 09:33 device_blocked
-rw-r--r-- 1 root root 4096 Mar 21 09:33 dh_state
lrwxrwxrwx 1 root root 0 Mar 21 09:33 driver -> ../../../../../../../bus/scsi/drivers/sd
-r--r--r-- 1 root root 4096 Mar 21 09:33 evt_media_change
lrwxrwxrwx 1 root root 0 Mar 21 09:30 generic -> scsi_generic/sg4
-r--r--r-- 1 root root 4096 Mar 21 09:33 iocounterbits
-r--r--r-- 1 root root 4096 Mar 21 09:33 iodone_cnt
-r--r--r-- 1 root root 4096 Mar 21 09:33 ioerr_cnt
-r--r--r-- 1 root root 4096 Mar 21 09:33 iorequest_cnt
-r--r--r-- 1 root root 4096 Mar 21 09:33 modalias
-r--r--r-- 1 root root 4096 Mar 21 09:33 model
drwxr-xr-x 2 root root 0 Mar 21 09:33 power
-r--r--r-- 1 root root 4096 Mar 21 09:33 queue_depth
-r--r--r-- 1 root root 4096 Mar 21 09:33 queue_type
--w------- 1 root root 4096 Mar 21 09:33 rescan
-r--r--r-- 1 root root 4096 Mar 21 09:30 rev
drwxr-xr-x 3 root root 0 Mar 21 09:22 scsi_device
drwxr-xr-x 3 root root 0 Mar 21 09:33 scsi_disk
drwxr-xr-x 3 root root 0 Mar 21 09:33 scsi_generic
-r--r--r-- 1 root root 4096 Mar 21 09:30 scsi_level
-rw-r--r-- 1 root root 4096 Mar 21 09:33 state
lrwxrwxrwx 1 root root 0 Mar 21 09:33 subsystem -> ../../../../../../../bus/scsi
-rw-r--r-- 1 root root 4096 Mar 21 09:33 timeout
-r--r--r-- 1 root root 4096 Mar 21 09:33 type
-rw-r--r-- 1 root root 4096 Mar 21 09:33 uevent
-r--r--r-- 1 root root 4096 Mar 21 09:33 vendor
[root@iscsi 2:0:3:0]#
  • The file we’re interested in is called “rescan”.  The way these generally work is you can poke a value into the kernel by echoing that value into the file like you were appending something to a text file.  Depending on the kernel parameter you’re working with, the value you poke into it will determine what action it takes.  They generally take a 1 or 0 for true or false.  In this case, we want the kernel to rescan this device so we echo “1” > rescan.  This tells the kernel to take another look at the device itself and register any changes that have been made since the system first became aware of it at boot time.
[root@iscsi 2:0:3:0]# echo "1" > rescan
[root@iscsi 2:0:3:0]# fdisk -l /dev/sdd

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/sdd: 32.2 GB, 32212254720 bytes
256 heads, 63 sectors/track, 3900 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sdd1 1 2081 16777215+ ee GPT


Notice the new size of /dev/sdd is now 32GB where it was 16GB before.  Congratulations!  Ok don’t get all excited just yet, your journey has just begun.  Now you have to update the partition table to reflect the new number of sectors so the OS can use the new space.  Then you have to resize whatever filesystem resides on that disk.  If you’re using LVM or software RAID, you’ll need to make some changes at that level first before messing with the filesystem.  The good news is most if not all of this stuff can be done online without having to reboot.


I hope this helps and let me know if you have any questions or input as to how to do this better!


Oracle Announcements Coming

With Oracle OpenWorld coming up in a week or so, we’re used to Oracle using this event to make major announcements about new products, features or generally good stuff.  They’ve already quietly announced their next SPARC chip This year- keep your ears peeled for some very impactful but not all that shocking news about one of their current products. There will be a new product in particular that I’m particularly interested in.  It’s called the SuperCluster M8.  From what I’ve heard, it should be able to run circles around the current exadata.  Additionally, there have been announcements about the support life for Solaris as well.


I can’t say much more until it’s public knowledge, but I would enjoy some spirited discussion about it after it comes out.



Stay tuned!!

Oracle VM for x86: Hard Partitioning Hands On

As most of you likely know, Oracle has stringent licensing rules when it comes to running their software in a virtual environment.  With anything other than Oracle VM Server for x86, you basically have to license every core in the cluster (VMware, Hyper-V, etc).  With OVM, Oracle does accept a specific configuration that satisfies their definition of a “hard partition” where processor licensing is concerned.  This means that if you own 2 processor licenses for Oracle Database EE for example, and are running on a platform that has a .5 license multiplier (such as x86), you are entitled to run that software on 4 cores.

Here are the requirements to satisfy the hard partition I mentioned above (taken from a document that is linked in InfoDoc 1529408.1):

To conform to the Oracle hard partition licensing requirement, you must follow the instructions described in this white paper to bind vCPUs to physical CPU threads or cores.

Live migration of CPU pinned virtual machines to another Oracle VM Server is not permitted under the terms of the hard partitioning license. Consequently, for Oracle VM Release 3, any servers running CPU pinned guests must not be included in DRS (Distributed Resource Scheduler) and DPM (Distributed Power Management) policies.

When live migration is used in an Oracle VM server pool, hard partition licensing is not applicable. You must determine the number of virtual machines running the Oracle Software and then license the same number of physical servers (starting with the largest servers based on the CPU core count) up to the total number of the physical servers in the pool. For example, if a customer has a server pool with 32 servers and 20 virtual machines running Oracle Software within the server pool, the customer must license the 20 largest physical servers in the pool. If the customer is running 50 virtual machines with Oracle Software in a pool of 32 physical servers, they need only to license the 32 physical servers in the pool.

Live migration of other virtual machines with non-Oracle software within the server pool is not relevant to Oracle software hard partitioning or has no impact to how Oracle software license is calculated.

“Trusted Partitions” allow subset licensing without limitation on live migration, but only available on the approved Oracle Engineered Systems listed on Oracle licensing policies for partitioned environments.

There is more information in that document on how to actually perform the CPU pinning but we don’t need to get into that level of detail just yet.  To summarize- here are the key takeaways you should be aware of when considering using OVM for hard partitioning:

  • The use of hyperthreading or no hyperthreading is irrelevant to Oracle from a licensing perspective
  • vCPUs are bound or “pinned” to physical cores using an OVM Manager utility that must be downloaded and installed on your OVM Manager
  • Live Migration, DRS and DPM is not allowed for pinned VMs
  • You have to choose which vCPUs you want to PIN your VM to.  Be careful that you don’t accidentally pin more than one VM to a given set of vCPUs- it’s a completely valid configuration but your performance will go to hell due to contention in the CPU scheduler.
  • Get in the habit of pinning your secondary workloads (applications that don’t require hard partitions) to a set of unused vCPUs.  This way they can’t potentially run on the same vCPU that you just pinned your production database VM to.
  • Make sure when you bind vCPUs that you don’t accidentally cross core boundaries.  It only takes 1 vCPU running on a separate core to mess up your licensing costs.  See my blog post here to get an idea of what I mean.

The Real World

Now I want to show you a few things that they don’t talk about in the licensing documents that you are likely to run across in your life as an OVM administrator.

  • live migrate a pinned VM from one OVM Server to another

Capture 2

As you can see above, we have 4 VMs running in this cluster.  Below is an overview of prod_db1.  Take note of the ID field, we’ll use it later to identify the VM:


We’re gonna use prod_db1 as our guinea pig for this experiment.  Currently prod_db1 is running on server OVM1 and is pinned to vCPUs 0-3 as noted in the vm.cfg snippet below:


I also have a VM running on server ovm2 that is pinned to the very same vCPUs:


One would think you cannot live migrate the VM from ovm1 to ovm2 because of the fact that prod_db3 is already pinned to the same vCPUs on ovm2?

Screenshot (7)

You certainly can perform the live migration.  Here’s what will happen:

  • The VM will successfully migrate to ovm2
  • prod_db1 will only run on vCPUs 0-3 on ovm2
  • prod_db3 will only run on vCPUs 0-3 on ovm2
  • your performance in both VMs will likely go down the drain
  • you will be out of compliance with Oracle hard partition licensing requirements

I’ve had a LOT of people ask me this question, so here’s your proof:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 20 -b- 3988.0 any cpu
0004fb00000600000632b8de1db5a014 3 1 21 -b- 133.8 any cpu
0004fb00000600008825773ba1661d01 2 0 0 -b- 3083.6 0-3
0004fb00000600008825773ba1661d01 2 1 3 -b- 308.1 0-3
Domain-0 0 0 0 r-- 63990.1 0
Domain-0 0 1 1 r-- 62421.0 1
Domain-0 0 2 2 -b- 16102.8 2
Domain-0 0 3 3 -b- 10355.7 3
Domain-0 0 4 4 -b- 2718.1 4
Domain-0 0 5 5 -b- 9427.4 5
Domain-0 0 6 6 -b- 5660.8 6
Domain-0 0 7 7 -b- 3932.0 7
Domain-0 0 8 8 -b- 2268.0 8
Domain-0 0 9 9 -b- 8477.9 9
Domain-0 0 10 10 -b- 4950.6 10
Domain-0 0 11 11 -b- 4304.6 11
Domain-0 0 12 12 -b- 2001.5 12
Domain-0 0 13 13 -b- 10321.1 13
Domain-0 0 14 14 -b- 5221.5 14
Domain-0 0 15 15 -b- 3515.0 15
Domain-0 0 16 16 -b- 2408.8 16
Domain-0 0 17 17 -b- 9905.2 17
Domain-0 0 18 18 -b- 6105.3 18
Domain-0 0 19 19 -b- 4504.2 19

[root@ovm2 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
Domain-0 0 0 0 -b- 54065.1 0
Domain-0 0 1 1 -b- 10110.4 1
Domain-0 0 2 2 -b- 4909.4 2
Domain-0 0 3 3 -b- 6344.0 3
Domain-0 0 4 4 -b- 1012.4 4
Domain-0 0 5 5 -b- 6506.3 5
Domain-0 0 6 6 -b- 4163.1 6
Domain-0 0 7 7 -b- 1564.5 7
Domain-0 0 8 8 -b- 1367.5 8
Domain-0 0 9 9 -b- 14307.2 9
Domain-0 0 10 10 -b- 4068.7 10
Domain-0 0 11 11 -b- 1799.4 11
Domain-0 0 12 12 -b- 1731.3 12
Domain-0 0 13 13 -b- 5478.0 13
Domain-0 0 14 14 -b- 6983.5 14
Domain-0 0 15 15 -b- 5781.6 15
Domain-0 0 16 16 -b- 723.4 16
Domain-0 0 17 17 r-- 4922.6 17
Domain-0 0 18 18 r-- 3585.3 18
Domain-0 0 19 19 -b- 1705.8 19
0004fb0000060000c9e5303a8dc2c675 3 0 0 -b- 5556.6 0-3
0004fb0000060000c9e5303a8dc2c675 3 1 3 -b- 144.4 0-3
  • Now I live migrate prod_db1 from ovm1 to ovm2

Screenshot (8)Screenshot (9)

Screenshot (10)

Here is the new vcpu-list post-migration:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 20 -b- 4007.2 any cpu
0004fb00000600000632b8de1db5a014 3 1 21 -b- 134.4 any cpu
Domain-0 0 0 0 r-- 64376.4 0
Domain-0 0 1 1 r-- 62793.1 1
Domain-0 0 2 2 -b- 16201.5 2
Domain-0 0 3 3 -b- 10418.6 3
Domain-0 0 4 4 -b- 2743.2 4
Domain-0 0 5 5 -b- 9486.1 5
Domain-0 0 6 6 -b- 5702.4 6
Domain-0 0 7 7 -b- 3955.7 7
Domain-0 0 8 8 -b- 2279.8 8
Domain-0 0 9 9 -b- 8530.4 9
Domain-0 0 10 10 -b- 4984.4 10
Domain-0 0 11 11 -b- 4328.3 11
Domain-0 0 12 12 -b- 2013.2 12
Domain-0 0 13 13 -b- 10390.7 13
Domain-0 0 14 14 -b- 5257.2 14
Domain-0 0 15 15 -b- 3542.0 15
Domain-0 0 16 16 -b- 2422.3 16
Domain-0 0 17 17 -b- 9969.5 17
Domain-0 0 18 18 -b- 6150.0 18
Domain-0 0 19 19 -b- 4532.5 19

[root@ovm2 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600008825773ba1661d01 5 0 2 -b- 1.9 0-3
0004fb00000600008825773ba1661d01 5 1 1 -b- 0.2 0-3
Domain-0 0 0 0 -b- 54418.2 0
Domain-0 0 1 1 -b- 10228.5 1
Domain-0 0 2 2 -b- 4939.8 2
Domain-0 0 3 3 -b- 6373.9 3
Domain-0 0 4 4 -b- 1024.7 4
Domain-0 0 5 5 -b- 6547.6 5
Domain-0 0 6 6 -b- 4218.0 6
Domain-0 0 7 7 -b- 1596.2 7
Domain-0 0 8 8 -b- 1374.9 8
Domain-0 0 9 9 -b- 14341.6 9
Domain-0 0 10 10 -b- 4099.5 10
Domain-0 0 11 11 -b- 1822.6 11
Domain-0 0 12 12 -b- 1737.6 12
Domain-0 0 13 13 r-- 5513.4 13
Domain-0 0 14 14 -b- 7016.8 14
Domain-0 0 15 15 -b- 5814.6 15
Domain-0 0 16 16 -b- 731.6 16
Domain-0 0 17 17 -b- 4960.6 17
Domain-0 0 18 18 -b- 3617.2 18
Domain-0 0 19 19 -b- 1714.2 19
0004fb0000060000c9e5303a8dc2c675 3 0 3 -b- 5590.3 0-3
0004fb0000060000c9e5303a8dc2c675 3 1 0 -b- 145.6 0-3

You can see that both VMs are pinned to the same vCPUs and they’re still running just fine.  Like I said- it will technically work but you’re shooting yourself in the foot in multiple ways if you do this.  Also keep in mind- if you turn on HA for prod_db1 and ovm1 goes down, the VM will fail to start on ovm2 because of the cpu pinning.  Don’t say I didn’t warn you!

  • Apply CPU pinning to a VM online with no reboot

In OVM 3.2 and 3.3, you were able to apply CPU pinning to a VM live without having to restart it.  A bug emerged in OVM 3.4.1 and 3.4.2 that broke this.  However it was fixed in OVM 3.4.3.  So depending on which version of OVM you’re running, you may be able to pin your VMs without having to take a reboot.  Watch and be amazed!

Currently running OVM 3.3.3:

[root@ovm1 ~]# cat /etc/ovs-release
Oracle VM server release 3.3.3

ovm_vmcontrol utilities are installed:

[root@ovmm ovm_util]# pwd
[root@ovmm ovm_util]# ls -la
total 44
drwxrwxr-x 5 root root 4096 Jul 2 2014 .
drwxr-xr-x 11 oracle dba 4096 Aug 29 13:04 ..
drwxrwxr-x 2 root root 4096 Jul 2 2014 class
drwxr-xr-x 2 root root 4096 Jul 2 2014 lib
drwxr-xr-x 3 root root 4096 Jul 2 2014 man
-rwxr-xr-x 1 root root 1229 Jul 2 2014 ovm_reporestore
-rwxr-xr-x 1 root root 1227 Jul 2 2014 ovm_vmcontrol
-rwxr-xr-x 1 root root 1245 Jul 2 2014 ovm_vmdisks
-rwxr-xr-x 1 root root 1245 Jul 2 2014 ovm_vmhostd
-rwxr-xr-x 1 root root 1246 Jul 2 2014 ovm_vmmessage
-rwxr-xr-x 1 root root 2854 Jul 2 2014 vm-dump-metrics

I have an existing VM that is currently allowed to run on any vCPU on the server:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 20 -b- 4012.8 any cpu
0004fb00000600000632b8de1db5a014 3 1 21 -b- 134.6 any cpu
Domain-0 0 0 0 -b- 64446.0 0
Domain-0 0 1 1 -b- 62820.1 1
Domain-0 0 2 2 -b- 16213.7 2
Domain-0 0 3 3 -b- 10426.0 3
Domain-0 0 4 4 -b- 2746.1 4
Domain-0 0 5 5 -b- 9499.3 5
Domain-0 0 6 6 -b- 5712.5 6
Domain-0 0 7 7 -b- 3960.2 7
Domain-0 0 8 8 -b- 2282.3 8
Domain-0 0 9 9 -b- 8541.0 9
Domain-0 0 10 10 -b- 4992.0 10
Domain-0 0 11 11 -b- 4334.6 11
Domain-0 0 12 12 -b- 2015.6 12
Domain-0 0 13 13 -b- 10404.4 13
Domain-0 0 14 14 -b- 5265.1 14
Domain-0 0 15 15 -b- 3546.7 15
Domain-0 0 16 16 -b- 2423.7 16
Domain-0 0 17 17 r-- 9983.8 17
Domain-0 0 18 18 -b- 6158.2 18
Domain-0 0 19 19 -b- 4536.8 19

Now let’s pin that VM to vcpu 8-11:

[root@ovmm ovm_util]# ./ovm_vmcontrol -u admin -p ******** -h localhost -v prod_db2 -c vcpuset -s 8-11
Oracle VM VM Control utility 2.0.1.
Connecting with a secure connection.
Command : vcpuset
Pinning virtual CPUs
Pinning of virtual CPUs to physical threads '8-11' 'prod_db2' completed.

And here’s our proof that the pinning is applied immediately with no reboot:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 10 -b- 4013.6 8-11
0004fb00000600000632b8de1db5a014 3 1 8 -b- 134.6 8-11
Domain-0 0 0 0 -b- 64454.8 0
Domain-0 0 1 1 -b- 62823.2 1
Domain-0 0 2 2 -b- 16215.2 2
Domain-0 0 3 3 -b- 10427.0 3
Domain-0 0 4 4 -b- 2746.3 4
Domain-0 0 5 5 r-- 9500.6 5
Domain-0 0 6 6 -b- 5713.6 6
Domain-0 0 7 7 -b- 3960.6 7
Domain-0 0 8 8 -b- 2282.5 8
Domain-0 0 9 9 -b- 8542.9 9
Domain-0 0 10 10 -b- 4992.8 10
Domain-0 0 11 11 -b- 4335.0 11
Domain-0 0 12 12 -b- 2015.8 12
Domain-0 0 13 13 -b- 10406.7 13
Domain-0 0 14 14 -b- 5266.4 14
Domain-0 0 15 15 -b- 3547.2 15
Domain-0 0 16 16 -b- 2424.2 16
Domain-0 0 17 17 -b- 9984.8 17
Domain-0 0 18 18 -b- 6159.6 18
Domain-0 0 19 19 -b- 4537.6 19

You’ll just have to take my word that I didn’t reboot the VM inbetween the steps- which should be validated by the time column for that VM (note that it increased a little, not reset to 0).

Well- happy hunting for now!

OVM CPU Pinning



Oracle has published a few documents (2240035.1 and 2213691.1 for starters) about CPU pinning in relation to hard partitions for VMs running on OVM.  This is to avoid having to license every core on the server (like you have to with VMware) for Oracle products that are licensed per core or per user.


I’m going to provide an excel spreadsheet at the end of this post that will help you visualize which VM is pinned to which CPU and if there is any overlap.  When a VM is not pinned to a given CPU, it is allowed to run on any cpu within the constraints of the Xen scheduler and where it wants the VM to run.  It will take into account things like NUMA and core boundaries to avoid scheduling a VM in a way that is inefficient.


You will need to modify this spreadsheet to fit your server configuration.  Use the information in the ovm-hardpart-168217 document to figure out what your systems CPU topology looks like.


A couple things to keep in mind:

  • You cannot live migrate a VM that is pinned.  Technically it will work and the VM will migrate. but Oracle does not allow this based on the terms of their hard partitioning license.  See attached document ovm-hardpart-168217 at the end of this post for more information.
  • When you pin a VM to a vCPU or range of vCPUs, that VM can only run on those vCPUs.  However, if you have other VMs that are not pinned, they can run on any vCPU on the system- including the ones that you just pinned your production database to!  If you have a combination of pinned and unpinned VMs, pin all the other VMs to the range of vCPUs that you want to lock them to.  This way, they can’t run on any vCPUs that you’ve already pinned VMs to.
  • Remember that DOM0 has to be scheduled to run just like the other resources.  Based on how big your system is, OVM will run DOM0 on the first few vCPUs.  This shouldn’t be a problem unless your DOM0 is extremely busy doing work such as processing I/O for the VMs that are running and handling interrupts.  In this case, if you have VMs that are pinned to the same vCPUs as DOM0 you might have some performance problems.  I’ve outlined where DOM0 runs by default on the size system in the example.
  • Realize that you can pin more than one VM to a vCPU.  I wouldn’t recommend this for obvious performance reasons but it’s possible to do.  This is where the spreadsheet comes in handy.
  • If you’re installing the ovm utilities which provides ovm_vmcontrol, you may need to enable remote connections first.  If you get an error message stating that there is an error connecting to localhost, perform the steps below.  You have to pay attention to the version of the ovm utilites that you install.  The readme will show you which of the three (currently) versions to install based on the version of OVM you’re running.
  • Below are the steps to enable remote connections (this was taken from Douglas Hawthorne’s blog here).  Note that the steps below should be performed as the root user, not oracle:
[root@melbourne ~]# cd /u01/app/oracle/ovm-manager-3/bin
[root@melbourne bin]# ./
Generate OVMM TCP over SSH key store by following steps:
Enter keystore password:
Re-enter new password:
What is your first and last name?
 [Unknown]: OVM
What is the name of your organizational unit?
 [Unknown]: melbourne
What is the name of your organization?
 [Unknown]: YAOCM
What is the name of your City or Locality?
 [Unknown]: Melbourne
What is the name of your State or Province?
 [Unknown]: Victoria
What is the two-letter country code for this unit?
 [Unknown]: AU
Is CN=OVM, OU=melbourne, O=YAOCM, L=Melbourne, ST=Victoria, C=AU correct?
 [no]: yes

Enter key password for <ovmm>
 (RETURN if same as keystore password):
Re-enter new password:
[root@melbourne bin]# ./
Enabling OVMM TCP over SSH service

Please enter the Oracle VM manager user name: admin

Please enter the Oracle VM manager user password:

Please enter the password for TCPS key store :

The job of enabling OVMM TCPS service is committed, please restart OVMM to take effect.

[root@melbourne ~]# service ovmm restart
Stopping Oracle VM Manager [ OK ]
Starting Oracle VM Manager [ OK ]


If you have any questions- feel free to post them here.  Good luck!



CPU pinning example


OVM Manager Cipher Mismatch fix

I was installing a virtual OVM 3.3.3 test environment the other day and when I got to logging into OVM Manager for the first time I got this error:


This has to due with the fact that most modern browsers have dropped support for the older RC4 encryption cipher which is what OVM Manager uses.  There is a “fix” until you update to a newer version that has this bug patched.  See InfoDoc 2099148.1 for all the details, but here’s the meat of it:


  • Make a backup of the Weblogic config file
# cd /u01/app/oracle/ovm-manager-3/domains/ovm_domain/config
# cp config.xml config.xml.bak


  • Add the following line to the cihpersuite section (search for ciphersuite)


  • Restart the ovm manager service and all is well
# service ovmm restart

Configure simple DNS server on RHEL 6

Sometimes when setting up hardware for a customer, it makes things a lot easier if I can simulate their network in our lab.  This allows me to deploy the solution plug and play without having to re-ip a bunch of stuff or wait until I’m on their network to do most of the install.  A couple problems I’ve come across are access to the internet for patches/updates and DNS.


I’ve generally used an old netgear or linksys router to front the customer’s internal network inside my lab environment and just connect it to the back of our cable modem.  This solves the first problem- internet access.  The other problem is a bit more involved, since you have to have a DNS server on that network (preferrably on the same IP address as in the real network when it’s deployed) I’ve taken to using Linux as a stepping stone.  It’s really simple to install Linux or grab one that’s already there and plug it into my private sandbox.  Once that’s done, you just need to install and configure a DNS server.  Here is the step by step process (your IP network will be different, just substitute where appropriate). FYI- I’m running Oracle Linux 6.7 with the Red Hat Compatible Kernel for this tutorial. CentOS 6.7 and RHEL 6.7 are no different other than the repositories you point to in order to get your patches.

Let’s install BIND (Berkley Internet Name Domain) better known as DNS

# yum install -y bind bind-utils
[root@tempDNS ~]# yum install -y bind bind-utils
Loaded plugins: refresh-packagekit, security, ulninfo
Setting up Install Process
public_ol6_latest                                                                                            | 1.4 kB     00:00
Resolving Dependencies
--> Running transaction check
---> Package bind.x86_64 32:9.8.2-0.62.rc1.el6_9.2 will be installed
--> Processing Dependency: bind-libs = 32:9.8.2-0.62.rc1.el6_9.2 for package: 32:bind-9.8.2-0.62.rc1.el6_9.2.x86_64
---> Package bind-utils.x86_64 32:9.8.2-0.37.rc1.el6 will be updated
---> Package bind-utils.x86_64 32:9.8.2-0.62.rc1.el6_9.2 will be an update
--> Running transaction check
---> Package bind-libs.x86_64 32:9.8.2-0.37.rc1.el6 will be updated
---> Package bind-libs.x86_64 32:9.8.2-0.62.rc1.el6_9.2 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

 Package                   Arch                  Version                                     Repository                        Size
 bind                      x86_64                32:9.8.2-0.62.rc1.el6_9.2                   public_ol6_latest                4.0 M
 bind-utils                x86_64                32:9.8.2-0.62.rc1.el6_9.2                   public_ol6_latest                188 k
Updating for dependencies:
 bind-libs                 x86_64                32:9.8.2-0.62.rc1.el6_9.2                   public_ol6_latest                891 k

Transaction Summary
Install       1 Package(s)
Upgrade       2 Package(s)

Total download size: 5.1 M
Downloading Packages:
(1/3): bind-9.8.2-0.62.rc1.el6_9.2.x86_64.rpm                                                                | 4.0 MB     00:00
(2/3): bind-libs-9.8.2-0.62.rc1.el6_9.2.x86_64.rpm                                                           | 891 kB     00:00
(3/3): bind-utils-9.8.2-0.62.rc1.el6_9.2.x86_64.rpm                                                          | 188 kB     00:00
Total                                                                                               3.8 MB/s | 5.1 MB     00:01
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Importing GPG key 0xEC551F03:
 Userid : Oracle OSS group (Open Source Software group) 
 Package: 6:oraclelinux-release-6Server-7.0.5.x86_64 (@anaconda-OracleLinuxServer-201507280245.x86_64/6.7)
 From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : 32:bind-libs-9.8.2-0.62.rc1.el6_9.2.x86_64                                                                       1/5
  Updating   : 32:bind-utils-9.8.2-0.62.rc1.el6_9.2.x86_64                                                                      2/5
  Installing : 32:bind-9.8.2-0.62.rc1.el6_9.2.x86_64                                                                            3/5
  Cleanup    : 32:bind-utils-9.8.2-0.37.rc1.el6.x86_64                                                                          4/5
  Cleanup    : 32:bind-libs-9.8.2-0.37.rc1.el6.x86_64                                                                           5/5
  Verifying  : 32:bind-utils-9.8.2-0.62.rc1.el6_9.2.x86_64                                                                      1/5
  Verifying  : 32:bind-9.8.2-0.62.rc1.el6_9.2.x86_64                                                                            2/5
  Verifying  : 32:bind-libs-9.8.2-0.62.rc1.el6_9.2.x86_64                                                                       3/5
  Verifying  : 32:bind-libs-9.8.2-0.37.rc1.el6.x86_64                                                                           4/5
  Verifying  : 32:bind-utils-9.8.2-0.37.rc1.el6.x86_64                                                                          5/5

  bind.x86_64 32:9.8.2-0.62.rc1.el6_9.2

  bind-utils.x86_64 32:9.8.2-0.62.rc1.el6_9.2

Dependency Updated:
  bind-libs.x86_64 32:9.8.2-0.62.rc1.el6_9.2


Ok, now that we have that done, let’s do a system update to make sure we have all the latest bits and bytes. If this is a production system, consult your companys policy on updates and patches before doing this. I don’t want to be responsible for making the other applications on this server potentially not work for any reason.

[root@tempDNS ~]# yum update -y
Loaded plugins: refresh-packagekit, security, ulninfo
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package ConsoleKit.x86_64 0:0.4.1-3.el6 will be updated
---> Package ConsoleKit.x86_64 0:0.4.1-6.el6 will be an update
---> Package ConsoleKit-libs.x86_64 0:0.4.1-3.el6 will be updated

At this point, I generally recommend a reboot so any updates that had prerequisites for a reboot are taken care of. Also it just makes sure the system is at a known good place for our work.

Let’s edit the /etc/named.conf file and replace the options section with our own custom code:

options {
    listen-on port 53 {;; };
        #listen-on-v6 port 53 { ::1; };
        directory   "/var/named";
        dump-file   "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        allow-query { any; };
        allow-transfer     { localhost; };
        recursion yes;

        dnssec-enable yes;
        dnssec-validation yes;
        dnssec-lookaside auto;

        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";

        managed-keys-directory "/var/named/dynamic";

Note above that I have added my local IP address to the end of the listen-on line. Now let’s add a couple zone files.

zone "" IN {
                type master;
                file "";
                allow-update { none; };

zone "" IN {
                type master;
                file "";
                allow-update { none; };

Obviously change the domain name to your own on the zone line and the file line. Leave the .zone at the end though.

Here are the two files you want to put into /var/named/

$TTL 86400
@   IN  SOA (
        2017062601  ;Serial
        3600        ;Refresh
        1800        ;Retry
        604800      ;Expire
        86400       ;Minimum TTL
; Specify our nameserver
                IN      NS    

; Resolve nameserver hostname to IP
ns1             IN      A     

; Define hostname -> IP pairs which you wish to resolve
gateway         IN      A     

$TTL 86400
@       IN      SOA (
                        21600      ; refresh after 6 hours
                        3600       ; retry after 1 hour
                        604800     ; expire after 1 week
                        86400 )    ; minimum TTL of 1 day
@       IN      NS
1       IN      PTR

There are a few things I’d like you to note.

1) You have to update the serial number any time you make a change to the zone file (forward or reverse). I usually use the format YYYYMMDD## where ## is a sequential number starting with 01. This way if you make multiple updates on the same day, the root servers on the internet will know which version is current.

2) Take notice of the . at the end of the entries in the reverse zone file. These have to be there- they terminate the domain hierarchy and tell the server that this is the root so it doesn’t try to keep looking any further.

3) In my example above, I also have an entry for which has an IP address of This is not normally something you would need or want to do but I wanted to show the syntax of how to do it.

4) For every record you want to add to DNS, it’s a good idea to make sure you also add a reverse record. This lets you do an nslookup or dig against the IP address and it will return the name. A lot of stuff will break or at the very least give you problems if it’s not in place so just get in the habit of doing it.

That’s pretty much it. There are a lot of other nuances that I don’t need to get into here. I almost didn’t write this because there are so many tutorials out there that IMHO are written better than mine. Mainly I wanted to keep it for my own use so I know right where to go when I need to install a quick and dirty DNS server. Hopefully one of you will benefit from this.


Virtualized ODA X6-2HA – working with VMs

It’s been awhile since I built a virtualized ODA with VMs on a shared repo so I thought I’d go through the basic steps.

  1. install the OS
    1. install Virtual ISO image
    2. configure networking
    3. install ODA_BASE patch
    4. deploy ODA_BASE
    5. configure networking in ODA_BASE
    6. deploy ODA_BASE with configurator
  2. create shared repository.  This is where your specific situation plays out.  Depending on your hardware you may have less or more space in DATA or RECO.  Your DBA will be able to tell you how much they need for each and where you can borrow a few terabytes (or however much you need) for your VMs
  3. (optionally) create a separate shared repository to store your templates.  This all depends on how many of the same kind of VM you’ll be deploying.  If it makes no sense to keep the templates around once you create your VMs then don’t bother with this step
  4. import template into repository
    1. download the assembly file from Oracle (it will unzip into an .ova archive file)
    2. ***CRITICAL*** copy the .ova to /OVS on either nodes’ DOM0, not into ODA_BASE
    3. import the assembly (point it to the file sitting in DOM0 /OVS)
  5. modify template config as needed (# of vCPUs, Memory, etc)
  6. clone the template to a VM
  7. add network to VM (usually net1 for first public network, net2 for second and net3+ for any VLANs you’ve created
  8. boot VM and start console (easiest way is to VNC into ODA_BASE and launch it from there)
  9. set up your hostname, networking, etc the way you want it
  10. reboot VM to ensure changes persist
  11. rinse and repeat as needed

If you need to configure HA, preferred node or any other things, this is the time to do it.


ODA Software – Closed for Business!

I’ve deployed a number of these appliances over the last couple years both virtualized and bare metal.  When people realize that Oracle Linux is running under the hood they sometimes think it’s ok to throw rpmforge up in there and have at it.  What’s worse is a customer actually tried to do a yum update on the OS itself from the Oracle public YUM repo!   Ack….


I guess I can see wanting to stay patched to the latest available kernel or version of tools, but it needs to be understood that this appliance is a closed ecosystem.  The beauty of patching the ODA is the fact that I don’t have to chase down all the firmware updates for HDD/SSD/NVM disks, ILOM, BIOS, etc…  That legwork has already been done for me.  Plus the fact that all the patches are tested as a unit together on each platform makes me able to sleep better at night.  Sure- the patches take about 4-5 hours all said and done, but when you’re done, you’re done!  I’m actually wondering if Oracle will eventually implement busybox or something like it for the command line interface to hide the OS layer from end users.  With their move to a web interface for provisioning of the ODA X6-2S/M/L it seems they’ve taken a step in that direction.


If you decide to add repositories to your ODA in order to install system utilities like sysstat and such, it’s generally ok, but I need to say this:  the Oracle hard line states that no additional software should be installed on the ODA at all.  In support of that statement, I will say that I’ve had problems patching when the Oracle public YUM repo is configured and I also ran into the expired RHN key error that started rearing its ugly head at the beginning of 2017.  Both of these are easily fixed, but why put yourself in that position in the first place?


Also, in closing I’d like to recommend to all my customers/readers that you make it a priority to patch your ODA at least once a year.  There are actual ramifications to being out of date that have bitten folks.  I can think of one case where the customers’ ODA hadn’t been updated in 3-4 years.  The customer experienced multiple Hard Drive failures within a week or two and because they had their ODA loaded to the kilt, the ASM rebuild was impacting performance dramatically.  The reason the drives failed so close to eachother and more importantly the way they failed was because of outdated disk firmware.  Newer firmware was available that changed the way disk failure was performed in that it was more sensitive to “blips” and failed out the disk instead of letting it continue to stay in service.  As a result, the disk was dying for awhile and causing degraded performance.  Another reason the disks probably failed early-ish is the amount of load they were placing on the system.  Anywho… just remember to patch ok?