Oracle VM for x86: Hard Partitioning Hands On

As most of you likely know, Oracle has stringent licensing rules when it comes to running their software in a virtual environment.  With anything other than Oracle VM Server for x86, you basically have to license every core in the cluster (VMware, Hyper-V, etc).  With OVM, Oracle does accept a specific configuration that satisfies their definition of a “hard partition” where processor licensing is concerned.  This means that if you own 2 processor licenses for Oracle Database EE for example, and are running on a platform that has a .5 license multiplier (such as x86), you are entitled to run that software on 4 cores.

 

Here are the requirements to satisfy the hard partition I mentioned above (taken from a document that is linked in InfoDoc 1529408.1):

To conform to the Oracle hard partition licensing requirement, you must follow the instructions described in this white paper to bind vCPUs to physical CPU threads or cores.

<

p style=”padding-left:30px;”>Live migration of CPU pinned virtual machines to another Oracle VM Server is not permitted under the terms of the hard partitioning license. Consequently, for Oracle VM Release 3, any servers running CPU pinned guests must not be included in DRS (Distributed Resource Scheduler) and DPM (Distributed Power Management) policies.
When live migration is used in an Oracle VM server pool, hard partition licensing is not applicable. You must determine the number of virtual machines running the Oracle Software and then license the same number of physical servers (starting with the largest servers based on the CPU core count) up to the total number of the physical servers in the pool. For example, if a customer has a server pool with 32 servers and 20 virtual machines running Oracle Software within the server pool, the customer must license the 20 largest physical servers in the pool. If the customer is running 50 virtual machines with Oracle Software in a pool of 32 physical servers, they need only to license the 32 physical servers in the pool.

Live migration of other virtual machines with non-Oracle software within the server pool is not relevant to Oracle software hard partitioning or has no impact to how Oracle software license is calculated.

“Trusted Partitions” allow subset licensing without limitation on live migration, but only available on the approved Oracle Engineered Systems listed on Oracle licensing policies for partitioned environments.

 

There is more information in that document on how to actually perform the CPU pinning but we don’t need to get into that level of detail just yet.  To summarize- here are the key takeaways you should be aware of when considering using OVM for hard partitioning:

  • The use of hyperthreading or no hyperthreading is irrelevant to Oracle from a licensing perspective
  • vCPUs are bound or “pinned” to physical cores using an OVM Manager utility that must be downloaded and installed on your OVM Manager
  • Live Migration, DRS and DPM is not allowed for pinned VMs
  • You have to choose which vCPUs you want to PIN your VM to.  Be careful that you don’t accidentally pin more than one VM to a given set of vCPUs- it’s a completely valid configuration but your performance will go to hell due to contention in the CPU scheduler.
  • Get in the habit of pinning your secondary workloads (applications that don’t require hard partitions) to a set of unused vCPUs.  This way they can’t potentially run on the same vCPU that you just pinned your production database VM to.
  • Make sure when you bind vCPUs that you don’t accidentally cross core boundaries.  It only takes 1 vCPU running on a separate core to mess up your licensing costs.  See my blog post here to get an idea of what I mean.

 

The Real World

Now I want to show you a few things that they don’t talk about in the licensing documents that you are likely to run across in your life as an OVM administrator.

  • live migrate a pinned VM from one OVM Server to another

Capture 2

As you can see above, we have 4 VMs running in this cluster.  Below is an overview of prod_db1.  Take note of the ID field, we’ll use it later to identify the VM:

Capture1

We’re gonna use prod_db1 as our guinea pig for this experiment.  Currently prod_db1 is running on server OVM1 and is pinned to vCPUs 0-3 as noted in the vm.cfg snippet below:

Capture3

I also have a VM running on server ovm2 that is pinned to the very same vCPUs:

Capture4

One would think you cannot live migrate the VM from ovm1 to ovm2 because of the fact that prod_db3 is already pinned to the same vCPUs on ovm2?

Screenshot (7)

 

You certainly can perform the live migration.  Here’s what will happen:

  • The VM will successfully migrate to ovm2
  • prod_db1 will only run on vCPUs 0-3 on ovm2
  • prod_db3 will only run on vCPUs 0-3 on ovm2
  • your performance in both VMs will likely go down the drain
  • you will be out of compliance with Oracle hard partition licensing requirements

 

I’ve had a LOT of people ask me this question, so here’s your proof:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 20 -b- 3988.0 any cpu
0004fb00000600000632b8de1db5a014 3 1 21 -b- 133.8 any cpu
0004fb00000600008825773ba1661d01 2 0 0 -b- 3083.6 0-3
0004fb00000600008825773ba1661d01 2 1 3 -b- 308.1 0-3
Domain-0 0 0 0 r-- 63990.1 0
Domain-0 0 1 1 r-- 62421.0 1
Domain-0 0 2 2 -b- 16102.8 2
Domain-0 0 3 3 -b- 10355.7 3
Domain-0 0 4 4 -b- 2718.1 4
Domain-0 0 5 5 -b- 9427.4 5
Domain-0 0 6 6 -b- 5660.8 6
Domain-0 0 7 7 -b- 3932.0 7
Domain-0 0 8 8 -b- 2268.0 8
Domain-0 0 9 9 -b- 8477.9 9
Domain-0 0 10 10 -b- 4950.6 10
Domain-0 0 11 11 -b- 4304.6 11
Domain-0 0 12 12 -b- 2001.5 12
Domain-0 0 13 13 -b- 10321.1 13
Domain-0 0 14 14 -b- 5221.5 14
Domain-0 0 15 15 -b- 3515.0 15
Domain-0 0 16 16 -b- 2408.8 16
Domain-0 0 17 17 -b- 9905.2 17
Domain-0 0 18 18 -b- 6105.3 18
Domain-0 0 19 19 -b- 4504.2 19



[root@ovm2 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
Domain-0 0 0 0 -b- 54065.1 0
Domain-0 0 1 1 -b- 10110.4 1
Domain-0 0 2 2 -b- 4909.4 2
Domain-0 0 3 3 -b- 6344.0 3
Domain-0 0 4 4 -b- 1012.4 4
Domain-0 0 5 5 -b- 6506.3 5
Domain-0 0 6 6 -b- 4163.1 6
Domain-0 0 7 7 -b- 1564.5 7
Domain-0 0 8 8 -b- 1367.5 8
Domain-0 0 9 9 -b- 14307.2 9
Domain-0 0 10 10 -b- 4068.7 10
Domain-0 0 11 11 -b- 1799.4 11
Domain-0 0 12 12 -b- 1731.3 12
Domain-0 0 13 13 -b- 5478.0 13
Domain-0 0 14 14 -b- 6983.5 14
Domain-0 0 15 15 -b- 5781.6 15
Domain-0 0 16 16 -b- 723.4 16
Domain-0 0 17 17 r-- 4922.6 17
Domain-0 0 18 18 r-- 3585.3 18
Domain-0 0 19 19 -b- 1705.8 19
0004fb0000060000c9e5303a8dc2c675 3 0 0 -b- 5556.6 0-3
0004fb0000060000c9e5303a8dc2c675 3 1 3 -b- 144.4 0-3
  • Now I live migrate prod_db1 from ovm1 to ovm2

Screenshot (8)Screenshot (9)

Screenshot (10)

 

Here is the new vcpu-list post-migration:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 20 -b- 4007.2 any cpu
0004fb00000600000632b8de1db5a014 3 1 21 -b- 134.4 any cpu
Domain-0 0 0 0 r-- 64376.4 0
Domain-0 0 1 1 r-- 62793.1 1
Domain-0 0 2 2 -b- 16201.5 2
Domain-0 0 3 3 -b- 10418.6 3
Domain-0 0 4 4 -b- 2743.2 4
Domain-0 0 5 5 -b- 9486.1 5
Domain-0 0 6 6 -b- 5702.4 6
Domain-0 0 7 7 -b- 3955.7 7
Domain-0 0 8 8 -b- 2279.8 8
Domain-0 0 9 9 -b- 8530.4 9
Domain-0 0 10 10 -b- 4984.4 10
Domain-0 0 11 11 -b- 4328.3 11
Domain-0 0 12 12 -b- 2013.2 12
Domain-0 0 13 13 -b- 10390.7 13
Domain-0 0 14 14 -b- 5257.2 14
Domain-0 0 15 15 -b- 3542.0 15
Domain-0 0 16 16 -b- 2422.3 16
Domain-0 0 17 17 -b- 9969.5 17
Domain-0 0 18 18 -b- 6150.0 18
Domain-0 0 19 19 -b- 4532.5 19



[root@ovm2 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600008825773ba1661d01 5 0 2 -b- 1.9 0-3
0004fb00000600008825773ba1661d01 5 1 1 -b- 0.2 0-3
Domain-0 0 0 0 -b- 54418.2 0
Domain-0 0 1 1 -b- 10228.5 1
Domain-0 0 2 2 -b- 4939.8 2
Domain-0 0 3 3 -b- 6373.9 3
Domain-0 0 4 4 -b- 1024.7 4
Domain-0 0 5 5 -b- 6547.6 5
Domain-0 0 6 6 -b- 4218.0 6
Domain-0 0 7 7 -b- 1596.2 7
Domain-0 0 8 8 -b- 1374.9 8
Domain-0 0 9 9 -b- 14341.6 9
Domain-0 0 10 10 -b- 4099.5 10
Domain-0 0 11 11 -b- 1822.6 11
Domain-0 0 12 12 -b- 1737.6 12
Domain-0 0 13 13 r-- 5513.4 13
Domain-0 0 14 14 -b- 7016.8 14
Domain-0 0 15 15 -b- 5814.6 15
Domain-0 0 16 16 -b- 731.6 16
Domain-0 0 17 17 -b- 4960.6 17
Domain-0 0 18 18 -b- 3617.2 18
Domain-0 0 19 19 -b- 1714.2 19
0004fb0000060000c9e5303a8dc2c675 3 0 3 -b- 5590.3 0-3
0004fb0000060000c9e5303a8dc2c675 3 1 0 -b- 145.6 0-3

 

You can see that both VMs are pinned to the same vCPUs and they’re still running just fine.  Like I said- it will technically work but you’re shooting yourself in the foot in multiple ways if you do this.  Also keep in mind- if you turn on HA for prod_db1 and ovm1 goes down, the VM will fail to start on ovm2 because of the cpu pinning.  Don’t say I didn’t warn you!

 

  • Apply CPU pinning to a VM online with no reboot

In OVM 3.2 and 3.3, you were able to apply CPU pinning to a VM live without having to restart it.  A bug emerged in OVM 3.4.1 and 3.4.2 that broke this.  However it was fixed in OVM 3.4.3.  So depending on which version of OVM you’re running, you may be able to pin your VMs without having to take a reboot.  Watch and be amazed!

 

Currently running OVM 3.3.3:

[root@ovm1 ~]# cat /etc/ovs-release
Oracle VM server release 3.3.3

 

ovm_vmcontrol utilities are installed:

[root@ovmm ovm_util]# pwd
/u01/app/oracle/ovm-manager-3/ovm_util
[root@ovmm ovm_util]# ls -la
total 44
drwxrwxr-x 5 root root 4096 Jul 2 2014 .
drwxr-xr-x 11 oracle dba 4096 Aug 29 13:04 ..
drwxrwxr-x 2 root root 4096 Jul 2 2014 class
drwxr-xr-x 2 root root 4096 Jul 2 2014 lib
drwxr-xr-x 3 root root 4096 Jul 2 2014 man
-rwxr-xr-x 1 root root 1229 Jul 2 2014 ovm_reporestore
-rwxr-xr-x 1 root root 1227 Jul 2 2014 ovm_vmcontrol
-rwxr-xr-x 1 root root 1245 Jul 2 2014 ovm_vmdisks
-rwxr-xr-x 1 root root 1245 Jul 2 2014 ovm_vmhostd
-rwxr-xr-x 1 root root 1246 Jul 2 2014 ovm_vmmessage
-rwxr-xr-x 1 root root 2854 Jul 2 2014 vm-dump-metrics

 

I have an existing VM that is currently allowed to run on any vCPU on the server:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 20 -b- 4012.8 any cpu
0004fb00000600000632b8de1db5a014 3 1 21 -b- 134.6 any cpu
Domain-0 0 0 0 -b- 64446.0 0
Domain-0 0 1 1 -b- 62820.1 1
Domain-0 0 2 2 -b- 16213.7 2
Domain-0 0 3 3 -b- 10426.0 3
Domain-0 0 4 4 -b- 2746.1 4
Domain-0 0 5 5 -b- 9499.3 5
Domain-0 0 6 6 -b- 5712.5 6
Domain-0 0 7 7 -b- 3960.2 7
Domain-0 0 8 8 -b- 2282.3 8
Domain-0 0 9 9 -b- 8541.0 9
Domain-0 0 10 10 -b- 4992.0 10
Domain-0 0 11 11 -b- 4334.6 11
Domain-0 0 12 12 -b- 2015.6 12
Domain-0 0 13 13 -b- 10404.4 13
Domain-0 0 14 14 -b- 5265.1 14
Domain-0 0 15 15 -b- 3546.7 15
Domain-0 0 16 16 -b- 2423.7 16
Domain-0 0 17 17 r-- 9983.8 17
Domain-0 0 18 18 -b- 6158.2 18
Domain-0 0 19 19 -b- 4536.8 19

 

Now let’s pin that VM to vcpu 8-11:

[root@ovmm ovm_util]# ./ovm_vmcontrol -u admin -p ******** -h localhost -v prod_db2 -c vcpuset -s 8-11
Oracle VM VM Control utility 2.0.1.
Connecting with a secure connection.
Connected.
Command : vcpuset
Pinning virtual CPUs
Pinning of virtual CPUs to physical threads '8-11' 'prod_db2' completed.

 

And here’s our proof that the pinning is applied immediately with no reboot:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 10 -b- 4013.6 8-11
0004fb00000600000632b8de1db5a014 3 1 8 -b- 134.6 8-11
Domain-0 0 0 0 -b- 64454.8 0
Domain-0 0 1 1 -b- 62823.2 1
Domain-0 0 2 2 -b- 16215.2 2
Domain-0 0 3 3 -b- 10427.0 3
Domain-0 0 4 4 -b- 2746.3 4
Domain-0 0 5 5 r-- 9500.6 5
Domain-0 0 6 6 -b- 5713.6 6
Domain-0 0 7 7 -b- 3960.6 7
Domain-0 0 8 8 -b- 2282.5 8
Domain-0 0 9 9 -b- 8542.9 9
Domain-0 0 10 10 -b- 4992.8 10
Domain-0 0 11 11 -b- 4335.0 11
Domain-0 0 12 12 -b- 2015.8 12
Domain-0 0 13 13 -b- 10406.7 13
Domain-0 0 14 14 -b- 5266.4 14
Domain-0 0 15 15 -b- 3547.2 15
Domain-0 0 16 16 -b- 2424.2 16
Domain-0 0 17 17 -b- 9984.8 17
Domain-0 0 18 18 -b- 6159.6 18
Domain-0 0 19 19 -b- 4537.6 19

 

You’ll just have to take my word that I didn’t reboot the VM inbetween the steps- which should be validated by the time column for that VM (note that it increased a little, not reset to 0).

 

 

Well- happy hunting for now!

Advertisements

OVM CPU Pinning

shutterstock_90181546

 

Oracle has published a few documents (2240035.1 and 2213691.1 for starters) about CPU pinning in relation to hard partitions for VMs running on OVM.  This is to avoid having to license every core on the server (like you have to with VMware) for Oracle products that are licensed per core or per user.

 

I’m going to provide an excel spreadsheet at the end of this post that will help you visualize which VM is pinned to which CPU and if there is any overlap.  When a VM is not pinned to a given CPU, it is allowed to run on any cpu within the constraints of the Xen scheduler and where it wants the VM to run.  It will take into account things like NUMA and core boundaries to avoid scheduling a VM in a way that is inefficient.

 

You will need to modify this spreadsheet to fit your server configuration.  Use the information in the ovm-hardpart-168217 document to figure out what your systems CPU topology looks like.

 

A couple things to keep in mind:

  • You cannot live migrate a VM that is pinned.  Technically it will work and the VM will migrate. but Oracle does not allow this based on the terms of their hard partitioning license.  See attached document ovm-hardpart-168217 at the end of this post for more information.
  • When you pin a VM to a vCPU or range of vCPUs, that VM can only run on those vCPUs.  However, if you have other VMs that are not pinned, they can run on any vCPU on the system- including the ones that you just pinned your production database to!  If you have a combination of pinned and unpinned VMs, pin all the other VMs to the range of vCPUs that you want to lock them to.  This way, they can’t run on any vCPUs that you’ve already pinned VMs to.
  • Remember that DOM0 has to be scheduled to run just like the other resources.  Based on how big your system is, OVM will run DOM0 on the first few vCPUs.  This shouldn’t be a problem unless your DOM0 is extremely busy doing work such as processing I/O for the VMs that are running and handling interrupts.  In this case, if you have VMs that are pinned to the same vCPUs as DOM0 you might have some performance problems.  I’ve outlined where DOM0 runs by default on the size system in the example.
  • Realize that you can pin more than one VM to a vCPU.  I wouldn’t recommend this for obvious performance reasons but it’s possible to do.  This is where the spreadsheet comes in handy.
  • If you’re installing the ovm utilities which provides ovm_vmcontrol, you may need to enable remote connections first.  If you get an error message stating that there is an error connecting to localhost, perform the steps below.  You have to pay attention to the version of the ovm utilites that you install.  The readme will show you which of the three (currently) versions to install based on the version of OVM you’re running.
  • Below are the steps to enable remote connections (this was taken from Douglas Hawthorne’s blog here).  Note that the steps below should be performed as the root user, not oracle:
[root@melbourne ~]# cd /u01/app/oracle/ovm-manager-3/bin
[root@melbourne bin]# ./secureOvmmTcpGenKeyStore.sh
Generate OVMM TCP over SSH key store by following steps:
Enter keystore password:
Re-enter new password:
What is your first and last name?
 [Unknown]: OVM
What is the name of your organizational unit?
 [Unknown]: melbourne
What is the name of your organization?
 [Unknown]: YAOCM
What is the name of your City or Locality?
 [Unknown]: Melbourne
What is the name of your State or Province?
 [Unknown]: Victoria
What is the two-letter country code for this unit?
 [Unknown]: AU
Is CN=OVM, OU=melbourne, O=YAOCM, L=Melbourne, ST=Victoria, C=AU correct?
 [no]: yes

Enter key password for <ovmm>
 (RETURN if same as keystore password):
Re-enter new password:
[root@melbourne bin]# ./secureOvmmTcp.sh
Enabling OVMM TCP over SSH service

Please enter the Oracle VM manager user name: admin

Please enter the Oracle VM manager user password:

Please enter the password for TCPS key store :

The job of enabling OVMM TCPS service is committed, please restart OVMM to take effect.





[root@melbourne ~]# service ovmm restart
Stopping Oracle VM Manager [ OK ]
Starting Oracle VM Manager [ OK ]

 

If you have any questions- feel free to post them here.  Good luck!

 

 

CPU pinning example

ovm-hardpart-168217

OVM Manager Cipher Mismatch fix

I was installing a virtual OVM 3.3.3 test environment the other day and when I got to logging into OVM Manager for the first time I got this error:

3ssOL

This has to due with the fact that most modern browsers have dropped support for the older RC4 encryption cipher which is what OVM Manager uses.  There is a “fix” until you update to a newer version that has this bug patched.  See InfoDoc 2099148.1 for all the details, but here’s the meat of it:

 

  • Make a backup of the Weblogic config file
# cd /u01/app/oracle/ovm-manager-3/domains/ovm_domain/config
# cp config.xml config.xml.bak

 

  • Add the following line to the cihpersuite section (search for ciphersuite)
<ciphersuite>TLS_RSA_WITH_AES_128_CBC_SHA</ciphersuite>

 

  • Restart the ovm manager service and all is well
# service ovmm restart

Virtualized ODA X6-2HA – working with VMs

It’s been awhile since I built a virtualized ODA with VMs on a shared repo so I thought I’d go through the basic steps.

  1. install the OS
    1. install Virtual ISO image
    2. configure networking
    3. install ODA_BASE patch
    4. deploy ODA_BASE
    5. configure networking in ODA_BASE
    6. deploy ODA_BASE with configurator
  2. create shared repository.  This is where your specific situation plays out.  Depending on your hardware you may have less or more space in DATA or RECO.  Your DBA will be able to tell you how much they need for each and where you can borrow a few terabytes (or however much you need) for your VMs
  3. (optionally) create a separate shared repository to store your templates.  This all depends on how many of the same kind of VM you’ll be deploying.  If it makes no sense to keep the templates around once you create your VMs then don’t bother with this step
  4. import template into repository
    1. download the assembly file from Oracle (it will unzip into an .ova archive file)
    2. ***CRITICAL*** copy the .ova to /OVS on either nodes’ DOM0, not into ODA_BASE
    3. import the assembly (point it to the file sitting in DOM0 /OVS)
  5. modify template config as needed (# of vCPUs, Memory, etc)
  6. clone the template to a VM
  7. add network to VM (usually net1 for first public network, net2 for second and net3+ for any VLANs you’ve created
  8. boot VM and start console (easiest way is to VNC into ODA_BASE and launch it from there)
  9. set up your hostname, networking, etc the way you want it
  10. reboot VM to ensure changes persist
  11. rinse and repeat as needed

If you need to configure HA, preferred node or any other things, this is the time to do it.

 

Create VM in Oracle VM for x86 using NFS share

I’m using OVM Manager 3.4.2 and OVM Server 3.3.2 to test an upgrade for one of our customers.  I am using Starwind iSCSI server to present the shared storage to the cluster but in production you should use enterprise grade hardware to do this.  There’s an easier way to do this- create an HVM VM and install from an ISO stored in a repository.  Then power the VM off and change the type to PVM then power on.  This may not work with all operating systems however so I’m going over how to create a new PVM VM from an ISO image shared from an NFS server.

* Download ISO (I'm using Oracle Linux 6.5 64bit for this example)
* Copy ISO image to OVM Manager (any NFS server is fine)
* Mount ISO on the loopback device
# mount -o loop /var/tmp/V41362-01.iso /mnt

* Share the folder via NFS
# service nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS mountd: [ OK ]
Starting NFS daemon: [ OK ]
Starting RPC idmapd: [ OK ]

# exportfs *:/mnt/

# showmount -e
Export list for ovmm:
/mnt *

* Create new VM in OVM Manager
* Edit VM properties and configure as PVM
* Set additional properties such as memory, cpu and network
* At the boot order tab, enter the network boot path formatted like this:
  nfs:{ip address or FQDN of NFS host}:/{path to ISO image top level directory}

For example, our NFS server is 10.2.3.4 and the path where I mounted the ISO is at /mnt.  Leave the {}'s off of course:

  nfs:10.2.3.4:/mnt 

You should be able to boot your VM at this point and perform the install of the OS.

Using VVOLs with vSphere 6 and Nimble

VMware Virtual Volumes is a concept that represents a major paradigm shift from the way storage is used in VMware today.

Below is a short 5 minute video that explains the basic concept of VVOLs.

 

Additionally, knowing the difference between communicating with LUNs as in the old world and communicating with PEs (Protocol Endpoints) is crucial to understanding what VVOLs brings to the table and why.

In short, PE’s are actually a special LUN on the storage array that the ESXi server uses to communicate with the array.  It’s not a LUN in the traditional sense, but more like a logical gateway to talk to the array.  I would say in some ways it’s similar in function to a gatekeeper LUN on an EMC array.  That LUN in turn maps to multiple sub-luns that make up the VM’s individual storage related components (vmdk, vswp, vmsd, vmsn etc).  When the host wants to talk to a LUN, it sends the request to the address of the PE “LUN” with an offset address of the actual LUN on the storage array.  Two things immediately came to mind once I understood this concept:

  1. Since all communication related to the sub-volumes is a VASA function, what happens when vCenter craps the bed?
  2. If I only have 1 PE, isn’t that going to be a huge bottleneck for storage I/O?

The answers to these and other questions are handily dealt with in a post here by VMware vExpert Paul Meehan.  Again- the short version is that vCenter is not needed after the host boots and gets information on PE’s and address offsets.  When it IS needed however is during a host boot.  Secondly, I/O traffic actually goes through the individual volumes, not the PE.  Remember, the PE is a logical LUN that serves as a map to the actual volumes underneath.

This brings me to the next video- understanding PEs.  This link starts about 12 minutes into an hour long presentation where PE’s are talked about.  Feel free to watch the entire video if you want to learn more!

 

Finally, let’s walk through how to set up VVOLs on your Nimble array.  There are a few pre-requisites before you can start:

  • NOS version 3.x or newer
  • vSphere 6.x or newer

Here’s the step by step process:

  1. Connect to web interface of local array
  2. Click on Administration -> VMware integration
  3. Fill in the following information
    • vCenter Name (this can be a vanity name- doesn’t have to be the address of the host)
    • choose the proper subnet on your Nimble array to communicate with vCenter
    • vCenter Host (FQDN or IP address)
    • Credentials
    • Check Web Client and VASA Provider
    • Click Save (This registers vCenter with the storage array and installs the VASA 2.0 provider)
  4. Navigate to Manage -> Storage Pools
  5. Select the Pool in which you want to create the VVOLs (for most sites this will be default)
  6. Click New Folder
  7. Change the Management Type to VMware Virtual Volumes
  8. Give the folder a Name and Description
  9. Set the size of the folder
  10. Choose the vCenter that you registered above, then click Create

Now you have a storage container on the Nimble array that you can use to create VVOLs.  Let’s look at the VMware side now:

  1. Connect to the vSphere web client for your vCenter 6 instance (this will not work with the thick client)
  2. Navigate to Storage and highlight your ESX server
  3. Click on Datastores on the right side of the window
  4. Click on the icon to create a new datastore
  5. Select your location (datacenter) then click next
  6. Select type VVOL then click next
  7. You should see at least one container- click next.  If not, try rescanning your HBA’s in the web client and start again
  8. Assign which host(s) will need access to the VVOL and click next
  9. On the summary screen- click finish

You should now see a new datastore.  Now let’s create a VM in the datastore and see what it looks like in the Nimble web interface!

  1. In vCenter, navigate to hosts and clusters
  2. Right click on your host to create a new virtual machine
  3. Click next under creation type to create a new virtual machine
  4. Give the VM a name, select the location where it should be created and click next
  5. Select the VVOL no requirements policy under VM storage policy
  6. Select the VVOL datastore that is compatible and click next
  7. Select ESXI 6.0 and later under the VM compatibility dtrop down and click next
  8. Choose the appropriate guest OS family and version then click next
  9. Adjust the virtual hardware to meet your needs and click next
  10. At the summary screen, verify all settings are correct and click Finish

Now if you navigate to Manage volumes in your Nimble web interface you will see multiple volumes for each VM you created.  Instead of putting all the .vmdk, .vmx, .vswp and other files inside a single datastore on a single LUN, each object is it’s own volume.  This is what allows you to set performance policies on a per VM basis because each volume can be treated differently.  You can set high performance policy on your production VM’s and low performance on dev/test for example.  Normally you would have to split your VMs into separate datastores and manage the performance policies on a per datastore level.  The problem with this is that you still have no visibility into each VM in that datastore at the storage layer.  With VVOLs, you can see latency, throughput and even noisy neighbor information on a per VM basis in the Nimble web interface!

 

Adding new hosts to vSphere cluster with RDM disks

33I was working with a customer to assist them in upgrading their cluster from 5.1 to 6.o u2.  They had started off with a 2 host cluster, then added a third node to the cluster.  The SAN is an HP EVA6000.  When VMware was first set up and volumes were provisioned and presented, there were only the first two hosts in the cluster.  After the third host was added, naturally the SAN admin presented the volumes to the new host.  What was missed was making sure the LUN number for each volume that was previously presented to just the first two hosts was the same LUN number when presented to the third host.

 

We were running into some problems performing a vMotion of a VM with RDM’s to the new host.  It was complaining that the target host couldn’t see the disks, even though I was able to verify both in the GUI and CLI that it absolutely could see it.  I was able to vMotion between the two original hosts however so this had me stumped.  I had the SAN admin double check the presentation for the RDM disk on that VM and that’s when I saw the LUN number mismatch.

 

The fix was to power off the host, unpresent the volumes, present the volumes making sure to use the same LUN number as the other two, then power the host back up.  After doing this, our problems were solved!

OVM Server for x86 version 3.4.2 released!

downloadOracle has just released the latest version of Oracle VM for x86 and announced it at OpenWorld.  There are some really cool additions that enhance the stability and useability of Oracle VM.  Here are some of the new features:

 

Installation and Upgrades

Oracle VM Manager support for previous Oracle VM Server releases
As of Oracle VM Release 3.4.2, Oracle VM Manager supports current and previous Oracle VM Server releases. For more information, see Chapter 6, Oracle VM Manager Support for Previous Oracle VM Server releases.

Infrastructure

Support for NVM Express (NVMe) devices
Oracle VM Server now discovers NVMe devices and presents them to Oracle VM Manager, where the NVMe device is available as a local disk that you can use to store virtual machine disks or create storage repositories.

The following rules apply to NVMe devices:

Oracle VM Server for x86
  • To use the entire NVMe device as a storage repository or for a single virtual machine physical disk, you should not partition the NVMe device.
  • To provision the NVMe device into multiple physical disks, you should partition it on the Oracle VM Server where the device is installed. If an NVMe device is partitioned then Oracle VM Manager displays each partition as a physical disk, not the entire device.

    You must partition the NVMe device outside of the Oracle VM environment. Oracle VM Manager does not provide any facility for partitioning NVMe devices.

  • NVMe devices can be discovered if no partitions exist on the device.
  • If Oracle VM Server is installed on an NVMe device, then Oracle VM Server does not discover any other partitions on that NVMe device.
Oracle VM Server for SPARC
  • Oracle VM Manager does not display individual partitions on an NVMe device but only a single device.

    Oracle recommends that you create a storage repository on the NVMe device if you are using Oracle VM Server for SPARC. You can then create as many virtual disks as required in the storage repository. However, if you plan to create logical storage volumes for virtual machine disks, you must manually create ZFS volumes on the NVMe device. See Creating ZFS Volumes on NVMe Devices in the Oracle VM Administration Guide.

Using Oracle Ksplice to update the dom0 kernel
Oracle Ksplice capabilities are now available that allow you to update the dom0 kernel for Oracle VM Server without requiring a reboot. Your systems remain up to date with their OS vulnerability patches and downtime is minimized. A Ksplice update takes effect immediately when it is applied. It is not an on-disk change that only takes effect after a subsequent reboot.

Note

This does not impact the underlying Xen hypervisor.

Depending on your level of support, contact your Oracle support representative for assistance before using Oracle Ksplice to update the dom0 kernel for Oracle VM Server. For more information, see Oracle VM: Using Ksplice Uptrack Document ID 2115501.1, on My Oracle Support at: https://support.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=2115501.1.

Extended SCSI functionality available for virtual machines
Oracle VM now provides additional support for SCSI functionality to virtual machines:

  • Linux guests can now retrieve vital product data (VPD) page 0x84 information from physical disks if the device itself makes it available.
  • Microsoft Windows Server guests can use SCSI-3 persistent reservation to form a Microsoft Failover Cluster in an upcoming Oracle VM Paravirtual Drivers for Microsoft Windows release. See the Oracle VM Paravirtual Drivers for Microsoft Windowsdocumentation for information about the availability of failover cluster capabilities on specific Microsoft Operating System versions.
Dom0 kernel upgraded
The dom0 kernel for Oracle VM Server is updated to Oracle Unbreakable Enterprise Kernel Release 4 Quarterly Update 2 in this release.

Package additions and updates
  • The ovmport-1.0-1.el6.4.src.rpm package is added to the Oracle VM Server ISO to support Microsoft Clustering and enable communication between Dom0 and DomU processes using the libxenstore API.
  • The Perl package is updated to perl-5.10.1-141.el6_7.1.src.rpm.
  • The Netscape Portable Runtime (NSPR) package is updated to nspr-4.11.0-1.el6.x86_64.rpm.
  • The openSCAP package is updated to openscap-1.2.8-2.0.1.el6.rpm.
  • The Linux-firmware package is updated to linux-firmware-20160616-44.git43e96a1e.0.12.el6.src.rpm.

Performance and Scalability

Oracle VM Manager performance enhancements
This release enhances the performance of Oracle VM Manager by reducing the number of non-critical events that Oracle VM Server sends to Oracle VM Manager when a system goes down.

Note

If you are running a large Oracle VM environment, it is recommended to increase the amount of memory allocated to the Oracle WebLogic Server. This ensures that adequate memory is available when required. See Increasing the Memory Allocated to Oracle WebLogic Server in the Oracle VM Administration Guide for more information.

Oracle VM Server for x86 performance optimization
For information on performance optimization goals and techniques for Oracle VM Server for x86, see Optimizing Oracle VM Server for x86 Performance, on Oracle Technology Network at: http://www.oracle.com/technetwork/server-storage/vm/ovm-performance-2995164.pdf.

Xen 4.4.4 performance and scalability updates
  • Improved memory allocation: Host system performance is improved by releasing memory more efficiently when tearing down domains, for example, migrating a virtual machine from one Oracle VM Server to another or deleting a virtual machine. This ensures that the host system can manage other guest systems more effectively without experiencing issues with performance.
  • Improved aggregate performance: Oracle VM Server now uses ticket locks for spinlocks, which improves aggregate performance on large scale machines with more than four sockets.
  • Improved performance for Windows and Solaris guests: Microsoft Windows and Oracle Solaris guests with the HVM or PVHVM domain type can now specify local APIC vectors to use as upcall notifications for specific vCPUs. As a result, the guests can more efficiently bind event channels to vCPUs.
  • Improved workload performance: Changes to the Linux scheduler ensure that workload performance is optimized in this release.
  • Improved grant locking: Xen-netback multi-queue improvements take advantage of the grant locking enhancements that are now available in Oracle VM Server Release 3.4.2.
  • Guest disk I/O performance improvements: Block scalability is improved through the implementation of the Xen block multi-queue layer.

Usability

Oracle VM Manager Rule for Live Migration
To prevent failure of live migration, and subsequent issues with the virtual machine environment, a rule has been added to Oracle VM Manager, as follows:

Oracle VM Manager does not allow you to perform a live migration of a virtual machine to or from any instance of Oracle VM Server with a Xen release earlier than xen-4.3.0-55.el6.22.18. This rule applies to any guest OS.

Table 3.1 Live Migration Paths between Oracle VM Server Releases using Oracle VM Manager Release 3.4.2

capture
 

Where the live migration path depends on the Xen release, you should review the following details:

Xen Release (from) Xen Release (to) Live Migration Available?
xen-4.3.0-55.el6.x86_64 xen-4.3.0-55.el6.0.17.x86_64 No
xen-4.3.0-55.el6.22.18.x86_64 and newer xen-4.3.0-55 Yes

For example, as a result of this live migration rule, all virtual machines in an Oracle VM server pool running Oracle VM Server Release 3.3.2 with Xen version xen-4.3.0-55.el6.22.9.x86_64 must be stopped before migrating to Oracle VM Server Release 3.4.2.

Tip

Run the following command on Oracle VM Server to find the Xen version:

# rpm -qa | grep "xen"
PVHVM hot memory modification
As of this release, it is possible to modify the memory allocated to running PVHVM guests without a reboot. Additionally, Oracle VM Manager now allows you to set the allocated memory to a value that is different to the maximum memory available.

Note
  • Hot memory modification is supported on x86-based PVHVM guests running on Linux OS and guests running on Oracle VM Server for SPARC. For x86-based PVHVM guests running on Oracle Solaris OS, you cannot change the memory if the virtual machine is running.
  • See the Oracle VM Paravirtual Drivers for Microsoft Windows documentation for information about the availability of hot memory modification on PVHVM guests that are running a Microsoft Windows OS. You must use a Windows PV Driver that supports hot memory modification or you must stop the guest before you modify the memory.
  • Oracle VM supports hot memory modification through Oracle VM Manager only. If you have manually created unsupported configurations, such as device passthrough, hot memory modification is not supported.

Security

  • Oracle MySQL patch update: This release of Oracle VM includes the July 2016 Critical Patch Update for MySQL. (23087189)
  • Oracle WebLogic patch update: This release of Oracle VM includes the July 2016 Critical Patch Update for WebLogic. (23087185)
  • Oracle Java patch update: This release of Oracle VM includes the July 2016 Critical Patch Update for Java. (23087198).
  • Xen security advisories: The following Xen security advisories are included in this release:
    • XSA-154 (CVE-2016-2270)
    • XSA-170 (CVE-2016-2271)
    • XSA-172 (CVE-2016-3158 and CVE-2016-3159)
    • XSA-173 (CVE-2016-3960)
    • XSA-175 (CVE-2016-4962)
    • XSA-176 (CVE-2016-4480)
    • XSA-178 (CVE-2016-4963)
    • XSA-179 (CVE-2016-3710 and CVE-2016-3712)
    • XSA-180 (CVE-2014-3672)
    • XSA-182 (CVE-2016-6258)
    • XSA-185 (CVE-2016-7092)
    • XSA-187 (CVE-2016-7094)
    • XSA-188 (CVE-2016-7154)

 

 

Hands on with FireEye

images

I recently had a chance to get some soak time with some of FireEye’s suite of cyber security hardware at a customer site.  They deployed NX, HX and CM appliances into their network.  DTI (Dynamic Threat Intelligence) was also purchased, I’ll go into that more in a later post.  Following is an eye chart of FireEye’s comprehensive suite of products as well as a more in depth description of the products that were deployed at this particular customer site:

 

1470248677864

NX

The NX appliance is responsible for monitoring and stopping web based attacks, zero day web exploits and multi-protocol callbacks.  What this means is that the appliance is constantly monitoring traffic coming into your network.  It looks for suspicious activity based on known exploits and how they work (i.e. modify the registry to turn off the firewall, turn off anti-virus or spawn multiple processes and delete the original executable to hide itself).  Once it finds something suspicious, it can analyze the behavior of the potential threat using it’s Multi-Vector Virtual Execution (MVX) engine.  The MVX engine will detonate the payload in an isolated and heavily instrumented VM environment where it can log exactly what the exploit does and how it does it.  Once it has this information and it has identified the exploit, it can automatically block it from getting into your network.

 

HX/HXD

The HX/HXD appliances are used to monitor endpoints (windows desktops/laptops, servers or even cellphones and tablets) for compromises.  It monitors all endpoints across the entire organization at once and is able to correlate suspicious activity.  Once a threat is identified, you then have the option of downloading a triage package that consists of detailed information about what the system was doing or even isolating or containing an endpoint from the network so it can’t cause any additional harm to the environment.  The appliances are typically deployed both in the internal network and the DMZ.  This gives the additional ability to protect remote endpoints that connect externally as well as internal ones.

 

CM

The CM appliance is basically the command center for FireEye that is able to communicate with all other appliances and provide a comprehensive view of what is going on.  It has the ability to reach into email, file storage, endpoint, network and mobile platforms and correlate activities in a single pane of glass.  One of the big benefits of this product is it’s ability to stop multi-vector attacks that span multiple platforms.  By deploying the FireEye NX, EX, FX, HX and AX series together with the FireEye CM series, the analysis of blended threats, such as pinpointing a spear-phishing email used to distribute malicious URLs, and correlating a perimeter alert to the endpoint, becomes possible.

 

The vast majority of customers that purchase HX appliances also purchase DTI for its obvious advantages.  The MVX engine is the really cool part of what FireEye has to offer.  Below is a description of MVX:

The FireEye Malware Protection System features dynamic, real-time analysis for advanced malware using our patent-pending, multi-flow Multi-Vector Virtual Execution (MVX) engine. The MVX engine captures and confirms zero-day, and targeted APT attacks by detonating suspicious files, Web objects, and email attachments within instrumented virtual machine environments.

The MVX engine performs multi-flow analysis to understand the full context of an advanced targeted attack. Stateful attack analysis is critical to trigger analysis of the entire attack lifecycle, from initial exploit to data exfiltration. This is why point products that focus on a single attack object (e.g., malware executable (EXE), dynamic linked library (DLL), or portable document format (PDF) file types) will miss the vast majority of advanced attacks as they are blind to the full attack lifecycle.

 

The customer was able to deploy the endpoint software to hundreds of agents automatically by using Group Policy profiles to push out the installer and run it silently.  Once that was done, we tested containment which essentially takes the machine off the network until you can decide how you want to react.  If the endpoint has been determined to be safe, you can then un-contain it through the GUI.
 
We did run into what seems to be a rather interesting glitch during testing of the containment process.  We contained a machine that was originally on the internal network.  We then placed it on the VPN and due to some initial configuration issues, it was unable to contact the HXD appliance to receive the un-contain instruction.  The result left the machine unable to communicate on the network and no way to fix it.  Surprisingly, we were able to revert to a previous system restore point when the agent hadn’t been installed yet.  Thus we circumvented the whole containment process.
 
I’ve not yet decided if this truly is a bad thing since by reverting back to a point before the agent install, it would essentially rid the machine of the exploit as well.  Regardless, I was a bit dismayed at how easy it was to bypass containment.  Assuming the end user was not malicious (they wouldn’t have infected their own machine) I’m not sure this is a really viable scenario.  One potential would be an exploit that was aware of the mechanics of how the agent works and communicates- in which case it could theoretically block communication to the HX.  This would manifest as an endpoint that hasn’t checked in for awhile and would probably arouse suspicion as well.
 
In summary- I’m extremely impressed with FireEye’s ability to detect and block very complex and coordinated attacks where other products fall down completely.  The MVX engine in particular is something to behold- the level of instrumentation of an exploit is truly incredible.  To take this a step further, if you purchase the AX appliance, this also gives you the ability to do forensics against the exploit including a video of the exploit during detonation (along with all the other telemetry that is captured).  This could prove to be invaluable to root cause analysis in situations where it’s required to determine exactly how an exploit works.

ODA Patching – get ahead of yourself?

I was at a customer site deploying an X5-2 ODA.  They are standardizing on the 12.1.2.6.0 patch level.  Even though 12.1.2.7.0 is currently the latest, they don’t want to be on the bleeding edge.  Recall that the 12.1.2.6.0 patch doesn’t include infrastructure patches (mostly firmware) so you have to install 12.1.2.5.0 first, run the –infra patch to get the firmware and then update to 12.1.2.6.0.

 

We unpacked the 12.1.2.5.0 patch on both systems and then had an epiphany.  Why don’t we just unpack the 12.1.2.6.0 patch as well and save some time later?  What could possibly go wrong?  Needless to say, when we went to install or even verify the 12.1.2.5.0 patch it complained as follows:

ERROR: Patch version must be 12.1.2.6.0

 

Ok, so there has to be a way to clean that patch off the system so I can use 12.1.2.5.0 right?  I stumbled across the oakcli manage cleanrepo command and thought for sure that would fix things up nicely.  Ran it and I got this output:

 


[root@CITX-5ODA-ODABASE-NODE0 tmp]# oakcli manage cleanrepo --ver 12.1.2.6.0
Deleting the following files...
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/OAK/12.1.2.6.0/Base
Deleting the files under /DOM0OAK/12.1.2.6.0/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/Seagate/ST95000N/SF04/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/Seagate/ST95001N/SA03/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/WDC/WD500BLHXSUN/5G08/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HGST/H101860SFSUN600G/A770/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/Seagate/ST360057SSUN600G/0B25/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HITACHI/H106060SDSUN600G/A4C0/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HITACHI/H109060SESUN600G/A720/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HITACHI/HUS1560SCSUN600G/A820/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HGST/HSCAC2DA6SUN200G/A29A/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HGST/HSCAC2DA4SUN400G/A29A/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/STEC/ZeusIOPs-es-G3/E12B/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/STEC/Z16IZF2EUSUN73G/9440/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Expander/ORACLE/DE2-24P/0018/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Expander/ORACLE/DE2-24C/0018/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Expander/ORACLE/DE3-24C/0291/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x0072/11.05.03.00/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x0072/11.05.03.00/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Ilom/SUN/X4370-es-M2/3.0.16.22.f-es-r100119/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HITACHI/H109090SESUN900G/A720/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/STEC/Z16IZF4EUSUN200G/944A/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HGST/H7240AS60SUN4.0T/A2D2/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HGST/H7240B520SUN4.0T/M554/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HGST/H7280A520SUN8.0T/P554/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Expander/SUN/T4-es-Storage/0342/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x0072/11.05.03.00/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x005d/4.230.40-3739/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x0097/06.00.02.00/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/Mellanox/0x1003/2.11.1280/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Ilom/SUN/X4170-es-M3/3.2.4.26.b-es-r101722/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Ilom/SUN/X4-2/3.2.4.46.a-es-r101689/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Ilom/SUN/X5-2/3.2.4.52-es-r101649/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/HMP/2.3.4.0.1/Base
Deleting the files under /DOM0HMP/2.3.4.0.1/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/IPMI/1.8.12.4/Base
Deleting the files under /DOM0IPMI/1.8.12.4/Base
Deleting the files under /JDK/1.7.0_91/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/ASR/5.3.1/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/OPATCH/12.1.0.1.0/Patches/6880880
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/OPATCH/12.0.0.0.0/Patches/6880880
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/OPATCH/11.2.0.4.0/Patches/6880880
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/GI/12.1.0.2.160119/Patches/21948354
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/DB/12.1.0.2.160119/Patches/21948354
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/DB/11.2.0.4.160119/Patches/21948347
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/DB/11.2.0.3.15/Patches/20760997
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/DB/11.2.0.2.12/Patches/17082367
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/OEL/6.7/Patches/6.7.1
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/OVM/3.2.9/Patches/3.2.9.1
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/OVS/12.1.2.6.0/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x0072/11.05.02.00/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x0072/11.05.02.00/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/GI/12.1.0.2.160119/Base

 

So I assumed that this fixed the problem.  Nope…

 


[root@CITX-5ODA-ODABASE-NODE0 tmp]# oakcli update -patch 12.1.2.5.0 --verify

ERROR: Patch version must be 12.1.2.6.0

 

 

Ok so more searching the CLI manual and the oakcli help pages came up with bupkiss.  So I decided to do an strace of the oakcli command I had just ran.  As ususal- there was a LOT of garbage I didn’t care about or didn’t know what it was doing.  I did find however that it was reading the contents of a file that looked interesting to me:

 


[pid 5509] stat("/opt/oracle/oak/pkgrepos/System/VERSION", {st_mode=S_IFREG|0777, st_size=19, ...}) = 0
[pid 5509] open("/opt/oracle/oak/pkgrepos/System/VERSION", O_RDONLY) = 3
[pid 5509] read(3, "version=12.1.2.6.0\n", 8191) = 19
[pid 5509] read(3, "", 8191) = 0
[pid 5509] close(3) = 0
[pid 5509] fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
[pid 5509] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f159799d000
[pid 5509] write(1, "\n", 1
) = 1
[pid 5509] write(1, "ERROR: Patch version must be 12."..., 40ERROR: Patch version must be 12.1.2.6.0
) = 40
[pid 5509] exit_group(0) = ?

 

There were a dozen or so lines after that, but I had what I needed.  Apparently /opt/oracle/oak/pkgrepos/System/VERSION contains the current version of the latest patch that has been unpacked.  The system software version is kept somewhere else because after I unpacked the 12.1.2.6.0 patch, I ran an oakcli show version and it reported 12.1.2.5.0.  But the VERSION file referenced earlier said 12.1.2.6.0.  I assume when I unpacked the 12.1.2.6.0 patch, it updates this file.  So what I wound up doing is changing the VERSION file back to 12.1.2.5.0 as well as deleting the folder /opt/oracle/oak/pkgrepos/System/12.1.2.6.0.  Once I did this, everything worked as I expected.  I was able to verify and install the –infra portion of 12.1.2.5.0 and continue on my merry way.

 

This highlights the fact that there isn’t a known way (to me at least) to delete an unpacked patch via oakcli or any python scripts I’ve been able to find yet.  Also- as an aside I tried just deleting the VERSION file assuming it would be rebuilt by oakcli and it didn’t.  I got this:

 


[root@CITX-5ODA-ODABASE-NODE0 System]# oakcli update -patch 12.1.2.5.0 --verify
ERROR : Couldn't find the VERSION file to extract the current allowed version

 

So I just recreated the file and all was good.  I was hoping that the oak software didn’t maintain some sort of binary formatted database that kept track of all this information- I think I got lucky in this case.  Hope this helps someone out in a pinch!

X5-2 ODA upgrade from 12.1.2.5.0 to 12.1.2.6.0 observations

Word on keyboard

More fun with patching!  So this time I’m doing a fresh virtualized install and I decided to take my own sage advice of installing 12.1.2.5.0 first to get the firmware patches.  I ran into a bunch of other issues which will be the topic of a different post but I digress.  I got 12.1.2.5.0 fully installed, ODA_BASE deployed, everything was happy.

 

Remember that starting with version 12.1.2.6.0, you have to patch each node separately with the –local option for the infra patches.  So I started the patch on node 0 and it got almost all the way to the end at step 12 where oakd is being patched.  I ran into the “known issue” in 888888.1 item 9:

9.  During the infra patching, after step 12 completed, IPMI, HMP done, if it appeared to be hang during Patching OAK with the following two lines
                               INIT: Sending processes the TERM signal
                               INIT: no more processes left in this runlevel
JDK is not patched, the infra patching is not complete to the end.  
Workaround:  To reboot the appeared hang node manually, then run 
# oakcli update -patch 12.1.2.6 –clean

# oakcli update -patch 12.1.2.6.0 –infra –local
To let it complete the infra patch cleanly.  

I waited about 30 minutes at this step before I started to wonder, and sure enough after checking some log files in /opt/oracle/oak/onecmd/tmp/ it thought oakd was fully patched.  What I found is that oakd gets whacked because the patch doesn’t fully complete.  After doing the reboot that’s recommended in the workaround above, sure enough oakd is not running.  What’s more- now when I boot ODA_BASE the console doesn’t get to the login prompt and you can’t do anything even though you can ssh in just fine.  So I ran the –clean option then kicked off the patch again.  This time it complained that oakd wasn’t running on the remote node.  It was in fact running on node1 but node0 oakd was not.  I suspect that when the ODA communicates to oakd between nodes it’s using the local oakd to do so.

 

So I manually restarted oakd by running /etc/init.d/init.oak restart and then oakd was running.  I rebooted ODA_BASE on node0 just to be sure everything was clean then kicked off the infra patch again.  This time it went all the way through and finished.  The problem now is that the ODA_BASE console is non responsive no matter what I do so I’ll be opening a case with Oracle support to get a WTF.  I’ll update this post with their answer/solution.  If I were a betting man I’d say they’ll tell me to update to 12.1.2.7.0 to fix it.  We’ll see…

 

As an aside- one of the things that 12.1.2.6.0 does is do an in-place upgrade of Oracle Linux 5.11 to version 6.7 for ODA_BASE.  I’ve never done a successful update that way and in fact, Red Hat doesn’t support it.  I guess I can see why they would want to do an update rather than a fresh install but it still feels very risky to me.

ODA Software v12.1.2.6.0 possible bug

I’ve been updating some X5-2 ODA’s for a customer of mine to version 12.1.2.6.0 in preparation for deployment.  I came across a stubborn bug that proved to be a little tricky to solve.  I was having a problem with ODA_BASE not fully completing the boot cycle after initial deployment and as a result I couldn’t get into the ODA_BASE console to configure firstnet.

 

The customer has some strict firewall rules for the network that these ODA’s sit in so I also couldn’t connect to the VNC console on port 5900 as a result.  If you’re gonna implement 12.1.2.6.0 on an X5-2 ODA, I’d recommend installing 12.1.2.5.0 first then update to 12.1.2.6.0..  I’ve not been able to determine for sure what the problem was- I originally thought it had something to do with firmware because 12.1.2.6.0 doesn’t update any of the firmware due to a big ODA_BASE OS version update from 5.11 to 6.7.  Apparently the thought was that the update would either be too big or take too long to download/install so they skip firmware in this release.  Here is the readme for the 12.1.2.6.0 update:

 

This Patch bundle consists of the Jan 2016 12.1.0.2.160119 GI Infrastructure and RDBMS – 12.1.0.2.160119, 11.2.0.4.160119, and 11.2.0.3.15.  The Grid Infrastructure release 12.1.0.2.160119 upgrade is included in this patch bundle.  The database patches 12.1.0.2.160119, 11.2.0.4.160119, 11.2.0.3.15 and 11.2.0.2.12 are included in this patch bundle. Depending on the current version of the system being patched, usually all other infrastructure components like Controller, ILOM, BIOS, and disk firmware etc will also be patched; due to this release focus on the major OS update from OL5 to OL6.7; all other infrastructure components will not be patches.  In a virtualized environment, usually all other infrastructure components on dom0 will also be patched; in this release, we skip them.  To avoid all other infrastructure components version too far behind, the minimum version required is 12.1.2.5.0 for infra and GI.  As part of the Appliance Manager 12.1.2.6, a new parameter has been introduced to control the rolling of ODA patching from one node to another.  This is the first release to provide this functionality to allow you to control when the second node to be patched.

 

I wound up having to re-image to 12.1.2.5.0 and then upgraded as I stated above.  That fixed the problem.  I’m not sure- it may have been a bad download or a glitch in the ODA_BASE bundle because I checked against our own X5-2 ODA and it has the same problem with a fresh install of 12.1.2.6.0 and all of the firmware is up to date.  In hindsight, I probably should have given more credence to this message but it would have added hours onto the install process.  As it is, it more than doubled the time because of the troubleshooting needed.  Lesson learned…

OVM 3.4 released!

VirtualizationOracle releases OVM 3.4!

 

 

 

 

Oracle released last Thursday the latest iteration of their flagship Type 1 hypervisor, Oracle VM for x86.  Some of the new features include:

  • Xen 4.4 hypervisor
  • Storage Live Migration
  • FCoE and UEFI boot support
  • KDump in DOM0
  • Direct OVA import
  • Automatic VNC/Serial Console on OVM Manager
  • OSWatcher installed and configured to run at boot on OVM Server
  • Up to 256 vCPU’s in a single VM
  • General performance improvements in administration tools
  • Simple name persistance in Repositories (keep original name when moving to new OVM Manager)
  • SNMP MIB’s for monitoring OVM Server
  • VIP for server pools deprecated

I’ll be kicking the tires soon and provide some more insight on the implications of these new features.  Stay Tuned!

How to create VLANs in DOM0 on a virtualized ODA

Capture

I’ve been working with a local customer the last week or so to help them set up a pair of ODA’s in virtualized mode.  In one of the datacenters, they needed it to be on a VLAN- including DOM0.  Normally, I just configure net1 for the customer’s network and I’m off to the races.  In this case, there are a few additional steps we have to do.

First thing you’ll need to do is install the ODA software from the install media.  Once this is done, you need to log into the console since we don’t have any IP information configured yet.  Below is a high level checklist of the steps needed to complete this activity:

 

  • Determine which VLAN DOM0 needs to be on
  • Pick a name for the VLAN interface.  It doesn’t have to be eth2 or anything like that.  I usually go with “VLAN456” if my VLAN ID is 456 so it’s self descriptive.
  • Run the following command in DOM0 on node 0 (assuming your VLAN ID is 456)

# oakcli create vlan VLAN456 -vlanid 456 -if bond0

 

At this point, you’ll have the following structures in place on each compute node:

VLAN 1

 

We now have networking set up so that eth2 and eth3 are bonded together (bond0).  Then we put a VLAN bond interface (bond0.456) on top of the bond pair.  Finally we create a VLAN bridge (VLAN456) that can be used to forward that network into the VM, and also allow DOM0 to talk on that VLAN.   I’ve shown in the example above what it looks like to connect more than one VLAN to a bond pair.  If you need access to both VLAN’s from within DOM0 then each VLAN interface on each node will need an IP address assigned to it.  You’ll need to rerun configure firstnet for each interface.  Note also that if you need to access more than one VLAN from a bond pair,  you’ll need to set the switch ports that eth2 and eth3 are connected to into trunked mode so they can pass more than a single VLAN.  Your network administrator will know what this means.

 

 

After that’s in place, you can continue to deploy ODA_BASE, do a configure firstnet in ODA_BASE (remember to assign the VLAN interface to ODA_BASE), yadda yadda…

 

Then, as you configure ODA_BASE and create your VM(s), the NetBack and NetFront drivers are created that are responsible for plumbing the network into the VM.  Here’s a completed diagram with a VM that has access to both VLAN’s:

VLAN final

 

Happy Hunting!

 

 

UPDATE: The way this customer wound up configuring their switches at the end of the day was to put the ODA and ODA_BASE on the Native VLAN.  In this case, even though the switch port is trunked to have access to one or more VLAN’s at a time, the Native VLAN traffic is actually passed untagged down to the server.  This implies that you do not need a special VLAN interface on the ODA to talk on this network, just use the regular net1 or net2 interface.  Now, if you want to talk on any other VLANs through that switch port, you will need to follow the procedure above and configure a VLAN interface for that VLAN.

OVM 3.3.4 Released

OVM 3.3.4 has finally been released after what seems like months since the last update.  Even so- it appears that there are only minor enhancements and mostly bug fixes in this release.  You can find them under patch 20492240 and 20492250.

 

I was hoping for some new features or major updates in this release since it’s been so long.  Will have to hold my breath a little longer I guess :).