Oracle VM for x86: Hard Partitioning Hands On

As most of you likely know, Oracle has stringent licensing rules when it comes to running their software in a virtual environment.  With anything other than Oracle VM Server for x86, you basically have to license every core in the cluster (VMware, Hyper-V, etc).  With OVM, Oracle does accept a specific configuration that satisfies their definition of a “hard partition” where processor licensing is concerned.  This means that if you own 2 processor licenses for Oracle Database EE for example, and are running on a platform that has a .5 license multiplier (such as x86), you are entitled to run that software on 4 cores.

 

Here are the requirements to satisfy the hard partition I mentioned above (taken from a document that is linked in InfoDoc 1529408.1):

To conform to the Oracle hard partition licensing requirement, you must follow the instructions described in this white paper to bind vCPUs to physical CPU threads or cores.

<

p style=”padding-left:30px;”>Live migration of CPU pinned virtual machines to another Oracle VM Server is not permitted under the terms of the hard partitioning license. Consequently, for Oracle VM Release 3, any servers running CPU pinned guests must not be included in DRS (Distributed Resource Scheduler) and DPM (Distributed Power Management) policies.
When live migration is used in an Oracle VM server pool, hard partition licensing is not applicable. You must determine the number of virtual machines running the Oracle Software and then license the same number of physical servers (starting with the largest servers based on the CPU core count) up to the total number of the physical servers in the pool. For example, if a customer has a server pool with 32 servers and 20 virtual machines running Oracle Software within the server pool, the customer must license the 20 largest physical servers in the pool. If the customer is running 50 virtual machines with Oracle Software in a pool of 32 physical servers, they need only to license the 32 physical servers in the pool.

Live migration of other virtual machines with non-Oracle software within the server pool is not relevant to Oracle software hard partitioning or has no impact to how Oracle software license is calculated.

“Trusted Partitions” allow subset licensing without limitation on live migration, but only available on the approved Oracle Engineered Systems listed on Oracle licensing policies for partitioned environments.

 

There is more information in that document on how to actually perform the CPU pinning but we don’t need to get into that level of detail just yet.  To summarize- here are the key takeaways you should be aware of when considering using OVM for hard partitioning:

  • The use of hyperthreading or no hyperthreading is irrelevant to Oracle from a licensing perspective
  • vCPUs are bound or “pinned” to physical cores using an OVM Manager utility that must be downloaded and installed on your OVM Manager
  • Live Migration, DRS and DPM is not allowed for pinned VMs
  • You have to choose which vCPUs you want to PIN your VM to.  Be careful that you don’t accidentally pin more than one VM to a given set of vCPUs- it’s a completely valid configuration but your performance will go to hell due to contention in the CPU scheduler.
  • Get in the habit of pinning your secondary workloads (applications that don’t require hard partitions) to a set of unused vCPUs.  This way they can’t potentially run on the same vCPU that you just pinned your production database VM to.
  • Make sure when you bind vCPUs that you don’t accidentally cross core boundaries.  It only takes 1 vCPU running on a separate core to mess up your licensing costs.  See my blog post here to get an idea of what I mean.

 

The Real World

Now I want to show you a few things that they don’t talk about in the licensing documents that you are likely to run across in your life as an OVM administrator.

  • live migrate a pinned VM from one OVM Server to another

Capture 2

As you can see above, we have 4 VMs running in this cluster.  Below is an overview of prod_db1.  Take note of the ID field, we’ll use it later to identify the VM:

Capture1

We’re gonna use prod_db1 as our guinea pig for this experiment.  Currently prod_db1 is running on server OVM1 and is pinned to vCPUs 0-3 as noted in the vm.cfg snippet below:

Capture3

I also have a VM running on server ovm2 that is pinned to the very same vCPUs:

Capture4

One would think you cannot live migrate the VM from ovm1 to ovm2 because of the fact that prod_db3 is already pinned to the same vCPUs on ovm2?

Screenshot (7)

 

You certainly can perform the live migration.  Here’s what will happen:

  • The VM will successfully migrate to ovm2
  • prod_db1 will only run on vCPUs 0-3 on ovm2
  • prod_db3 will only run on vCPUs 0-3 on ovm2
  • your performance in both VMs will likely go down the drain
  • you will be out of compliance with Oracle hard partition licensing requirements

 

I’ve had a LOT of people ask me this question, so here’s your proof:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 20 -b- 3988.0 any cpu
0004fb00000600000632b8de1db5a014 3 1 21 -b- 133.8 any cpu
0004fb00000600008825773ba1661d01 2 0 0 -b- 3083.6 0-3
0004fb00000600008825773ba1661d01 2 1 3 -b- 308.1 0-3
Domain-0 0 0 0 r-- 63990.1 0
Domain-0 0 1 1 r-- 62421.0 1
Domain-0 0 2 2 -b- 16102.8 2
Domain-0 0 3 3 -b- 10355.7 3
Domain-0 0 4 4 -b- 2718.1 4
Domain-0 0 5 5 -b- 9427.4 5
Domain-0 0 6 6 -b- 5660.8 6
Domain-0 0 7 7 -b- 3932.0 7
Domain-0 0 8 8 -b- 2268.0 8
Domain-0 0 9 9 -b- 8477.9 9
Domain-0 0 10 10 -b- 4950.6 10
Domain-0 0 11 11 -b- 4304.6 11
Domain-0 0 12 12 -b- 2001.5 12
Domain-0 0 13 13 -b- 10321.1 13
Domain-0 0 14 14 -b- 5221.5 14
Domain-0 0 15 15 -b- 3515.0 15
Domain-0 0 16 16 -b- 2408.8 16
Domain-0 0 17 17 -b- 9905.2 17
Domain-0 0 18 18 -b- 6105.3 18
Domain-0 0 19 19 -b- 4504.2 19



[root@ovm2 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
Domain-0 0 0 0 -b- 54065.1 0
Domain-0 0 1 1 -b- 10110.4 1
Domain-0 0 2 2 -b- 4909.4 2
Domain-0 0 3 3 -b- 6344.0 3
Domain-0 0 4 4 -b- 1012.4 4
Domain-0 0 5 5 -b- 6506.3 5
Domain-0 0 6 6 -b- 4163.1 6
Domain-0 0 7 7 -b- 1564.5 7
Domain-0 0 8 8 -b- 1367.5 8
Domain-0 0 9 9 -b- 14307.2 9
Domain-0 0 10 10 -b- 4068.7 10
Domain-0 0 11 11 -b- 1799.4 11
Domain-0 0 12 12 -b- 1731.3 12
Domain-0 0 13 13 -b- 5478.0 13
Domain-0 0 14 14 -b- 6983.5 14
Domain-0 0 15 15 -b- 5781.6 15
Domain-0 0 16 16 -b- 723.4 16
Domain-0 0 17 17 r-- 4922.6 17
Domain-0 0 18 18 r-- 3585.3 18
Domain-0 0 19 19 -b- 1705.8 19
0004fb0000060000c9e5303a8dc2c675 3 0 0 -b- 5556.6 0-3
0004fb0000060000c9e5303a8dc2c675 3 1 3 -b- 144.4 0-3
  • Now I live migrate prod_db1 from ovm1 to ovm2

Screenshot (8)Screenshot (9)

Screenshot (10)

 

Here is the new vcpu-list post-migration:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 20 -b- 4007.2 any cpu
0004fb00000600000632b8de1db5a014 3 1 21 -b- 134.4 any cpu
Domain-0 0 0 0 r-- 64376.4 0
Domain-0 0 1 1 r-- 62793.1 1
Domain-0 0 2 2 -b- 16201.5 2
Domain-0 0 3 3 -b- 10418.6 3
Domain-0 0 4 4 -b- 2743.2 4
Domain-0 0 5 5 -b- 9486.1 5
Domain-0 0 6 6 -b- 5702.4 6
Domain-0 0 7 7 -b- 3955.7 7
Domain-0 0 8 8 -b- 2279.8 8
Domain-0 0 9 9 -b- 8530.4 9
Domain-0 0 10 10 -b- 4984.4 10
Domain-0 0 11 11 -b- 4328.3 11
Domain-0 0 12 12 -b- 2013.2 12
Domain-0 0 13 13 -b- 10390.7 13
Domain-0 0 14 14 -b- 5257.2 14
Domain-0 0 15 15 -b- 3542.0 15
Domain-0 0 16 16 -b- 2422.3 16
Domain-0 0 17 17 -b- 9969.5 17
Domain-0 0 18 18 -b- 6150.0 18
Domain-0 0 19 19 -b- 4532.5 19



[root@ovm2 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600008825773ba1661d01 5 0 2 -b- 1.9 0-3
0004fb00000600008825773ba1661d01 5 1 1 -b- 0.2 0-3
Domain-0 0 0 0 -b- 54418.2 0
Domain-0 0 1 1 -b- 10228.5 1
Domain-0 0 2 2 -b- 4939.8 2
Domain-0 0 3 3 -b- 6373.9 3
Domain-0 0 4 4 -b- 1024.7 4
Domain-0 0 5 5 -b- 6547.6 5
Domain-0 0 6 6 -b- 4218.0 6
Domain-0 0 7 7 -b- 1596.2 7
Domain-0 0 8 8 -b- 1374.9 8
Domain-0 0 9 9 -b- 14341.6 9
Domain-0 0 10 10 -b- 4099.5 10
Domain-0 0 11 11 -b- 1822.6 11
Domain-0 0 12 12 -b- 1737.6 12
Domain-0 0 13 13 r-- 5513.4 13
Domain-0 0 14 14 -b- 7016.8 14
Domain-0 0 15 15 -b- 5814.6 15
Domain-0 0 16 16 -b- 731.6 16
Domain-0 0 17 17 -b- 4960.6 17
Domain-0 0 18 18 -b- 3617.2 18
Domain-0 0 19 19 -b- 1714.2 19
0004fb0000060000c9e5303a8dc2c675 3 0 3 -b- 5590.3 0-3
0004fb0000060000c9e5303a8dc2c675 3 1 0 -b- 145.6 0-3

 

You can see that both VMs are pinned to the same vCPUs and they’re still running just fine.  Like I said- it will technically work but you’re shooting yourself in the foot in multiple ways if you do this.  Also keep in mind- if you turn on HA for prod_db1 and ovm1 goes down, the VM will fail to start on ovm2 because of the cpu pinning.  Don’t say I didn’t warn you!

 

  • Apply CPU pinning to a VM online with no reboot

In OVM 3.2 and 3.3, you were able to apply CPU pinning to a VM live without having to restart it.  A bug emerged in OVM 3.4.1 and 3.4.2 that broke this.  However it was fixed in OVM 3.4.3.  So depending on which version of OVM you’re running, you may be able to pin your VMs without having to take a reboot.  Watch and be amazed!

 

Currently running OVM 3.3.3:

[root@ovm1 ~]# cat /etc/ovs-release
Oracle VM server release 3.3.3

 

ovm_vmcontrol utilities are installed:

[root@ovmm ovm_util]# pwd
/u01/app/oracle/ovm-manager-3/ovm_util
[root@ovmm ovm_util]# ls -la
total 44
drwxrwxr-x 5 root root 4096 Jul 2 2014 .
drwxr-xr-x 11 oracle dba 4096 Aug 29 13:04 ..
drwxrwxr-x 2 root root 4096 Jul 2 2014 class
drwxr-xr-x 2 root root 4096 Jul 2 2014 lib
drwxr-xr-x 3 root root 4096 Jul 2 2014 man
-rwxr-xr-x 1 root root 1229 Jul 2 2014 ovm_reporestore
-rwxr-xr-x 1 root root 1227 Jul 2 2014 ovm_vmcontrol
-rwxr-xr-x 1 root root 1245 Jul 2 2014 ovm_vmdisks
-rwxr-xr-x 1 root root 1245 Jul 2 2014 ovm_vmhostd
-rwxr-xr-x 1 root root 1246 Jul 2 2014 ovm_vmmessage
-rwxr-xr-x 1 root root 2854 Jul 2 2014 vm-dump-metrics

 

I have an existing VM that is currently allowed to run on any vCPU on the server:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 20 -b- 4012.8 any cpu
0004fb00000600000632b8de1db5a014 3 1 21 -b- 134.6 any cpu
Domain-0 0 0 0 -b- 64446.0 0
Domain-0 0 1 1 -b- 62820.1 1
Domain-0 0 2 2 -b- 16213.7 2
Domain-0 0 3 3 -b- 10426.0 3
Domain-0 0 4 4 -b- 2746.1 4
Domain-0 0 5 5 -b- 9499.3 5
Domain-0 0 6 6 -b- 5712.5 6
Domain-0 0 7 7 -b- 3960.2 7
Domain-0 0 8 8 -b- 2282.3 8
Domain-0 0 9 9 -b- 8541.0 9
Domain-0 0 10 10 -b- 4992.0 10
Domain-0 0 11 11 -b- 4334.6 11
Domain-0 0 12 12 -b- 2015.6 12
Domain-0 0 13 13 -b- 10404.4 13
Domain-0 0 14 14 -b- 5265.1 14
Domain-0 0 15 15 -b- 3546.7 15
Domain-0 0 16 16 -b- 2423.7 16
Domain-0 0 17 17 r-- 9983.8 17
Domain-0 0 18 18 -b- 6158.2 18
Domain-0 0 19 19 -b- 4536.8 19

 

Now let’s pin that VM to vcpu 8-11:

[root@ovmm ovm_util]# ./ovm_vmcontrol -u admin -p ******** -h localhost -v prod_db2 -c vcpuset -s 8-11
Oracle VM VM Control utility 2.0.1.
Connecting with a secure connection.
Connected.
Command : vcpuset
Pinning virtual CPUs
Pinning of virtual CPUs to physical threads '8-11' 'prod_db2' completed.

 

And here’s our proof that the pinning is applied immediately with no reboot:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 10 -b- 4013.6 8-11
0004fb00000600000632b8de1db5a014 3 1 8 -b- 134.6 8-11
Domain-0 0 0 0 -b- 64454.8 0
Domain-0 0 1 1 -b- 62823.2 1
Domain-0 0 2 2 -b- 16215.2 2
Domain-0 0 3 3 -b- 10427.0 3
Domain-0 0 4 4 -b- 2746.3 4
Domain-0 0 5 5 r-- 9500.6 5
Domain-0 0 6 6 -b- 5713.6 6
Domain-0 0 7 7 -b- 3960.6 7
Domain-0 0 8 8 -b- 2282.5 8
Domain-0 0 9 9 -b- 8542.9 9
Domain-0 0 10 10 -b- 4992.8 10
Domain-0 0 11 11 -b- 4335.0 11
Domain-0 0 12 12 -b- 2015.8 12
Domain-0 0 13 13 -b- 10406.7 13
Domain-0 0 14 14 -b- 5266.4 14
Domain-0 0 15 15 -b- 3547.2 15
Domain-0 0 16 16 -b- 2424.2 16
Domain-0 0 17 17 -b- 9984.8 17
Domain-0 0 18 18 -b- 6159.6 18
Domain-0 0 19 19 -b- 4537.6 19

 

You’ll just have to take my word that I didn’t reboot the VM inbetween the steps- which should be validated by the time column for that VM (note that it increased a little, not reset to 0).

 

 

Well- happy hunting for now!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s