Oracle VM for x86: Hard Partitioning Hands On

As most of you likely know, Oracle has stringent licensing rules when it comes to running their software in a virtual environment.  With anything other than Oracle VM Server for x86, you basically have to license every core in the cluster (VMware, Hyper-V, etc).  With OVM, Oracle does accept a specific configuration that satisfies their definition of a “hard partition” where processor licensing is concerned.  This means that if you own 2 processor licenses for Oracle Database EE for example, and are running on a platform that has a .5 license multiplier (such as x86), you are entitled to run that software on 4 cores.

 

Here are the requirements to satisfy the hard partition I mentioned above (taken from a document that is linked in InfoDoc 1529408.1):

To conform to the Oracle hard partition licensing requirement, you must follow the instructions described in this white paper to bind vCPUs to physical CPU threads or cores.

<

p style=”padding-left:30px;”>Live migration of CPU pinned virtual machines to another Oracle VM Server is not permitted under the terms of the hard partitioning license. Consequently, for Oracle VM Release 3, any servers running CPU pinned guests must not be included in DRS (Distributed Resource Scheduler) and DPM (Distributed Power Management) policies.
When live migration is used in an Oracle VM server pool, hard partition licensing is not applicable. You must determine the number of virtual machines running the Oracle Software and then license the same number of physical servers (starting with the largest servers based on the CPU core count) up to the total number of the physical servers in the pool. For example, if a customer has a server pool with 32 servers and 20 virtual machines running Oracle Software within the server pool, the customer must license the 20 largest physical servers in the pool. If the customer is running 50 virtual machines with Oracle Software in a pool of 32 physical servers, they need only to license the 32 physical servers in the pool.

Live migration of other virtual machines with non-Oracle software within the server pool is not relevant to Oracle software hard partitioning or has no impact to how Oracle software license is calculated.

“Trusted Partitions” allow subset licensing without limitation on live migration, but only available on the approved Oracle Engineered Systems listed on Oracle licensing policies for partitioned environments.

 

There is more information in that document on how to actually perform the CPU pinning but we don’t need to get into that level of detail just yet.  To summarize- here are the key takeaways you should be aware of when considering using OVM for hard partitioning:

  • The use of hyperthreading or no hyperthreading is irrelevant to Oracle from a licensing perspective
  • vCPUs are bound or “pinned” to physical cores using an OVM Manager utility that must be downloaded and installed on your OVM Manager
  • Live Migration, DRS and DPM is not allowed for pinned VMs
  • You have to choose which vCPUs you want to PIN your VM to.  Be careful that you don’t accidentally pin more than one VM to a given set of vCPUs- it’s a completely valid configuration but your performance will go to hell due to contention in the CPU scheduler.
  • Get in the habit of pinning your secondary workloads (applications that don’t require hard partitions) to a set of unused vCPUs.  This way they can’t potentially run on the same vCPU that you just pinned your production database VM to.
  • Make sure when you bind vCPUs that you don’t accidentally cross core boundaries.  It only takes 1 vCPU running on a separate core to mess up your licensing costs.  See my blog post here to get an idea of what I mean.

 

The Real World

Now I want to show you a few things that they don’t talk about in the licensing documents that you are likely to run across in your life as an OVM administrator.

  • live migrate a pinned VM from one OVM Server to another

Capture 2

As you can see above, we have 4 VMs running in this cluster.  Below is an overview of prod_db1.  Take note of the ID field, we’ll use it later to identify the VM:

Capture1

We’re gonna use prod_db1 as our guinea pig for this experiment.  Currently prod_db1 is running on server OVM1 and is pinned to vCPUs 0-3 as noted in the vm.cfg snippet below:

Capture3

I also have a VM running on server ovm2 that is pinned to the very same vCPUs:

Capture4

One would think you cannot live migrate the VM from ovm1 to ovm2 because of the fact that prod_db3 is already pinned to the same vCPUs on ovm2?

Screenshot (7)

 

You certainly can perform the live migration.  Here’s what will happen:

  • The VM will successfully migrate to ovm2
  • prod_db1 will only run on vCPUs 0-3 on ovm2
  • prod_db3 will only run on vCPUs 0-3 on ovm2
  • your performance in both VMs will likely go down the drain
  • you will be out of compliance with Oracle hard partition licensing requirements

 

I’ve had a LOT of people ask me this question, so here’s your proof:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 20 -b- 3988.0 any cpu
0004fb00000600000632b8de1db5a014 3 1 21 -b- 133.8 any cpu
0004fb00000600008825773ba1661d01 2 0 0 -b- 3083.6 0-3
0004fb00000600008825773ba1661d01 2 1 3 -b- 308.1 0-3
Domain-0 0 0 0 r-- 63990.1 0
Domain-0 0 1 1 r-- 62421.0 1
Domain-0 0 2 2 -b- 16102.8 2
Domain-0 0 3 3 -b- 10355.7 3
Domain-0 0 4 4 -b- 2718.1 4
Domain-0 0 5 5 -b- 9427.4 5
Domain-0 0 6 6 -b- 5660.8 6
Domain-0 0 7 7 -b- 3932.0 7
Domain-0 0 8 8 -b- 2268.0 8
Domain-0 0 9 9 -b- 8477.9 9
Domain-0 0 10 10 -b- 4950.6 10
Domain-0 0 11 11 -b- 4304.6 11
Domain-0 0 12 12 -b- 2001.5 12
Domain-0 0 13 13 -b- 10321.1 13
Domain-0 0 14 14 -b- 5221.5 14
Domain-0 0 15 15 -b- 3515.0 15
Domain-0 0 16 16 -b- 2408.8 16
Domain-0 0 17 17 -b- 9905.2 17
Domain-0 0 18 18 -b- 6105.3 18
Domain-0 0 19 19 -b- 4504.2 19



[root@ovm2 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
Domain-0 0 0 0 -b- 54065.1 0
Domain-0 0 1 1 -b- 10110.4 1
Domain-0 0 2 2 -b- 4909.4 2
Domain-0 0 3 3 -b- 6344.0 3
Domain-0 0 4 4 -b- 1012.4 4
Domain-0 0 5 5 -b- 6506.3 5
Domain-0 0 6 6 -b- 4163.1 6
Domain-0 0 7 7 -b- 1564.5 7
Domain-0 0 8 8 -b- 1367.5 8
Domain-0 0 9 9 -b- 14307.2 9
Domain-0 0 10 10 -b- 4068.7 10
Domain-0 0 11 11 -b- 1799.4 11
Domain-0 0 12 12 -b- 1731.3 12
Domain-0 0 13 13 -b- 5478.0 13
Domain-0 0 14 14 -b- 6983.5 14
Domain-0 0 15 15 -b- 5781.6 15
Domain-0 0 16 16 -b- 723.4 16
Domain-0 0 17 17 r-- 4922.6 17
Domain-0 0 18 18 r-- 3585.3 18
Domain-0 0 19 19 -b- 1705.8 19
0004fb0000060000c9e5303a8dc2c675 3 0 0 -b- 5556.6 0-3
0004fb0000060000c9e5303a8dc2c675 3 1 3 -b- 144.4 0-3
  • Now I live migrate prod_db1 from ovm1 to ovm2

Screenshot (8)Screenshot (9)

Screenshot (10)

 

Here is the new vcpu-list post-migration:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 20 -b- 4007.2 any cpu
0004fb00000600000632b8de1db5a014 3 1 21 -b- 134.4 any cpu
Domain-0 0 0 0 r-- 64376.4 0
Domain-0 0 1 1 r-- 62793.1 1
Domain-0 0 2 2 -b- 16201.5 2
Domain-0 0 3 3 -b- 10418.6 3
Domain-0 0 4 4 -b- 2743.2 4
Domain-0 0 5 5 -b- 9486.1 5
Domain-0 0 6 6 -b- 5702.4 6
Domain-0 0 7 7 -b- 3955.7 7
Domain-0 0 8 8 -b- 2279.8 8
Domain-0 0 9 9 -b- 8530.4 9
Domain-0 0 10 10 -b- 4984.4 10
Domain-0 0 11 11 -b- 4328.3 11
Domain-0 0 12 12 -b- 2013.2 12
Domain-0 0 13 13 -b- 10390.7 13
Domain-0 0 14 14 -b- 5257.2 14
Domain-0 0 15 15 -b- 3542.0 15
Domain-0 0 16 16 -b- 2422.3 16
Domain-0 0 17 17 -b- 9969.5 17
Domain-0 0 18 18 -b- 6150.0 18
Domain-0 0 19 19 -b- 4532.5 19



[root@ovm2 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600008825773ba1661d01 5 0 2 -b- 1.9 0-3
0004fb00000600008825773ba1661d01 5 1 1 -b- 0.2 0-3
Domain-0 0 0 0 -b- 54418.2 0
Domain-0 0 1 1 -b- 10228.5 1
Domain-0 0 2 2 -b- 4939.8 2
Domain-0 0 3 3 -b- 6373.9 3
Domain-0 0 4 4 -b- 1024.7 4
Domain-0 0 5 5 -b- 6547.6 5
Domain-0 0 6 6 -b- 4218.0 6
Domain-0 0 7 7 -b- 1596.2 7
Domain-0 0 8 8 -b- 1374.9 8
Domain-0 0 9 9 -b- 14341.6 9
Domain-0 0 10 10 -b- 4099.5 10
Domain-0 0 11 11 -b- 1822.6 11
Domain-0 0 12 12 -b- 1737.6 12
Domain-0 0 13 13 r-- 5513.4 13
Domain-0 0 14 14 -b- 7016.8 14
Domain-0 0 15 15 -b- 5814.6 15
Domain-0 0 16 16 -b- 731.6 16
Domain-0 0 17 17 -b- 4960.6 17
Domain-0 0 18 18 -b- 3617.2 18
Domain-0 0 19 19 -b- 1714.2 19
0004fb0000060000c9e5303a8dc2c675 3 0 3 -b- 5590.3 0-3
0004fb0000060000c9e5303a8dc2c675 3 1 0 -b- 145.6 0-3

 

You can see that both VMs are pinned to the same vCPUs and they’re still running just fine.  Like I said- it will technically work but you’re shooting yourself in the foot in multiple ways if you do this.  Also keep in mind- if you turn on HA for prod_db1 and ovm1 goes down, the VM will fail to start on ovm2 because of the cpu pinning.  Don’t say I didn’t warn you!

 

  • Apply CPU pinning to a VM online with no reboot

In OVM 3.2 and 3.3, you were able to apply CPU pinning to a VM live without having to restart it.  A bug emerged in OVM 3.4.1 and 3.4.2 that broke this.  However it was fixed in OVM 3.4.3.  So depending on which version of OVM you’re running, you may be able to pin your VMs without having to take a reboot.  Watch and be amazed!

 

Currently running OVM 3.3.3:

[root@ovm1 ~]# cat /etc/ovs-release
Oracle VM server release 3.3.3

 

ovm_vmcontrol utilities are installed:

[root@ovmm ovm_util]# pwd
/u01/app/oracle/ovm-manager-3/ovm_util
[root@ovmm ovm_util]# ls -la
total 44
drwxrwxr-x 5 root root 4096 Jul 2 2014 .
drwxr-xr-x 11 oracle dba 4096 Aug 29 13:04 ..
drwxrwxr-x 2 root root 4096 Jul 2 2014 class
drwxr-xr-x 2 root root 4096 Jul 2 2014 lib
drwxr-xr-x 3 root root 4096 Jul 2 2014 man
-rwxr-xr-x 1 root root 1229 Jul 2 2014 ovm_reporestore
-rwxr-xr-x 1 root root 1227 Jul 2 2014 ovm_vmcontrol
-rwxr-xr-x 1 root root 1245 Jul 2 2014 ovm_vmdisks
-rwxr-xr-x 1 root root 1245 Jul 2 2014 ovm_vmhostd
-rwxr-xr-x 1 root root 1246 Jul 2 2014 ovm_vmmessage
-rwxr-xr-x 1 root root 2854 Jul 2 2014 vm-dump-metrics

 

I have an existing VM that is currently allowed to run on any vCPU on the server:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 20 -b- 4012.8 any cpu
0004fb00000600000632b8de1db5a014 3 1 21 -b- 134.6 any cpu
Domain-0 0 0 0 -b- 64446.0 0
Domain-0 0 1 1 -b- 62820.1 1
Domain-0 0 2 2 -b- 16213.7 2
Domain-0 0 3 3 -b- 10426.0 3
Domain-0 0 4 4 -b- 2746.1 4
Domain-0 0 5 5 -b- 9499.3 5
Domain-0 0 6 6 -b- 5712.5 6
Domain-0 0 7 7 -b- 3960.2 7
Domain-0 0 8 8 -b- 2282.3 8
Domain-0 0 9 9 -b- 8541.0 9
Domain-0 0 10 10 -b- 4992.0 10
Domain-0 0 11 11 -b- 4334.6 11
Domain-0 0 12 12 -b- 2015.6 12
Domain-0 0 13 13 -b- 10404.4 13
Domain-0 0 14 14 -b- 5265.1 14
Domain-0 0 15 15 -b- 3546.7 15
Domain-0 0 16 16 -b- 2423.7 16
Domain-0 0 17 17 r-- 9983.8 17
Domain-0 0 18 18 -b- 6158.2 18
Domain-0 0 19 19 -b- 4536.8 19

 

Now let’s pin that VM to vcpu 8-11:

[root@ovmm ovm_util]# ./ovm_vmcontrol -u admin -p ******** -h localhost -v prod_db2 -c vcpuset -s 8-11
Oracle VM VM Control utility 2.0.1.
Connecting with a secure connection.
Connected.
Command : vcpuset
Pinning virtual CPUs
Pinning of virtual CPUs to physical threads '8-11' 'prod_db2' completed.

 

And here’s our proof that the pinning is applied immediately with no reboot:

[root@ovm1 ~]# xm vcpu-list
Name ID VCPU CPU State Time(s) CPU Affinity
0004fb00000600000632b8de1db5a014 3 0 10 -b- 4013.6 8-11
0004fb00000600000632b8de1db5a014 3 1 8 -b- 134.6 8-11
Domain-0 0 0 0 -b- 64454.8 0
Domain-0 0 1 1 -b- 62823.2 1
Domain-0 0 2 2 -b- 16215.2 2
Domain-0 0 3 3 -b- 10427.0 3
Domain-0 0 4 4 -b- 2746.3 4
Domain-0 0 5 5 r-- 9500.6 5
Domain-0 0 6 6 -b- 5713.6 6
Domain-0 0 7 7 -b- 3960.6 7
Domain-0 0 8 8 -b- 2282.5 8
Domain-0 0 9 9 -b- 8542.9 9
Domain-0 0 10 10 -b- 4992.8 10
Domain-0 0 11 11 -b- 4335.0 11
Domain-0 0 12 12 -b- 2015.8 12
Domain-0 0 13 13 -b- 10406.7 13
Domain-0 0 14 14 -b- 5266.4 14
Domain-0 0 15 15 -b- 3547.2 15
Domain-0 0 16 16 -b- 2424.2 16
Domain-0 0 17 17 -b- 9984.8 17
Domain-0 0 18 18 -b- 6159.6 18
Domain-0 0 19 19 -b- 4537.6 19

 

You’ll just have to take my word that I didn’t reboot the VM inbetween the steps- which should be validated by the time column for that VM (note that it increased a little, not reset to 0).

 

 

Well- happy hunting for now!

Advertisements

Virtualized ODA X6-2HA – working with VMs

It’s been awhile since I built a virtualized ODA with VMs on a shared repo so I thought I’d go through the basic steps.

  1. install the OS
    1. install Virtual ISO image
    2. configure networking
    3. install ODA_BASE patch
    4. deploy ODA_BASE
    5. configure networking in ODA_BASE
    6. deploy ODA_BASE with configurator
  2. create shared repository.  This is where your specific situation plays out.  Depending on your hardware you may have less or more space in DATA or RECO.  Your DBA will be able to tell you how much they need for each and where you can borrow a few terabytes (or however much you need) for your VMs
  3. (optionally) create a separate shared repository to store your templates.  This all depends on how many of the same kind of VM you’ll be deploying.  If it makes no sense to keep the templates around once you create your VMs then don’t bother with this step
  4. import template into repository
    1. download the assembly file from Oracle (it will unzip into an .ova archive file)
    2. ***CRITICAL*** copy the .ova to /OVS on either nodes’ DOM0, not into ODA_BASE
    3. import the assembly (point it to the file sitting in DOM0 /OVS)
  5. modify template config as needed (# of vCPUs, Memory, etc)
  6. clone the template to a VM
  7. add network to VM (usually net1 for first public network, net2 for second and net3+ for any VLANs you’ve created
  8. boot VM and start console (easiest way is to VNC into ODA_BASE and launch it from there)
  9. set up your hostname, networking, etc the way you want it
  10. reboot VM to ensure changes persist
  11. rinse and repeat as needed

If you need to configure HA, preferred node or any other things, this is the time to do it.

 

ODA Software – Closed for Business!

I’ve deployed a number of these appliances over the last couple years both virtualized and bare metal.  When people realize that Oracle Linux is running under the hood they sometimes think it’s ok to throw rpmforge up in there and have at it.  What’s worse is a customer actually tried to do a yum update on the OS itself from the Oracle public YUM repo!   Ack….

 

I guess I can see wanting to stay patched to the latest available kernel or version of tools, but it needs to be understood that this appliance is a closed ecosystem.  The beauty of patching the ODA is the fact that I don’t have to chase down all the firmware updates for HDD/SSD/NVM disks, ILOM, BIOS, etc…  That legwork has already been done for me.  Plus the fact that all the patches are tested as a unit together on each platform makes me able to sleep better at night.  Sure- the patches take about 4-5 hours all said and done, but when you’re done, you’re done!  I’m actually wondering if Oracle will eventually implement busybox or something like it for the command line interface to hide the OS layer from end users.  With their move to a web interface for provisioning of the ODA X6-2S/M/L it seems they’ve taken a step in that direction.

 

If you decide to add repositories to your ODA in order to install system utilities like sysstat and such, it’s generally ok, but I need to say this:  the Oracle hard line states that no additional software should be installed on the ODA at all.  In support of that statement, I will say that I’ve had problems patching when the Oracle public YUM repo is configured and I also ran into the expired RHN key error that started rearing its ugly head at the beginning of 2017.  Both of these are easily fixed, but why put yourself in that position in the first place?

 

Also, in closing I’d like to recommend to all my customers/readers that you make it a priority to patch your ODA at least once a year.  There are actual ramifications to being out of date that have bitten folks.  I can think of one case where the customers’ ODA hadn’t been updated in 3-4 years.  The customer experienced multiple Hard Drive failures within a week or two and because they had their ODA loaded to the kilt, the ASM rebuild was impacting performance dramatically.  The reason the drives failed so close to eachother and more importantly the way they failed was because of outdated disk firmware.  Newer firmware was available that changed the way disk failure was performed in that it was more sensitive to “blips” and failed out the disk instead of letting it continue to stay in service.  As a result, the disk was dying for awhile and causing degraded performance.  Another reason the disks probably failed early-ish is the amount of load they were placing on the system.  Anywho… just remember to patch ok?

 

 

Putting the Oracle SPARC M7 Chip through its paces

From time to time I get an opportunity to dive under the hood of some pretty cool technologies in my line of work.  Being an Oracle Platinum Partner, Collier IT specializes in Oracle based hardware and software solutions.  On the hardware side we work with Exadata, Oracle Database Appliance and the Oracle ZFS Appliance just to name a few.  We have a pretty nice lab that includes our own Exatada and ODA, and just recently a T7-2.

 

download (1)Featuring the new SPARC M7 chip released in October of 2015 with Software in Silicon technology, the M7-x and T7-x server line represents a huge leap forward in Oracle Database performance.  The difference between the M7 and T7 servers is basically size and power.  The chip itself is called M7, not to be confused with the server model M7-x.  The T7-x servers also use the same M7 processor.  Hopefully that clears up any confusion on this going forward.  Here’s a link to a datasheet that outlines the server line in more detail.

 

In addition to faster on-chip encryption and real time data integrity checking, SQL query acceleration provides an extremely compelling use case for consolidation while maintaining a high level of performance and security with virtually no overhead.  The SPARC line of processors has come a very long way indeed since it’s infancy.  Released in late 1987, it was designed from the start to provide a highly scalable architecture around which to build a compute package that ranged from embedded processors all the way up to large server based CPU’s while utilizing the same core instruction set.  The name SPARC itself stands for Scalable Processor ARChitecture.  Based on the RISC (Reduced Instruction Set Computer) architecture, operations are designed to be as simple as possible.  This helps achieve nearly one instruction per CPU cycle which allows for greater speed and simplicity of hardware.  Furthermore this helps promote consolidation of other functions such as memory management or Floating Point operations on the same chip.

 

Some of what the M7 chip is doing has actually been done in principle for decades.  Applications such as Hardware Video Acceleration or Cryptographic Acceleration leverage instruction sets hard coded into the processor itself yielding incredible performance.  Think of it as a CPU that has only one job in life- to do one thing and do it very fast.  Modern CPUs such as the Intel x86 cpu have many many jobs to perform and they have to juggle all of them at once.  They are very powerful however because of the sheer number of jobs they are asked to perform, they don’t really excel at any one thing.  Call them a jack of all trades and master of none.  The concept of what a dedicated hardware accelerator is doing for Video playback for example, is what Oracle is doing with Database Instructions such as SQL in the M7 chip.  The M7 processor is still a general purpose CPU, however with the ability to perform in hardware database related instructions at machine level speeds with little to no overhead.  Because of this, the SPARC M7 is able to outperform all other general purpose processors that have to timeshare those types of instructions along with all the other workloads they’re being asked to perform.

 

sprinting-runnerA great analogy would be comparing an athlete who competes in a decathlon to a sprint runner.  The decathlete is very good at running fast, however he needs to be proficient in 9 other areas of competition.  Because of this, the decathlete cannot possibly be as good at running fast as the sprinter because the sprinter is focusing on doing just one thing and being the best at it.  In the same vein, the M7 chip also performs SQL instructions like a sprinter.  The same applies to encryption and real time data compression.

 

Having explained this concept, we can now get into practical application.  The most common use case will be for accelerating Oracle Database workloads.  I’ll spend some time digging into that in my next article.  Bear in mind that there are also other applications such as crypto acceleration and hardware data compression that are accelerated as well.

 

Over the past few weeks, we’ve been doing some benchmark comparisons between 3 very different Oracle Database hardware configurations.  The Exadata (x5), the Oracle Database Appliance (x5) and an Oracle T7-2 are the three platforms that were chosen.  There is a white paper that Collier IT is in the process of developing which I will be a part of.  Because the data is not yet fully analyzed, I can’t go into specifics on the results.  What I can say is that the T7-2 performed amazingly well from a price/performance perspective compared to the other two platforms.

 

Stay tuned for more details on a new test with the S7 and a Nimble CS-500 array as well as a more in depth look at how the onboard acceleration works including some practical examples.

 

 

 

 

 

 

hjh