Log into sftp server with filezilla using ssl key

Prerequisites:

  1. create account on sftp server (optionally in a chroot sftp only environment for safety)
  2. generate an rsa ssl key with puttygen
  3. save the key in both public and private key (.ppk) format
  4. copy the public key to the remote system you’re connecting to in the appropriate location (usually ~username/.ssh/authorized_keys)
  • copy the private .ppk key you created to the local system
  • open filezilla
  • click file -> Site Manager
  • click New Site
    • Host: {address of sftp server}
    • Port: 22
    • Protocol: sftp
    • Logon Type: key file
    • user: {username created on the sftp server}
    • Key file: {browse to location where you saved the .ppk file}
    • click connect
  • In the left pane, cd to the location of the file you want to upload
  • Drag the file you want to upload from the left pane to the right pane and wait for the upload to complete
  • Close the program

 

DONE!

Advertisements

Log into UNIX servers with key via Putty

  • Create an rsa keypair with no passphrase using puttygen
  • Save the public key
  • Save the private key (.ppk format)
  • Configure PuTTy to use your private key
    • Connection -> SSH -> Auth -> Private key for authentication
  • Configure PuTTy to automatically log you in with your username
    • Connection -> Data -> Auto-login Username
  • Save the profile in PuTTy
  • Copy your public key to ~/.ssh/authorized_hosts on each server you want to connect to

Oracle X7-2 HA ODA fiber interface issues

I was working with a customer to deploy an X7-2HA ODA awhile back.  They opted to use 10gbE fiber (as most customers do) for their public network interface.  One problem I ran into quite early on is what turned out to be a bug in the driver for those onboard SFP ports.  They actually can negotiate up to 25gb in addition to 10gb.  This is what caused my problem- the fiber switch ports couldn’t do 25gb and that’s what the onboard adapters was trying to negotiate.

I applied the updated driver to the NIC and rebooted.  Nothing.  No link no packets.  I verified with ethtool that the NIC was only advertising 10gb as its max speed and not 25 like before.  After some troubleshooting, I disabled autonegotiation and forced each of the two adapters to 10gb.  A few seconds after this the link came up and I was able to ping my default gateway!

After reboot however, same thing- no link.

33

 

I wound up having to put the following lines at the end of /etc/rc.local on each compute node:

# btbond1 - force em2/em3 to 10 gig Ethernet speed
/sbin/ethtool -s em2 autoneg off speed 10000
/sbin/ethtool -s em3 autoned off speed 10000
sleep 10
ping -c 5 {default gateway}
sleep 10
ping -c 5 {default gateway}

This basically ensures the adapters get forced to 10gb and makes them ping to “wake up” the interface at the end of the system boot process.  Make sure you don’t forget to limit the ping to 5 packets or something reasonable, otherwise guess what you’re going to see on your console every second until eternity?

I’ve been assured this has been fixed in future driver releases- and I’m pretty confident that it will be or there are going to be quite a few pissed off Oracle customers out there!

Rescan virtual disk in a VMware linux VM without reboot

I’ve run into this situation a number of times.  I get a request from a user to resize a filesystem and add some space to it.  Thankfully, through the magic of virtualization I can change the size of the disk on the fly (assuming there are no snapshots in effect).

 

Resizing the disk in VMware goes fine, so I log into the VM.  Normally for a physical machine with SAN or even SCSI disks, I’d go to /sys/class/scsi_host/ and figure out which adapter the disk I want to resize is sitting on.  Usually a combination of fdisk and lsscsi will give me the info I need here.  Good to go!  So I cd into the hostX folder that represents the right controller.  Here’s where things go south.  I’ve had luck with sending an all HCTL scan and the disk recognizes the new size:
 

echo "- - -" > scan

 

By the way, when I mentioned sending an all HCTL scan, let me explain what that means. HCTL stands for scsi_Host, Channel, Target and Lun. When you’re looking at the output of lsscsi as you’ll see below shortly, you’ll see some numbers separated by colons like such:

[2:0:3:0] disk VMware Virtual disk 1.0 /dev/sdd

The four numbers here represent the following
2 = scsi_host
    This is the numeric iteration of the host bus adapter that the scsi device is connected to.  Think of a scsi card or fiber channel card here.  The first scsi_host is usually the internal ones built into most servers as they usually get enumerated first during POST.

0 = Channel
    This is the channel on the HBA that is being referred to.  Think of a dual channel SCSI card or a dual port Fiber Channel HBA.  

3 = Target
    This refers to the SCSI target of the device we're looking at.  In this case, a good description would be an internal SCSI card that has a tape drive, CDROM and a couple hard drives attached to it.  Each of those devices would have a separate "target" to address that device specifically.

0 = Logical Unit Number or LUN
    This is the representation of a sub unit of a target.  A good example would be an optical disk library where the drive itself gets a SCSI target and assumes LUN 0, then the optical disks themselves get assigned LUNs so they can be addressed.  This also more commonly comes into play when you have a SAN that is exporting multiple disks (a.k.a. LUNs).  Say you have an EMC SAN that is presenting 30 disks to a server.  Based on conventional SCSI limitations, most cards can only address up to 15 targets per HBA channel (some will do up to 24 but they are extremely rare).  In this scenario you would need a couple SCSI HBAs to see all those disks.  Now picture thousands of disks... You see where I'm going with this.

I’ve always had luck seeing newly presented disks by doing an all HCTL scan, even in VMware.  But I always wound up having to reboot the damn VM just to get it to recognize the new size of the disk.  Well I stumbled upon a slightly different process today that lets me do what I’ve been trying to do.  Here’s the breakdown:

  • Determine which disk you want to resize.  fdisk -l usually does the trick:
[root@iscsi ~]# fdisk -l

Disk /dev/sda: 32.2 GB, 32212254720 bytes
64 heads, 32 sectors/track, 30720 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0009e4a5

Device Boot Start End Blocks Id System
/dev/sda1 * 2 501 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 502 30720 30944256 8e Linux LVM
Partition 2 does not end on cylinder boundary.

Disk /dev/mapper/VolGroup-lv_root: 27.5 GB, 27455913984 bytes
255 heads, 63 sectors/track, 3337 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/VolGroup-lv_swap: 4227 MB, 4227858432 bytes
255 heads, 63 sectors/track, 514 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb: 1099.5 GB, 1099511627776 bytes
255 heads, 63 sectors/track, 133674 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x02020202

Disk /dev/sdc: 12.9 GB, 12884901888 bytes
64 heads, 32 sectors/track, 12288 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/sdd: 16.6 GB, 32212254720 bytes
256 heads, 63 sectors/track, 3900 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sdd1 1 2081 16777215+ ee GPT
[root@iscsi ~]#
  • Ok so /dev/sdd is the disk I want to resize.  I go resize it in VMware and re-run fdisk but still get the same size- no shock there.
  • Now let’s use the lsscsi command to show us some information about the scsi devices that the OS sees.  You may have to install this tool first.
[root@iscsi ~]# lsscsi -v
[1:0:0:0] cd/dvd NECVMWar VMware IDE CDR10 1.00 /dev/sr0
dir: /sys/bus/scsi/devices/1:0:0:0 [/sys/devices/pci0000:00/0000:00:07.1/host1/target1:0:0/1:0:0:0]
[2:0:0:0] disk VMware Virtual disk 1.0 /dev/sda
dir: /sys/bus/scsi/devices/2:0:0:0 [/sys/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host2/target2:0:0/2:0:0:0]
[2:0:1:0] disk Nimble Server 1.0 /dev/sdb
dir: /sys/bus/scsi/devices/2:0:1:0 [/sys/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host2/target2:0:1/2:0:1:0]
[2:0:2:0] disk Nimble Server 1.0 /dev/sdc
dir: /sys/bus/scsi/devices/2:0:2:0 [/sys/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host2/target2:0:2/2:0:2:0]
[2:0:3:0] disk VMware Virtual disk 1.0 /dev/sdd
dir: /sys/bus/scsi/devices/2:0:3:0 [/sys/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host2/target2:0:3/2:0:3:0]

 

  • I used the -v (verbose) flag to have it tell me more information about each scsi device.  This gives us a great shortcut to decoding where in /sys/class/scsi_device our target resides.
  • To tell the scsi subsystem to rescan the scsi device, we simply echo a 1 to the rescan file which is located inside the folder identified above.  In our case, the folder is /sys/devices/pci0000:00/0000:00:15.0/0000:03:00.0/host2/target2:0:3/2:0:3:0.  If you cd into this folder, you see a bunch of entries.  BE CAREFUL here, these file entries represent in most cases a live view of what the system is seeing or doing.  If you do the wrong thing, you could tell the disk to power off, or maybe something even more destructive.  There’s no failsafe, the OS isn’t going to ask you if you’re sure you want to do this- we’re poking around in the live kernel here through the “back door” if you will.  Here’s a listing of what my systems shows:
[root@iscsi 2:0:3:0]# ls -la
total 0
drwxr-xr-x 8 root root 0 Mar 21 09:22 .
drwxr-xr-x 4 root root 0 Mar 21 09:22 ..
drwxr-xr-x 3 root root 0 Mar 21 09:22 block
drwxr-xr-x 3 root root 0 Mar 21 09:33 bsg
--w------- 1 root root 4096 Mar 21 09:33 delete
-r--r--r-- 1 root root 4096 Mar 21 09:33 device_blocked
-rw-r--r-- 1 root root 4096 Mar 21 09:33 dh_state
lrwxrwxrwx 1 root root 0 Mar 21 09:33 driver -> ../../../../../../../bus/scsi/drivers/sd
-r--r--r-- 1 root root 4096 Mar 21 09:33 evt_media_change
lrwxrwxrwx 1 root root 0 Mar 21 09:30 generic -> scsi_generic/sg4
-r--r--r-- 1 root root 4096 Mar 21 09:33 iocounterbits
-r--r--r-- 1 root root 4096 Mar 21 09:33 iodone_cnt
-r--r--r-- 1 root root 4096 Mar 21 09:33 ioerr_cnt
-r--r--r-- 1 root root 4096 Mar 21 09:33 iorequest_cnt
-r--r--r-- 1 root root 4096 Mar 21 09:33 modalias
-r--r--r-- 1 root root 4096 Mar 21 09:33 model
drwxr-xr-x 2 root root 0 Mar 21 09:33 power
-r--r--r-- 1 root root 4096 Mar 21 09:33 queue_depth
-r--r--r-- 1 root root 4096 Mar 21 09:33 queue_type
--w------- 1 root root 4096 Mar 21 09:33 rescan
-r--r--r-- 1 root root 4096 Mar 21 09:30 rev
drwxr-xr-x 3 root root 0 Mar 21 09:22 scsi_device
drwxr-xr-x 3 root root 0 Mar 21 09:33 scsi_disk
drwxr-xr-x 3 root root 0 Mar 21 09:33 scsi_generic
-r--r--r-- 1 root root 4096 Mar 21 09:30 scsi_level
-rw-r--r-- 1 root root 4096 Mar 21 09:33 state
lrwxrwxrwx 1 root root 0 Mar 21 09:33 subsystem -> ../../../../../../../bus/scsi
-rw-r--r-- 1 root root 4096 Mar 21 09:33 timeout
-r--r--r-- 1 root root 4096 Mar 21 09:33 type
-rw-r--r-- 1 root root 4096 Mar 21 09:33 uevent
-r--r--r-- 1 root root 4096 Mar 21 09:33 vendor
[root@iscsi 2:0:3:0]#
  • The file we’re interested in is called “rescan”.  The way these generally work is you can poke a value into the kernel by echoing that value into the file like you were appending something to a text file.  Depending on the kernel parameter you’re working with, the value you poke into it will determine what action it takes.  They generally take a 1 or 0 for true or false.  In this case, we want the kernel to rescan this device so we echo “1” > rescan.  This tells the kernel to take another look at the device itself and register any changes that have been made since the system first became aware of it at boot time.
[root@iscsi 2:0:3:0]# echo "1" > rescan
[root@iscsi 2:0:3:0]# fdisk -l /dev/sdd

WARNING: GPT (GUID Partition Table) detected on '/dev/sdd'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/sdd: 32.2 GB, 32212254720 bytes
256 heads, 63 sectors/track, 3900 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot Start End Blocks Id System
/dev/sdd1 1 2081 16777215+ ee GPT

 

Notice the new size of /dev/sdd is now 32GB where it was 16GB before.  Congratulations!  Ok don’t get all excited just yet, your journey has just begun.  Now you have to update the partition table to reflect the new number of sectors so the OS can use the new space.  Then you have to resize whatever filesystem resides on that disk.  If you’re using LVM or software RAID, you’ll need to make some changes at that level first before messing with the filesystem.  The good news is most if not all of this stuff can be done online without having to reboot.

 

I hope this helps and let me know if you have any questions or input as to how to do this better!

 

OVM Manager Cipher Mismatch fix

I was installing a virtual OVM 3.3.3 test environment the other day and when I got to logging into OVM Manager for the first time I got this error:

3ssOL

This has to due with the fact that most modern browsers have dropped support for the older RC4 encryption cipher which is what OVM Manager uses.  There is a “fix” until you update to a newer version that has this bug patched.  See InfoDoc 2099148.1 for all the details, but here’s the meat of it:

 

  • Make a backup of the Weblogic config file
# cd /u01/app/oracle/ovm-manager-3/domains/ovm_domain/config
# cp config.xml config.xml.bak

 

  • Add the following line to the cihpersuite section (search for ciphersuite)
<ciphersuite>TLS_RSA_WITH_AES_128_CBC_SHA</ciphersuite>

 

  • Restart the ovm manager service and all is well
# service ovmm restart

Configure simple DNS server on RHEL 6

Sometimes when setting up hardware for a customer, it makes things a lot easier if I can simulate their network in our lab.  This allows me to deploy the solution plug and play without having to re-ip a bunch of stuff or wait until I’m on their network to do most of the install.  A couple problems I’ve come across are access to the internet for patches/updates and DNS.

 

I’ve generally used an old netgear or linksys router to front the customer’s internal network inside my lab environment and just connect it to the back of our cable modem.  This solves the first problem- internet access.  The other problem is a bit more involved, since you have to have a DNS server on that network (preferrably on the same IP address as in the real network when it’s deployed) I’ve taken to using Linux as a stepping stone.  It’s really simple to install Linux or grab one that’s already there and plug it into my private sandbox.  Once that’s done, you just need to install and configure a DNS server.  Here is the step by step process (your IP network will be different, just substitute where appropriate). FYI- I’m running Oracle Linux 6.7 with the Red Hat Compatible Kernel for this tutorial. CentOS 6.7 and RHEL 6.7 are no different other than the repositories you point to in order to get your patches.

Let’s install BIND (Berkley Internet Name Domain) better known as DNS

# yum install -y bind bind-utils
[root@tempDNS ~]# yum install -y bind bind-utils
Loaded plugins: refresh-packagekit, security, ulninfo
Setting up Install Process
public_ol6_latest                                                                                            | 1.4 kB     00:00
Resolving Dependencies
--> Running transaction check
---> Package bind.x86_64 32:9.8.2-0.62.rc1.el6_9.2 will be installed
--> Processing Dependency: bind-libs = 32:9.8.2-0.62.rc1.el6_9.2 for package: 32:bind-9.8.2-0.62.rc1.el6_9.2.x86_64
---> Package bind-utils.x86_64 32:9.8.2-0.37.rc1.el6 will be updated
---> Package bind-utils.x86_64 32:9.8.2-0.62.rc1.el6_9.2 will be an update
--> Running transaction check
---> Package bind-libs.x86_64 32:9.8.2-0.37.rc1.el6 will be updated
---> Package bind-libs.x86_64 32:9.8.2-0.62.rc1.el6_9.2 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================================
 Package                   Arch                  Version                                     Repository                        Size
====================================================================================================================================
Installing:
 bind                      x86_64                32:9.8.2-0.62.rc1.el6_9.2                   public_ol6_latest                4.0 M
Updating:
 bind-utils                x86_64                32:9.8.2-0.62.rc1.el6_9.2                   public_ol6_latest                188 k
Updating for dependencies:
 bind-libs                 x86_64                32:9.8.2-0.62.rc1.el6_9.2                   public_ol6_latest                891 k

Transaction Summary
====================================================================================================================================
Install       1 Package(s)
Upgrade       2 Package(s)

Total download size: 5.1 M
Downloading Packages:
(1/3): bind-9.8.2-0.62.rc1.el6_9.2.x86_64.rpm                                                                | 4.0 MB     00:00
(2/3): bind-libs-9.8.2-0.62.rc1.el6_9.2.x86_64.rpm                                                           | 891 kB     00:00
(3/3): bind-utils-9.8.2-0.62.rc1.el6_9.2.x86_64.rpm                                                          | 188 kB     00:00
------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                               3.8 MB/s | 5.1 MB     00:01
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Importing GPG key 0xEC551F03:
 Userid : Oracle OSS group (Open Source Software group) 
 Package: 6:oraclelinux-release-6Server-7.0.5.x86_64 (@anaconda-OracleLinuxServer-201507280245.x86_64/6.7)
 From   : /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : 32:bind-libs-9.8.2-0.62.rc1.el6_9.2.x86_64                                                                       1/5
  Updating   : 32:bind-utils-9.8.2-0.62.rc1.el6_9.2.x86_64                                                                      2/5
  Installing : 32:bind-9.8.2-0.62.rc1.el6_9.2.x86_64                                                                            3/5
  Cleanup    : 32:bind-utils-9.8.2-0.37.rc1.el6.x86_64                                                                          4/5
  Cleanup    : 32:bind-libs-9.8.2-0.37.rc1.el6.x86_64                                                                           5/5
  Verifying  : 32:bind-utils-9.8.2-0.62.rc1.el6_9.2.x86_64                                                                      1/5
  Verifying  : 32:bind-9.8.2-0.62.rc1.el6_9.2.x86_64                                                                            2/5
  Verifying  : 32:bind-libs-9.8.2-0.62.rc1.el6_9.2.x86_64                                                                       3/5
  Verifying  : 32:bind-libs-9.8.2-0.37.rc1.el6.x86_64                                                                           4/5
  Verifying  : 32:bind-utils-9.8.2-0.37.rc1.el6.x86_64                                                                          5/5

Installed:
  bind.x86_64 32:9.8.2-0.62.rc1.el6_9.2

Updated:
  bind-utils.x86_64 32:9.8.2-0.62.rc1.el6_9.2

Dependency Updated:
  bind-libs.x86_64 32:9.8.2-0.62.rc1.el6_9.2

Complete!

Ok, now that we have that done, let’s do a system update to make sure we have all the latest bits and bytes. If this is a production system, consult your companys policy on updates and patches before doing this. I don’t want to be responsible for making the other applications on this server potentially not work for any reason.

[root@tempDNS ~]# yum update -y
Loaded plugins: refresh-packagekit, security, ulninfo
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package ConsoleKit.x86_64 0:0.4.1-3.el6 will be updated
---> Package ConsoleKit.x86_64 0:0.4.1-6.el6 will be an update
---> Package ConsoleKit-libs.x86_64 0:0.4.1-3.el6 will be updated
..
..
..
Complete!

At this point, I generally recommend a reboot so any updates that had prerequisites for a reboot are taken care of. Also it just makes sure the system is at a known good place for our work.

Let’s edit the /etc/named.conf file and replace the options section with our own custom code:

options {
    listen-on port 53 { 127.0.0.1; 192.168.1.50; };
        #listen-on-v6 port 53 { ::1; };
        directory   "/var/named";
        dump-file   "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        allow-query { any; };
        allow-transfer     { localhost; };
        recursion yes;

        dnssec-enable yes;
        dnssec-validation yes;
        dnssec-lookaside auto;

        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";

        managed-keys-directory "/var/named/dynamic";
};

Note above that I have added my local IP address to the end of the listen-on line. Now let’s add a couple zone files.

zone "mydomain.com" IN {
                type master;
                file "mydomain.com.zone";
                allow-update { none; };
};

zone "1.168.192.in-addr.arpa" IN {
                type master;
                file "1.168.192.in-addr.arpa";
                allow-update { none; };
};

Obviously change the domain name to your own on the zone line and the file line. Leave the .zone at the end though.

Here are the two files you want to put into /var/named/

mydomain.com

$TTL 86400
@   IN  SOA     ns1.mydomain.com. root.mydomain.com. (
        2017062601  ;Serial
        3600        ;Refresh
        1800        ;Retry
        604800      ;Expire
        86400       ;Minimum TTL
)
; Specify our nameserver
                IN      NS              ns1.mydomain.com.

; Resolve nameserver hostname to IP
ns1             IN      A               192.168.1.50

; Define hostname -> IP pairs which you wish to resolve
gateway         IN      A               192.168.1.1

1.168.192.in-addr.arpa

$TTL 86400
@       IN      SOA     ns1.mydomain.com.        root.mydomain.com. (
                        2017062601
                        21600      ; refresh after 6 hours
                        3600       ; retry after 1 hour
                        604800     ; expire after 1 week
                        86400 )    ; minimum TTL of 1 day
;
@       IN      NS      ns1.mydomain.com.
;
1       IN      PTR     gateway.mydomain.com.

There are a few things I’d like you to note.

1) You have to update the serial number any time you make a change to the zone file (forward or reverse). I usually use the format YYYYMMDD## where ## is a sequential number starting with 01. This way if you make multiple updates on the same day, the root servers on the internet will know which version is current.

2) Take notice of the . at the end of the entries in the reverse zone file. These have to be there- they terminate the domain hierarchy and tell the server that this is the root so it doesn’t try to keep looking any further.

3) In my example above, I also have an entry for gateway.mydomain.com which has an IP address of 192.168.1.1. This is not normally something you would need or want to do but I wanted to show the syntax of how to do it.

4) For every record you want to add to DNS, it’s a good idea to make sure you also add a reverse record. This lets you do an nslookup or dig against the IP address and it will return the name. A lot of stuff will break or at the very least give you problems if it’s not in place so just get in the habit of doing it.

That’s pretty much it. There are a lot of other nuances that I don’t need to get into here. I almost didn’t write this because there are so many tutorials out there that IMHO are written better than mine. Mainly I wanted to keep it for my own use so I know right where to go when I need to install a quick and dirty DNS server. Hopefully one of you will benefit from this.

Enjoy!!

Virtualized ODA X6-2HA – working with VMs

It’s been awhile since I built a virtualized ODA with VMs on a shared repo so I thought I’d go through the basic steps.

  1. install the OS
    1. install Virtual ISO image
    2. configure networking
    3. install ODA_BASE patch
    4. deploy ODA_BASE
    5. configure networking in ODA_BASE
    6. deploy ODA_BASE with configurator
  2. create shared repository.  This is where your specific situation plays out.  Depending on your hardware you may have less or more space in DATA or RECO.  Your DBA will be able to tell you how much they need for each and where you can borrow a few terabytes (or however much you need) for your VMs
  3. (optionally) create a separate shared repository to store your templates.  This all depends on how many of the same kind of VM you’ll be deploying.  If it makes no sense to keep the templates around once you create your VMs then don’t bother with this step
  4. import template into repository
    1. download the assembly file from Oracle (it will unzip into an .ova archive file)
    2. ***CRITICAL*** copy the .ova to /OVS on either nodes’ DOM0, not into ODA_BASE
    3. import the assembly (point it to the file sitting in DOM0 /OVS)
  5. modify template config as needed (# of vCPUs, Memory, etc)
  6. clone the template to a VM
  7. add network to VM (usually net1 for first public network, net2 for second and net3+ for any VLANs you’ve created
  8. boot VM and start console (easiest way is to VNC into ODA_BASE and launch it from there)
  9. set up your hostname, networking, etc the way you want it
  10. reboot VM to ensure changes persist
  11. rinse and repeat as needed

If you need to configure HA, preferred node or any other things, this is the time to do it.

 

ODA Software – Closed for Business!

I’ve deployed a number of these appliances over the last couple years both virtualized and bare metal.  When people realize that Oracle Linux is running under the hood they sometimes think it’s ok to throw rpmforge up in there and have at it.  What’s worse is a customer actually tried to do a yum update on the OS itself from the Oracle public YUM repo!   Ack….

 

I guess I can see wanting to stay patched to the latest available kernel or version of tools, but it needs to be understood that this appliance is a closed ecosystem.  The beauty of patching the ODA is the fact that I don’t have to chase down all the firmware updates for HDD/SSD/NVM disks, ILOM, BIOS, etc…  That legwork has already been done for me.  Plus the fact that all the patches are tested as a unit together on each platform makes me able to sleep better at night.  Sure- the patches take about 4-5 hours all said and done, but when you’re done, you’re done!  I’m actually wondering if Oracle will eventually implement busybox or something like it for the command line interface to hide the OS layer from end users.  With their move to a web interface for provisioning of the ODA X6-2S/M/L it seems they’ve taken a step in that direction.

 

If you decide to add repositories to your ODA in order to install system utilities like sysstat and such, it’s generally ok, but I need to say this:  the Oracle hard line states that no additional software should be installed on the ODA at all.  In support of that statement, I will say that I’ve had problems patching when the Oracle public YUM repo is configured and I also ran into the expired RHN key error that started rearing its ugly head at the beginning of 2017.  Both of these are easily fixed, but why put yourself in that position in the first place?

 

Also, in closing I’d like to recommend to all my customers/readers that you make it a priority to patch your ODA at least once a year.  There are actual ramifications to being out of date that have bitten folks.  I can think of one case where the customers’ ODA hadn’t been updated in 3-4 years.  The customer experienced multiple Hard Drive failures within a week or two and because they had their ODA loaded to the kilt, the ASM rebuild was impacting performance dramatically.  The reason the drives failed so close to eachother and more importantly the way they failed was because of outdated disk firmware.  Newer firmware was available that changed the way disk failure was performed in that it was more sensitive to “blips” and failed out the disk instead of letting it continue to stay in service.  As a result, the disk was dying for awhile and causing degraded performance.  Another reason the disks probably failed early-ish is the amount of load they were placing on the system.  Anywho… just remember to patch ok?

 

 

Create VM in Oracle VM for x86 using NFS share

I’m using OVM Manager 3.4.2 and OVM Server 3.3.2 to test an upgrade for one of our customers.  I am using Starwind iSCSI server to present the shared storage to the cluster but in production you should use enterprise grade hardware to do this.  There’s an easier way to do this- create an HVM VM and install from an ISO stored in a repository.  Then power the VM off and change the type to PVM then power on.  This may not work with all operating systems however so I’m going over how to create a new PVM VM from an ISO image shared from an NFS server.

* Download ISO (I'm using Oracle Linux 6.5 64bit for this example)
* Copy ISO image to OVM Manager (any NFS server is fine)
* Mount ISO on the loopback device
# mount -o loop /var/tmp/V41362-01.iso /mnt

* Share the folder via NFS
# service nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS mountd: [ OK ]
Starting NFS daemon: [ OK ]
Starting RPC idmapd: [ OK ]

# exportfs *:/mnt/

# showmount -e
Export list for ovmm:
/mnt *

* Create new VM in OVM Manager
* Edit VM properties and configure as PVM
* Set additional properties such as memory, cpu and network
* At the boot order tab, enter the network boot path formatted like this:
  nfs:{ip address or FQDN of NFS host}:/{path to ISO image top level directory}

For example, our NFS server is 10.2.3.4 and the path where I mounted the ISO is at /mnt.  Leave the {}'s off of course:

  nfs:10.2.3.4:/mnt 

You should be able to boot your VM at this point and perform the install of the OS.

SSH Tunneling with PuTTY

From time to time I have a need to connect to a system inside another remote network (usually my work).  Normally I just ssh in and then jump to the machine I need to be on.  That’s all fine and dandy if you don’t need a GUI.  What if you need to be on the GUI console of the target machine inside the firewall and the firewall doesn’t allow the port you need to use?

 

Enter VNC and PuTTY.  You aren’t limited to doing this with PuTTY or VNC.  It’s just that a majority of my work is done from a windows machine and I refuse to install the bloated CYGWIN app on my machine just to get an ssh command line session.  Bah.. that’s a story for another day.  Anyway- SSH tunnels can be a bit confusing to the lay person so I thought I’d do a graphical illustration to help that out.

 

In this scenario, I will be using my laptop at home to connect into a landing pad UNIX machine at work.  I will then open a tunnel to another machine inside the remote network that will establish a connection to the VNC server running on that machine.  I won’t go into how to set up a VNC Server on linux as there are plenty of tutorials out there that will cover it.  The one thing I will say is make sure you use a password when you start it up.  This is a visual example of what the connection looks like:

 

capture

 

Here are some enlarged views so you can see what’s going on.  First we start PuTTY on the laptop.  I’ll show an example of what options you need to select inside the Putty connection later.  Once the tunnel is in place, fire up your favorite VNC client and point it to 127.0.0.1 or localhost on port 59001:

capture1

We pointed our VNC client to the address and port of the tunnel we just created, so the traffic is sent through the tunnel into the external Landing Pad and being forwarded on into the remote network:

capture2

Finally, the tunnel terminates on the server inside the remote network and connects the tunnel to port 5901 on that machine:

capture3

 

It may seem odd to connect your VNC client to the laptop’s localhost address in order to reach the target machine.  This is because you’re sending that traffic through the SSH tunnel that we set up rather than pointing it directly to the server you want to reach.

 

Now I’ll show you how to configure PuTTY to create the tunnel.  First, fire up Putty and populate the username and IP address of the landing pad server in our example (substitute yours of course).  Leave the port at 22:

capture4

 

Next, scroll down on the left hand side in the Category window and select Tunnels.  Here, populate the source port (59001 in my example), the IP address of the final destination server along with the port you want to connect to on that machine (5901 in my example).  Remember, you aren’t putting the IP address of the landing pad here- we want the target server in the Destination field. Once you have the Source port and Destination fields filled in, click Add and it will pop into the window as seen below:

capture5

 

To establish the tunnel, click Open. This will launch the PuTTY terminal and prompt you for your password.  In this screenshot, I’m using root to log in however generally it’s a good idea to use a non-privileged user to log into any machine:

 

capture6

Once you see the user prompt and you’re logged in, the tunnel is now in place.  Keep in mind that this SSH session you have open is the only thing keeping that tunnel open.  If you log out of the shell, it also tears down the tunnel so keep this window open while you’re using the tunnel.

 

The next step is to launch a VNC Viewer on your laptop and point it to your local machine on port 59001:

capture7

Click the connect button and you should see the next window prompting you for the password you set up earlier:

capture8

Finally, once you click OK you will be brought to your VNC Desktop on the machine inside the remote network!

capture9

 

So let’s take a step back and review what we’ve effectively done here:

 

Start VNC server:

We have to start a VNC server on the target computer, along with configuring a password to keep everyone else out.  This would have to be done separately.

 

Establish Tunnel:

We first establish the tunnel from the laptop, through the landing pad and finally to the remote server.  I’m making the obvious assumption here that you have the landing pad accessible to the internet on port 22 and that you have an SSH server running that will accept such connections.  You’re effectively logging into the landing pad just like you would on any other day.  The difference here is that we’re also telling PuTTY to set up a tunnel for us pointing to the remote server as well.  Aside from that- your login session will look and feel just the same.

 

Launch VNC Client:

We then start the VNC client on our laptop.  Normally, we would point it directly to the server we want to VNC into.  In our case, we created a tunnel that terminates on your laptop at port 59001.  So we connect our VNC client to the laptop (localhost or 127.0.0.1 should work) and point it to port 59001 instead of the standard port 5901.  The VNC client doesn’t care how the traffic is getting to the VNC server, it just does its job.

Think of this SSH tunnel as kind of a wormhole if that type of thing were to actually exist.  The traditional method of connecting to your remote endpoint would be similar to pointing our space shuttle towards the Andromeda galaxy which is about 2.5 million light years away.  It’s essentially not possible to get there- similar to a firewall that is blocking us.  But what if there were a wormhole that terminated near Earth that ended in the Andromeda galaxy?  If we were to point our space shuttle into the wormhole, theoretically we would pop out the other side at our target.

 

If you do plan on doing something like this, make sure you network administrator is ok with it.  They may detect the traffic as malicious if they’re not sure where it’s coming from and you may wind up in trouble.  I hope this helps give a basic understanding of how SSH Tunnels work.

 

 

 

 

 

Linux Exploit: How Making Your System More Secure Can Hurt You

51zah9831tlWith all the security exploits in the wild these days, it pays to protect your data.  One would think that encrypting your filesystems would be a good step in the right direction.  Normally it would, but in thstar_trek_ii_the_wrath_of_khan_2009_dvd_cover_region_1is case not so much.  Unlike Iron Maiden’s nod to the doomsday clock in the song 2 minutes to midnight, now it doesn’t even take that long to compromise a system.  No- watch out, here comes Cryptsetup.  We’ll do it for you in 30 seconds!  Ok, maybe too many references to British heavy metal bands and cult classic movies but you get the point.

 


CVE-2016-4484: Cryptsetup Initrd root Shell
was first revealed to “the public” about a week ago at the DeepSec security conference held in Austria and a few days later on the web at large.  The kicker is that it only applies if you have encrypted your system partition.  There is a much more detailed writeup here on how it works, how to tell if you’re vulnerable and how to fix it.  The good news is apparently not many people are encrypting their system partitions because I can’t believe that someone hasn’t even by accident ran across this until now (who hasn’t accidentally left something pressing the enter key on their keyboard like a book or something like that).

Dirty COW Linux Vulnerability – CVE-2016-5195

dirty_cow

A newly reported exploit in the memory mapping section of the Kernel has been reported.  It’s actually been in the kernel for years but just recently became much more dangerous due to recent changes in the kernel structure.  Here’s the alert from Red Hat’s website:

 

Red Hat Product Security has been made aware of a vulnerability in the Linux kernel that has been assigned CVE-2016-5195. This issue was publicly disclosed on October 19, 2016 and has been rated as Important.

Background Information

A race condition was found in the way the Linux kernel’s memory subsystem handled the copy-on-write (COW) breakage of private read-only memory mappings. An unprivileged local user could use this flaw to gain write access to otherwise read-only memory mappings and thus increase their privileges on the system.

This could be abused by an attacker to modify existing setuid files with instructions to elevate privileges. An exploit using this technique has been found in the wild.

 

Here’s a great description of how the exploit works in a 12 minute youtube video

 

Patch patch patch!!

ODA Patching – get ahead of yourself?

I was at a customer site deploying an X5-2 ODA.  They are standardizing on the 12.1.2.6.0 patch level.  Even though 12.1.2.7.0 is currently the latest, they don’t want to be on the bleeding edge.  Recall that the 12.1.2.6.0 patch doesn’t include infrastructure patches (mostly firmware) so you have to install 12.1.2.5.0 first, run the –infra patch to get the firmware and then update to 12.1.2.6.0.

 

We unpacked the 12.1.2.5.0 patch on both systems and then had an epiphany.  Why don’t we just unpack the 12.1.2.6.0 patch as well and save some time later?  What could possibly go wrong?  Needless to say, when we went to install or even verify the 12.1.2.5.0 patch it complained as follows:

ERROR: Patch version must be 12.1.2.6.0

 

Ok, so there has to be a way to clean that patch off the system so I can use 12.1.2.5.0 right?  I stumbled across the oakcli manage cleanrepo command and thought for sure that would fix things up nicely.  Ran it and I got this output:

 


[root@CITX-5ODA-ODABASE-NODE0 tmp]# oakcli manage cleanrepo --ver 12.1.2.6.0
Deleting the following files...
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/OAK/12.1.2.6.0/Base
Deleting the files under /DOM0OAK/12.1.2.6.0/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/Seagate/ST95000N/SF04/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/Seagate/ST95001N/SA03/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/WDC/WD500BLHXSUN/5G08/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HGST/H101860SFSUN600G/A770/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/Seagate/ST360057SSUN600G/0B25/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HITACHI/H106060SDSUN600G/A4C0/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HITACHI/H109060SESUN600G/A720/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HITACHI/HUS1560SCSUN600G/A820/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HGST/HSCAC2DA6SUN200G/A29A/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HGST/HSCAC2DA4SUN400G/A29A/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/STEC/ZeusIOPs-es-G3/E12B/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/STEC/Z16IZF2EUSUN73G/9440/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Expander/ORACLE/DE2-24P/0018/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Expander/ORACLE/DE2-24C/0018/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Expander/ORACLE/DE3-24C/0291/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x0072/11.05.03.00/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x0072/11.05.03.00/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Ilom/SUN/X4370-es-M2/3.0.16.22.f-es-r100119/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HITACHI/H109090SESUN900G/A720/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/STEC/Z16IZF4EUSUN200G/944A/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HGST/H7240AS60SUN4.0T/A2D2/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HGST/H7240B520SUN4.0T/M554/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Disk/HGST/H7280A520SUN8.0T/P554/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Expander/SUN/T4-es-Storage/0342/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x0072/11.05.03.00/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x005d/4.230.40-3739/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x0097/06.00.02.00/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/Mellanox/0x1003/2.11.1280/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Ilom/SUN/X4170-es-M3/3.2.4.26.b-es-r101722/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Ilom/SUN/X4-2/3.2.4.46.a-es-r101689/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Ilom/SUN/X5-2/3.2.4.52-es-r101649/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/HMP/2.3.4.0.1/Base
Deleting the files under /DOM0HMP/2.3.4.0.1/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/IPMI/1.8.12.4/Base
Deleting the files under /DOM0IPMI/1.8.12.4/Base
Deleting the files under /JDK/1.7.0_91/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/ASR/5.3.1/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/OPATCH/12.1.0.1.0/Patches/6880880
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/OPATCH/12.0.0.0.0/Patches/6880880
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/OPATCH/11.2.0.4.0/Patches/6880880
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/GI/12.1.0.2.160119/Patches/21948354
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/DB/12.1.0.2.160119/Patches/21948354
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/DB/11.2.0.4.160119/Patches/21948347
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/DB/11.2.0.3.15/Patches/20760997
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/DB/11.2.0.2.12/Patches/17082367
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/OEL/6.7/Patches/6.7.1
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/OVM/3.2.9/Patches/3.2.9.1
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/OVS/12.1.2.6.0/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x0072/11.05.02.00/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/thirdpartypkgs/Firmware/Controller/LSI-es-Logic/0x0072/11.05.02.00/Base
Deleting the files under $OAK_REPOS_HOME/pkgrepos/orapkgs/GI/12.1.0.2.160119/Base

 

So I assumed that this fixed the problem.  Nope…

 


[root@CITX-5ODA-ODABASE-NODE0 tmp]# oakcli update -patch 12.1.2.5.0 --verify

ERROR: Patch version must be 12.1.2.6.0

 

 

Ok so more searching the CLI manual and the oakcli help pages came up with bupkiss.  So I decided to do an strace of the oakcli command I had just ran.  As ususal- there was a LOT of garbage I didn’t care about or didn’t know what it was doing.  I did find however that it was reading the contents of a file that looked interesting to me:

 


[pid 5509] stat("/opt/oracle/oak/pkgrepos/System/VERSION", {st_mode=S_IFREG|0777, st_size=19, ...}) = 0
[pid 5509] open("/opt/oracle/oak/pkgrepos/System/VERSION", O_RDONLY) = 3
[pid 5509] read(3, "version=12.1.2.6.0\n", 8191) = 19
[pid 5509] read(3, "", 8191) = 0
[pid 5509] close(3) = 0
[pid 5509] fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
[pid 5509] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f159799d000
[pid 5509] write(1, "\n", 1
) = 1
[pid 5509] write(1, "ERROR: Patch version must be 12."..., 40ERROR: Patch version must be 12.1.2.6.0
) = 40
[pid 5509] exit_group(0) = ?

 

There were a dozen or so lines after that, but I had what I needed.  Apparently /opt/oracle/oak/pkgrepos/System/VERSION contains the current version of the latest patch that has been unpacked.  The system software version is kept somewhere else because after I unpacked the 12.1.2.6.0 patch, I ran an oakcli show version and it reported 12.1.2.5.0.  But the VERSION file referenced earlier said 12.1.2.6.0.  I assume when I unpacked the 12.1.2.6.0 patch, it updates this file.  So what I wound up doing is changing the VERSION file back to 12.1.2.5.0 as well as deleting the folder /opt/oracle/oak/pkgrepos/System/12.1.2.6.0.  Once I did this, everything worked as I expected.  I was able to verify and install the –infra portion of 12.1.2.5.0 and continue on my merry way.

 

This highlights the fact that there isn’t a known way (to me at least) to delete an unpacked patch via oakcli or any python scripts I’ve been able to find yet.  Also- as an aside I tried just deleting the VERSION file assuming it would be rebuilt by oakcli and it didn’t.  I got this:

 


[root@CITX-5ODA-ODABASE-NODE0 System]# oakcli update -patch 12.1.2.5.0 --verify
ERROR : Couldn't find the VERSION file to extract the current allowed version

 

So I just recreated the file and all was good.  I was hoping that the oak software didn’t maintain some sort of binary formatted database that kept track of all this information- I think I got lucky in this case.  Hope this helps someone out in a pinch!

ODA X6-2 in the wild!

cw20v1-x62-3-2969092

It looks like Oracle has deployed their newest server (the X6-2) into the ODA appliance lineup now.  It’s already an option on the ExaData, BDA and ZDLRA.  There are now 3 different configurations available, 2 of which don’t include shared storage and have a much lower price point.  You can also run Oracle Database SE2 or EE on the two smaller configurations however neither one offers the virtualization option that’s been around since the original V1 ODA.

 

Here are the 3 options:

Oracle Database Appliance X6-2S ($18k):
One E5-2630 v4 2.2GHz 10 core CPU
6.4 TB (2 x 3.2 TB) NVMe SSDs *
128 GB (4 x 32 GB) DDR4-2400 Main Memory **
Two 480 GB SATA SSDs (mirrored) for OS
Two onboard 10GBase-T Ethernet ports
Dual-port 10GbE SFP+ PCIe

Notes: 
* You can add up to 2 more NVMe SSD’s for a total of 4
** An optional memory expansion kit is available that brings this configuration up to 384GB

 

Oracle Database Appliance X6-2M ($24k):
Two E5-2630 v4 2.2GHz 10 core CPUs
6.4 TB (2 x 3.2 TB) NVMe SSDs *
256 GB (8 x 32 GB) DDR4-2400 Main Memory **
Two 480 GB SATA SSDs (mirrored) for OS
Four onboard 10GBase-T Ethernet ports
Dual-port 10GbE SFP+ PCIe

Notes:
* You can add up to 2 more NVMe SSD’s for a total of 4
** An optional memory expansion kit is available that brings this configuration up to 768GB

 

Oracle Database Appliance X6-2HA (?):
TBD – information about this configuration isn’t available yet.  More info coming soon!

X5-2 ODA upgrade from 12.1.2.5.0 to 12.1.2.6.0 observations

Word on keyboard

More fun with patching!  So this time I’m doing a fresh virtualized install and I decided to take my own sage advice of installing 12.1.2.5.0 first to get the firmware patches.  I ran into a bunch of other issues which will be the topic of a different post but I digress.  I got 12.1.2.5.0 fully installed, ODA_BASE deployed, everything was happy.

 

Remember that starting with version 12.1.2.6.0, you have to patch each node separately with the –local option for the infra patches.  So I started the patch on node 0 and it got almost all the way to the end at step 12 where oakd is being patched.  I ran into the “known issue” in 888888.1 item 9:

9.  During the infra patching, after step 12 completed, IPMI, HMP done, if it appeared to be hang during Patching OAK with the following two lines
                               INIT: Sending processes the TERM signal
                               INIT: no more processes left in this runlevel
JDK is not patched, the infra patching is not complete to the end.  
Workaround:  To reboot the appeared hang node manually, then run 
# oakcli update -patch 12.1.2.6 –clean

# oakcli update -patch 12.1.2.6.0 –infra –local
To let it complete the infra patch cleanly.  

I waited about 30 minutes at this step before I started to wonder, and sure enough after checking some log files in /opt/oracle/oak/onecmd/tmp/ it thought oakd was fully patched.  What I found is that oakd gets whacked because the patch doesn’t fully complete.  After doing the reboot that’s recommended in the workaround above, sure enough oakd is not running.  What’s more- now when I boot ODA_BASE the console doesn’t get to the login prompt and you can’t do anything even though you can ssh in just fine.  So I ran the –clean option then kicked off the patch again.  This time it complained that oakd wasn’t running on the remote node.  It was in fact running on node1 but node0 oakd was not.  I suspect that when the ODA communicates to oakd between nodes it’s using the local oakd to do so.

 

So I manually restarted oakd by running /etc/init.d/init.oak restart and then oakd was running.  I rebooted ODA_BASE on node0 just to be sure everything was clean then kicked off the infra patch again.  This time it went all the way through and finished.  The problem now is that the ODA_BASE console is non responsive no matter what I do so I’ll be opening a case with Oracle support to get a WTF.  I’ll update this post with their answer/solution.  If I were a betting man I’d say they’ll tell me to update to 12.1.2.7.0 to fix it.  We’ll see…

 

As an aside- one of the things that 12.1.2.6.0 does is do an in-place upgrade of Oracle Linux 5.11 to version 6.7 for ODA_BASE.  I’ve never done a successful update that way and in fact, Red Hat doesn’t support it.  I guess I can see why they would want to do an update rather than a fresh install but it still feels very risky to me.