Solaris Commands – off the beaten path

Oracle blogger Giri Mandalika posted some useful and somewhat obscure commands that can be very useful in troubleshooting Solaris.  These commands mostly work with Solaris 10 however some of them are specific to Solaris 11:

 

Interrupt Statistics : intrstat utility
intrstat utility can be used to monitor interrupt activity generated by various hardware devices along with the CPU that serviced the interrupt and the CPU time spent in servicing those interrupts on a system. On a busy system, intrstat reported stats may help figure out which devices are busy at work, and keeping the system busy with interrupts.
eg.,
.. [idle system] showing the interrupt activity on first two vCPUs ..

# intrstat -c 0-1 5

device | cpu0 %tim cpu1 %tim
-------------+------------------------------
cnex#0 | 0 0.0 0 0.0
ehci#0 | 0 0.0 0 0.0
hermon#0 | 0 0.0 0 0.0
hermon#1 | 0 0.0 0 0.0
hermon#2 | 0 0.0 0 0.0
hermon#3 | 0 0.0 0 0.0
igb#0 | 0 0.0 0 0.0
ixgbe#0 | 0 0.0 0 0.0
mpt_sas#0 | 18 0.0 0 0.0
vldc#0 | 0 0.0 0 0.0

device | cpu0 %tim cpu1 %tim
-------------+------------------------------
cnex#0 | 0 0.0 0 0.0
ehci#0 | 0 0.0 0 0.0
hermon#0 | 0 0.0 0 0.0
hermon#1 | 0 0.0 0 0.0
hermon#2 | 0 0.0 0 0.0
hermon#3 | 0 0.0 0 0.0
igb#0 | 0 0.0 0 0.0
ixgbe#0 | 0 0.0 0 0.0
mpt_sas#0 | 53 0.2 0 0.0
vldc#0 | 0 0.0 0 0.0
^C


Check the outputs of the following as well.
# echo ::interrupts | mdb -k
# echo ::interrupts -d | mdb -k

 
Physical Location of Disk : croinfo & diskinfo commands
Both croinfo and diskinfo commands provide information about the chassis, receptacle, and occupant relative to all disks or a specific disk. Note that croinfo and diskinfo utilities share the same executable binary and function in a identical manner. The main difference being the defaults used by each of the utilities.
eg.,

# croinfo
D:devchassis-path t:occupant-type c:occupant-compdev
------------------------------ --------------- ---------------------
/dev/chassis//SYS/MB/HDD0/disk disk c0t5000CCA0125411FCd0
/dev/chassis//SYS/MB/HDD1/disk disk c0t5000CCA0125341F0d0
/dev/chassis//SYS/MB/HDD2 - -
/dev/chassis//SYS/MB/HDD3 - -
/dev/chassis//SYS/MB/HDD4/disk disk c0t5000CCA012541218d0
/dev/chassis//SYS/MB/HDD5/disk disk c0t5000CCA01248F0B8d0
/dev/chassis//SYS/MB/HDD6/disk disk c0t500151795956778Ed0
/dev/chassis//SYS/MB/HDD7/disk disk c0t5001517959567690d0

# diskinfo -oDcpd
D:devchassis-path c:occupant-compdev p:occupant-paths d:occupant-devices
------------------------------ --------------------- ----------------------------------------------------------------------------- -----------------------------------------
/dev/chassis//SYS/MB/HDD0/disk c0t5000CCA0125411FCd0 /devices/pci@400/pci@1/pci@0/pci@0/LSI,sas@0/iport@1/disk@w5000cca0125411fd,0 /devices/scsi_vhci/disk@g5000cca0125411fc
/dev/chassis//SYS/MB/HDD1/disk c0t5000CCA0125341F0d0 /devices/pci@400/pci@1/pci@0/pci@0/LSI,sas@0/iport@2/disk@w5000cca0125341f1,0 /devices/scsi_vhci/disk@g5000cca0125341f0
/dev/chassis//SYS/MB/HDD2 - - -
/dev/chassis//SYS/MB/HDD3 - - -
/dev/chassis//SYS/MB/HDD4/disk c0t5000CCA012541218d0 /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@1/disk@w5000cca012541219,0 /devices/scsi_vhci/disk@g5000cca012541218
/dev/chassis//SYS/MB/HDD5/disk c0t5000CCA01248F0B8d0 /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@2/disk@w5000cca01248f0b9,0 /devices/scsi_vhci/disk@g5000cca01248f0b8
/dev/chassis//SYS/MB/HDD6/disk c0t500151795956778Ed0 /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@4/disk@w500151795956778e,0 /devices/scsi_vhci/disk@g500151795956778e
/dev/chassis//SYS/MB/HDD7/disk c0t5001517959567690d0 /devices/pci@700/pci@1/pci@0/pci@0/LSI,sas@0/iport@8/disk@w5001517959567690,0 /devices/scsi_vhci/disk@g5001517959567690

 
Monitoring Network Traffic Statistics : dlstat command
dlstat command reports network traffic statistics for all datalinks or a specific datalink on a system.
eg.,

# dlstat -i 5 net0
LINK IPKTS RBYTES OPKTS OBYTES
net0 163.12M 39.93G 206.14M 43.63G
net0 312 196.59K 146 370.80K
net0 198 172.18K 121 121.98K
net0 168 91.23K 93 195.57K
^C

For the complete list of options along with examples, please consult the Solaris Documentation.

 
Fault Management : fmstat utility
Solaris Fault Manager gathers and diagnoses problems detected by the system software, and initiates self-healing activities such as disabling faulty components. fmstat utility can be used to check the statistics associated with the Fault Manager.
fmadm config lists out all active fault management modules that are currently participating in fault management. -m option can be used to report the diagnostic statistics related to a specific fault management module. fmstat without any option report stats from all fault management modules.
eg.,

# fmstat 5
module ev_recv ev_acpt wait svc_t %w %b open solve memsz bufsz
cpumem-retire 0 0 1.0 8922.5 96 0 0 0 12b 0
disk-diagnosis 1342 0 1.1 8526.0 96 0 0 0 0 0
disk-transport 0 0 1.0 8600.3 96 1 0 0 56b 0
...
...
zfs-diagnosis 139 75 1.0 8864.5 96 0 4 12 672b 608b
zfs-retire 608 0 0.0 15.2 0 0 0 0 4b 0
...
...
# fmstat -m cpumem-retire 5
NAME VALUE DESCRIPTION
auto_flts 0 auto-close faults received
bad_flts 0 invalid fault events received
cacheline_fails 0 cacheline faults unresolveable
cacheline_flts 0 cacheline faults resolved
cacheline_nonent 0 non-existent retires
cacheline_repairs 0 cacheline faults repaired
cacheline_supp 0 cacheline offlines suppressed
...
...

 
InfiniBand devices : List & Show Information about each device
ibv_devices lists out all available IB devices whereas ibv_devinfo shows information about all devices or a specific IB device.
eg.,

# ibv_devices
device node GUID
------ ----------------
mlx4_0 0021280001cee63a
mlx4_1 0021280001cee492
mlx4_2 0021280001cee4aa
mlx4_3 0021280001cee4ea

# ibv_devinfo -d mlx4_0
hca_id: mlx4_0
transport: InfiniBand (0)
fw_ver: 2.7.8130
node_guid: 0021:2800:01ce:e63a
sys_image_guid: 0021:2800:01ce:e63d
vendor_id: 0x02c9
vendor_part_id: 26428
hw_ver: 0xB0
board_id: SUN0160000002
phys_port_cnt: 2
port: 1
state: PORT_ACTIVE (4)
max_mtu: 2048 (4)
active_mtu: 2048 (4)
sm_lid: 56
port_lid: 95
port_lmc: 0x00
link_layer: IB

port: 2
state: PORT_ACTIVE (4)
max_mtu: 2048 (4)
active_mtu: 2048 (4)
sm_lid: 56
port_lid: 96
port_lmc: 0x00
link_layer: IB

Other commands and utilities such as ibstatus, fwflash or cfgadm can also be used to retrieve similar information.

PCIe Hot-Plugging : hotplug command
When the hotplug service is enabled on a Solaris system, hotplug command to bring hot pluggable devices online or offline without physically adding or removing the device from the system.
The following command lists out the all physical [hotplug] connectors along with the current status.
eg.,

# hotplug list -c
Connection State Description
________________________________________________________________________________
IOU2-EMS2 ENABLED PCIe-Native
IOU2-PCIE6 ENABLED PCIe-Native
IOU2-PCIE7 EMPTY PCIe-Native
IOU2-PCIE4 EMPTY PCIe-Native
IOU2-PCIE1 EMPTY PCIe-Native

For detailed instructions to hotplug a device, check the Solaris documentation out.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s