How to Install & Configure the OSWATCHER (OSWbb) in RAC Database
Posted by Mir Sayeed Hassan on January 16th, 2019
How to Install & COnfigure the OSWATCHER (OSWbb) in RAC Database
This Oracle OSWatcher Black Box (OSWbb) is mainly used in RAC Database for collects & archive OS & Network metrics which use to diagnose performance issue & it Should be install in every NODE of RAC DB.
It will invoke the unix utilities are vmstat, netstat, iostat, and top etc, this feature is useful for Oracle RAC (Real Application Clusters) and Oracle Grid Infrastructure configurations.
Download the oswatcher software file from below link:
https://support.oracle.com/epmos/main/downloadattachmentprocessor?attachid=301137.1:OSW_file.
Login to the root & copy the file into the software folder & extract tar as shown below.
[root@trac1 software]# ls oswbb801.tar
[root@trac1 software]# tar -xvf oswbb801.tar oswbb/ oswbb/docs/ oswbb/docs/DB_OSWatcher_Overview.pdf oswbb/docs/The_Analyzer/ oswbb/docs/The_Analyzer/oswbba_README.txt oswbb/docs/The_Analyzer/oswbba_User_Guide.html oswbb/docs/The_Analyzer/AnalyzerWalkThrough.pdf oswbb/docs/OSWatcher/ oswbb/docs/OSWatcher/oswbb_README.txt oswbb/docs/OSWatcher/OSWatcher_User_Guide.html oswbb/docs/OSWatcher/RAC_Traceroute_Automation_Readme.txt oswbb/oswbba.jar oswbb/nfssub.sh oswbb/iosub.sh oswbb/oswnet.sh oswbb/xtop.sh oswbb/stopOSWbb.sh oswbb/genprvnet.sh oswbb/psmemsub.sh oswbb/startOSWbb.sh oswbb/Example_extras.txt oswbb/Exampleprivate.net oswbb/tar_up_partial_archive.sh oswbb/src/ oswbb/src/js/ oswbb/src/js/modernizr.js oswbb/src/js/jquery-migrate-1.2.1.min.js oswbb/src/js/jquery.lightbox.js oswbb/src/js/scripts.js oswbb/src/js/bootstrap.js oswbb/src/js/templatemo_custom.js oswbb/src/js/tabs.js oswbb/src/js/jquery-1.10.2.min.js oswbb/src/META-INF/ oswbb/src/META-INF/application-client.xml oswbb/src/Thumbs.db oswbb/src/oswbba_input.txt oswbb/src/css/ oswbb/src/css/templatemo_style.css oswbb/src/css/animate.css oswbb/src/css/font-awesome.min.css oswbb/src/css/bootstrap.min.css oswbb/src/css/style.css oswbb/src/css/templatemo_misc.css oswbb/src/fonts/ oswbb/src/fonts/fontawesome-webfont.svg oswbb/src/fonts/fontawesome-webfont.eot oswbb/src/fonts/fontawesome-webfont.woff2 oswbb/src/fonts/fontawesome-webfont.ttf oswbb/src/fonts/fontawesome-webfont.woff oswbb/src/fonts/FontAwesome.otf oswbb/src/images/ oswbb/src/images/close.png oswbb/src/images/next.png oswbb/src/images/previous.png oswbb/src/images/loading.gif oswbb/src/images/Thumbnail_Placeholder.png oswbb/src/images/bx_loader.gif oswbb/ltop.sh oswbb/OSWatcher.sh oswbb/tar_up_full_archive.sh oswbb/OSWatcherFM.sh oswbb/call_uptime.sh oswbb/ifconfigsub.sh oswbb/mpsub.sh oswbb/call_du.sh oswbb/call_sar.sh oswbb/oswsub.sh oswbb/oswrds.sh oswbb/topaix.sh oswbb/oswib.sh oswbb/vmsub.sh
Verify the mandatory RPM Package as “KSH”, In case if allready not install download it
Check the KSH RPM Package
[root@trac1 software]# rpm -qa ksh ksh-20120801-28.el6.x86_64
Start the Installation Process
[root@trac1 software]# ls oswbb oswbb801.tar
[root@trac1 software]# cd oswbb
[root@trac1 oswbb]# pwd /software/oswbb
[root@trac1 oswbb]# ls call_du.sh call_uptime.sh Example_extras.txt genprvnet.sh iosub.sh mpsub.sh OSWatcherFM.sh oswbba.jar oswnet.sh oswsub.sh src stopOSWbb.sh tar_up_partial_archive.sh vmsub.sh call_sar.sh docs Exampleprivate.net ifconfigsub.sh ltop.sh nfssub.sh OSWatcher.sh oswib.sh oswrds.sh psmemsub.sh startOSWbb.sh tar_up_full_archive.sh topaix.sh xtop.sh
[root@trac1 oswbb]# ./startOSWbb.sh Info...You did not enter a value for snapshotInterval. Info...Using default value = 30 Info...You did not enter a value for archiveInterval. Info...Using default value = 48 Setting the archive log directory to/software/oswbb/archive Testing for discovery of OS Utilities... VMSTAT found on your system. IOSTAT found on your system. MPSTAT found on your system. IFCONFIG found on your system. NETSTAT found on your system. TOP found on your system. TRACEROUTE found on your system. OS User running OSwatcher is not the same as Oracle Clusterware software owner..private.net will not be configured. private.net file is required to monitor private Networks in RAC environment. Kindly configure & run OSWatcher as Oracle Clusterware software owner.. If you want to run OSWatcher as a different OS User than Oracle Clusterware software owner then configure private.net manually as per OSWatcher README. Discovery of CPU CORE COUNT CPU CORE COUNT will be used by oswbba to automatically look for cpu problems Warning... CPU CORE COUNT not found on your system. Defaulting to CPU CORE COUNT = 1 To correctly specify CPU CORE COUNT 1. Correct the error listed above for your unix platform or 2. Manually set core_count on OSWatcher.sh line 16 or 3. Do nothing and accept default value = 1 Discovery completed. Starting OSWatcher v8.0.1 on Wed Jan 16 17:30:04 IRST 2019 With SnapshotInterval = 30 With ArchiveInterval = 48 OSWatcher - Written by Carl Davis, Center of Expertise, Oracle Corporation For questions on install/usage please go to MOS (Note:301137.1) If you need further assistance or have comments or enhancement requests you can email me Carl.Davis@Oracle.com Data is stored in directory: /software/oswbb/archive Starting Data Collection... oswbb heartbeat:Wed Jan 16 17:30:10 IRST 2019 oswbb heartbeat:Wed Jan 16 17:30:40 IRST 2019 oswbb heartbeat:Wed Jan 16 17:31:10 IRST 2019 oswbb heartbeat:Wed Jan 16 17:31:40 IRST 2019 oswbb heartbeat:Wed Jan 16 17:32:10 IRST 2019 oswbb heartbeat:Wed Jan 16 17:32:40 IRST 2019 oswbb heartbeat:Wed Jan 16 17:33:10 IRST 2019 oswbb heartbeat:Wed Jan 16 17:33:40 IRST 2019 ..
Here the Stop the process or else it will continue the extraction of data
[root@trac1 oswbb]# ./stopOSWbb.sh
Check the status of the report of Individual Services
[root@trac1 oswbb]# cd archive/ [root@trac1 archive]# ls oswifconfig oswiostat oswmeminfo oswmpstat oswnetstat oswprvtnet oswps oswslabinfo oswtop oswvmstat
The below data describe are corresponsing to the result of above tested process
OSWIFCONFIG – It include the output from the oswifconfig utility
[root@trac1 archive]# cd oswifconfig/ [root@trac1 oswifconfig]# ls trac1_ifconfig_19.01.16.1700.dat [root@trac1 oswifconfig]# tail -10 trac1_ifconfig_19.01.16.1700.dat lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:877301 errors:0 dropped:0 overruns:0 frame:0 TX packets:877301 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:417235263 (397.9 MiB) TX bytes:417235263 (397.9 MiB)
OSWIOSTAT – It include the output from the iostat utility
[root@trac1 archive]# cd oswiostat/ [root@trac1 oswiostat]# ls trac1_iostat_19.01.16.1700.dat [root@trac1 oswiostat]# tail -10 trac1_iostat_19.01.16.1700.dat Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 0.00 0.00 5.15 2.06 50.52 17.01 18.71 0.00 0.00 0.00 0.00 sdc 0.00 0.00 0.00 1.03 0.00 0.52 1.00 0.00 1.00 1.00 0.10 sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 sde 0.00 0.00 0.00 2.06 0.00 17.01 16.50 0.00 0.50 0.50 0.10 fd0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
OSWMEMINFO – It include a listing of the contents of /proc/meminfo.
[root@trac1 archive]# cd oswmeminfo [root@trac1 oswmeminfo]# ls trac1_meminfo_19.01.16.1700.dat [root@trac1 oswmeminfo]# tail -10 trac1_meminfo_19.01.16.1700.dat VmallocUsed: 143680 kB VmallocChunk: 34359588688 kB HardwareCorrupted: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 12288 kB DirectMap2M: 4083712 kB
OSWMPSTAT – It include the output from the mpstat utility
[root@trac1 archive]# cd oswmpstat/ [root@trac1 oswmpstat]# ls trac1_mpstat_19.01.16.1700.dat [root@trac1 oswmpstat]# tail -10 trac1_mpstat_19.01.16.1700.dat 17:38:42 all 15.31 0.00 20.41 0.00 0.00 0.00 0.00 0.00 64.29 17:38:42 0 15.31 0.00 20.41 0.00 0.00 0.00 0.00 0.00 64.29 17:38:42 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 17:38:43 all 9.28 0.00 12.37 0.00 0.00 0.00 0.00 0.00 78.35 17:38:43 0 9.28 0.00 12.37 0.00 0.00 0.00 0.00 0.00 78.35 Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle Average: all 12.31 0.00 16.41 0.00 0.00 0.00 0.00 0.00 71.28 Average: 0 12.31 0.00 16.41 0.00 0.00 0.00 0.00 0.00 71.28
OSWNETSTAT – It include the output from the netstat utility
[root@trac1 archive]# cd oswnetstat [root@trac1 oswnetstat]# ls trac1_netstat_19.01.16.1700.dat [root@trac1 oswnetstat]# tail -10 trac1_netstat_19.01.16.1700.dat InMcastPkts: 818 OutMcastPkts: 413 InBcastPkts: 17569 OutBcastPkts: 52 InOctets: 3040566819 OutOctets: 2344299751 InMcastOctets: 180533 OutMcastOctets: 99932 InBcastOctets: 1708880 OutBcastOctets: 19618
OSWPRVTNET – In case if you have enable the private network tracing for RAC, then it include the data about the status of the private networks
[root@trac1 archive]# cd oswprvtnet/ [root@trac1 oswprvtnet]# ls [EMPTY]
OSWPS – It include the output from the PS utility
[root@trac1 archive]# cd oswps/ [root@trac1 oswps]# ls trac1_ps_19.01.16.1700.dat [root@trac1 oswps]# tail -10 trac1_ps_19.01.16.1700.dat gdm 5658 1 19 0.0 0.0 354516 832 poll_s S Jan 15 00:00:00 /usr/libexec/bonobo-activation-server --ac-activate --ior-output-fd=12 gdm 5652 5627 19 0.0 0.0 120764 1480 poll_s S Jan 15 00:00:00 /usr/libexec/at-spi-registryd gdm 5635 1 19 0.0 0.0 133324 1352 poll_s S Jan 15 00:00:00 /usr/libexec/gconfd-2 gdm 5627 5398 19 0.0 0.0 267164 1532 poll_s S Jan 15 00:00:00 /usr/bin/gnome-session --autostart=/usr/share/gdm/autostart/LoginWindow/ gdm 5625 1 19 0.0 0.0 29812 648 poll_s S Jan 15 00:00:00 /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session gdm 5624 1 19 0.0 0.0 20092 168 poll_s S Jan 15 00:00:00 /usr/bin/dbus-launch --exit-with-session dbus 4286 1 19 0.0 0.0 30112 1388 poll_s S Jan 15 00:00:00 dbus-daemon --system USER PID PPID PRI %CPU %MEM VSZ RSS WCHAN S STARTED TIME COMMAND 68 4453 4380 19 0.0 0.0 18060 668 unix_s S Jan 15 00:00:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket 68 4379 1 19 0.0 0.0 37440 1840 poll_s S Jan 15 00:00:00 hald
OSWSLABINFO – It include a listing of the contents of /proc/slabinfo
[root@trac1 archive]# cd oswslabinfo/ [root@trac1 oswslabinfo]# ls trac1_slabinfo_19.01.16.1700.dat [root@trac1 oswslabinfo]# tail -10 trac1_slabinfo_19.01.16.1700.dat size-256 1125 1125 256 15 1 : tunables 120 60 0 : slabdata 75 75 0 size-192(DMA) 0 0 192 20 1 : tunables 120 60 0 : slabdata 0 0 0 size-192 2060 2180 192 20 1 : tunables 120 60 0 : slabdata 109 109 0 size-128(DMA) 0 0 128 30 1 : tunables 120 60 0 : slabdata 0 0 0 size-64(DMA) 0 0 64 59 1 : tunables 120 60 0 : slabdata 0 0 0 size-64 17994 18054 64 59 1 : tunables 120 60 0 : slabdata 306 306 0 size-32(DMA) 0 0 32 112 1 : tunables 120 60 0 : slabdata 0 0 0 size-128 7650 7650 128 30 1 : tunables 120 60 0 : slabdata 255 255 0 size-32 439056 440608 32 112 1 : tunables 120 60 0 : slabdata 3934 3934 0 kmem_cache 197 200 192 20 1 : tunables 120 60 0 : slabdata 10 10 0
OSWTOP – It include the output from the top utility
[root@trac1 archive]# cd oswtop/ [root@trac1 oswtop]# ls trac1_top_19.01.16.1700.dat [root@trac1 oswtop]# tail -10 trac1_top_19.01.16.1700.dat 22 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md 23 root 20 0 0 0 0 S 0.0 0.0 0:11.56 kworker/0:1 24 root 20 0 0 0 0 S 0.0 0.0 0:00.07 khungtaskd 25 root 20 0 0 0 0 S 0.0 0.0 0:04.54 kswapd0 26 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd 27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 fsnotify_mark 38 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld 40 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kpsmoused 41 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:2 42 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 deferwq
OSWVMSTAT – It include the output from the vmstat utility.
[root@trac1 archive]# cd oswvmstat/ [root@trac1 oswvmstat]# ls trac1_vmstat_19.01.16.1700.dat [root@trac1 oswvmstat]# more trac1_vmstat_19.01.16.1700.dat Linux OSWbb v8.0.1 trac1 SNAP_INTERVAL 30 CPU_CORES 1 VCPUS 1 OSWBB_ARCHIVE_DEST /software/oswbb/archive zzz ***Wed Jan 16 17:30:10 IRST 2019 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 8 0 462256 204068 36228 1583400 2 7 132 170 94 404 13 10 77 0 0 4 1 462072 199316 36228 1583972 288 0 1097 349 5853 9177 71 29 0 0 0 5 0 462036 197672 36240 1584104 64 0 169 422 5052 7836 72 28 0 0 0 zzz ***Wed Jan 16 17:30:40 IRST 2019 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
To de-install the OSWatcher, Just you need to remove the software folder of oswbb to de-install the service.
[root@trac1 software]# rm -rf oswbb
Hence tested & verified in our test env