Fixed Table x$ktfbue has not statistics

While playing around 12.1.0.2 db, I ran my usual script to check tablespace usage details and it took ~59 secs (Elapsed: 00:00:59.02), which was high.
Ran the sql using “gather_plan_statistics” hint and below lines in the execution plan were of interest –

|  34 |      HASH GROUP BY                     |                    |      1 |      6 |      3 |00:00:58.68 |     129K|  54801 |  1160K|  1160K|  758K (0)|
|  35 |       VIEW                             | DBA_FREE_SPACE     |      1 |  21019 |     13 |00:00:00.01 |     129K|  54801 |       |       |          |
|  36 |        JOIN FILTER USE                 | :BF0000            |      1 |        |     13 |00:00:00.01 |     129K|  54801 |       |       |          |
|  37 |         UNION-ALL                      |                    |      1 |        |     19 |00:00:00.01 |     129K|  54801 |       |       |          |
|  38 |          NESTED LOOPS                  |                    |      1 |      1 |      0 |00:00:00.01 |       7 |      0 |       |       |          |
|  39 |           NESTED LOOPS                 |                    |      1 |      1 |      0 |00:00:00.01 |       7 |      0 |       |       |          |
|* 40 |            INDEX FULL SCAN             | I_FILE2            |      1 |      4 |      4 |00:00:00.01 |       1 |      0 |       |       |          |
|* 41 |            TABLE ACCESS CLUSTER        | FET$               |      4 |      1 |      0 |00:00:00.01 |       6 |      0 |       |       |          |
|* 42 |             INDEX UNIQUE SCAN          | I_TS#              |      4 |      1 |      4 |00:00:00.01 |       2 |      0 |       |       |          |
|* 43 |           TABLE ACCESS CLUSTER         | TS$                |      0 |      1 |      0 |00:00:00.01 |       0 |      0 |       |       |          |
|* 44 |            INDEX UNIQUE SCAN           | I_TS#              |      0 |      1 |      0 |00:00:00.01 |       0 |      0 |       |       |          |
|  45 |          NESTED LOOPS                  |                    |      1 |     10 |     10 |00:00:00.01 |      25 |      0 |       |       |          |
|  46 |           NESTED LOOPS                 |                    |      1 |     10 |     10 |00:00:00.01 |      21 |      0 |       |       |          |
|* 47 |            TABLE ACCESS FULL           | TS$                |      1 |      4 |      4 |00:00:00.01 |       9 |      0 |       |       |          |
|* 48 |            FIXED TABLE FIXED INDEX     | X$KTFBFE (ind:1)   |      4 |      2 |     10 |00:00:00.01 |      12 |      0 |       |       |          |
|* 49 |           INDEX UNIQUE SCAN            | I_FILE2            |     10 |      1 |     10 |00:00:00.01 |       4 |      0 |       |       |          |
|  50 |          NESTED LOOPS                  |                    |      1 |  21007 |      9 |00:00:57.85 |     129K|  54801 |       |       |          |
|* 51 |           HASH JOIN                    |                    |      1 |      7 |      9 |00:00:00.01 |      13 |      0 |  1115K|  1115K|  471K (0)|
|  52 |            NESTED LOOPS                |                    |      1 |      7 |      9 |00:00:00.01 |      12 |      0 |       |       |          |
|  53 |             NESTED LOOPS               |                    |      1 |     32 |      9 |00:00:00.01 |      11 |      0 |       |       |          |
|* 54 |              TABLE ACCESS FULL         | TS$                |      1 |      4 |      4 |00:00:00.01 |       9 |      0 |       |       |          |
|* 55 |              INDEX RANGE SCAN          | RECYCLEBIN$_TS     |      4 |      8 |      9 |00:00:00.01 |       2 |      0 |       |       |          |
|  56 |             TABLE ACCESS BY INDEX ROWID| RECYCLEBIN$        |      9 |      2 |      9 |00:00:00.01 |       1 |      0 |       |       |          |
|* 57 |            INDEX FULL SCAN             | I_FILE2            |      1 |      4 |      4 |00:00:00.01 |       1 |      0 |       |       |          |
|* 58 |           FIXED TABLE FULL             | X$KTFBUE           |      9 |   3125 |      9 |00:00:58.67 |     129K|  54801 |       |       |          |
|  59 |          NESTED LOOPS                  |                    |      1 |      1 |      0 |00:00:00.01 |      21 |      0 |       |       |          |
|  60 |           NESTED LOOPS                 |                    |      1 |      1 |      0 |00:00:00.01 |      21 |      0 |       |       |          |

Note
-----
   - dynamic statistics used: dynamic sampling (level=2)
   - this is an adaptive plan
   - 3 Sql Plan Directives used for this statement

The E-Rows for X$KTFBUE is 3125 whereas A-Rows 9. Checking the stats on the table

SYS@orcl:1> set serveroutput on
SYS@orcl:1>
SYS@orcl:1>
SYS@orcl:1> exec print_table('select * from dba_tab_statistics where  table_name=''X$KTFBUE'' ');
OWNER                         : SYS
TABLE_NAME                    : X$KTFBUE
PARTITION_NAME                :
PARTITION_POSITION            :
SUBPARTITION_NAME             :
SUBPARTITION_POSITION         :
OBJECT_TYPE                   : FIXED TABLE
NUM_ROWS                      :
BLOCKS                        :
EMPTY_BLOCKS                  :
AVG_SPACE                     :
CHAIN_CNT                     :
AVG_ROW_LEN                   :
AVG_SPACE_FREELIST_BLOCKS     :
NUM_FREELIST_BLOCKS           :
AVG_CACHED_BLOCKS             :
AVG_CACHE_HIT_RATIO           :
SAMPLE_SIZE                   :
LAST_ANALYZED                 :
GLOBAL_STATS                  :
USER_STATS                    :
STATTYPE_LOCKED               :
STALE_STATS                   :
SCOPE                         : SHARED
-----------------

Looking at the directive details shows missing details on “F(SYS.TS$) – F(SYS.UNDO$) – F(SYS.X$KTFBUE)”

SYS@orcl:1> exec  print_table('select * from DBA_SQL_PLAN_DIR_OBJECTS where owner=''SYS'' and object_name=''X$KTFBUE'' ');
DIRECTIVE_ID                  : 16383234857226438245
OWNER                         : SYS
OBJECT_NAME                   : X$KTFBUE
SUBOBJECT_NAME                :
OBJECT_TYPE                   : TABLE
NOTES                         : NONONOYES
-----------------

PL/SQL procedure successfully completed.

SYS@orcl:1> exec  print_table('select * from DBA_SQL_PLAN_DIRECTIVES where DIRECTIVE_ID=''16383234857226438245'' ');
DIRECTIVE_ID                  : 16383234857226438245
TYPE                          : DYNAMIC_SAMPLING
ENABLED                       : NO
STATE                         : USABLE
AUTO_DROP                     : YES
REASON                        : JOIN CARDINALITY MISESTIMATE
CREATED                       : 28-MAY-16 09.21.20.000000 PM
LAST_MODIFIED                 : 28-MAY-16 11.12.20.000000 PM
LAST_USED                     : 28-MAY-16 10.21.29.000000000 PM
NOTES                         : MISSING_STATSNO{F(SYS.TS$) - F(SYS.UNDO$) - F(SYS.X$KTFBUE)}
-----------------

PL/SQL procedure successfully completed.

Lets gather fixed object stats :-

SYS@orcl:1> exec DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;

PL/SQL procedure successfully completed.

Elapsed: 00:02:47.92

Ran the script again and it still took the same time, so checked the stats on X$KTFBUE and its still missing

SYS@orcl:1> exec print_table('select * from dba_tab_statistics where  table_name=''X$KTFBUE'' ');
OWNER                         : SYS
TABLE_NAME                    : X$KTFBUE
PARTITION_NAME                :
PARTITION_POSITION            :
SUBPARTITION_NAME             :
SUBPARTITION_POSITION         :
OBJECT_TYPE                   : FIXED TABLE
NUM_ROWS                      :
BLOCKS                        :
EMPTY_BLOCKS                  :
AVG_SPACE                     :
CHAIN_CNT                     :
AVG_ROW_LEN                   :
AVG_SPACE_FREELIST_BLOCKS     :
NUM_FREELIST_BLOCKS           :
AVG_CACHED_BLOCKS             :
AVG_CACHE_HIT_RATIO           :
SAMPLE_SIZE                   :
LAST_ANALYZED                 :
GLOBAL_STATS                  :
USER_STATS                    :
STATTYPE_LOCKED               :
STALE_STATS                   :
SCOPE                         : SHARED
-----------------

PL/SQL procedure successfully completed.

Reviewed few documents

Query Against DBA_EXTENTS Performs Slowly After Upgrade to 11.2.0.3 (Doc ID 1453425.1)
Bug 5259025 – The fixed table x$ktfbue has no statistics (Doc ID 5259025.8)

Gathered the stats on fixed table x$ktfbue

SYS@orcl:1>  exec DBMS_STATS.GATHER_TABLE_STATS('SYS','X$KTFBUE');

PL/SQL procedure successfully completed.

Elapsed: 00:00:34.09
SYS@orcl:1> exec print_table('select * from dba_tab_statistics where  table_name=''X$KTFBUE'' ');
OWNER                         : SYS
TABLE_NAME                    : X$KTFBUE
PARTITION_NAME                :
PARTITION_POSITION            :
SUBPARTITION_NAME             :
SUBPARTITION_POSITION         :
OBJECT_TYPE                   : FIXED TABLE
NUM_ROWS                      : 9581
BLOCKS                        :
EMPTY_BLOCKS                  :
AVG_SPACE                     :
CHAIN_CNT                     :
AVG_ROW_LEN                   : 47
AVG_SPACE_FREELIST_BLOCKS     :
NUM_FREELIST_BLOCKS           :
AVG_CACHED_BLOCKS             :
AVG_CACHE_HIT_RATIO           :
SAMPLE_SIZE                   : 9581
LAST_ANALYZED                 : 28-may-2016 22:55:47
GLOBAL_STATS                  : YES
USER_STATS                    : NO
STATTYPE_LOCKED               :
STALE_STATS                   :
SCOPE                         : SHARED
-----------------

Re-ran the sql script and Elapsed: 00:00:00.41

Oracle ASMLib

This blog post is more of a note for myself on configuring ASMLib.

ASMLib is an optional utility that can be used on Linux systems to manage Oracle ASM devices.

ASMLib consists of the following components:

An open source (GPL) kernel module package: kmod-oracleasm
An open source (GPL) utilities package: oracleasm-support
A closed source (proprietary) library package: oracleasmlib

On my Oracle VMBox, I performed the below steps –

1. Check the installed packages related to oracleasm

[root@oel67-12c dev]# rpm -qa | grep oracleasm
[root@oel67-12c dev]#

2. As no rpm were installed, I started installing the packages

[root@oel67-12c dev]# yum install oracleasm-support
Loaded plugins: refresh-packagekit, security, ulninfo
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package oracleasm-support.x86_64 0:2.1.8-1.el6 will be installed
--> Finished Dependency Resolution
..........................
Installed:
  oracleasm-support.x86_64 0:2.1.8-1.el6

Complete!
[root@oel67-12c dev]#
[root@oel67-12c dev]# yum install kmod-oracleasm
Loaded plugins: refresh-packagekit, security, ulninfo
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package kmod-oracleasm.x86_64 0:2.0.8-13.el6_8 will be installed
--> Processing Dependency: kernel >= 2.6.32-642.el6 for package: kmod-oracleasm-2.0.8-13.el6_8.x86_64

...........................
Is this ok [y/N]: y
Downloading Packages:
(1/2): kernel-2.6.32-642.el6.x86_64.rpm
(2/2): kmod-oracleasm-2.0.8-13.el6_8.x86_64.rpm   

...........................

Installed:
  kmod-oracleasm.x86_64 0:2.0.8-13.el6_8

Dependency Installed:
  kernel.x86_64 0:2.6.32-642.el6

Complete!
[root@oel67-12c dev]#

3. Download the oracleasmlib rpm from http://www.oracle.com/technetwork/server-storage/linux/asmlib/rhel6-1940776.html and install it. Without this RPM, you will not have the oracleasm-discover executable


[root@oel67-12c dev]# ls -lrt /media/sf_12cR1/oracleasmlib*
-rwxrwx--- 1 root vboxsf    13336 May 18 21:44 oracleasmlib-2.0.12-1.el6.x86_64.rpm

[root@oel67-12c sf_12cR1]# rpm -ihv oracleasmlib-2.0.12-1.el6.x86_64.rpm
Preparing...                ########################################### [100%]
   1:oracleasmlib           ########################################### [100%]
[root@oel67-12c sf_12cR1]#

4. Verify all the components are installed

[root@oel67-12c sf_12cR1]# rpm -qa | grep oracleasm
oracleasm-support-2.1.8-1.el6.x86_64
oracleasmlib-2.0.12-1.el6.x86_64
kmod-oracleasm-2.0.8-13.el6_8.x86_64
[root@oel67-12c sf_12cR1]# cd 
[root@oel67-12c ~]# which oracleasm
/usr/sbin/oracleasm
[root@oel67-12c ~]#

5. Configure oracleasm — will configure the on-boot properties of the Oracle ASM library driver as well as set the appropriate file permissions.

[root@oel67-12c dev]# oracleasm configure -i
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: oinstall
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
[root@oel67-12c dev]#

The configuration setting done above, can be checked in /etc/sysconfig/oracleasm

[root@oel67-12c dev]# ls -lrt /etc/sysconfig/oracleasm
lrwxrwxrwx 1 root root 24 May 18 22:02 /etc/sysconfig/oracleasm -> oracleasm-_dev_oracleasm

6. Check the status

[root@oel67-12c ~]# oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@oel67-12c ~]#

7. Load oracleasm module with the init command

[root@oel67-12c ~]# oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm
[root@oel67-12c ~]#

8. Verify the oracleasm configuration

a. Perform ‘df –ha’, this should show you that /dev/oracleasmfs is mounted.


[root@oel67-12c ~]# df -ha
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_oel6712c-lv_root
                       45G  7.4G   36G  18% /
....................
oracleasmfs              0     0     0    - /dev/oracleasm

b. Execute the ‘lsmod’ command, to show the loaded oracleasm module.

[root@oel67-12c ~]# lsmod | grep oracleasm
Module                  Size  Used by
oracleasm              53591  1

9. Create the partition on disk , if not already done

[root@oel67-12c oracleasm]# lsblk
NAME                           MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                              8:0    0   50G  0 disk
├─sda1                           8:1    0  500M  0 part /boot
└─sda2                           8:2    0 49.5G  0 part
  ├─vg_oel6712c-lv_root (dm-0) 252:0    0 45.6G  0 lvm  /
  └─vg_oel6712c-lv_swap (dm-1) 252:1    0  3.9G  0 lvm  [SWAP]
sdb                              8:16   0   10G  0 disk
sdc                              8:32   0   12G  0 disk
sdd                              8:48   0   12G  0 disk
sde                              8:64   0   10G  0 disk
sdf                              8:80   0   12G  0 disk
sr0                             11:0    1 55.5M  0 rom  /media/VBOXADDITIONS_5.0.20_106931


[root@oel67-12c disks]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x50e6a869.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4):
Value out of range.
Partition number (1-4): 1
First cylinder (1-1305, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1305, default 1305):
Using default value 1305

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@oel67-12c disks]#
[root@oel67-12c disks]#
[root@oel67-12c disks]#
[root@oel67-12c disks]# lsblk
NAME                           MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                              8:0    0   50G  0 disk
├─sda1                           8:1    0  500M  0 part /boot
└─sda2                           8:2    0 49.5G  0 part
  ├─vg_oel6712c-lv_root (dm-0) 252:0    0 45.6G  0 lvm  /
  └─vg_oel6712c-lv_swap (dm-1) 252:1    0  3.9G  0 lvm  [SWAP]
sdb                              8:16   0   10G  0 disk
└─sdb1                           8:17   0   10G  0 part
sdc                              8:32   0   12G  0 disk
sdd                              8:48   0   12G  0 disk
sde                              8:64   0   10G  0 disk
sdf                              8:80   0   12G  0 disk
sr0                             11:0    1 55.5M  0 rom /media/VBOXADDITIONS_5.0.20_106931

Perform the same above steps for remaining disk.

10. Once the disk are created, label the disks as ASM disks and give then an ASM name.


[root@oel67-12c disks]# oracleasm createdisk data0101 /dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@oel67-12c disks]# oracleasm createdisk data0102 /dev/sde1
Writing disk header: done
Instantiating disk: done
[root@oel67-12c disks]# ls -lrt
total 0
brw-rw---- 1 oracle oinstall 8, 17 May 18 22:49 DATA0101
brw-rw---- 1 oracle oinstall 8, 65 May 18 22:50 DATA0102

11. Check the configured disk using listdisks

[root@oel67-12c disks]# oracleasm listdisks
DATA0101

12. You can query the disk to check if its a valid ASM Disk

[root@oel67-12c disks]# oracleasm querydisk -d DATA0101
Disk "DATA0101" is a valid ASM disk on device [8,17]

[root@oel67-12c disks]# more /proc/partitions
major minor  #blocks  name
   8    17   10482381 sdb1

[root@oel67-12c sbin]# blkid|grep asm
/dev/sdb1: LABEL="DATA0101" TYPE="oracleasm"
/dev/sde1: LABEL="DATA0102" TYPE="oracleasm"
[root@oel67-12c sbin]#

The [8,17] is the major,minor number of the device with this partition. The value comes from /proc/partitions

major — The major number of the device with this partition. The major number in the /proc/partitions, (3), corresponds with the block device ide0, in /proc/devices.

minor — The minor number of the device with this partition. This serves to separate the partitions into different physical devices and relates to the number at the end of the name of the partition.

#blocks — Lists the number of physical disk blocks contained in a particular partition.

name — The name of the partition.

13. oracleasm-discover


[root@oel67-12c sbin]# oracleasm-discover
-bash: /usr/sbin/oracleasm-discover: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory

For the above error install/lib/ld-linux.so.2

[root@oel67-12c ~]# yum install /lib/ld-linux.so.2
Loaded plugins: refresh-packagekit, security, ulninfo
Setting up Install Process
public_ol6_latest/filelists                                                                                                                                                             |  61 MB     00:43
Resolving Dependencies
--> Running transaction check
---> Package glibc.i686 0:2.12-1.192.el6 will be installed
Dependencies Resolved

..................
Complete!
[root@oel67-12c ~]#

[root@oel67-12c ~]# ls -lrt /lib/ld-linux.so.2
lrwxrwxrwx 1 root root 10 May 18 23:25 /lib/ld-linux.so.2 -> ld-2.12.so
[root@oel67-12c ~]#
[root@oel67-12c ~]# oracleasm-discover
Using ASMLib from /opt/oracle/extapi/32/asm/orcl/1/libasm.so
[ASM Library - Generic Linux, version 2.0.4 (KABI_V2)]
Discovered disk: ORCL:DATA0101 [20964762 blocks (10733958144 bytes), maxio 512]
Discovered disk: ORCL:DATA0102 [20964762 blocks (10733958144 bytes), maxio 512]
[root@oel67-12c ~]#

Tuned-adm and Oracle

“Tuned” in RHEL7/OEL7 is tuning daemon for automatically tuning the system via the use of tuning profile. It can also be configured to react to changes to improve performance of the server and also to make system settings persistent.

“tuned-adm” is a command line tool that provides a number of different profiles to improve performance.

Below are the profiles provided and supported in RHEL 7 :-

  1. throughput-performance
  2. latency-performance
  3. network-latency
  4. network-throughput
  5. virtual-guest
  6. virtual-host

Apart from the provided profile, we can create custom profiles. All the profile are stored in /usr/lib/tuned/ in RHEL7.

The recommended profile for Oracle database workloads is “throughput-performance”.

In my virtualbox, by default “vitual-guest” was set as active profile

[root@racnode1 ~]# cd /usr/lib/tuned/
[root@racnode1 tuned]# tuned-adm active
Current active profile: virtual-guest

If tuned is not installed, install it using yum

#yum install tuned

Enabled tuned to ensure it is started upon boot time

# systemctl enable tuned.service

Start the tuned service

#systemctl start tuned.service

To check the status of tuned service

#systemctl status tuned.service

Now, create new “oracle” profile to be used

1. Create oracle directory –

[root@racnode1 ~]# mkdir /usr/lib/tuned/oracle

2. Create tuned.conf –

[root@racnode1 ~]# cd /usr/lib/tuned/oracle
[root@racnode1 oracle]# vi tuned.conf
[root@racnode1 oracle]# more tuned.conf
#
# tuned configuration
#
[main]
include=throughput-performance

[sysctl]
vm.swappiness = 1
vm.dirty_background_ratio = 3
vm.dirty_ratio = 80
vm.dirty_expire_centisecs = 500
vm.dirty_writeback_centisecs = 100
kernel.shmmax = 4398046511104
kernel.shmall = 1073741824
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
kernel.panic_on_oops = 1

[vm]
transparent_hugepages=never

[root@racnode1 oracle]#

3. Activate the newly added oracle profile –

[root@racnode1 oracle]# tuned-adm profile oracle
[root@racnode1 oracle]# sysctl -a | grep vm.swappiness
vm.swappiness = 1
[root@racnode1 oracle]# sysctl -a | grep vm.dirty_ratio
vm.dirty_ratio = 80
[root@racnode1 oracle]#

4. To check the profile list –

[root@racnode1 ~]# tuned-adm list

One of the benefits of tuned is the profiles can be applied dynamically.
To configure dynamic tuning behavior, edit the dynamic_tuning parameter in the /etc/tuned/tuned-main.conf file.

If we are using tuned profile (as shown above) which make system settings persistent, it is recommended to remove all the oracle related entries from /etc/sysctl.conf as the entries are susceptible of being overwrittern, replaced or removed all together.

Hope the article is helpful!!

12cR1 RAC Installation on OEL7

To build Oracle Clusterware Database at Home, I believe , RAC ATTACK is the best place to learn. Its is a free curriculum and platform for hands-on learning labs related to Oracle RAC. While reviewing the article, I thought to perform 12cR1 RAC installation on OEL 7.2.

Attached is the document :- 12c_RAC_on_OEL7

The attached article is inspired by

RAC ATTACK :- https://en.wikibooks.org/wiki/RAC_Attack_-_Oracle_Cluster_Database_at_Home/RAC_Attack_12c

Tim Hall’s article :- https://oracle-base.com/articles/12c/oracle-db-12cr1-rac-installation-on-oracle-linux-7-using-virtualbox 

Deploying Oracle RAC Database 12c on RHEL 7 – Best Practices :- https://www.redhat.com/en/resources/deploying-oracle-rac-database-12c-rhel-7-best-practices

Big Thank you to RAC Attack members!!!

I hope the document helps some of you. Please feel free to comment.

Its all about learning:)

Online Partition Move fails with ORA-00932 – 12.1.0.2

Oracle 12c introduced feature to move table partitions and sub-partitions as online operations.So, this blog is related to this feature and the issue I faced.

The database was upgraded from 11.2.0.4 to 12.1.0.2 and this issue is seen in upgraded databases only.

Lets start!!!

SQL> create table sales_part
    (product char(25),channel_id number,cust_id number,
    amount_sold number, time_id date)
    partition by range (time_id)
    (
    partition sale_jan values less than (to_date('01-02-2015','dd-mm-yyyy')),
    partition sale_feb values less than (to_date('01-03-2015','dd-mm-yyyy')),
    partition sale_mar values less than (to_date('01-04-2015','dd-mm-yyyy')),
    partition sale_apr values less than (to_date('01-05-2015','dd-mm-yyyy')),
    partition sale_may values less than (to_date('01-06-2015','dd-mm-yyyy')),
    partition sale_jun values less than (to_date('01-07-2015','dd-mm-yyyy')),
    partition sale_jul values less than (to_date('01-08-2015','dd-mm-yyyy')),
    partition sale_aug values less than (to_date('01-09-2015','dd-mm-yyyy')),
    partition sale_sep values less than (to_date('01-10-2015','dd-mm-yyyy')),
    partition sale_oct values less than (to_date('01-11-2015','dd-mm-yyyy')),
    partition sale_nov values less than (to_date('01-12-2015','dd-mm-yyyy')),
    partition sale_dec values less than (to_date('01-01-2016','dd-mm-yyyy'))
   ) 
   TABLESPACE USERS;

SQL> insert into sales_part
    select
    'Oracle Enterprise Edition' as product,
     mod(rownum,5) as channel_id,
     mod(rownum,1000) as cust_id ,
     5000 as amount_sold,
     to_date
     ('01.' || lpad(to_char(mod(rownum,6)+4),2,'0') || '.2015' ,'dd.mm.yyyy')
     as time_id
     from dual connect by level commit;

Commit complete.

SQL> create index idx_sale_local on sales_part (cust_id,time_id)local tablespace users;

Index created.

SQL> create index idx_sale_global on sales_part (channel_id)  tablespace users;

Index created.

SQL>                        
SQL> exec dbms_stats.gather_table_stats('ANAND','SALES_PART', cascade => true);

PL/SQL procedure successfully completed.

SQL>

Table and Index details

SQL>@partition_table 

TABLE_NAME      PARTITION_NAME            HIGH_VALUE                                                                            LAST_ANALYZED      TABLESPACE_NAM   NUM_ROWS
--------------- ------------------------- ------------------------------------------------------------------------------------- ------------------ -------------- ----------
SALES_PART      SALE_JAN                  TO_DATE(' 2015-02-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA      02-OCT-15          USERS                   0
SALES_PART      SALE_FEB                  TO_DATE(' 2015-03-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA      02-OCT-15          USERS                   0
SALES_PART      SALE_MAR                  TO_DATE(' 2015-04-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA      02-OCT-15          USERS                   0
SALES_PART      SALE_APR                  TO_DATE(' 2015-05-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA      02-OCT-15          USERS               33333
SALES_PART      SALE_MAY                  TO_DATE(' 2015-06-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA      02-OCT-15          USERS               33334
SALES_PART      SALE_JUN                  TO_DATE(' 2015-07-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA      02-OCT-15          USERS               33334
SALES_PART      SALE_JUL                  TO_DATE(' 2015-08-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA      02-OCT-15          USERS               33333
SALES_PART      SALE_AUG                  TO_DATE(' 2015-09-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA      02-OCT-15          USERS               33333
SALES_PART      SALE_SEP                  TO_DATE(' 2015-10-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA      02-OCT-15          USERS               33333
SALES_PART      SALE_OCT                  TO_DATE(' 2015-11-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA      02-OCT-15          USERS                   0
SALES_PART      SALE_NOV                  TO_DATE(' 2015-12-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA      02-OCT-15          USERS                   0
SALES_PART      SALE_DEC                  TO_DATE(' 2016-01-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA      02-OCT-15          USERS                   0

12 rows selected.

SQL> @index
Enter value for owner: ANAND
Enter value for table_name: SALES_PART

TABLE_OWNER     TABLE_NAME            INDEX_OWNER     INDEX_NAME            TABLESPACE_NAME   NUM_ROWS      CLUST STATUS                   INDEX_TYPE
--------------- --------------------- --------------- --------------------- --------------- ---------- ---------- ------------------------ ------------
ANAND           SALES_PART            ANAND           IDX_SALE_GLOBAL       USERS               200000       6084 VALID                    NORMAL
ANAND           SALES_PART            ANAND           IDX_SALE_LOCAL                            200000     199784 N/A                      NORMAL

SQL> @partition_index
Enter value for index_name: IDX_SALE_LOCAL
Enter value for owner: ANAND

Index                                                                                                                   Number
Name                  PARTITION_NA STATUS                   TABLESPACE_NAME INITIAL_EXTENT NEXT_EXTENT PCT_INCREASE    of Rows
--------------------- ------------ ------------------------ --------------- -------------- ----------- ------------ ----------
IDX_SALE_LOCAL        SALE_JAN     USABLE                   USERS                                                            0
                      SALE_FEB     USABLE                   USERS                                                            0
                      SALE_MAR     USABLE                   USERS                                                            0
                      SALE_APR     USABLE                   USERS                    65536     1048576                   33333
                      SALE_MAY     USABLE                   USERS                    65536     1048576                   33334
                      SALE_JUN     USABLE                   USERS                    65536     1048576                   33334
                      SALE_JUL     USABLE                   USERS                    65536     1048576                   33333
                      SALE_AUG     USABLE                   USERS                    65536     1048576                   33333
                      SALE_SEP     USABLE                   USERS                    65536     1048576                   33333
                      SALE_OCT     USABLE                   USERS                                                            0
                      SALE_NOV     USABLE                   USERS                                                            0
                      SALE_DEC     USABLE                   USERS                                                            0

Now I plan to move Partition SALE_JUN to tablespace DEMO.

SQL> alter table anand.sales_part move partition sale_jun online tablespace demo
  2  update indexes (
  3  anand.idx_sale_local (partition sale_jun tablespace demo)
  4  );
alter table anand.sales_part move partition sale_jun online tablespace demo
                  *
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-00932: inconsistent datatypes: expected NUMBER got BINARY

After searching for the issue in MOS and reviewing DOC Id 2028583.1, I applied patch 15894842 and 20123899

Oracle Database 12c                                                  12.1.0.2.0
There are 1 products installed in this Oracle Home.

List of Bugs fixed by Installed Patches:

Bug        Fixed by  Installed at                   Description
            Patch
---        --------  ------------                   -----------
20123899   20123899  Fri Oct 02 10:32:54 UTC 2015   INSERT IN TO STAT TABLE FAILS WITH ORA-932
                                                    INCONSISTENT DATATYPES
15894842   15894842  Fri Oct 02 10:31:52 UTC 2015   INSERT IN TO STAT TABLE FAILS WITH ORA-00932
                                                    INCONSISTENT DATATYPES

Even after applying the above patches, the online partition move failed with the same error. This time I traced the session –

SQL> alter session set events '932 trace name errorstack level 3';

Session altered.

SQL>alter table anand.sales_part move partition sale_jun online tablespace demo
    update indexes (
     anand.idx_sale_local (partition sale_jun tablespace demo)
     );
 alter table anand.sales_part move partition sale_jun online tablespace demo
                   *
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-00932: inconsistent datatypes: expected NUMBER got BINARY

The trace file showed

*** 2015-10-02 10:56:40.306
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=3, mask=0x0)
----- Error Stack Dump -----
ORA-00932: inconsistent datatypes: expected NUMBER got BINARY
----- Current SQL Statement for this session (sql_id=8qwf072kqbt3g) -----
insert into histgrm$ select :1, col#, row#, bucket, endpoint, intcol#, epvalue, ep_repeat_count, epvalue_raw, spare1, spare2 from histgrm$ where obj# = :2

----- Call Stack Trace -----

So the issue is related to HISTGRM$ table. After spending few minutes searching for issue in MOS, I found the below useful documents:-

Bug 20703000 : INSERT INTO STAT TABLE HISTGRM$ FAILS WITH ORA-00932: INCONSISTENT DATATYPES
Moving Table Partition With CLOB Creates ORA-604 , ORA-932 (Doc ID 2040742.1)

After applying the patch 20703000, online partition move worked fine and below is the changed sql on histgrm$

PARSING IN CURSOR #140348537646088 len=257 dep=1 uid=0 oct=2 lid=0 tim=41782225111 hv=216722248 ad='8635af20' sqlid='66x6dxn6fpuu8'
insert into histgrm$ (obj#, col#, row#, bucket, endpoint, intcol#, epvalue,  ep_repeat_count, epvalue_raw, spare1, spare2 ) 
select :1, col#, row#, bucket, endpoint, intcol#, epvalue, ep_repeat_count, epvalue_raw, spare1, spare2 from histgrm$ where obj# = :2
END OF STMT

Hope this helps!!