Friday, 14 July 2017

Bug 18371441 - RMAN-11003 / .. / ORA-19755 from RMAN recover of file with missing BCT file

Rman duplicate failed with below error
Errors with log +FRA/JAYTST/archivelog/2017_07_14/thread_2_seq_201199.2127.949341265
Recovery interrupted!
Recovered data files to a consistent state at change 97471797575
Media Recovery failed with error 19755
Errors in file /oracle/product/diag/rdbms/JAYTST/JAYTST/trace/JAYTST_pr00_7863.trc:
ORA-00283: recovery session canceled due to errors
ORA-19755: could not open change tracking file
ORA-19750: change tracking file: '+DATA/hr91prod/changetracking/rman_change_track.f'
ORA-17503: ksfdopn:2 Failed to open file +DATA/hr91prod/changetracking/rman_change_track.f
ORA-15173: entry 'JAYPROD' does not exist in directory '/'
ORA-10877 signalled during: alter database recover logfile '+FRA/JAYTST/archivelog/2017_07_14/thread_2_seq_201199.2127.949341265'...
Fri Jul 14 19:13:58 2017


Check any data files need recovery
select status, to_char(resetlogs_change#), resetlogs_time,   to_char(checkpoint_change#), to_char(checkpoint_time, 'DD-MON-YYYY HH24:MI:SS') as checkpoint_time,
count(*) from v$datafile_header group by status, resetlogs_change#, resetlogs_time, checkpoint_change#, checkpoint_time
 order by status, checkpoint_change#, checkpoint_time ;
STATUS  TO_CHAR(RESETLOGS_CHANGE#)  RESETLOGS TO_CHAR(CHECKPOINT_CHANGE#)  CHECKPOINT_TIME        COUNT(*)
------- --------------------------  --------- ---------------------------- -------------------- ----------
ONLINE  1                           17-MAY-12 97471797575                   13-JUL-2017 09:44:46      100 - different (scn) not consistent

ONLINE  1                           17-MAY-12 97495746853                   13-JUL-2017 17:59:21       41

Note: -- Datafile(s) are at different checkpoint_change# (scn), so not consistent and need recovery

  Recovery failed due to BCT not available at target side
  Copy bct file from source to target
· At the source, check the location of BCT file
SELECT filename bct_file, status, bytes FROM   v$block_change_tracking;
BCT_FILE                                                            STATUS       BYTES
------------------------------------------------------------------ ---------- ----------
+DATA/JAYPROD/changetracking/rman_change_track.f                     ENABLED    147915264
1 row selected.

· Copy the BCT file from ASM to a file system
ASMCMD> cp +DATA/JAYPROD/changetracking/rman_change_track.f /tmp/.
copying +DATA/JAYPROD/changetracking/rman_change_track.f -> /tmp/./rman_change_track.f
ASMCMD>

· Transfer the file to the auxiliary site using scp, ftp etc
· Auxiliary site, copy the file from the file system back into ASM:
ASMCMD> cd +DATA/JAYPROD/changetracking
ASMCMD> ls
ASMCMD> cp /tmp/rman_change_track.f .
copying /tmp/rman_change_track.f -> +DATA/JAYPROD/changetracking/rman_change_track.f

  Recover until data files are consistent
SQL> RECOVER DATABASE USING BACKUP CONTROLFILE UNTIL CANCEL;
ORA-00279: change 97471797575 generated at 07/13/2017 09:44:46 needed for thread 2
ORA-00289: suggestion : +FRA/JAYTST/archivelog/2017_07_14/thread_2_seq_201199.2127.949341265
ORA-00280: change 97471797575 for thread 2 is in sequence #201199

Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
+FRA/JAYTST/archivelog/2017_07_14/thread_2_seq_201199.2127.949341265
ORA-00279: change 97471797575 generated at 07/13/2017 09:28:24 needed for thread 1
ORA-00289: suggestion : +FRA/JAYTST/archivelog/2017_07_14/thread_1_seq_200803.2075.949341807
ORA-00280: change 97471797575 for thread 1 is in sequence #200803

Check datafile check point status:
STATUS  TO_CHAR(RESETLOGS_CHANGE#)               RESETLOGS TO_CHAR(CHECKPOINT_CHANGE#)              CHECKPOINT_TIME        COUNT(*)
------- ---------------------------------------- --------- ---------------------------------------- -------------------- ----------
ONLINE  1                                        17-MAY-12 97495852724                              13-JUL-2017 18:26:31        141

   All data files are in consistent , Open database with reset logs
(control file type is in backup , recreate it to make as current  to allow open with reset logs )
SQL> create pfile='/tmp/tmp1.ora' from spfile;
File created.

Edit DB name as JAYPROD.

SQL> shutdown immediate;
ORA-01507: database not mounted

SQL> startup mount pfile=/tmp/tmp1.ora;

SQL> alter database disable block change tracking;
Database altered.

SQL> alter database backup controlfile to trace as '/tmp/tmp2.sql'; 
Database altered.

SQL> shutdown immediate;
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.

Re-control file (second case remove all the rest of the character tmp2.sql)

CREATE CONTROLFILE REUSE DATABASE "JAYPROD" RESETLOGS FORCE LOGGING ARCHIVELOG
    MAXLOGFILES 192
    MAXLOGMEMBERS 3
    MAXDATAFILES 1024
    MAXINSTANCES 32
    MAXLOGHISTORY 4144


oracle> sqlplus "/as sysdba"  
SQL> startup nomount pfile=/tmp/tmp1.ora
ORACLE instance started.

Total System Global Area 2138521600 bytes
Fixed Size                  2161024 bytes
Variable Size             905971328 bytes
Database Buffers         1207959552 bytes
Redo Buffers               22429696 bytes

SQL> @tmp2.sql

SQL> alter database open resetlogs;

  Present database name is JAYPROD need to rename to target as JAYTST

SQL> ALTER DATABASE RENAME GLOBAL_NAME TO JAYTST;
Database altered.

SQL> shutdown immediate ;
SQL>startup  mount pfile=/tmp/tmp1.ora

nid TARGET=SYS DBNAME=JAYTST   (Change DB name as per requirement )

 (Sys password  manager)

  Before restart  and resetlogs  rename redolog  ****

Edit db name JAYTST in tmp1.ora

SQL> startup  mount pfile=/tmp/tmp1.ora

  Rename redologos   
SQL> select 'Alter database rename file ''' || member || ''' to '''||member||' '||''';' from v$logfile where member like '%+DATA%';
SQL> select 'Alter database rename file ''' || member || ''' to '''||member||' '||''';' from v$logfile where member like '%+FRA%';

SQL> alter database open resetlogs;

  Add temporary space
SQL> ALTER TABLESPACE TEMP ADD TEMPFILE '+DATA/' REUSE;
SQL> ALTER TABLESPACE TEMP ADD TEMPFILE '+DATA/' REUSE;

Tuesday, 4 July 2017

EXADATA Flash Cache Compression only on X3 & X4 Machines

Exadata Flash Cache Compression was introduced in Exadata Storage Software version 11.2.3.3.0
·   Flash cache compression dynamically increases the logical capacity of the flash cache by transparently compressing user data as it is loaded into the flash cache.
·   This allows much more data to be kept in flash, and decreases the need to access data on disk drives.
·   The compression and decompression operations are completely transparent to the application and database, and have no performance overhead, even when running at rates of millions of I/Os per second.
·   Depending on the user data compress ability, Oracle Exadata Storage Server Software dynamically expands the flash cache size up to two times. Compression benefits vary based on the redundancy in the data. Tables and indexes that are uncompressed have the largest space reductions. Tables and indexes that are OLTP compressed have significant space reductions. Tables that use Hybrid Columnar Compression have minimal space reductions.
·         Oracle Advanced Compression Option is required to enable flash cache compression.

How to enable Flash Cache compression in Rolling method
1. Flush the dirty blocks from Flash Disks to the Grid Disks
CellCLI> ALTER FLASHCACHE all flush
To monitor the flush operation, open a new cell cellcli session and run command:
CellCLI> LIST CELLDISK attributes name, flushstatus, flusherror
While flush operation is working, the attribute flushstatus will report working
CellCLI> LIST CELLDISK attributes name, flushstatus, flusherror
         CD_00_celadm08              
         CD_01_celadm08              
         CD_02_celadm08              
         CD_03_celadm08              
         CD_04_celadm08              
         CD_05_celadm08              
         CD_06_celadm08              
         CD_07_celadm08              
         CD_08_celadm08              
         CD_09_celadm08              
         CD_10_celadm08              
         CD_11_celadm08              
         FD_00_celadm08    working   
         FD_01_celadm08    working   
         FD_02_celadm08    working   
         FD_03_celadm08    working   
         FD_04_celadm08    working   
         FD_05_celadm08    working   
         FD_06_celadm08    working   
         FD_07_celadm08    working   
         FD_08_celadm08    working   
         FD_09_celadm08    working     

To validate that flush operation was successful use any of following methods, opening a new cellcli session
Attribute flushstatus reports complete

CellCLI> LIST CELLDISK attributes name, flushstatus, flusherror
         CD_00_celadm08              
         CD_01_celadm08              
         CD_02_celadm08              
         CD_03_celadm08              
         CD_04_celadm08              
         CD_05_celadm08              
         CD_06_celadm08              
         CD_07_celadm08              
         CD_08_celadm08              
         CD_09_celadm08              
         CD_10_celadm08              
         CD_11_celadm08              
         FD_00_celadm08    complete  
         FD_01_celadm08    complete  
         FD_02_celadm08    complete  
         FD_03_celadm08    complete  
         FD_04_celadm08    complete  
         FD_05_celadm08    complete  
         FD_06_celadm08    complete  
         FD_07_celadm08    complete  
         FD_08_celadm08    complete  
         FD_09_celadm08    complete   
     
Verify cachedby attribute is null.  If a FlashDisk is reported, flush is still in progress

CellCLI> LIST GRIDDISK attributes name,cachedby
CellCLI> list griddisk attributes name,cachedby
         DATA_DMORL_CD_00_celadm08
         DATA_DMORL_CD_01_celadm08
         DATA_DMORL_CD_02_celadm08
         DATA_DMORL_CD_03_celadm08
         DATA_DMORL_CD_04_celadm08
         DATA_DMORL_CD_05_celadm08
         DATA_DMORL_CD_06_celadm08
         DATA_DMORL_CD_07_celadm08
         DATA_DMORL_CD_08_celadm08
         DATA_DMORL_CD_09_celadm08
         DATA_DMORL_CD_10_celadm08
         DATA_DMORL_CD_11_celadm08
         DBFS_DG_CD_02_celadm08
         DBFS_DG_CD_03_celadm08
         DBFS_DG_CD_04_celadm08
         DBFS_DG_CD_05_celadm08
         DBFS_DG_CD_06_celadm08
         DBFS_DG_CD_07_celadm08
         DBFS_DG_CD_08_celadm08
         DBFS_DG_CD_09_celadm08
         DBFS_DG_CD_10_celadm08
         DBFS_DG_CD_11_celadm08
         RECO_DMORL_CD_00_celadm08
         RECO_DMORL_CD_01_celadm08
         RECO_DMORL_CD_02_celadm08
         RECO_DMORL_CD_03_celadm08
         RECO_DMORL_CD_04_celadm08
         RECO_DMORL_CD_05_celadm08
         RECO_DMORL_CD_06_celadm08
         RECO_DMORL_CD_07_celadm08
         RECO_DMORL_CD_08_celadm08
         RECO_DMORL_CD_09_celadm08
         RECO_DMORL_CD_10_celadm08
         RECO_DMORL_CD_11_celadm08
The number of dirty buffers (unflushed) will be 0 after flush is complete
CellCLI> LIST METRICCURRENT FC_BY_DIRTY
CellCLI> LIST METRICCURRENT FC_BY_DIRTY
         FC_BY_DIRTY     FLASHCACHE      0.000 MB

2. Drop the FlashCache
CellCLI> DROP FLASHCACHE all

3. Drop the FlashLog
CellCLI> DROP FLASHLOG all
4. Drop the celldisks on the FlashDisks
CellCLI> DROP CELLDISK all flashdisk

5. Enable flash cache compression using the following commands, based on the system:
If Exadata Storage Cell Server image is 11.2.3.3.0 and the Storage Cell is X3-2:
CellCLI> ALTER CELL flashCacheCompX3Support=true
CellCLI> ALTER CELL flashCacheCompress=true

If Exadata Storage Cell server image is 11.2.3.3.1 or higher:
CellCLI> ALTER CELL flashCacheCompress=true
6. Validate the Physical Disk/Luns have the new size (Reference Table 2. Physical Disk size for FlashDisks)
CellCLI>LIST PHYSICALDISK attributes name, physicalSize where disktype=flashdisk

AURA 2.0 / F40/X3
AURA 2.1 / F80 / X4

Compression OFF
Compression ON
Compression OFF
Compression ON
Physical Disk Size
93.13 G
186.26 G
186.26 G
372.53 G
Flash Cache Size
1489 G
2979 G
2979 G
5959 G
Table 1. Physical Disk size for FlashDisks

7. Create the celldisks on the FlashDisks
CellCLI> CREATE CELLDISK all flashdisk
Validate all sixteen FlashDisks are present
CellCLI> LIST CELLDISK where disktype=flashdisk
8. Create the FlashLog
CellCLI> CREATE FLASHLOG all
Validate FlashLog is in normal mode
CellCLI> LIST FLASHLOG detail
9.  Create the FlashCache
CellCLI> CREATE FLASHCACHE all
Validate FlashCache is in normal mode
CellCLI > LIST FLASHCACHE detail

How to disable Flash Cache compression:
 1. Flush the dirty blocks from Flash Disks to the GridDisks
CellCLI> ALTER FLASHCACHE all flush
To validate that flush operation was successful use any of following methods, opening a new cellcli session
Verify cachedby attribute is null.  If a FlashDisk is reported, flush is still in progress
CellCLI> LIST GRIDDISK attributes name,cachedby
The number of dirty buffers (unflushed) will be 0 after flush is complete
CellCLI> LIST METRICCURRENT fc_by_dirty
2. Drop the FlashCache
CellCLI> DROP FLASHCACHE all
3. Drop the FlashLog
CellCLI> DROP FLASHLOG all
4. Drop the celldisks on the FlashDisks
CellCLI> DROP CELLDISK all flashdisk
5. Disable Flash Cache Compression
If Exadata Storage Cell Server image is 11.2.3.3.0 and the Storage Cell is X3-2:
CellCLI> ALTER CELL flashCacheCompX3Support=true
CellCLI> ALTER CELL flashCacheCompress=false
If Exadata Storage Cell server image is 11.2.3.3.1 or higher:
CellCLI> ALTER CELL flashCacheCompress=false
 Validate it has been disabled by checking cell attributes:
CellCLI> LIST CELL attributes name,flashCacheCompress
Correct values are FALSE or a null string.
6. Validate the Physical Disks have been shrunk
CellCLI> LIST PHYSICALDISK attributes name,physicalSize, status where disktype=flashdisk
The status should be normal.  
Use the following table to validate the expected size when Compression is OFF

AURA 2.0 / F40/X3
AURA 2.1 / F80 / X4

Compression OFF
Compression ON
Compression OFF
Compression ON
Physical Disk Size
93.13 G
186.26 G
186.26 G
372.53 G
Flash Cache Size
1489 G
2979 G
2979 G
5959 G
 Table 2.PhysicalDisk size for FlashDisks
 7. Create the celldisks on the FlashDisks
CellCLI> create celldisk all flashdisk
Validate all sixteen FlashDisks are present
CellCLI> list celldisk where disktype=flashdisk
8. Create the FlashLog
CellCLI> create flashlog all
Validate FlashLog is in normal mode
CellCLI> list flashlog detail
9.  Create the FlashCache
CellCLI> create flashcache all
Validate FlashCache is in normal mode
CellCLI > list flashcache detail

Enable Exadata Flash Cache Compression – Non-Rolling Method
Create a cell_group file with other storage cell name
·  cat cell_group
    Storage server1
    Storage server2

·  dcli -g cell_group -l root cellcli -e list flashcache attributes size
·  dcli -g cell_group -l root cellcli -e alter flashcache all flush
·  dcli -g cell_group -l root cellcli -e drop flashcache all
·  dcli -g cell_group -l root cellcli -e drop flashlog all
·  dcli -g cell_group -l root cellcli -e drop celldisk all flashdisk
     
    For Exadata Storage Software version 11.2.3.3.1 & above and Exadata X4
    dcli -g cell_group -l root cellcli -e alter cell FlashCacheCompress=true

    For Exadata Storage Software version 11.2.3.3.0 and Exadata X3
   dcli -g cell_group -l root cellcli -e alter cell flashCacheCompX3Support=true
   dcli -g cell_group -l root cellcli -e alter cell flashCacheCompress=true

   dcli -g cell_group -l root cellcli -e create celldisk all flashdisk
   dcli -g cell_group -l root cellcli -e create flashlog all size=512M
   dcli -g cell_group -l root cellcli -e create flashcache all
   dcli -g cell_group -l root cellcli -e list cell attributes flashCacheCompress
   

   Note: Exadata Flash Cache Compression is discontinued for Exadata Storage Server X5.
   Reference: Oracle support Document (ID 1664257.1)