Friday, 29 May 2020

ORA-39511: Start of CRS resource for instance '223' failed with error:[CRS-5702: Resource 'ora.asm' is already running on 'xvm009'

Problem abstract: 

While trying to startup a new instance manually in order to start duplicating one database I was getting below error:

SQL> startup nomount pfile=initxtd1.ora 
ORA-39511: Start of CRS resource for instance '223' failed with error:[CRS-5702: Resource 'ora.asm' is already running on 'xvm009'
CRS-0223: Resource 'ora.asm' has placement error.
clsr_start_resource:260 status:223
clsrapi_start_asm:start_asmdbs status:223


This is on a 19c database setup on a cluster environment with ASM.

Before I issue this command I've manually dropped a database having same name xtd, I didn't use dbca for this job because I had a problem login to the server with VNC, so I took the easy approach and dropped it manually and deleted all its files under ASM groups.

Fix:

It's always recommended dropping the database using DBCA, specially on a cluster/Oracle Restart environment. This will maintain to clean up all database resources on the server and avoid having above error.

For my scenario, because my job was not complete, I had to remove the remaining database cluster resources manually using the following commands:

1- Remove the database and its instances resources from OCR:
# srvctl remove instance -d xtd -i xtd1
# srvctl remove instance -d xtd -i xtd2
# srvctl remove database -d xtd


2- Remove all the database files inside ASM including datafiles, redo log files, controlfiles, password file, spfile from asmcmd console.
3- Remove the database entry from /etc/oratab

4- Make sure there are no resources belong to the dropped database:

# crsctl stat res -p


ORA-00600: internal error code, arguments: [krfg_mgen_coord2], [82], [2], [2], [], [], [], [], [], [], [], []

Problem: 

On one of 19c active data guard RAC database the recovery process was keep failing with below error:

ORA-00600: internal error code, arguments: [krfg_mgen_coord2], [82], [2], [2], [], [], [], [], [], [], [], []
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
ORA-10877: error signaled in parallel recovery slave
Incident details in: /u01/oracle/diag/rdbms/sp/sp1/incident/incdir_1639057/sp1_mrp0_4991_i1639057.trc 


Fix:

I came to know it's Bug 12407536 | MRP PROCRESS CRASHES DUE TO ORA-600 [KRFG_MGEN_COORD2]
The workaround is to disable the FLASHBACK feature which was already enabled on that database:

SQL> ALTER DATABASE FLASHBACK OFF;
SQL> RECOVER MANAGED STANDBY DATABASE NODELAY DISCONNECT;