Wednesday, September 16, 2009

PART 1 :- RMAN with Oracle Data Guard in Oracle Database 10g

INTRODUCTION
==================
Data Guard enables and automates the management of a disaster recovery solution for Oracle databases located on the same campus or across the continent. Data Guard consists of a production database (also known as the primary database) and one or more standby database(s), which are transactionally consistent copies of the production database. As transactions occur in the primary database, redo data is generated and is written to the local redo logs. Data Guard automatically transfers this redo data to the standby sites and applies it to the standby databases, synchronizing them with the primary database


RMAN and Data Guard are part of the integrated Oracle High Availability technology stack, RMAN backups can be seamlessly offloaded to a physical standby database, allowing customers to gain more value out of their disaster recovery investment. Backups do not impact normal Data Guard operation – they can be taken while the standby database is in recovery or read-only mode. Backups can be used to recover either primary or standby database servers
 
Data Guard and RMAN are both fully supported features of the Oracle Database Enterprise Edition (RMAN is also provided with Oracle Database Standard Edition).
 
The assumptions for this setup are :-




•The standby database is a physical standby database and backups are only taken on the standby database.


•The data file directories on the primary and standby database are identical.This simplifies the RMAN backup and recovery operations no matter which host is used.


•RMAN Recovery Catalog is required so that backups taken on one database server can be restored onto another database server. Using just the control file as the RMAN repository is not sufficient, as the primary database will have no knowledge of backups taken on the standby database.


•Primary database does not use Oracle Managed Files (OMF). When using OMF, standby database filenames can vary from those on the primary. Refer to the Appendix for modifications to the restore procedures when standby database filenames are different than those on the primary.


Recommended Oracle Database Configuration
    On primary and standby databases:


Configure Flash Recovery Area :-
     Configure the Flash Recovery Area, by setting the following init.ora
parameters:


DB_RECOVERY_FILE_DEST = <mount point or ASM Disk Group>
DB_RECOVERY_FILE_DEST_SIZE = <disk space quota>


Use a system parameter file (SPFILE) so that it can be used with any database in the Data Guard configuration. This allows restoring of the SPFILE from a backup taken on another database.



Uniquely name archived log and backup directories for each database.


For example, on a primary database named ‘PROD’, archived logs are written to ‘/arch/PROD’ directory, whereas on a standby database named ‘STDBY’, archived logs are written to ‘/arch/STDBY’ directory. This allows RMAN maintenance commands, such as CROSSCHECK and DELETE, to be used with the LIKE option to select the appropriate archived logs for a particular database.


Uniquely tag backups,


 If backups are taken on primary and standby databases. For example, primary database full backups can be tagged ‘BOS_FULL_BACKUP’, while standby database full backups can be tagged ‘SF_FULL_BACKUP’. This allows for proper selection of backups that were taken on a particular database when performing RMAN maintenance operations.



Enable Flashback Database on primary and standby databases.


When Flashback Database is enabled, Oracle maintains flashback logs in the Flash Recovery Area. These logs can be used to ‘rewind’ the database back to an earlier point in time, without requiring a complete restore.

Friday, September 11, 2009

Creating a Duplicate Database on a New Host

Duplicate database from server A to server B (Non ASM)
=============================================================


Assumed database names:


Primary Database SID: PROD
Duplicate Database SID: AUX
RMAN Catalog SID: RMAN

Steps


1. Backup the primary database.


2. Determine how much disk space will be required.


3. Ensuring you have enough space on your target server.


4. Making the backup available for the duplicate process.


5. Creating init.ora & admin directories for the duplicate database.


6. Checking connections to primary database and RMAN catalog.


7. Prepare RMAN duplicate script.


8. Execute the RMAN script.


=========================================================


Backup of the primary database


Host A (Target)


Prior to creating a duplicate database you must have a backup of the target database. In this example we assume the backup media is disk. Find appropriate disk space on host A and take a full backup including archive logs and current controlfile.


If you are planning to duplicate a subset of the database refer to second script illustrates the RMAN command to backing up certain tablespaces.


[A]This command will perform a full database backup including archivelogs and the current controlfile


[oracle@linux] export ORACLE_SID=PROD


[oracle@linux] rman target=/ catalog=rman/rman@RMAN


RMAN> run {
allocate channel d1 type disk;
backup format '/backups/PROD/df_t%t_s%s_p%p' database;
sql 'alter system archive log current';
backup format '/backups/PROD/al_t%t_s%s_p%p' archivelog all;
release channel d1;
}



[B]This command will perform a tablespace backup ( SYSTEM, SYSAUX, UNDO & USERS) including archive logs and the current controlfile


[oracle@linux] export ORACLE_SID=PROD


[oracle@linux] rman target=/ catalog=rman/rman@RMAN


RMAN> run {
allocate channel d1 type disk;
backup format '/backups/PROD/df_t%t_s%s_p%p' tablespace SYSTEM, SYSAUX, UNDO, USERS;
sql 'alter system archive log current';
backup format '/backups/PROD/al_t%t_s%s_p%p' archivelog all;
release channel d1;
}



2. Determine how much disk space will be required


[A] This query will calculte the size of the whole DB instead of subset of the Database...If you are going for Subset then pls check the Script..B


select DF.TOTAL/1048576 "DataFile Size Mb",
LOG.TOTAL/1048576 "Redo Log Size Mb",
CONTROL.TOTAL/1048576 "Control File Size Mb",
(DF.TOTAL + LOG.TOTAL + CONTROL.TOTAL)/1048576 "Total Size Mb" from dual,
(select sum(a.bytes) TOTAL from dba_data_files a) DF,
(select sum(b.bytes) TOTAL from v$log b) LOG,
(select sum((cffsz+1)*cfbsz) TOTAL from x$kcccf c) CONTROL;

Output will be like this:-


DataFile Size Mb Redo Log Size Mb Control File Size Mb Total Size Mb
---------------- ---------------- -------------------- -------------
13491 75 22.140625 13588.1406


[B] This script will calculate the Size of the subset of the database..only for the sapce required for these tablespaces ['SYSTEM','UNDOTBS1', 'SYSAUX', 'USERS']


select DF.TOTAL/1048576 "DataFile Size Mb",
LOG.TOTAL/1048576 "Redo Log Size Mb",
CONTROL.TOTAL/1048576 "Control File Size Mb",
(DF.TOTAL + LOG.TOTAL + CONTROL.TOTAL)/1048576 "Total Size Mb" from dual,
(select sum(a.bytes) TOTAL from dba_data_files a where tablespace_name in ('SYSTEM','UNDOTBS1', 'SYSAUX', 'USERS')) DF,
(select sum(b.bytes) TOTAL from v$log b) LOG,
(select sum((cffsz+1)*cfbsz) TOTAL from x$kcccf c) CONTROL;



Output will be like this


DataFile Size Mb Redo Log Size Mb Control File Size Mb Total Size Mb
---------------- ---------------- -------------------- -------------
1062 75 22.140625 1159.14063


You can check the diff in the output of the script related to the size of files and Database..


3. Ensuring you have enough space on your target server [HOST B]


Prior to starting the duplicate process you must ensure you have enough disk space within file system to hold the database


--Sample output showing the space available on your filesystem--


[oracle@linux] df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/vg01/root 9.9G 2.8G 6.6G 30% /
/dev/sda1 145M 16M 122M 12% /boot
none 999M 0 999M 0% /dev/shm
/dev/vg01/tmp 2.0G 383M 1.5G 20% /tmp
/dev/vg01/u01 20G 12G 7.0G 62% /u01
/dev/vg01/u02 4.9G 1010M 3.6G 22% /u02
/dev/vg01/backups 5.9G 1.2G 4.4G 22% /backups
/dev/vg01/oradata 15G 13G 2.0G 87% /oradata


4. Making the backup available for the duplicate process


If your backup resides on disk you will need to copy this back up from host A to host B. Ensure you place it in the same directory as where it was created. In the example below the backup piece resides in ‘/backups/PROD’ these files need to be copied into the same directory on host B.


Also ensure than all archive log backups have also been moved and any archive logs in the archive log directory that may be require for the recovery. The archive logs required will depend on the point in time of the recovery


RMAN> list backup;


BS Key Type LV Size Device Type Elapsed Time Completion Time
------- ---- -- ---------- ----------- ------------ -------------------
22 Full 529M DISK 00:00:51 2006/05/16 11:12:54
BP Key: 22 Status: AVAILABLE Compressed: NO Tag: TAG20060516T111203
Piece Name: /backups/PROD/df_t590584323_s23_p1
List of Datafiles in backup set 22
File LV Type Ckp SCN Ckp Time Name
---- -- ---- ---------- ------------------- ----
1 Full 1393845 2006/05/16 11:12:03 /oradata/PROD/system01.dbf
2 Full 1393845 2006/05/16 11:12:03 /oradata/PROD/undotbs01.dbf
3 Full 1393845 2006/05/16 11:12:03 /oradata/PROD/sysaux01.dbf
4 Full 1393845 2006/05/16 11:12:03 /oradata/PROD/users01.dbf

BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ -------------------
24 48M DISK 00:00:06 2006/05/16 11:13:07
BP Key: 24 Status: AVAILABLE Compressed: NO Tag: TAG20060516T111301
Piece Name: /backups/PROD/al_t590584381_s25_p1


List of Archived Logs in backup set 24
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- ------------------- ---------- ---------
1 78 1344750 2006/05/15 04:55:34 1353208 2006/05/15 10:00:19
1 79 1353208 2006/05/15 10:00:19 1353213 2006/05/15 10:00:20
1 80 1353213 2006/05/15 10:00:20 1371564 2006/05/15 22:00:11
1 81 1371564 2006/05/15 22:00:11 1373291 2006/05/15 22:00:59
1 82 1373291 2006/05/15 22:00:59 1381066 2006/05/16 03:00:05
1 83 1381066 2006/05/16 03:00:05 1390685 2006/05/16 09:03:00
1 84 1390685 2006/05/16 09:03:00 1393870 2006/05/16 11:13:00
1 85 1393870 2006/05/16 11:13:00 1393929 2006/05/16 11:13:00


5. Creating the init.ora & administration directories for the duplicate database


Host B(AUX)


Create the pfile [initAUX.ora] parameter file in the $ORACLE_HOME/dbs directory for the auxiliary database with bare minimum settings.


Note you may require to set parameters which are the same as your production database (refer to you primary init.ora)


# +----------------------------------------+
# | FILE : initAUX.ora |
# | DATABASE NAME : AUX |
# +----------------------------------------+
# Set the below to location of the clone Duplicate database / name of
# clone database.


audit_file_dest =/oradata/AUX/adump
background_dump_dest =/oradata/AUX/bdump
core_dump_dest =/oradata/AUX/cdump
user_dump_dest =/oradata/AUX/udump
db_name ="AUX"
instance_name =AUX
cluster_database=false (in case the production is a rac environment)

# Set the below to the location of the duplicate clone control file.


control_files =('/oradata/AUX/control01.ctl','/oradata/AUX/control02.ctl','/oradata/AUX/control03.ctl')

# Set the below for the from and to location for all data files / redo
# logs to be cloned.


db_file_name_convert =("/oradata/PROD/", "/oradata/AUX/")
log_file_name_convert =("/oradata/PROD/", "/oradata/AUX/")



#Set the below to the same as the production target


undo_management =AUTO
undo_retention =10800
undo_tablespace =UNDOTBS1
db_block_size = 8192
compatible = 10.2.0.1.0
------------EOF----------
Following the creation of the initAUX.ora startup nomount the auxiliary instance.


[oracle@linux]export ORACLE_SID=AUX


[oracle@linux] sqlplus '/as sysdba'


SQLPLUS> startup nomount;

6. Ensuring SQL*NET connections to primary database and RMAN catalog are working.


Host B(AUX)


Ensure the production target database is open or at least mounted. If using a catalog database this must also be open. Test your SQL*NET connections: From Host B you must be able to connect to the primary DB as sysdba and make an rman connection to RMAN. Perform a basic test using sqlplus. When you have successfully connected exit sqlplus and move onto the next step.


[oracle@linux]% sqlplus ‘sys/oracle@PROD as sysdba’


[oracle@linux]% sqlplus rman/rman@RMAN (not mandatory)


7. Prepare RMAN duplicate script.


In a working directory on Host B create an RMAN script file duplicate.rcv. The example below


[A] shows the command for a complete duplicate


run {
allocate auxiliary channel C1 device type disk;
duplicate target database to AUX;
}


[B] skips the tablespaces which are not required in the duplicate and


run {
allocate auxiliary channel C1 device type disk;
duplicate target database to AUX skip tablespace ABC, XYZ;
}


[C] provide the syntax required for a point in time duplicate.


run {
set until time "to_date('Jan 01 2000 12:00:00','Mon DD YYYY HH24:MI:SS')";
allocate auxiliary channel C1 device type disk;
duplicate target database to AUX;
}



8. Execute the RMAN script.


Start RMAN, connect to the production target, the catalog instance and also the auxiliary clone. Run the RMAN duplicate script as shown below. Before doing this ensure that the Oracle SID environment variable is set to the duplicate clone database.


[oracle@linux] export ORACLE_SID=AUX


[oracle@linux] rman target sys/sys@PROD catalog rman/rman@rman auxiliary /


RMAN> @duplicate.rcv




Vijay Kumar...

RMAN Stored Scripts in the Recovery Catalog

Stored scripts offer an alternative to command files for managing frequently used sequences of RMAN commands.

The advantage of a stored script over a command file is that a stored script is always available to any RMAN client that can connect to the target database and recovery catalog, whereas command files are only available if the RMAN client has access to the file system on which they are stored.

Stored scripts can be global or local. A local stored script is associated with the target database to which RMAN is connected when the script is created, and can only be executed when you are connected to that target database. A global stored script can be run against any database registered in the recovery catalog, if the RMAN client is connected to the recovery catalog and a target database.

Note that to work with stored scripts, even global ones, you must be connected to both a recovery catalog and a target instance

Creating Stored Scripts: CREATE SCRIPT

Make sure RMAN is connected to the right target database and the recovery catalog. Then run the CREATE SCRIPT command, as shown in this example:

CREATE SCRIPT full_db_backup
{
BACKUP DATABASE PLUS ARCHIVELOG;
DELETE OBSOLETE;
}

Examine the output. If no errors are displayed, then the script was successfully created and stored in the recovery catalog

For a global script, the syntax is similar:
CREATE GLOBAL SCRIPT global_full_db_backup
{
BACKUP DATABASE PLUS ARCHIVELOG;
DELETE OBSOLETE;
}


You can also provide a COMMENT with descriptive information:

CREATE GLOBAL SCRIPT global_full_db_backup
COMMENT 'use only with ARCHIVELOG mode databases'
{
BACKUP DATABASE PLUS ARCHIVELOG;
DELETE OBSOLETE;
}


Finally, you can create a local or global script, reading its contents from a text file:

CREATE SCRIPT full_db_backup FROM FILE 'script_file.txt';

The file must begin with a { character, contain a series of commands valid within a RUN block, and end with a } character. Otherwise, a syntax error is signalled, just as if the commands were entered at the keyboard

Running Stored Scripts: EXECUTE SCRIPT
============================================
To run a stored script, connect to the target database and recovery catalog, and use EXECUTE SCRIPT. EXECUTE SCRIPT requires a RUN block, as shown:

RUN { EXECUTE SCRIPT full_db_backup; }

This command invokes a local script if one is with the name specified. If no local script is found, but there is a global script with the name specified, RMAN will execute the global script.

You can also use EXECUTE GLOBAL SCRIPT to control which script is invoked if a local and a global script have the same name

Assuming there is no local script called global_full_db_backup, the following two commands have the same effect:

RUN { EXECUTE GLOBAL SCRIPT global_full_db_backup; }
RUN { EXECUTE SCRIPT global_full_db_backup; }

Executing a global script only affects the connected target database; to run a global script across multiple databases, you must connect the RMAN client to each one separately and execute the script.

Your script will use the automatic channels configured at the time you execute the script. Use ALLOCATE CHANNEL commands in the script if you need to override the configured channels

Note If because of the RUN block, if an RMAN command in the script fails, subsequent RMAN commands in the script will not execute.

Displaying a Stored Script: PRINT SCRIPT
===============================================

The PRINT SCRIPT command displays a stored script or writes it out to a file. With RMAN connected to the target database and recovery catalog, use the PRINT SCRIPT command as shown here:

PRINT SCRIPT full_db_backup;

To send the contents of a script to a file, use this form of the command:

PRINT SCRIPT full_db_backup TO FILE 'my_script_file.txt';

For global scripts, the analogous syntax would be:

PRINT GLOBAL SCRIPT global_full_db_backup;

and

PRINT GLOBAL SCRIPT global_full_db_backup TO FILE 'script_file.txt';


Listing Stored Scripts: LIST SCRIPT NAMES
===============================================

Use the LIST SCRIPT NAMES command to display the names of scripts defined in the recovery catalog. This command displays the names of all stored scripts, both global and local, that can be executed for the currently connected target database:
LIST SCRIPT NAMES;

If RMAN is not connected to a target database when the LIST SCRIPT NAMES command is run, then RMAN will respond with an error.

To view only global script names, use this form of the command:

LIST GLOBAL SCRIPT NAMES;

To view the names of all scripts stored in the current recovery catalog, including global scripts and local scripts for all target databases registered in the recovery catalog, use this form of the command:

LIST ALL SCRIPT NAMES;

The output will indicate for each script listed which target database the script is defined for (or whether a script is global).

Note:
LIST GLOBAL SCRIPT NAMES and LIST ALL SCRIPT NAMES are the only commands that work when RMAN is connected to a recovery catalog without connecting to a target instance.

Updating Stored Scripts: REPLACE SCRIPT
==============================================

To update stored scripts, connect to the target database and recovery catalog and use the REPLACE SCRIPT command. If the script does not already exist, then RMAN creates it.

This command updates stored script script full_backup with new contents:

REPLACE SCRIPT full_db_backup
{
BACKUP DATABASE PLUS ARCHIVELOG;
}


Global scripts can be updated using the REPLACE GLOBAL SCRIPT command when connected to a recovery catalog, as follows:

REPLACE GLOBAL SCRIPT global_db_full_backup
COMMENT 'A script for full backup to be used with any database'
{
BACKUP AS BACKUPSET DATABASE PLUS ARCHIVELOG;
}


Deleting Stored Scripts: DELETE SCRIPT
=============================================

To delete a stored script from the recovery catalog, connect to the catalog and a target database, and use the DELETE SCRIPT command:
DELETE SCRIPT 'full_backup';

To delete a global stored script, use DELETE GLOBAL SCRIPT:

DELETE GLOBAL SCRIPT 'global_full_db_backup';

If you use DELETE SCRIPT without GLOBAL, and there is no stored script for the target database with the specified name, RMAN will look for a global stored script by the specified name and delete the global script if it exists. So, if you were to enter the command

DELETE SCRIPT 'global_full_db_backup';

RMAN would look for a script 'global_full_db_backup' defined for the connected target database, and if it did not find one, it would search the global scripts for a script called 'global_full_backup' and delete that script.

Starting the RMAN Client and Running a Stored Script
===========================================================
To run the RMAN client and start a stored script in the recovery catalog on startup, use the SCRIPT argument when starting the RMAN client.

% rman TARGET SYS/oracle@trgt CATALOG rman/cat@catdb SCRIPT 'full_backup';

You must connect to a recovery catalog (which contains the stored script) and target database (to which the script will apply) when starting the RMAN client

Restrictions on Stored Script Names
==========================================

There are some issues to be aware of about how RMAN resolves script names, especially when a local and global script share the same name.

• RMAN permits but generally does not require that you use quotes around the name of a stored script. However, if the name begins with a digit or if the name is an RMAN reserved word, you will have to put quotes around the name to use it as a stored script name. Consider avoiding stored script names that begin with characters other than A-Z or that are the same as RMAN reserved words.

• When starting the RMAN client with a SCRIPT argument on the command line, if local and global scripts are defined with the same name, then RMAN will always execute the local script.

• For the EXECUTE SCRIPT, DELETE SCRIPT and PRINT SCRIPT commands, if the script name passed as an argument is not the name of a script defined for the connected target instance, RMAN will look for a global script by the same name to execute, delete or print. For example, if the a stored script global_full_backup is in the recovery catalog as a global script, but no local stored script global_full_backup is defined for the target database, the following command will delete the global script:

• DELETE SCRIPT global_full_db_backup;

Consider using some naming convention to avoid mistakes due to confusion between global stored scripts and local stored scripts


Vijay Kumar

Thursday, September 10, 2009

CRSCTL

Find below various commands which can be used to administer Oracle Clusterware using crsctl.


There are options for CRSCTL which can be seen using the fllowing command



#crsctl

Or

#crsctl help



To determine software version (binary version of the software/operating version on a particular cluster node) use


#crsctl query crs softwareversion [] - lists the version of CRS software installed

Oracle Clusterware version on node [Rac01] is [x.x.x.x.x]


#crsctl query crs activeversion - lists the CRS software operating version

Oracle Clusterware active version on the cluster is [x.x.x.x.x]


Start Oracle Clusterware --> #crsctl start crs


Stop Oracle Clusterware --> #crsctl stop crs


Enable Oracle Clusterware --> #crsctl enable crs


It enables automatic startup of Clusterware daemons


Disable Oracle Clusterware --> #crsctl disable crs

NOTE:- 1) This command disables OC from being started in
a subsequent restart

2) This Does not stop the currently running OC Stack




@===== Check CRS Status =====@


#crsctl check crs

Cluster Synchronization Services appears healthy
Cluster Ready Services appears healthy
Event Manager appears healthy



@===== To see particular daemon status =====@


$crsctl check cssd

Cluster Synchronization Services appears healthy


$crsctl check crsd

Cluster Ready Services appears healthy


$crsctl check evmd

Event Manager appears healthy


You can also check Clusterware status on the nodes using



$crsctl check cluster

Rac01 ONLINE
Rac02 ONLINE
..... ......



@===== Checking Voting disk Location =====@


$crsctl query css votedisk
0. 0 /dev/sda3
1. 0 /dev/sda5
2. 0 /dev/sda6

Located 3 voting disk(s).



Note: -Any command which just needs to query information can be run using oracle user. But anything which alters Oracle Clusterware requires root privileges.



Add Voting disk --> #crsctl add css votedisk path



Remove Voting disk --> #crsctl delete css votedisk path




@===== Enabling Debugging of Oracle Clusterware Components =====@



You can enable debugging for the Oracle Cluster daemons by running crsctl commands [debugging_level is a number from 1 to 5]



# crsctl debug log component module:debugging_level


To enable tracing for the EVMD module of the css component, you could use the following command:


# crsctl debug log css EVMD:1



@===== Locating the Oracle Clusterware Alert Log =====@


The location of the Oracle Clusterware log file is CRS_home/log/hostname/alerthostname.log, where CRS_home is the directory in which Oracle Clusterware was installed and hostname is the host name of the local node

To obtain a list of the resources available for debugging, use the following command:


# crs_stat

=====@@@@@@@@@@@@@@@@@@@=====
USAGE
============================

crsctl
Usage: crsctl check crs - checks the viability of the CRS stack
crsctl check cssd - checks the viability of CSS
crsctl check crsd - checks the viability of CRS
crsctl check evmd - checks the viability of EVM
crsctl set css - sets a parameter override
crsctl get css - gets the value of a CSS parameter
crsctl unset css - sets CSS parameter to its default
crsctl query css votedisk - lists the voting disks used by CSS
crsctl add css votedisk - adds a new voting disk
crsctl delete css votedisk - removes a voting disk
crsctl enable crs - enables startup for all CRS daemons
crsctl disable crs - disables startup for all CRS daemons
crsctl start crs - starts all CRS daemons.
crsctl stop crs - stops all CRS daemons. Stops CRS resources in case of cluster.
crsctl start resources - starts CRS resources.
crsctl stop resources - stops CRS resources.
crsctl debug statedump evm - dumps state info for evm objects
crsctl debug statedump crs - dumps state info for crs objects
crsctl debug statedump css - dumps state info for css objects
crsctl debug log css [module:level]{,module:level} ...


- Turns on debugging for CSS

crsctl debug trace css - dumps CSS in-memory tracing cache
crsctl debug log crs [module:level]{,module:level} ...

- Turns on debugging for CRS

crsctl debug trace crs - dumps CRS in-memory tracing cache
crsctl debug log evm [module:level]{,module:level} ...

- Turns on debugging for EVM

crsctl debug trace evm - dumps EVM in-memory tracing cache
crsctl debug log res turns on debugging for resources
crsctl query crs softwareversion [] - lists the version of CRS software installed
crsctl query crs activeversion - lists the CRS software operating version
crsctl lsmodules css - lists the CSS modules that can be used for debugging
crsctl lsmodules crs - lists the CRS modules that can be used for debugging
crsctl lsmodules evm - lists the EVM modules that can be used for debugging

If necesary any of these commands can be run with additional tracing by
adding a "trace" argument at the very front.


Example: crsctl trace check css


Vijay Kumar....