Friday, August 31, 2007

Connect to multiple Applications 11i Instance using ADI

I had issue with connecting to ADI to individual instance when more than one Oracle Applications were running on same server. I performed following to overcome issue. 1. Defined the RRA: Service Prefix in each of your instances at site level as FNDFS_ntest. 2. Added in the Desktop PC tnsnames.ora in $ORACLE_HOME/network/admin and $ORACLE_HOME/net80/admin should be equal and include definitions for FNDFS which name should be FNDFS__

FNDFS_ntest_nfinox = (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=nfinox)(PORT=1522)) (CONNECT_DATA=(SID=FNDFS)) )

3. Added given two services in each tnsnames in the 8.0.6 ORACLE HOME of each instance in the applications server is generated by AutoConfig including these two services: FNDFS_ntest = (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=nfinox)(PORT=1522)) (CONNECT_DATA=(SID=FNDFS)) ) FNDFS_ntest_nfinox = (DESCRIPTION= (ADDRESS=(PROTOCOL=tcp)(HOST=nfinox)(PORT=1522)) (CONNECT_DATA=(SID=FNDFS)) ) 4. Restarted the Concurrent Managers -- All the instances 5. Restarted the APPS Listener -- All the instances

Create Global oraInventory for Oracle Applications 11i

Some time I am not able to apply database patch (using opatch) to RDBMS home due to global inventory issue. So I thought of putting all the steps together to recreate global inventory problem. 1. Login as oracle user and identify the oracle Inventory location by referring to content of /var/opt/oracle/oraInst.loc.

$ cat /var/opt/oracle/oraInst.loc inventory_loc=/local/oracle/admin/ntst/oraInventory inst_group=dba
2. Renamed old inventory
$cd /local/oracle/admin/ntst
$mv oraInventory oraInventory.old
$chmod 777 oraInventory
3. Downloaded patch Patch 5035661 for HP-UX 32 (my os is HP-UX)
4. Login as
$ . $APPL_TOP/APPSORA.env
$ pwd /reports/ntrg/patches/secjuly07 $ echo $IAS_ORACLE_HOME /apps/OEBS/ntrg/iAS $ unzip -od /apps/OEBS/ntrg/iAS p5035661_11i_HPUX11.zip
$ cd $IAS_ORACLE_HOME/appsoui/setup $ pwd /apps/OEBS/ntst/iAS/appsoui/setup $ perl OUIsetup.pl
Check log from above script and ensure it was successful.
5. Go to parent directory of oraInventory and change its permission to 777 recursively.
$ pwd /local/oracle/admin/ntrg/oraInventory
$ chmod -R 777 *
6. Login as oracle and set enviroment
$cd /reports/ntrg/patches/secjuly07 $unzip -od $ORACLE_HOME p5035661_11i_HPUX11.zip
$ cd $ORACLE_HOME/appsoui/setup $pwd /local/oracle/product/9.2.0.8.2/appsoui/setup
$perl OUIsetup.pl
Log file : /local/oracle/product/9.2.0.8.2/appsoui/setup/OUIsetup.log
7. Check if you can list oraInventory
$cd $ORACLE_HOME/OPatch/
$./opatch lsinventory
8. You can also invoke universal installer to check the home registration validity in global inventory.
$export DISPLAY=:0.0
$cd $ORACLE_HOME/appsoui/oui/install
$./runInstaller.sh
It will invoke universal installer and you can see RDBS home registered there.
Subject: How to Recreate the Global oraInventory Note: 295185.1

Custom Forms/Menu Failes with Listener Servlet with Oracle Applications 11i

Last week we configured oracle application forms from socket mode to servelete move in production. We were not able to start any forms/menus from oracle applications. When we further investigated issue and did following. 1. Go to $APACHE_TOP/Jserv/etc/ 2. Add all custom tops in the formservlet.ini file. Put custom top between “#Begin Customization” and “End Customization”. #Begin Customization SEXX_TOP=/apps/OEBS//appltop/sexx/custom #End Customization 3. Begin Customization and end Customization entries are needed if instance is autoconfig enabled else SEXX_TOP will be overwritten by autoconfig. 4. You need not to restart services. You can log off and login again.

Thursday, August 30, 2007

How to turn on trace for apache and Jserv to troubleshoot Selfservice Issue?

Most of time Oracle ask us to generate trace file of Apache and Jserv so I though of putting all steps together. 1. Please enable debug in your Apache Conf files Bring down Apache - Set LogLevel debug in the following files httpd.conf, httpd.conf_pls, and jserv.conf - Edit the .../iAS/Apache/Jserv/etc/jserv.properties to enable debug for the java portion of the Apache jserv. example log=true / log.timestamp=true 2. Delete all existing logs from $IAS_ORACLE_HOME/A* ./Apache/logs/ ./Jserv/logs/ ./Jserv/jvm/logs/ 3. Restart Apache 4. Recreate your issue 5. Provide all log and config files with the following script To be executed from the .../iAS/Apache directory: zip logfile.zip \ ./Apache/logs/*_log* \ ./Apache/conf/*.conf \ ./Jserv/logs/*.log \ ./Jserv/logs/jvm/* \ ./Jserv/etc/*.properties \ ./Jserv/etc/*.conf \ ./Jserv/etc/*.ini \ ./Jserv/etc/*.txt

Oracle Applications Clear Cache

Refer given below to clear cache in E-Business Suite. 1. Apache - shutdown iAS server- go to $OA_HTML or $COMMON_TOP (11.5.10) directory- remove the directory _pages- restart iAS server 2. Browser -Netscape :- go to menu Edit => Preferences…, - choose 'Cache' in 'Advanced' category,- click on buttons 'Clear Memory Cache' and 'Clear Disk Cache'.- close all Netscape windows and restart new browser session. -Internet Explorer :- go to menu Tools => Internet Options…,- select 'General' tab,- click on button 'Delete Files' in 'Temporary Internet files' zone- close all IE windows and restart new browser session. 3. Jinitiator -Two possibilities depending of Jinitiator version, for instance: - delete all files in directory : C:\Program Files\Oracle\Jinitiator 1.1.7.27\jcache\ (Jinitiator version directory may be different) or - go to Start => Parameters => Control Panel- double-click on Jinitiator 1.3.1.18 (or other version)- in the new pop-up window, click on "Cache" Tab- click on "Clear Jar Cache " button. On prompt, click Yes. (you can also delete directly all files under directory:C:\Documents and Settings\\Oracle Jar Cache) After clearing cache bound Apache and Forms server.

Friday, July 20, 2007

Oracle Critical Patch Update–July 2007

As part of security concern DBA has to apply Oracle Critical Patch regularly as and when there is critical security patch release from Oracle. Oracle started critical security patch in 2005 and since then there is four-security patches release in year. For year 2007 following are four dates:

- 17 July 2007

- 16 October 2007

- 15 January 2008

- 15 April 2008

Last we applied critical security patch July 2006. As part of regular critical security patch update we are going to apply “Oracle Critical Patch Update – July 2007” by referring given URL.

Oracle E-Business Suite Critical Patch Update Note July 2007

Note: 432882.1

Before analyzing Patch requirement I would list down product installed.

- Oracle E-Business Suite Release 11i 11.5.10 CU2

- Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production

- Oracle Application Server 10g Release 2 (10.1.2), versions 10.1.2.0.2

We have running Oracle E-Business Suite Rel. 11i so I am referring “Section 2, "Oracle E-Business Suite Release 11i"” of Note: 432882.1

As per oracle guideline we should apply updates to systems that have highest risk first e.g. update systems that are external to firewall before upgrading internal network system. It our environment we do not have any external system.

Oracle has recommended patching Oracle environment in given order and I follow same.

1. Oracle Database

2. Oracle Application Server

3. Oracle E-Business Suite

Patching Considerations for Oracle E-Business Suite Release 11i

Oracle Database and Oracle Application Server critical patches are cumulative, which means it includes all patches for fixes from earlier Oracle critical security patch updates. CPU patches for Oracle E-Business Suite 11i are generally not cumulative, that means one has to refer earlier critical updates and apply all the e-business suite critical patches in chronological order.

In my case I had applied Critical update patch July 2006 and now I need to apply Critical update patch July 2007. So in this case I should apply patches as given below.

1. Oracle Database Critical Update patches is cumulative so I need to apply only critical update patch July 2007.

2. Oracle Applications Server Critical Update patches is cumulative so I need not to apply Critical update patch July 2007.

3. Oracle E-Business Suite Critical Update patches is not cumulative so I need to apply Critical update patch Oct 2006, Jan 2007, Apr 2007 before apply Critical update patch July 2007.

As per Note: 432882.1 Release 11.5.10 applications upgraded to release 11.5.10 CU2 without apply CPUApr2007 must now apply CPUApr2007. E-Business Suite are not cumulative, so we one should apply prior CPUs as appropriate before applying CPUApr2007.

Stop all process running using that Oracle home before patching. Patch one oracle home at a time.

Oracle recommends updating database tier and the middle tier at same time, one after the other. After applying patch start services in following order. We have only these two components in our environment.

1. Oracle Database

2. Oracle Application Server

‘opatch’ is required to apply database patch.

$ pwd

/local/oracle/product/9.2.0.8/OPatch

$ opatch -version

Oracle Interim Patch Installer version 1.0.0.0.56

Oracle recommends to use OPatch release 1.0.0.0.57 so need to install patch 2617419

Patches to be applied in Ebusiness Suit.

Noteà We are Running Ebusiness Suite in HP-UX 11i environment.

Oracle Database Patches

Oracle Database 9.2.0.8 Rel Patch 6079579

Oracle HTTP Server Patche

There are no new patches for CPUJul2007. Please uptake the patches listed for this release in CPUJan2007. See Note 402670.1: Oracle E-Business Suite Critical Patch Update Note January 2007.

As per Note 402670.1

1. If you have not already done so, upgrade to release 1.0.2.2.x. See Note 146468.1.

2. Upgrade to Oracle8i Database release 8.1.7.4 using patch 2376472 if your Oracle Database Client release is earlier than release 8.1.7.4. See Note 309026.1.

3. Apply CPUJan2007 patch 5700129 to the Oracle Application Server Oracle home.

We have applied upgraded to Orace8i 8.1.7.4 home using patch 2376472 so I need to apply only CPUJan2007 patch 5700129.

Oracle Developer Suite Patches

We are at Oracle Developer Suite Release 6.0.8.27.0. According to Note 402670.1 “There are no new patches for CPUApr2007. Please uptake the following patches which were listed for this release in CPUJan2007. “

· Oracle Forms (HP-UX): 5687261

· Oracle Reports (HP-UX): 5686997

· Release 8.0.6 Oracle home: 5078711

JInitiator Patches

We are using Jinitiator Release 1.3.1.26 and as Note 402670.1 “Per Oracle policy, customers on Oracle JInitiator Release 1.3 must move to version 1.3.1.26 or higher for CPUApr2007.” Refer to Note 124606.1 and Note 232200.1 for more information.

We’ll upgrade Jinitator Release from 1.3.1.26 to 1.3.1.28.

As per Note 124606.1 I need to apply following patches to upgrade Jinitiator to 1.3.1.28.

JInitiator Version JInitiator Patch Interop Patch

1.3.1.28 5882294 5117525

Apart from above patches following patches are required for respective product users. I am listing only those patches, which are applicable in our environment.

- Oracle Applications Framework Navigator Page

If you are using Oracle Applications prior to 11.5.10.2CU, please apply patch 4517707.

- General Ledger Hierarchy Manager / Financial Intelligence - Financial Dimension Hierarchy Manager Users

If you are upgrading to JInitiator 1.3.1.x and are using Oracle Applications Framework 5.7 or later, you must apply patch 3698893.

Oracle E-Business Suite Release 11i Patches

Oracle CPUApr2007 patches for E-Business Suite Release 11i are not generally cumulative. We need to apply the previous critical patch updates and Oracle security alerts related to ebusiness Suite before applying this critical patch update.

In my case I need to apply following patches.

E-Business Suite Critical Patch October 2006

· 5486407

· 5479643 Superceded by 5473858 (11i.ATG_PF.H.RUP5)

· 5500118 Superceded by 5473858 (11i.ATG_PF.H.RUP5)

· 5335967

E-Business Suite Critical Patch January 2007

· 5571208 Superceded by 5473858

· If your instance is at 11i.BNE.D, then apply patch 5518587 Supercede by patch 5473858

· 5658489

· 3748835

· 5711485

E-Business Suite Critical Patch April 2007

Patch 5021981 (FND)

Release 11.5.10 CU2

· 5893391

· 5900224

· 5352601

· 5720979

· 5738134

· 5909233

· 5873313

· 5904576

· 5967405

Oracle E-Business Suite 11i with 11i.ATG_PF.H RUP4 (4676589) installed

· 5873313

· 5904576

· 5900224

· 5938358

· Any other patches listed for the Oracle E-Business Suite Release 11i base release on which 11i.ATG_PF.H RUP4 is installed.

E-Business Suite Critical Patch July 2007

11i with 11i.ATG_PF.H RUP4 (4676589) installed

· 5973659

· 6045931

Release 11.5.10 CU2

· 6117954

· 6045983

· 5973898

· 5973651

· 5973629

· 6185758

· 6082483

· 6110945

· 6111066

· 6111008

· 6188705

Reference:

Oracle E-Business Suite Critical Patch Update Note January 2007 Note:402670.1

Oracle E-Business Suite Critical Patch Update Note April 2007

Note:420072.1

Oracle E-Business Suite Critical Patch Update Note July 2007

Note:432882.1

Wednesday, July 18, 2007

Generate SQL tkprof

Very often we need to generate tkprof output of sql/plsql to troubleshoot performance issue so I though of putting all steps together.

  • Login as SQLPLUS.
SQL>connect apps/appsndev6 Connected.
  • Set SQL Trace ON
SQL> ALTER SESSION SET SQL_TRACE = TRUE; SQL> alter session set events '10046 trace name context forever, level 8';
  • Get SPID number to identify trace file generate at UDUMP location.
SQL> SELECT spid FROM v$process WHERE addr = ( SELECT paddr FROM v$session WHERE audsid=userenv('SESSIONID') ); SPID ------------ 18979
  • Run SQL to generate trace
SQL> begin BIS_MV_REFRESH.refresh_wrapper ('FII_GL_BASE_MAP_MV', 'f', 0); end;
  • Go to RDBMS udump location and list *SPID* return from previous query.
nysesapd:/local/oracle/admin/ndev6/udump $ls -l *18979* -rw-rw-r-- 1 oracle dba 238565 Jul 18 14:56 ndev6_ora_18979.trc
  • Generate tkprof output as given below.
tkprof ndev6_ora_18979.trc ndev6_ora_18979.tkp sort=exeela,prsela,fchela mailto:explain=apps/appsndev6@ndev6

Tuesday, July 17, 2007

Forms Listener Servlet with Oracle Applications 11i

As part of DMZ configuration we were suppose to convert Forms to Forms Listener Servlet.

As part of configuration I performed following steps.

  1. Ensured Forms Patchset level is 16 or higher. We are at patchset level 18.

  2. Ensured Instance is Autoconfig enable.

  3. Ensured patch 3594604 has been applied. We have applied 11.5.10 Rel 2 maintenance patch 3480000 which is superseded patch of 3594604.

  4. Logged in to OAM. OAM Site Map -> AutoConfig-> Configuration Wizard -> Forms Listener Servlet and Choose the Enable or Disable button to enable or disable "Forms Listener Servlet".

  5. Run autoconfig.

$ ./adautocfg.sh Enter the APPS user password: AutoConfig is configuring the Applications environment...

Tuesday, June 26, 2007

Copy Putty along with saved server

How to copy window based putty from one server to another along with its saved server configuration?

Very recently I got new desktop with high configuration. I copied all data and installed new software also. Only problem was how to copy putty along with saved server configuration? I performed following to copy putty along with save session.

1. Executed given putt.bat file on source server at .

c:\temp:\>putt.bat

Conetnt of putt.bat is given below

ECHO OFF regedit /ea new.reg HKEY_CURRENT_USERSoftwareSimonTathamPuTTY

2. Copied putty.exe and new.reg (registry) file to new server.

3. Double click on new.reg and click on yes on dialog box

4. You can now start putty and connect to server.

Note--> You need to ensure network connectivity before trying to connect to server :-).

Sunday, June 24, 2007

Automatic Storage Management

Automatic Storage Management provides easy file management. DBA need not to manage huge physical file manual. ASM manage database physical files in disk. ASM uses Oracle-managed files and provides benefits of mirroring and striping. It divides files into 1 MB extents and extents get store in all disk available in disk group. So manual intervention to tune disk IO is not required. ASM provides the redundancy level on a per-file basis. In this way two file (original and mirror) can be stored on same disk group. ASM provides mirroring at extent level. ASM provides three mirroring types.

· 2-Way Mirroring I mirror copy of every extent

· 3-Way Mirroring 2 mirror copy of every extent

· Unprotected No mirroring. (Mirroring provided by the disk subsystem)

ASM provides features to manage storage without stopping database. When you add/drop disk in/from disk group ASM re balances file data across all the disks of disk group. ASM can be used with old database file system. ASM requires one Oracle instance with SGA(smaller SGA) and background process. ASM instance can be used for more that one database. ASM diskgroup can be shared among all databases running on that server.

In RAC, each instance will have an ASM instance, which communicate to other node ASM instance. ASM instance in RAC environment can manage one or more that one RAC database or one or more than one single instance database. ASM supports database data files, control files, log files, archive files. You need to use RMAN to backup ASM database.

ASM Components

· Groups

A disk group is group of disk managed together as one unit. Disk group may store various database's files. Single database may use storage from multiple disk groups. Disk group type specifies disk group mirroring. There are three type of redundancy

Normal Redundancy (2 way) (Default Redundancy)

High Redundancy (3 way)

External Redundancy (none or unprotected)

· Disk

ADM disks is disks in disk group, witch is a partition of a logical unit number (LUN) or a network attached file.

· Failure Group

Failure group is applicable only to normal and high redundancy disk groups. Failure group define common potential failure mechanism.

· Files

Files are written on ASM disk called ASM files, which are spaced across all the ASM disks in the disk group. ASM automatically generate file names. User-friendly file name also can be specified. A ASM file can be stored in only one disk group.

· Templates

Collections of file attributes, which are used to set mirroring/striping of database file are created in ASM disk group are called templates

ASM Two new Background Process

ASM instace should be started before starting datbase instance. When you start ASM instace it does not mount ASM database it manages meta data required to make ASM files are available to all ASM storage database instances. . ASM starts background process and allocate SGA (SGA is very less compated to regulare database SGA). All the database running on ASM storage communicate to ASM database to get the disk group infomation. ASM instance starts two more background process RBAL and ASMB, which are not availagle in regualr oracle database or RAC database. RBAL coordinate rebalances activity for disk groups whereas ASMB performs the actual rebalance activity for AU movements. Rest of ASM background process are same as Oracle instance. When ASM instance starts, ASMB connects as a foreground process into the ASM instance and RBAL performs global open of disk groups.

ASM and Recovery feautre

In RAC enviorment where multiple ASM instance mounts same disk groups. If one instance fails other instance automatically recovers ASM metadata changes caused by failed instance.

In single ASM instance where multiple ASM instance mounts different disk groups, if ASM instance failes during its uses, disk groups are not recoverered util they are monted again. When ASM mounts failed disk groups it reads logs and recovers all changes. Such senario is called ASM crash recovery. So in RAC invironment it is always recommened to mount same set of disk group on all ASM instance.

ASMLibs

ASMLibs is ASM library to support ASM feature. It provides efficeint machanism for accessing block devices used by the ASM disk groups. ASM library provides ASM I/O to Linux disk without the limitations of the startard UNIX I/O API.

Detail installation of ASMLibs and ASM i will cover in my other post. I would like to discuss here only basic aspect of ASMLib.

Following three package are required to install ASMLib which can be downloaded from http://www.oracle.com/technology/tech/linux/asmlib/index.html

  • oracleasm-support-2.0.3-2

  • oracleasm-2.6.9-42.0.0.0.1-Elsmp-2.0.3-2

  • oracleasm-2.6.9-42.0.0.0.1.EL-2.0.3-2

Exact name depends on version of Linux(OS), which I will discuss in my next blog.

After installing ASMLib package you can configure ASMLib by executing following

# /etc/init.d/oracleasm configure

You can create ASM disk as given below

# /etc/init.d/oracleasm createdisk VAOL1 /dev/sdc1

You can scan ASM on other node in RAC

#/etc/init.d/oracleasm scandisks

To list ASM disk

#/etc/init.d/oracleasm listdisks

Install ASMLib

Check kernal version and system architecutre

# uname -r

2.6.9-42.0.0.0.1.ELsmp

I downloaded following ASMLib package

  • oracleasm-support-2.0.3-2

  • oracleasm-2.6.9-42.0.0.0.1-Elsmp-2.0.3-2

  • oracleasm-2.6.9-42.0.0.0.1.EL-2.0.3-2

Install ASM package.

$rpm -Uvh oracleasm-support-2.0.3-2 \

oracleasm-2.6.9-42.0.0.0.1-Elsmp-2.0.3-2 \

oracleasm-2.6.9-42.0.0.0.1.EL-2.0.3-2

Configure ASM

# /etc/init.d/oracleasm configure

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle

Default group to own the driver interface []: dba

Start Oracle ASM library driver on boot (y/n) [n]: y

Fix permissions of Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: [ OK ]

Creating /dev/oracleasm mount point: [ OK ]

Loading module "oracleasm": [ OK ]

Mounting ASMlib driver filesystem: [ OK ]

Scanning system for ASM disks: [ OK ]

"oracleasm configure" creates "/etc/sysconfig/oracleasm" configuration file and "/dev/oracleasm" mount points. It loads oracleasm kernel module and mounts ASM library driver file system.

ASM Disk Creation

Lists disk name

# /sbin/fdisk -l

Create ASM disk

# /etc/init.d/oracleasm createdisk disk1 /dev/sdb1

Make available asm disk to other node.

# /etc/init.d/oracleasm scandisks

# /etc/init.d/oracleasm -help

Usage: /etc/init.d/oracleasm {startstoprestartenabledisableconfigurecreatediskdeletediskquerydisklistdisksscandiskstatus} start/stop/restart: To start/stop/restart ASM library driver without restarting the system. enable/disable: To enable/disable ASM library driver when system starts. configure: To configure the ASM library driver createdisk: To use disk with ASM library deletedisk: To unmark a name disk querydisk: To check if disk name is being used by ASM Library driver. listdisks: To list disk names of marked ASM library driver disksscandisks: To inform RAC node which shared disks are marked as ASM library disks on another node.

Once disk is prapared I have set ASM_DISKSTRING parameter as /dev/oracleasm/disks/*, default value is "ORCL:DISK*".

ASM Database Instance Creation

To create ASM based database DBCA checkes if ASM instance is running on current node, if yes, it lists disk groups. If ASM is not running DBCA creates a new ASM instance. ASM instance are very small, 64MB SGA is enough for largets ASM installations.

ASM Instance Initialization Parameters

INSTANCE_TYPE=ASM # Set to ASM for ASM instance DB_UNIQUE_NAME=+ASM # Uniqe DB Name, defaul value +ASM ASM_POWER_LIMIT = 1 # Controls rebalance operation speed, Range 1 to 11(fastest), default value is 1. ASM_DISKSTRING= '/dev/oracleasm/disks/*', , ,

# ASM instance discovers disk mounted using this parameter if specified value is NULL ASM_DISKGROUPS=ORCL_DATA1,FLASH_RECOVERY_ARE # List of disk groups which should be mounted when instance starts or when "ALTER DISKGROUP ALL MOUNT is used"LARGE_POOL_SIZE=8MB

ASM instance & RAC

Select DBCA "Oracle RAC" database option and then select all nodes. DBCA create one ASM instance on each node automatically.

CLUSTER_DATABASE=TURE # Set this parameter in each ASM RAC instace to start global cache services.

ASM_DISKGROUP= ' ' # Instances can have different value but it is recomended to mount same value. Shared disk must be mounted on all ASM instance.

ASM_DISKSTRING=' ' * Multiple instance may have different value

Startup/Shotdown ASM Instance

SQL> connect / as sysdba

SQL>startup

Mount option mounts disk groups specified by ASM_DISKGROUPS init parameter. OPEN is invalid and NOMOUNT starts up ASM without mounting any disk group.

You can not stop ASM instance till its all client database instance are stopped. In case of SHUTDOWN NORMAL ASM waits for all session to disconnect . ASM SHUTDOWN IMMEDIATE, TRANSACTIONA, or ABORT, ASM immediately terminates its database instance connections hence all database immediately abort. When IMMEDIATE or TRANSACTIONAL is uses, ASM waits for in-progress ASM SQL to complete before shutting down ASM instance.

SQL> connect / as sysdba

SQL> SHUTDOWN IMMEDIATE

Create Disk Group

SQL>CREATE DISKGROUP ORCL_DATA1 NORMAL REDUNDANCY DISK 'ORCL:VOL1';

Drop disk group

SQL> DROP DISKGROUP ORCL_DATA1 INCLUDING CONTENTS;

Add Disk

SQL> ALTER DISKGROUP ORCL_DATA1 ADD DISK

'ORCL:VOL2';

SQL> ALTER DISKGROP ORCL_DATA1 ADD DISK '/dev/oracleasm/disks/*';

Above SQL will add disks (/dev/oracleasm/disks/*), which were not not added. ASM instance ensure that disk is usable and then it formates and reblances. It moves AUs from every file onto the new disks.

Drop disks.

SQL> ALTER DISKGROUP ORCL_DATA1 DROP DISK '/dev/oracleasm/disks/VOL4';

Cancel Previous Drop, it works for only pending drops not after drop completion.

SQL> ALTER DISKGROUP ORCL_DATA1 UNDROP DISKS;

To rebalance disk. POWER level of "0" halts rebalancing until command is reinvoked.

SQL> ALTER DISKGROUP ORCL_DATA1 REBALANCE POWER 5;

Mount/Dismount DISK GROUP.

SQL> ALTER DISKGROUP ORCL_DATA1 DISMOUNT;

To very internal consistency of disk group metadata and repair if error found.

SQL> ALTER DISKGROUP ORCL_DATA1 CHECK ALL;

Some time it takes long time to rebalance AU across disk. Such operation can be monitored using V$ASM_OPERATION

ASM Files

ASM files, oracle managed files are stored in ASM disk groups. ASM oracle managed files are automatically deleted when not required. ASM files created using alias are not deleted automatically. Manual intervention is required to delete such files. Each file in a disk group is spread across all disks in the DISK group, so backup of single disk not not useful. RMAN must be use to backup ASM files.

ASM does not manage binaries, alert logs, trace files, password files, and Cluster Ready Services files.

ASM File Name are of following forms

  • Fully Qualified ASM file names are used to refer existing ASM files. It specify a disk group name, database name, a file type, a type-specific tag, a file number and an incarnation number. When file is created even using alias, a fully qualified name is also created. Fully qualified ASM file names are also called system aliases,

<group>/<dbname>/<file_type>/<tag>.<file#>.<incarnation#>

+ORCL_DATA1/orcl/controlfile/current.256.623848779

  • Numeric ASM File name are used for refererencing existing ASM files. It specifis a disk group name, a file number and an incarnation number.

+ORCL_DATA1.257.8675309

  • Alias ASM file name can be used for referncing existing ASM file and for creating new ASM files. Alias name specify a disk group name and inclu a user-fiendly string insteand of file and incarnation number. Alias file name uses hierarchical directory structure. Total length of Alias ASM file name is limited to 256 bytes. File name can have space in between but not at begining or at end. Alias ASM file names are case-insensitive.

+ORCL_DATA1/myfiles/control_file1

Using "ALTER DISKGROUP ADD ALIAS" an alias can be created for existing file.

  • Alias ASM file name with templates are used only while ASM file creation. It specifies a disk group name, an alias name, and a file creation template name. e.g. +ORCL_DATA/config1(spfile).

  • Incomplete ASM file names used during ASM file creation. It uses disk group name only. It uses a default template for incomplete ASM file name specified by their file type. +ORCL_DATA1

  • Incomplete ASM file names with templates are used for file creation. It uses disk group name followed by a template name. File creation attributes are determined by template name. e.g. +dgroupA(datafile).

Template and Alias

SQL>ALTER DISKGROUP ORCL_DATA1

ADD TEMPLATE DATA1TEMP ATTRIBUTES (MIRROR);

SQL> ALTER DISKGROUP ORCL_DATA1 DROP TEMPLATE DATA1TEMP;

SQL> ALTER DISKGROUP ORCL_DATA1 ADD DIRECTORY '+ORCL_DATA1/jai';

SQL> ALTER DISKGROUP ORCL_DATA1 RENAME DIRECTORY '+ORCL_DATA1/jai' TO '+ORCL_DATA1/jai';

SQL> ALTER DISKGROUP ORCL_DATA1 ADD ALIAS '+ORCL_DATA1/jai/users.dbf' FOR '+ORCL_DATA1/orcl/datafile/users.265.623849191';

SQL> ALTER DISKGROUP ORCL_DATA1 RENAME ALIAS '+ORCL_DATA1/jai/users.dbf' TO '+ORCL_DATA1/jai/users1.dbf' ;

SQL> ALTER DISKGROUP ORCL_DATA1 DROP ALIAS '+ORCL_DATA1/jai/users1.dbf' ;

SQL> ALTER DISKGROUP ORCL_DATA1 DROP DIRECTORY '+ORCL_DATA1/jai';

SQL> ALTER DISKGROUP ORCL_DATA1 DROP FILE '+ORCL_DATA1/jai/users1.dbf' ;

List Aliases Information

SQL> SELECT reference_index INTO :al_id FROM V$ASM_ALIAS WHERE NAME='+ORCL_DATA1';

SQL> SELECT reference_index INTO :al_id FROM V$ASM_ALIAS WHERE PARENT_INDEX=:al_id AND NAME='jai' ; REM jai is direcotry name

SQL> SELECT name FROM V$ASM_ALIAS WHERE parent_index=al_id;

ASM Data Dictionary View

V$ASM_DISKGROUP To check disk group information (Number, name, size, state, redundancy type). in DB instance one row for every disk group mounted by ASM instance.

V$ASM_DISK One row for every discovered disk in ASM instance. In DB instance, disk used by database.

V$ASM_FILE In ASM instance one row for every ASM file in all disk group mounted by ASM instance. DB instance contains no rows.

V$ASM_TEMPLATE In ASM and DB, one row for every templated present in disk group mounted by ASM instance.

V$ASM_ALIAS In ASM instance one row for every alias present in every disk mounted by ASM

V$ASM_OPERATION, To check long running ASM opertaion in ASM instance

V$ASM_CLIENT In ASM instance lists databases using disk group managed by the ASM instance. In DB instance list one row if database has any open ASM files.

Tuesday, June 12, 2007

Oracle 10g Real Application Cluster Concept



Very recently I installed two node Oracle 10g RAC Database using ASM on Oracle Enterprise Linux 4.0. I had to struggle for almost two weeks to get it done and finally I was successful. Before I jump to explain installation part of 10g RAC Database using ASM , I would like to discuss basic concept and understanding of Oracle Real Application Cluster. Whenever we think about RAC first questions comes to our mind is what is Cluster?

What is Cluster?

A Cluster is combination of more than one connected computer (or server). From applications prospective it seems to be single server processing applications requests. Multiple connected servers refer to physical files stored on shared area and are interconnected using physical network.

Oracle Real Application Cluster

Oracle Real Application Cluster is software that is installed on all interconnected server running on same operating system. It acts as if they were single server. Oracle clusterware has two components, a voting disk and OCR (Oracle Cluster Registry). A voting disk keeps record of all nodes’ membership whereas OCR keeps track of cluster configuration. These two component Voting disk and OCR must be on shared storage, which can be access by all interconnected cluster servers. As part of Oracle cluster ware each server (node) should be connected to private network as private interconnect. Oracle Real Application cluster provides feature to run multiple instance on various node on single database.

Advantage of Oracle Real Application Cluster

Oracle RAC is mainly for high availability. If one node goes down connection would get routed through other server. Service can be configured to run against multiple nodes so if one node goes down it would switch over to another node. You can add various nodes to increase processing power and you can remove node when you do not use node for processing any more. Services load management happens automatically across various nodes. RAC allows more concurrent batch process, great parallel execution with more degree of parallelism. It also allows more number of user connections.

Oracle Cluster Software Processes

There are several Oracle Cluster background processes for cluster related operations. Cluster background process are given below.

  • Cluster Synchronization Services (CSS)
This background process manages oracle cluster configuration file. It keeps record of all nodes which are cluster members and inform other members when a node (server) join cluster or leaves the cluster. If cluster is from third party, then CSS process interacts and interfaces with clusterware to keep track of node information and manage node.
  • Cluster Ready Services (CRS)

CRS manages high availability within a cluster. CRS manage cluster resources which are database, instance, service, listener, a virtual IP(VIP address), application process etc. CRS refers resource’s configuration information that is stored in the OCR (Oracle Cluster Registry) on the shared storage to manage start, stop, monitor, failover operations clusterware resources (database, instance, service, listener, VIP).

  • Event Management (EVM)
This process publishes events that crs creates. The evm process scans a designated directory and invokes all of the scripts in that directory when an even occurs.
  • Oracle Notification Services (ONS)
ONS publishes and subscribes services for communication Fast Application Notification (FAN) events.
  • RACG
Helps clussterware to support Oracle-Specific requirements and complex resources. Runs server callout scripts when FAN events occurs.
  • Process Monitor Daemon (OPROCD)
This process is locked in memory to monitor the cluster and provide I/O fencing. OPROCD performs its check, stops running. If the wake up is beyond the expected
time, then OPROCD resents the processor and reboots the node. An OPROCD failure results in the Oracle Clusterware restarting the node. OPROCD uses the hangcheck
timer on Linux platforms.
Oracle Real Application Cluster Physical Files and Cluster Processing

In Oracle RAC database all datafiles, controlfiles, PFILEs(SPFILE) and redo log files must reside on shared disk, which should be cluster compatible. So cluster node can access these physical file stored on cluster aware shared storage area.

A RAC database can have maximum 100 instances. Each RAC database has at least one additional thread of redo for each instance and undo talbespace for individual instance. For shared store following option can be used.
  • OCFS (Oracle Cluster File System)
  • A network file system
  • Raw devices
  • ASM à Oracle recommends to use ASM

RAC database has more than one instance. Each oracle RAC instance has its own memory and background process. RAC database has some additional memory structure and background processes than non RAC single instance database.

RAC instance uses bigger cache in SGA. It uses Cache Fusion so SGA size for RAC database is more than SGA size of single instance database.

To ensure data integrity for individual RAC instance, RAC use to background processes, GCS (Global Cache Service) and GES (Global Enqueue Service) . The Global Cache Service and Global Enqueue Service maintain records/status of each data file and each cached block using a GRD (Global Resource Directory).

Global Resource Directory contents are distributed across all the active instances, which increases SGA size of a RAC instance.

When one of RAC instance caches data, other same RAC database instance can obtain a block from this instance instead of reading the block from disk. Cache Fusion moves current blocks between instances rather re-reading the block from disk. If changed block is needed by another RAC instance, Cache Fusion transfers the block image directly between affected instances.

Real application uses private interconnect to communicate between instances and to transfer block. The GES monitor and the Instance Enqueue Process manages access to Cache Fusion resource and enqueue recovery.

There are additional background process for RAC instance which are used to maintain database coherency among each instance. They manage global resources:

  • LMSx – Global Cache Service Process, where x can range from 0 to j
  • LMD0 – Global Enqueue Service Daemon
  • LMON – Global Enqueue Service Monitor
  • LCK0 – Instance Enqueue Process
  • DIAG – Diagnosiblity Process


Monday, June 11, 2007

Exclude SQL Net Connection from particular node (or list of nodes).

As I mentioned in previous post, we had tough time recovering production database which happened due to some SQL script update against production instead of development.

To prevent any sql client connection from development boxes to production server we did following.

1. Add following entry in $TNS_ADMIN/sqlnet.ora file on production database server.

>pwd

/local/oracle/product/9.2.0.8/network/admin

nxtffind:/local/oracle/product/9.2.0.8/network/admin

>cat sqlnet.ora

TCP.EXCLUDED_NODES= (nxtfsapq)

TCP.VALIDNODE_CHECKING = YES

nysefind:/local/oracle/product/9.2.0.8/network/admin

2. Restarted listener

$export ORACLE_SID=<sid>

$export ORACLE_HOME=/local/oracle/product/9.2.0.8

$export $TNS_ADMIN=$ORACLE_HOME/network/admin

$export PATH=$ORACLE_HOME/bin:$PATH

$lsnrctl stop ndev2

$lsnrctl start ndev2

Noteà Listener reload will not work so you need to stop listener and start it again.

3. Tried to connect to production database from development server which were included in sqlnet.ora file on RDBMS server to exclude sql * net connection coming from these server SQL client.

$hostname

nxtfsapq

$cat $TNS_ADMIN/network/tnsnames.ora

ndev2 = (DESCRIPTION=

(ADDRESS=(PROTOCOL=tcp)(HOST=nxtffind)(PORT=1527))

(CONNECT_DATA=(SID=ndev2))

)

$ sqlplus apps@ndev2

SQL*Plus: Release 9.2.0.8.0 - Production on Mon Jun 11 22:34:13 2007

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

Enter password:

ERROR:

ORA-12537: TNS:connection closed

Reference:

Metalink Note

What is Validnode Verification?

Doc ID: Note:1006500.6

Oracle Ebusiness Suite 11.5.10 Point In Time Recovery

Friday morning was not good for us. I got call around 8 a.m. and one of my colleagues informed me that “Debug instance post clone script” was run against Production database. Lot of production database tables was updated and it was not possible to rollback changes except restoring backup.

We take HP BC (Business Copy) backup every evening by 7.30. Before starting back up we generally pause database and initiate backup.

Whenever we get HP BC, we generally shutdown database and restart before copying BC backup set for database and applications clone.

We run database and applications servers on HP-UX 11 and discoverer server on Linux. I am not mentioning actual server names due to security concern.

We decided to use HP BC backup (taken at 7.30 p.m.) as restore and apply archive log file from 7.30 p.m. to 11.59 p.m. Our database size is close to 200G and we wanted to avoid copy time of 200G of data over Ethernet. We also planned to change server hostname from <dev_server> to <prod_db_servername> and put DNS entry. We decided all these to avoid production down time.

We took backup of existing HP BC copy (as of 7.30 p.m. Thursday) and performed following to recover database using point in time recovery (till 11.59 p.m.).

We performed following.

1. Copied HP BC database and applications to tape

2. Change archive log location in

log_archive_dest = /orastage/oracle/data/<sid>/

3. Copied password file from production DB server to development server when we wanted to recover production database $ORACLE_HOME/dbs/orapw<sid> to $ORACLE_HOME/dbs/

4. Copied $TNS_ADMIN/listener.ora, $TNS_ADMIN/tnsnames.ora files from production server to development server

5. Started production listener with same name and port on development server

Login as oracle

Set oracle environment

$export ORACLE_SID=<product_DB>

$export ORACLE_HOME=/local/oracle/product/9.2.0.8

$export TNS_ADMIN=$ORACLE_HOME/network/admin

$export PATH=$ORACLE_HOME/bin:$PATH

$lsnrctl start <production_listener>

6. Started database in nomount mode and performed following to recover database

Set environment as I mentioned above.

$sqlplus /nolog

SQL> connect / as sysdba

SQL> startup nomount

SQL> alter database mount

SQL> RECOVER DATABASE UNTIL TIME '2007-06-07:23:59:59' USING BACKUP CONTROLFILE;

ORA-00279: change 25163467125 generated at 06/08/2007 00:00:00 needed for thread 1

ORA-00289: suggestion : /orastage/oracle/data/<sid>/arch/8093_1.arc

ORA-00280: change 25 163 467 125 for thread 1 is in sequence #8093

It applied all log file and after apply last log file with sequence #8093 it failed with following error.

SQL> RECOVER DATABASE UNTIL TIME '2007-06-07:23:59:59' USING BACKUP CONTROLFILE

ORA-00279: change 25163467125 generated at 06/08/2007 00:00:00 needed for thread 1

ORA-00289: suggestion : /orastage/oracle/data/<sid>/arch/8093_1.arc

ORA-00280: change 25 163 467 125 for thread 1 is in sequence #8093

Specify log: {<RET>=suggested filename AUTO CANCEL}

/orastage/oracle/data/<sid>/arch/8093_1.arc

ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below

ORA-01194: file 1 needs more recovery to be consistent

ORA-01110: data file 1: '/u03004.05/oracle/data/<sid>/system01.dbf'

Referred metalink note

Subject: DIAGNOSING ORA-1547 & ORA-1194 ERRORS DURING RECOVERY

Doc ID: Note:435201.1

Most of database data files were in fuzzy mode.

SQL> select file#,STATUS, FUZZY from v$datafile_header;

7. We concluded that backup was not good so we planned to restore Last week Sunday backup and apply all archive files.

8. We requested unix team to restore last Sunday’s cold backup.

9. Recover database from Sunday Cold backup set by apply all archive files (150)

Login as oracle and set environment Changed archive_log_destination parameter in $ORACLE_HOME/dbs/init<sid>.ora file to reflect new archive log file location.

log_archive_dest =/ORASTAGE/oracle/data/<sid>/arch

$sqlplus /nolog

SQL> connect / as sysdba

SQL> startup nomount

SQL> show parameter archive

SQL> show parameter archive

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

archive_lag_target integer 0

log_archive_dest string /ORASTAGE/oracle/data/<sid>/arch

SQL> RECOVER DATABASE UNTIL TIME '2007-06-07:23:59:59' USING BACKUP CONTROLFILE;

AUTO

Content from $bdump/alert<sid>.log file.

Sat Jun 9 11:48:15 2007

Database mounted in Exclusive Mode.

Completed: ALTER DATABASE MOUNT

Sat Jun 9 11:49:19 2007

ALTER DATABASE RECOVER DATABASE UNTIL TIME '2007-06-07:23:59:59' USING BACKUP CONTROLFILE

Sat Jun 9 11:49:19 2007

Media Recovery Start

ORA-279 signalled during: ALTER DATABASE RECOVER DATABASE UNTIL TIME '2007-...

Sat Jun 9 11:49:48 2007

ALTER DATABASE RECOVER CONTINUE DEFAULT

Sat Jun 9 11:49:48 2007

Media Recovery Log /ORASTAGE/oracle/data/<sid>/arch/7943_1.arc

.

.

.

ALTER DATABASE RECOVER CONTINUE DEFAULT

Sat Jun 9 20:30:28 2007

Media Recovery Log /ORASTAGE/oracle/data/<sid>/arch/8093_1.arc

.

.

.

Sat Jun 9 20:34:17 2007

Terminal Recovery: done UNTIL CHANGE 25163467125

Media Recovery Complete

Completed: ALTER DATABASE RECOVER CONTINUE DEFAULT

Sat Jun 9 20:34:21 2007

ARC1: Cannot archive online log based on backup controlfile

Sat Jun 9 20:35:21 2007

ARC1: Cannot archive online log based on backup controlfile

Sat Jun 9 20:36:14 2007

SQL> ALTER DATABASE OPEN RESETLOGS;

ALTER DATABASE OPEN RESETLOGS

Sat Jun 9 20:36:14 2007

RESETLOGS after incomplete recovery UNTIL CHANGE 25163467125

Resetting resetlogs activation ID 1687126238 (0x648f80de)

Sat Jun 9 20:39:02 2007

.

.

Sat Jun 9 20:44:25 2007

replication_dependency_tracking turned off (no async multimaster replication found)

Completed: ALTER DATABASE OPEN RESETLOGS

It took almost 9 hour to apply 150 archive file (size 300M).

10. Started database listener and did basic checking to validate database restore and recovery.

11. Logged in as applmgr and tried to start oracle application services as given below

$ . $APPL_TOP/APPSORA.env

$ cd $COMMON_TOP/admin/scripts/<sid>_<hostname>/

$adstrtal.sh apps/<password>

It started all services.

I tried to log in to applications using given URL.

http://<hostname>.<domainname>:<PORT>/

Click on “Apps Logon Links”

Click on “E-Business Home Page”

Logged as “SYSADMIN” user

Login failed with given error.

"YOU HAVE ENCOUNTERED AN UNEXPECTED ERROR"

As part of troubleshooting I did following.

1. Gave wrong password and it failed with authentication error.

http://<hostname>.<domainname>:<PORT>/OA_HTML/jsp/fnd/aoljtest.jsp

All AOLJ test was passed except jsp pages. We doubt problem could be with jsp pages

2. Turned on debug for Apache and JSP as given below

a. Stopped apache services

$apps/OEBS/ndev2/commontop/admin/scripts/<sid>_<hostname>/adapcctl.sh stop

b. Deleted $IAS_ORACLE_HOME/Apache/Apache/logs and $IAS_ORACLE_HOME/Apache/Jserv/log

c. Edited $IAS_ORACLE_HOME/Apache/Apache/conf/httpd.conf

LogLevel debug

d. Edited $IAS_ORACLE_HOME/Apache/Jserv/conf/jserv.conf

ApJServLogLevel debug

e. Edited $IAS_ORACLE_HOME/Apache/Jserv/conf/jserv.properties

log=true

Notice following error

$IAS_ORACLE_HOME/Apache/Jserv/logs/ mod_jserv.log

[10/06/2007 12:59:36:265] (ERROR) ajp12: Servlet Error: OracleJSP: java.io.FileNotFoundException

$IAS_ORACLE_HOME/Apache/Jserv/logs/jserv.log

[10/06/2007 12:26:43:744 EDT] oracle.jsp.JspServlet: init

[Jun 10, 2007 12:27:04 PM EDT]:1181492824137:Thread[Thread-70,10,main]:-1:-1:hostname.domain name:<ipaddress>:8102:16020:UNEXPECTED:[fnd.common.logging.DebugEventManager.handlerException]

:java.lang.Exception: /aferror.log (Permission denied (errno:13))

at oracle.apps.fnd.common.logging.FileHandler.<init>(FileHandler.java:103)

at oracle.apps.fnd.common.logging.FileHandler.<init>(FileHandler.java:80)

at oracle.apps.fnd.common.logging.DebugEventManager.registerHandlers(DebugEventManager.java:

1241)

at oracle.apps.fnd.common.logging.DebugEventManager.init(DebugEventManager.java:904)

at oracle.apps.fnd.common.logging.DebugEventManager.<init>(DebugEventManager.java:194)

at oracle.apps.fnd.common.AppsLog.<init>(AppsLog.java:519)

at oracle.apps.fnd.common.AppsLog.<init>(AppsLog.java:489)

at oracle.apps.fnd.common.AppsLog.<clinit>(AppsLog.java:469)

at oracle.apps.fnd.common.AppsContext.makeLog(AppsContext.java:1153)

at oracle.apps.fnd.common.Context.setLog(Context.java:1400)

at oracle.apps.fnd.common.Context.setLog(Context.java:1381)

at oracle.apps.fnd.common.Context.getLog(Context.java:1353)

at oracle.apps.fnd.common.Context.isLoggingEnabled(Context.java:2119)

at oracle.apps.fnd.common.Context.makeConnectionManager(Context.java:655)

at oracle.apps.fnd.common.AppsContext.makeConnectionManager(AppsContext.java:1186)

at oracle.apps.fnd.common.Context.setConnectionManager(Context.java:615)

at oracle.apps.fnd.common.Context.setConnectionManager(Context.java:599)

at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:880)

at oracle.apps.fnd.common.AppsContext.<init>(AppsContext.java:742)

at oracle.apps.mwa.container.ApplicationsObjectLibrary.AOLInit(ApplicationsObjectLibrary.jav

a:174)

at oracle.apps.mwa.container.ApplicationsObjectLibrary.<init>(ApplicationsObjectLibrary.java

:78)

at oracle.apps.mwa.container.MWALib.setObjectLibrary(MWALib.java:339)

at oracle.apps.mwa.wap.engine.WapServlet.init(WapServlet.java:158)

at org.apache.jserv.JServServletManager.load_init(JServServletManager.java:755)

at org.apache.jserv.JServServletManager.loadServlet(JServServletManager.java:659)

at org.apache.jserv.JServServletManager.loadStartupServlets(JServServletManager.java:789)

at org.apache.jserv.JServServletManager.init(JServServletManager.java:447)

at org.apache.jserv.JServ.start(JServ.java:625)

at org.apache.jserv.JServ.main(JServ.java:234)

12. I got a hit and executed give below as advised by this metalink note.

Run the following to synchronize WF tables.

Begin

apps.wf_local_synch.BULKSYNCHRONIZATION (P_ORIG_SYSTEM=>'ALL',

P_PARALLEL_PROCESSES=>2,

P_LOGGING=>'LOGGING',

P_RAISEERRORS=>TRUE, P_TEMPTABLESPACE=>'APPS_TS_TX_DATA');

END;

/

Reference :

Unable To Login After Clone

Doc ID: Note:418130.1

It was verified by functional team and then we good database and applications backup before releasing production server for regular use.

Noteà Due to security Concern I have not mentioned actual server name, domain name, ipaddress and ORACLE SID.