Tuesday, June 26, 2007

Copy Putty along with saved server

How to copy window based putty from one server to another along with its saved server configuration?

Very recently I got new desktop with high configuration. I copied all data and installed new software also. Only problem was how to copy putty along with saved server configuration? I performed following to copy putty along with save session.

1. Executed given putt.bat file on source server at .

c:\temp:\>putt.bat

Conetnt of putt.bat is given below

ECHO OFF regedit /ea new.reg HKEY_CURRENT_USERSoftwareSimonTathamPuTTY

2. Copied putty.exe and new.reg (registry) file to new server.

3. Double click on new.reg and click on yes on dialog box

4. You can now start putty and connect to server.

Note--> You need to ensure network connectivity before trying to connect to server :-).

Sunday, June 24, 2007

Automatic Storage Management

Automatic Storage Management provides easy file management. DBA need not to manage huge physical file manual. ASM manage database physical files in disk. ASM uses Oracle-managed files and provides benefits of mirroring and striping. It divides files into 1 MB extents and extents get store in all disk available in disk group. So manual intervention to tune disk IO is not required. ASM provides the redundancy level on a per-file basis. In this way two file (original and mirror) can be stored on same disk group. ASM provides mirroring at extent level. ASM provides three mirroring types.

· 2-Way Mirroring I mirror copy of every extent

· 3-Way Mirroring 2 mirror copy of every extent

· Unprotected No mirroring. (Mirroring provided by the disk subsystem)

ASM provides features to manage storage without stopping database. When you add/drop disk in/from disk group ASM re balances file data across all the disks of disk group. ASM can be used with old database file system. ASM requires one Oracle instance with SGA(smaller SGA) and background process. ASM instance can be used for more that one database. ASM diskgroup can be shared among all databases running on that server.

In RAC, each instance will have an ASM instance, which communicate to other node ASM instance. ASM instance in RAC environment can manage one or more that one RAC database or one or more than one single instance database. ASM supports database data files, control files, log files, archive files. You need to use RMAN to backup ASM database.

ASM Components

· Groups

A disk group is group of disk managed together as one unit. Disk group may store various database's files. Single database may use storage from multiple disk groups. Disk group type specifies disk group mirroring. There are three type of redundancy

Normal Redundancy (2 way) (Default Redundancy)

High Redundancy (3 way)

External Redundancy (none or unprotected)

· Disk

ADM disks is disks in disk group, witch is a partition of a logical unit number (LUN) or a network attached file.

· Failure Group

Failure group is applicable only to normal and high redundancy disk groups. Failure group define common potential failure mechanism.

· Files

Files are written on ASM disk called ASM files, which are spaced across all the ASM disks in the disk group. ASM automatically generate file names. User-friendly file name also can be specified. A ASM file can be stored in only one disk group.

· Templates

Collections of file attributes, which are used to set mirroring/striping of database file are created in ASM disk group are called templates

ASM Two new Background Process

ASM instace should be started before starting datbase instance. When you start ASM instace it does not mount ASM database it manages meta data required to make ASM files are available to all ASM storage database instances. . ASM starts background process and allocate SGA (SGA is very less compated to regulare database SGA). All the database running on ASM storage communicate to ASM database to get the disk group infomation. ASM instance starts two more background process RBAL and ASMB, which are not availagle in regualr oracle database or RAC database. RBAL coordinate rebalances activity for disk groups whereas ASMB performs the actual rebalance activity for AU movements. Rest of ASM background process are same as Oracle instance. When ASM instance starts, ASMB connects as a foreground process into the ASM instance and RBAL performs global open of disk groups.

ASM and Recovery feautre

In RAC enviorment where multiple ASM instance mounts same disk groups. If one instance fails other instance automatically recovers ASM metadata changes caused by failed instance.

In single ASM instance where multiple ASM instance mounts different disk groups, if ASM instance failes during its uses, disk groups are not recoverered util they are monted again. When ASM mounts failed disk groups it reads logs and recovers all changes. Such senario is called ASM crash recovery. So in RAC invironment it is always recommened to mount same set of disk group on all ASM instance.

ASMLibs

ASMLibs is ASM library to support ASM feature. It provides efficeint machanism for accessing block devices used by the ASM disk groups. ASM library provides ASM I/O to Linux disk without the limitations of the startard UNIX I/O API.

Detail installation of ASMLibs and ASM i will cover in my other post. I would like to discuss here only basic aspect of ASMLib.

Following three package are required to install ASMLib which can be downloaded from http://www.oracle.com/technology/tech/linux/asmlib/index.html

  • oracleasm-support-2.0.3-2

  • oracleasm-2.6.9-42.0.0.0.1-Elsmp-2.0.3-2

  • oracleasm-2.6.9-42.0.0.0.1.EL-2.0.3-2

Exact name depends on version of Linux(OS), which I will discuss in my next blog.

After installing ASMLib package you can configure ASMLib by executing following

# /etc/init.d/oracleasm configure

You can create ASM disk as given below

# /etc/init.d/oracleasm createdisk VAOL1 /dev/sdc1

You can scan ASM on other node in RAC

#/etc/init.d/oracleasm scandisks

To list ASM disk

#/etc/init.d/oracleasm listdisks

Install ASMLib

Check kernal version and system architecutre

# uname -r

2.6.9-42.0.0.0.1.ELsmp

I downloaded following ASMLib package

  • oracleasm-support-2.0.3-2

  • oracleasm-2.6.9-42.0.0.0.1-Elsmp-2.0.3-2

  • oracleasm-2.6.9-42.0.0.0.1.EL-2.0.3-2

Install ASM package.

$rpm -Uvh oracleasm-support-2.0.3-2 \

oracleasm-2.6.9-42.0.0.0.1-Elsmp-2.0.3-2 \

oracleasm-2.6.9-42.0.0.0.1.EL-2.0.3-2

Configure ASM

# /etc/init.d/oracleasm configure

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library driver. The following questions will determine whether the driver is loaded on boot and what permissions it will have. The current values will be shown in brackets ('[]'). Hitting <ENTER> without typing an answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle

Default group to own the driver interface []: dba

Start Oracle ASM library driver on boot (y/n) [n]: y

Fix permissions of Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: [ OK ]

Creating /dev/oracleasm mount point: [ OK ]

Loading module "oracleasm": [ OK ]

Mounting ASMlib driver filesystem: [ OK ]

Scanning system for ASM disks: [ OK ]

"oracleasm configure" creates "/etc/sysconfig/oracleasm" configuration file and "/dev/oracleasm" mount points. It loads oracleasm kernel module and mounts ASM library driver file system.

ASM Disk Creation

Lists disk name

# /sbin/fdisk -l

Create ASM disk

# /etc/init.d/oracleasm createdisk disk1 /dev/sdb1

Make available asm disk to other node.

# /etc/init.d/oracleasm scandisks

# /etc/init.d/oracleasm -help

Usage: /etc/init.d/oracleasm {startstoprestartenabledisableconfigurecreatediskdeletediskquerydisklistdisksscandiskstatus} start/stop/restart: To start/stop/restart ASM library driver without restarting the system. enable/disable: To enable/disable ASM library driver when system starts. configure: To configure the ASM library driver createdisk: To use disk with ASM library deletedisk: To unmark a name disk querydisk: To check if disk name is being used by ASM Library driver. listdisks: To list disk names of marked ASM library driver disksscandisks: To inform RAC node which shared disks are marked as ASM library disks on another node.

Once disk is prapared I have set ASM_DISKSTRING parameter as /dev/oracleasm/disks/*, default value is "ORCL:DISK*".

ASM Database Instance Creation

To create ASM based database DBCA checkes if ASM instance is running on current node, if yes, it lists disk groups. If ASM is not running DBCA creates a new ASM instance. ASM instance are very small, 64MB SGA is enough for largets ASM installations.

ASM Instance Initialization Parameters

INSTANCE_TYPE=ASM # Set to ASM for ASM instance DB_UNIQUE_NAME=+ASM # Uniqe DB Name, defaul value +ASM ASM_POWER_LIMIT = 1 # Controls rebalance operation speed, Range 1 to 11(fastest), default value is 1. ASM_DISKSTRING= '/dev/oracleasm/disks/*', , ,

# ASM instance discovers disk mounted using this parameter if specified value is NULL ASM_DISKGROUPS=ORCL_DATA1,FLASH_RECOVERY_ARE # List of disk groups which should be mounted when instance starts or when "ALTER DISKGROUP ALL MOUNT is used"LARGE_POOL_SIZE=8MB

ASM instance & RAC

Select DBCA "Oracle RAC" database option and then select all nodes. DBCA create one ASM instance on each node automatically.

CLUSTER_DATABASE=TURE # Set this parameter in each ASM RAC instace to start global cache services.

ASM_DISKGROUP= ' ' # Instances can have different value but it is recomended to mount same value. Shared disk must be mounted on all ASM instance.

ASM_DISKSTRING=' ' * Multiple instance may have different value

Startup/Shotdown ASM Instance

SQL> connect / as sysdba

SQL>startup

Mount option mounts disk groups specified by ASM_DISKGROUPS init parameter. OPEN is invalid and NOMOUNT starts up ASM without mounting any disk group.

You can not stop ASM instance till its all client database instance are stopped. In case of SHUTDOWN NORMAL ASM waits for all session to disconnect . ASM SHUTDOWN IMMEDIATE, TRANSACTIONA, or ABORT, ASM immediately terminates its database instance connections hence all database immediately abort. When IMMEDIATE or TRANSACTIONAL is uses, ASM waits for in-progress ASM SQL to complete before shutting down ASM instance.

SQL> connect / as sysdba

SQL> SHUTDOWN IMMEDIATE

Create Disk Group

SQL>CREATE DISKGROUP ORCL_DATA1 NORMAL REDUNDANCY DISK 'ORCL:VOL1';

Drop disk group

SQL> DROP DISKGROUP ORCL_DATA1 INCLUDING CONTENTS;

Add Disk

SQL> ALTER DISKGROUP ORCL_DATA1 ADD DISK

'ORCL:VOL2';

SQL> ALTER DISKGROP ORCL_DATA1 ADD DISK '/dev/oracleasm/disks/*';

Above SQL will add disks (/dev/oracleasm/disks/*), which were not not added. ASM instance ensure that disk is usable and then it formates and reblances. It moves AUs from every file onto the new disks.

Drop disks.

SQL> ALTER DISKGROUP ORCL_DATA1 DROP DISK '/dev/oracleasm/disks/VOL4';

Cancel Previous Drop, it works for only pending drops not after drop completion.

SQL> ALTER DISKGROUP ORCL_DATA1 UNDROP DISKS;

To rebalance disk. POWER level of "0" halts rebalancing until command is reinvoked.

SQL> ALTER DISKGROUP ORCL_DATA1 REBALANCE POWER 5;

Mount/Dismount DISK GROUP.

SQL> ALTER DISKGROUP ORCL_DATA1 DISMOUNT;

To very internal consistency of disk group metadata and repair if error found.

SQL> ALTER DISKGROUP ORCL_DATA1 CHECK ALL;

Some time it takes long time to rebalance AU across disk. Such operation can be monitored using V$ASM_OPERATION

ASM Files

ASM files, oracle managed files are stored in ASM disk groups. ASM oracle managed files are automatically deleted when not required. ASM files created using alias are not deleted automatically. Manual intervention is required to delete such files. Each file in a disk group is spread across all disks in the DISK group, so backup of single disk not not useful. RMAN must be use to backup ASM files.

ASM does not manage binaries, alert logs, trace files, password files, and Cluster Ready Services files.

ASM File Name are of following forms

  • Fully Qualified ASM file names are used to refer existing ASM files. It specify a disk group name, database name, a file type, a type-specific tag, a file number and an incarnation number. When file is created even using alias, a fully qualified name is also created. Fully qualified ASM file names are also called system aliases,

<group>/<dbname>/<file_type>/<tag>.<file#>.<incarnation#>

+ORCL_DATA1/orcl/controlfile/current.256.623848779

  • Numeric ASM File name are used for refererencing existing ASM files. It specifis a disk group name, a file number and an incarnation number.

+ORCL_DATA1.257.8675309

  • Alias ASM file name can be used for referncing existing ASM file and for creating new ASM files. Alias name specify a disk group name and inclu a user-fiendly string insteand of file and incarnation number. Alias file name uses hierarchical directory structure. Total length of Alias ASM file name is limited to 256 bytes. File name can have space in between but not at begining or at end. Alias ASM file names are case-insensitive.

+ORCL_DATA1/myfiles/control_file1

Using "ALTER DISKGROUP ADD ALIAS" an alias can be created for existing file.

  • Alias ASM file name with templates are used only while ASM file creation. It specifies a disk group name, an alias name, and a file creation template name. e.g. +ORCL_DATA/config1(spfile).

  • Incomplete ASM file names used during ASM file creation. It uses disk group name only. It uses a default template for incomplete ASM file name specified by their file type. +ORCL_DATA1

  • Incomplete ASM file names with templates are used for file creation. It uses disk group name followed by a template name. File creation attributes are determined by template name. e.g. +dgroupA(datafile).

Template and Alias

SQL>ALTER DISKGROUP ORCL_DATA1

ADD TEMPLATE DATA1TEMP ATTRIBUTES (MIRROR);

SQL> ALTER DISKGROUP ORCL_DATA1 DROP TEMPLATE DATA1TEMP;

SQL> ALTER DISKGROUP ORCL_DATA1 ADD DIRECTORY '+ORCL_DATA1/jai';

SQL> ALTER DISKGROUP ORCL_DATA1 RENAME DIRECTORY '+ORCL_DATA1/jai' TO '+ORCL_DATA1/jai';

SQL> ALTER DISKGROUP ORCL_DATA1 ADD ALIAS '+ORCL_DATA1/jai/users.dbf' FOR '+ORCL_DATA1/orcl/datafile/users.265.623849191';

SQL> ALTER DISKGROUP ORCL_DATA1 RENAME ALIAS '+ORCL_DATA1/jai/users.dbf' TO '+ORCL_DATA1/jai/users1.dbf' ;

SQL> ALTER DISKGROUP ORCL_DATA1 DROP ALIAS '+ORCL_DATA1/jai/users1.dbf' ;

SQL> ALTER DISKGROUP ORCL_DATA1 DROP DIRECTORY '+ORCL_DATA1/jai';

SQL> ALTER DISKGROUP ORCL_DATA1 DROP FILE '+ORCL_DATA1/jai/users1.dbf' ;

List Aliases Information

SQL> SELECT reference_index INTO :al_id FROM V$ASM_ALIAS WHERE NAME='+ORCL_DATA1';

SQL> SELECT reference_index INTO :al_id FROM V$ASM_ALIAS WHERE PARENT_INDEX=:al_id AND NAME='jai' ; REM jai is direcotry name

SQL> SELECT name FROM V$ASM_ALIAS WHERE parent_index=al_id;

ASM Data Dictionary View

V$ASM_DISKGROUP To check disk group information (Number, name, size, state, redundancy type). in DB instance one row for every disk group mounted by ASM instance.

V$ASM_DISK One row for every discovered disk in ASM instance. In DB instance, disk used by database.

V$ASM_FILE In ASM instance one row for every ASM file in all disk group mounted by ASM instance. DB instance contains no rows.

V$ASM_TEMPLATE In ASM and DB, one row for every templated present in disk group mounted by ASM instance.

V$ASM_ALIAS In ASM instance one row for every alias present in every disk mounted by ASM

V$ASM_OPERATION, To check long running ASM opertaion in ASM instance

V$ASM_CLIENT In ASM instance lists databases using disk group managed by the ASM instance. In DB instance list one row if database has any open ASM files.

Tuesday, June 12, 2007

Oracle 10g Real Application Cluster Concept



Very recently I installed two node Oracle 10g RAC Database using ASM on Oracle Enterprise Linux 4.0. I had to struggle for almost two weeks to get it done and finally I was successful. Before I jump to explain installation part of 10g RAC Database using ASM , I would like to discuss basic concept and understanding of Oracle Real Application Cluster. Whenever we think about RAC first questions comes to our mind is what is Cluster?

What is Cluster?

A Cluster is combination of more than one connected computer (or server). From applications prospective it seems to be single server processing applications requests. Multiple connected servers refer to physical files stored on shared area and are interconnected using physical network.

Oracle Real Application Cluster

Oracle Real Application Cluster is software that is installed on all interconnected server running on same operating system. It acts as if they were single server. Oracle clusterware has two components, a voting disk and OCR (Oracle Cluster Registry). A voting disk keeps record of all nodes’ membership whereas OCR keeps track of cluster configuration. These two component Voting disk and OCR must be on shared storage, which can be access by all interconnected cluster servers. As part of Oracle cluster ware each server (node) should be connected to private network as private interconnect. Oracle Real Application cluster provides feature to run multiple instance on various node on single database.

Advantage of Oracle Real Application Cluster

Oracle RAC is mainly for high availability. If one node goes down connection would get routed through other server. Service can be configured to run against multiple nodes so if one node goes down it would switch over to another node. You can add various nodes to increase processing power and you can remove node when you do not use node for processing any more. Services load management happens automatically across various nodes. RAC allows more concurrent batch process, great parallel execution with more degree of parallelism. It also allows more number of user connections.

Oracle Cluster Software Processes

There are several Oracle Cluster background processes for cluster related operations. Cluster background process are given below.

  • Cluster Synchronization Services (CSS)
This background process manages oracle cluster configuration file. It keeps record of all nodes which are cluster members and inform other members when a node (server) join cluster or leaves the cluster. If cluster is from third party, then CSS process interacts and interfaces with clusterware to keep track of node information and manage node.
  • Cluster Ready Services (CRS)

CRS manages high availability within a cluster. CRS manage cluster resources which are database, instance, service, listener, a virtual IP(VIP address), application process etc. CRS refers resource’s configuration information that is stored in the OCR (Oracle Cluster Registry) on the shared storage to manage start, stop, monitor, failover operations clusterware resources (database, instance, service, listener, VIP).

  • Event Management (EVM)
This process publishes events that crs creates. The evm process scans a designated directory and invokes all of the scripts in that directory when an even occurs.
  • Oracle Notification Services (ONS)
ONS publishes and subscribes services for communication Fast Application Notification (FAN) events.
  • RACG
Helps clussterware to support Oracle-Specific requirements and complex resources. Runs server callout scripts when FAN events occurs.
  • Process Monitor Daemon (OPROCD)
This process is locked in memory to monitor the cluster and provide I/O fencing. OPROCD performs its check, stops running. If the wake up is beyond the expected
time, then OPROCD resents the processor and reboots the node. An OPROCD failure results in the Oracle Clusterware restarting the node. OPROCD uses the hangcheck
timer on Linux platforms.
Oracle Real Application Cluster Physical Files and Cluster Processing

In Oracle RAC database all datafiles, controlfiles, PFILEs(SPFILE) and redo log files must reside on shared disk, which should be cluster compatible. So cluster node can access these physical file stored on cluster aware shared storage area.

A RAC database can have maximum 100 instances. Each RAC database has at least one additional thread of redo for each instance and undo talbespace for individual instance. For shared store following option can be used.
  • OCFS (Oracle Cluster File System)
  • A network file system
  • Raw devices
  • ASM à Oracle recommends to use ASM

RAC database has more than one instance. Each oracle RAC instance has its own memory and background process. RAC database has some additional memory structure and background processes than non RAC single instance database.

RAC instance uses bigger cache in SGA. It uses Cache Fusion so SGA size for RAC database is more than SGA size of single instance database.

To ensure data integrity for individual RAC instance, RAC use to background processes, GCS (Global Cache Service) and GES (Global Enqueue Service) . The Global Cache Service and Global Enqueue Service maintain records/status of each data file and each cached block using a GRD (Global Resource Directory).

Global Resource Directory contents are distributed across all the active instances, which increases SGA size of a RAC instance.

When one of RAC instance caches data, other same RAC database instance can obtain a block from this instance instead of reading the block from disk. Cache Fusion moves current blocks between instances rather re-reading the block from disk. If changed block is needed by another RAC instance, Cache Fusion transfers the block image directly between affected instances.

Real application uses private interconnect to communicate between instances and to transfer block. The GES monitor and the Instance Enqueue Process manages access to Cache Fusion resource and enqueue recovery.

There are additional background process for RAC instance which are used to maintain database coherency among each instance. They manage global resources:

  • LMSx – Global Cache Service Process, where x can range from 0 to j
  • LMD0 – Global Enqueue Service Daemon
  • LMON – Global Enqueue Service Monitor
  • LCK0 – Instance Enqueue Process
  • DIAG – Diagnosiblity Process


Monday, June 11, 2007

Exclude SQL Net Connection from particular node (or list of nodes).

As I mentioned in previous post, we had tough time recovering production database which happened due to some SQL script update against production instead of development.

To prevent any sql client connection from development boxes to production server we did following.

1. Add following entry in $TNS_ADMIN/sqlnet.ora file on production database server.

>pwd

/local/oracle/product/9.2.0.8/network/admin

nxtffind:/local/oracle/product/9.2.0.8/network/admin

>cat sqlnet.ora

TCP.EXCLUDED_NODES= (nxtfsapq)

TCP.VALIDNODE_CHECKING = YES

nysefind:/local/oracle/product/9.2.0.8/network/admin

2. Restarted listener

$export ORACLE_SID=<sid>

$export ORACLE_HOME=/local/oracle/product/9.2.0.8

$export $TNS_ADMIN=$ORACLE_HOME/network/admin

$export PATH=$ORACLE_HOME/bin:$PATH

$lsnrctl stop ndev2

$lsnrctl start ndev2

Noteà Listener reload will not work so you need to stop listener and start it again.

3. Tried to connect to production database from development server which were included in sqlnet.ora file on RDBMS server to exclude sql * net connection coming from these server SQL client.

$hostname

nxtfsapq

$cat $TNS_ADMIN/network/tnsnames.ora

ndev2 = (DESCRIPTION=

(ADDRESS=(PROTOCOL=tcp)(HOST=nxtffind)(PORT=1527))

(CONNECT_DATA=(SID=ndev2))

)

$ sqlplus apps@ndev2

SQL*Plus: Release 9.2.0.8.0 - Production on Mon Jun 11 22:34:13 2007

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

Enter password:

ERROR:

ORA-12537: TNS:connection closed

Reference:

Metalink Note

What is Validnode Verification?

Doc ID: Note:1006500.6

Oracle Ebusiness Suite 11.5.10 Point In Time Recovery

Friday morning was not good for us. I got call around 8 a.m. and one of my colleagues informed me that “Debug instance post clone script” was run against Production database. Lot of production database tables was updated and it was not possible to rollback changes except restoring backup.

We take HP BC (Business Copy) backup every evening by 7.30. Before starting back up we generally pause database and initiate backup.

Whenever we get HP BC, we generally shutdown database and restart before copying BC backup set for database and applications clone.

We run database and applications servers on HP-UX 11 and discoverer server on Linux. I am not mentioning actual server names due to security concern.

We decided to use HP BC backup (taken at 7.30 p.m.) as restore and apply archive log file from 7.30 p.m. to 11.59 p.m. Our database size is close to 200G and we wanted to avoid copy time of 200G of data over Ethernet. We also planned to change server hostname from <dev_server> to <prod_db_servername> and put DNS entry. We decided all these to avoid production down time.

We took backup of existing HP BC copy (as of 7.30 p.m. Thursday) and performed following to recover database using point in time recovery (till 11.59 p.m.).

We performed following.

1. Copied HP BC database and applications to tape

2. Change archive log location in

log_archive_dest = /orastage/oracle/data/<sid>/

3. Copied password file from production DB server to development server when we wanted to recover production database $ORACLE_HOME/dbs/orapw<sid> to $ORACLE_HOME/dbs/

4. Copied $TNS_ADMIN/listener.ora, $TNS_ADMIN/tnsnames.ora files from production server to development server

5. Started production listener with same name and port on development server

Login as oracle

Set oracle environment

$export ORACLE_SID=<product_DB>

$export ORACLE_HOME=/local/oracle/product/9.2.0.8

$export TNS_ADMIN=$ORACLE_HOME/network/admin

$export PATH=$ORACLE_HOME/bin:$PATH

$lsnrctl start <production_listener>

6. Started database in nomount mode and performed following to recover database

Set environment as I mentioned above.

$sqlplus /nolog

SQL> connect / as sysdba

SQL> startup nomount

SQL> alter database mount

SQL> RECOVER DATABASE UNTIL TIME '2007-06-07:23:59:59' USING BACKUP CONTROLFILE;

ORA-00279: change 25163467125 generated at 06/08/2007 00:00:00 needed for thread 1

ORA-00289: suggestion : /orastage/oracle/data/<sid>/arch/8093_1.arc

ORA-00280: change 25 163 467 125 for thread 1 is in sequence #8093

It applied all log file and after apply last log file with sequence #8093 it failed with following error.

SQL> RECOVER DATABASE UNTIL TIME '2007-06-07:23:59:59' USING BACKUP CONTROLFILE

ORA-00279: change 25163467125 generated at 06/08/2007 00:00:00 needed for thread 1

ORA-00289: suggestion : /orastage/oracle/data/<sid>/arch/8093_1.arc

ORA-00280: change 25 163 467 125 for thread 1 is in sequence #8093

Specify log: {<RET>=suggested filename AUTO CANCEL}

/orastage/oracle/data/<sid>/arch/8093_1.arc

ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below

ORA-01194: file 1 needs more recovery to be consistent

ORA-01110: data file 1: '/u03004.05/oracle/data/<sid>/system01.dbf'

Referred metalink note

Subject: DIAGNOSING ORA-1547 & ORA-1194 ERRORS DURING RECOVERY

Doc ID: Note:435201.1

Most of database data files were in fuzzy mode.

SQL> select file#,STATUS, FUZZY from v$datafile_header;

7. We concluded that backup was not good so we planned to restore Last week Sunday backup and apply all archive files.

8. We requested unix team to restore last Sunday’s cold backup.

9. Recover database from Sunday Cold backup set by apply all archive files (150)

Login as oracle and set environment Changed archive_log_destination parameter in $ORACLE_HOME/dbs/init<sid>.ora file to reflect new archive log file location.

log_archive_dest =/ORASTAGE/oracle/data/<sid>/arch

$sqlplus /nolog

SQL> connect / as sysdba

SQL> startup nomount

SQL> show parameter archive

SQL> show parameter archive

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------

archive_lag_target integer 0

log_archive_dest string /ORASTAGE/oracle/data/<sid>/arch

SQL> RECOVER DATABASE UNTIL TIME '2007-06-07:23:59:59' USING BACKUP CONTROLFILE;

AUTO

Content from $bdump/alert<sid>.log file.

Sat Jun 9 11:48:15 2007

Database mounted in Exclusive Mode.

Completed: ALTER DATABASE MOUNT

Sat Jun 9 11:49:19 2007

ALTER DATABASE RECOVER DATABASE UNTIL TIME '2007-06-07:23:59:59' USING BACKUP CONTROLFILE

Sat Jun 9 11:49:19 2007

Media Recovery Start

ORA-279 signalled during: ALTER DATABASE RECOVER DATABASE UNTIL TIME '2007-...

Sat Jun 9 11:49:48 2007

ALTER DATABASE RECOVER CONTINUE DEFAULT

Sat Jun 9 11:49:48 2007

Media Recovery Log /ORASTAGE/oracle/data/<sid>/arch/7943_1.arc

.

.

.

ALTER DATABASE RECOVER CONTINUE DEFAULT

Sat Jun 9 20:30:28 2007

Media Recovery Log /ORASTAGE/oracle/data/<sid>/arch/8093_1.arc

.

.

.

Sat Jun 9 20:34:17 2007

Terminal Recovery: done UNTIL CHANGE 25163467125

Media Recovery Complete

Completed: ALTER DATABASE RECOVER CONTINUE DEFAULT

Sat Jun 9 20:34:21 2007

ARC1: Cannot archive online log based on backup controlfile

Sat Jun 9 20:35:21 2007

ARC1: Cannot archive online log based on backup controlfile

Sat Jun 9 20:36:14 2007

SQL> ALTER DATABASE OPEN RESETLOGS;

ALTER DATABASE OPEN RESETLOGS

Sat Jun 9 20:36:14 2007

RESETLOGS after incomplete recovery UNTIL CHANGE 25163467125

Resetting resetlogs activation ID 1687126238 (0x648f80de)

Sat Jun 9 20:39:02 2007

.

.

Sat Jun 9 20:44:25 2007

replication_dependency_tracking turned off (no async multimaster replication found)

Completed: ALTER DATABASE OPEN RESETLOGS

It took almost 9 hour to apply 150 archive file (size 300M).

10. Started database listener and did basic checking to validate database restore and recovery.

11. Logged in as applmgr and tried to start oracle application services as given below

$ . $APPL_TOP/APPSORA.env

$ cd $COMMON_TOP/admin/scripts/<sid>_<hostname>/

$adstrtal.sh apps/<password>

It started all services.

I tried to log in to applications using given URL.

http://<hostname>.<domainname>:<PORT>/

Click on “Apps Logon Links”

Click on “E-Business Home Page”

Logged as “SYSADMIN” user

Login failed with given error.

"YOU HAVE ENCOUNTERED AN UNEXPECTED ERROR"

As part of troubleshooting I did following.

1. Gave wrong password and it failed with authentication error.

http://<hostname>.<domainname>:<PORT>/OA_HTML/jsp/fnd/aoljtest.jsp

All AOLJ test was passed except jsp pages. We doubt problem could be with jsp pages

2. Turned on debug for Apache and JSP as given below

a. Stopped apache services

$apps/OEBS/ndev2/commontop/admin/scripts/<sid>_<hostname>/adapcctl.sh stop

b. Deleted $IAS_ORACLE_HOME/Apache/Apache/logs and $IAS_ORACLE_HOME/Apache/Jserv/log

c. Edited $IAS_ORACLE_HOME/Apache/Apache/conf/httpd.conf

LogLevel debug

d. Edited $IAS_ORACLE_HOME/Apache/Jserv/conf/jserv.conf

ApJServLogLevel debug

e. Edited $IAS_ORACLE_HOME/Apache/Jserv/conf/jserv.properties

log=true

Notice following error

$IAS_ORACLE_HOME/Apache/Jserv/logs/ mod_jserv.log

[10/06/2007 12:59:36:265] (ERROR) ajp12: Servlet Error: OracleJSP: java.io.FileNotFoundException

$IAS_ORACLE_HOME/Apache/Jserv/logs/jserv.log

[10/06/2007 12:26:43:744 EDT] oracle.jsp.JspServlet: init

[Jun 10, 2007 12:27:04 PM EDT]:1181492824137:Thread[Thread-70,10,main]:-1:-1:hostname.domain name:<ipaddress>:8102:16020:UNEXPECTED:[fnd.common.logging.DebugEventManager.handlerException]

:java.lang.Exception: /aferror.log (Permission denied (errno:13))

at oracle.apps.fnd.common.logging.FileHandler.<init>(FileHandler.java:103)

at oracle.apps.fnd.common.logging.FileHandler.<init>(FileHandler.java:80)

at oracle.apps.fnd.common.logging.DebugEventManager.registerHandlers(DebugEventManager.java:

1241)

at oracle.apps.fnd.common.logging.DebugEventManager.init(DebugEventManager.java:904)

at oracle.apps.fnd.common.logging.DebugEventManager.<init>(DebugEventManager.java:194)

at oracle.apps.fnd.common.AppsLog.<init>(AppsLog.java:519)

at oracle.apps.fnd.common.AppsLog.<init>(AppsLog.java:489)

at oracle.apps.fnd.common.AppsLog.<clinit>(AppsLog.java:469)

at oracle.apps.fnd.common.AppsContext.makeLog(AppsContext.java:1153)

at oracle.apps.fnd.common.Context.setLog(Context.java:1400)

at oracle.apps.fnd.common.Context.setLog(Context.java:1381)

at oracle.apps.fnd.common.Context.getLog(Context.java:1353)

at oracle.apps.fnd.common.Context.isLoggingEnabled(Context.java:2119)

at oracle.apps.fnd.common.Context.makeConnectionManager(Context.java:655)

at oracle.apps.fnd.common.AppsContext.makeConnectionManager(AppsContext.java:1186)

at oracle.apps.fnd.common.Context.setConnectionManager(Context.java:615)

at oracle.apps.fnd.common.Context.setConnectionManager(Context.java:599)

at oracle.apps.fnd.common.AppsContext.initializeContext(AppsContext.java:880)

at oracle.apps.fnd.common.AppsContext.<init>(AppsContext.java:742)

at oracle.apps.mwa.container.ApplicationsObjectLibrary.AOLInit(ApplicationsObjectLibrary.jav

a:174)

at oracle.apps.mwa.container.ApplicationsObjectLibrary.<init>(ApplicationsObjectLibrary.java

:78)

at oracle.apps.mwa.container.MWALib.setObjectLibrary(MWALib.java:339)

at oracle.apps.mwa.wap.engine.WapServlet.init(WapServlet.java:158)

at org.apache.jserv.JServServletManager.load_init(JServServletManager.java:755)

at org.apache.jserv.JServServletManager.loadServlet(JServServletManager.java:659)

at org.apache.jserv.JServServletManager.loadStartupServlets(JServServletManager.java:789)

at org.apache.jserv.JServServletManager.init(JServServletManager.java:447)

at org.apache.jserv.JServ.start(JServ.java:625)

at org.apache.jserv.JServ.main(JServ.java:234)

12. I got a hit and executed give below as advised by this metalink note.

Run the following to synchronize WF tables.

Begin

apps.wf_local_synch.BULKSYNCHRONIZATION (P_ORIG_SYSTEM=>'ALL',

P_PARALLEL_PROCESSES=>2,

P_LOGGING=>'LOGGING',

P_RAISEERRORS=>TRUE, P_TEMPTABLESPACE=>'APPS_TS_TX_DATA');

END;

/

Reference :

Unable To Login After Clone

Doc ID: Note:418130.1

It was verified by functional team and then we good database and applications backup before releasing production server for regular use.

Noteà Due to security Concern I have not mentioned actual server name, domain name, ipaddress and ORACLE SID.

Thursday, June 7, 2007

Oracle 10g BI Discoverer Viewer was getting Disconnect

Issue : Oracle 10g BI Discoverer Viewer was getting Disconnect with Given error for one discoverer server nfinoasd12 whereas it was working fine for other server infinoas11. We have same set of configuration for both discoverer servers nfinoas11 and nfinoas12.

“Contact with the Discover server has been lost To continue your work, please restart Discoverer Pluse. If this problem persists, please contact your Oracle Application Server Administrator"

Solution:

1. Checked "Timeout" in $ORACLE_HOME/discoverer/util/pref.txt at both server.

[oracle@nfinoas11 util]$ grep -i timeout pref.txt

Timeout = 28800 #1800 # Timeout in seconds if there is no client activity. Minimum allowed time is 180 secs

[oracle@nfinoas11 util]$ pwd

/local/oracle/product/10.1.2.0.2/discoverer/util

[oracle@nfinoas12 ~]$ cd /local/oracle/product/10.1.2.0.2/discoverer/util

[oracle@nfinoas12 util]$ grep -i timeout pref.txt

Timeout = 28800 # 1800 # Timeout in seconds if there is no client activity. Minimum allowed time is 180 secs

2. Checked $TNS_ADMIN/sqlnet.ora at database server and given entry was missing in sqlnet.ora file. SQLNET.EXPIRE_TIME= 10

Oracle Critical Patch Update – April 2007

As part of security concern DBA has to apply Oracle Critical Patch regularly as and when there is critical security patch release from Oracle. Oracle started critical security patch in 2005 and since then there is four-security patches release every year. Year 2007 release dates are given :

  • 17 July 2007
  • 16 October 2007
  • 15 January 2008
  • 15 April 2008

Last we applied critical security patch July 2006. We are going to apply “Oracle Critical Patch Update – April 2007” with reference to URL http://www.oracle.com/technology/deploy/security/critical-patch-updates/cpuapr2007.html. Let’s navigate though given note and see what patches I need to apply.

Note: 420072.1

Subject : Oracle E-Business Suite Critical Patch Update Note April 2007

have following product installed in our environment.

  • Oracle E-Business Suite Release 11i 11.5.10 CU2
  • Oracle9i Enterprise Edition Release 9.2.0.8.0 - 64bit Production
  • Oracle Application Server 10g Release 2 (10.1.2), versions 10.1.2.0.2

We have running Oracle E-Business Suite Rel. 11i so I am referring “Section 2, "Oracle E-Business Suite Release 11i"” of Note:420072.1. As per oracle guideline we should apply updates to systems that have highest risk first e.g. update systems that are external to firewall before upgrading internal network system. It our environment we do not have any external system but in future we are going to have (currently working on DMZ also).

Oracle has recommended patching Oracle environment in given order and I follow same.

  1. Oracle Database
  2. Oracle Application Server
  3. Oracle E-Business Suite

Patching Considerations for Oracle E-Business Suite Release 11i

Oracle Database and Oracle Application Server critical patches are cumulative, which means it includes all patches for fixes from earlier Oracle critical security patch updates. CPU patches for Oracle E-Business Suite 11i are generally not cumulative, that means one has to refer earlier critical updates and apply all the e-business suite critical patches in chronological order.

In my case I had applied Critical update patch July 2006 and now I need to apply Critical update patch Apr 2007. Oracle has released critical update patch Jan 2006. So in this case I should apply patches as given below.

  1. Database Critical Update patches is cumulative so I need not to apply Critical update patch Jan 2007. I should apply only critical update patch Apr 2007.
  2. Oracle Applications Server Critical Update patches is cumulative so I need not to apply Critical update patch Jan 2007. I should apply only critical update patch Apr 2007.
  3. Oracle E-Business Suite Critical Update patches is not cumulative so I need to apply Critical update patch Oct 2006 and Jan 2007 before apply Critical update patch Apr 2007.

As per Note: 420072.1 Release 11.5.10 applications upgraded to release 11.5.10 CU2 without apply CPUApr2007 must now apply CPUApr2007. E-Business Suite are not cumulative, so one should apply prior CPUs as appropriate before applying CPUApr2007.

Stop all process running using that Oracle home before patching. Patch one oracle home at a time.

Oracle recommends to update database tier and the middle tier at same time, one after the other. After applying patch start services in following order. We have only these two components in our environment.

  1. Oracle Database
  2. Oracle Application Server

‘opatch’ is required to apply database patch.

$ pwd

/local/oracle/product/9.2.0.8/OPatch

$ opatch -version

Oracle Interim Patch Installer version 1.0.0.0.56

Oracle recommends to use OPatch release 1.0.0.0.57 so need to install patch 2617419

Patches to be applied in Ebusiness Suit.

Noteà We are Running Ebusiness Suite in HP-UX 11i environment.

Oracle Database Patches

Oracle Database 9.2.0.8 Rel Patch 5901875

Oracle HTTP Server Patche

According to note “There are no new patches for CPUApr2007. Please uptake the patches listed for this release in CPUJan2007. See Note 402670.1: Oracle E-Business Suite Critical Patch Update Note January 2007

As per Note 402670.1

  1. If you have not already done so, upgrade to release 1.0.2.2.x. See Note 146468.1.
  2. Upgrade to Oracle8i Database release 8.1.7.4 using patch 2376472 if your Oracle Database Client release is earlier than release 8.1.7.4. See Note 309026.1.
  3. Apply CPUJan2007 patch 5700129 to the Oracle Application Server Oracle home.

We have applied upgraded to Orace8i 8.1.7.4 home using patch 2376472 so I need to apply only CPUJan2007 patch 5700129.

Oracle Developer Suite Patches

We are at Oracle Developer Suite Release 6.0.8.27.0. According to Note 402670.1 “There are no new patches for CPUApr2007. Please uptake the following patches which were listed for this release in CPUJan2007. “

JInitiator Patches

We are using Jinitiator Release 1.3.1.26 and as Note 402670.1 “Per Oracle policy, customers on Oracle JInitiator Release 1.3 must move to version 1.3.1.26 or higher for CPUApr2007.” Refer to Note 124606.1 and Note 232200.1 for more information.

We’ll upgrade Jinitator Release from 1.3.1.26 to 1.3.1.28.

As per Note 124606.1 I need to apply following patches to upgrade Jinitiator to 1.3.1.28.

JInitiatorVersion JInitiator Patch Interop Patch

1.3.1.28 5882294 5117525

Apart from above patches following patches are required for respective product users. I am listing only those patches, which are applicable in our environment.

Oracle Applications Framework Navigator Page

  • If you are using Oracle Applications prior to 11.5.10.2CU, please apply patch 4517707.

General Ledger Hierarchy Manager / Financial Intelligence - Financial Dimension Hierarchy Manager Users

  • If you are upgrading to JInitiator 1.3.1.x and are using Oracle Applications Framework 5.7 or later, you must apply patch 3698893.

Oracle E-Business Suite Release 11i Patches

Oracle CPUApr2007 patches for E-Business Suite Release 11i are not generally cumulative. We need to apply the previous critical patch updates and Oracle security alerts related to ebusiness Suite before applying this critical patch update.

In my case I need to apply following patches.

E-Business Suite Critical Patch October 2006

5486407

5479643 Superceded by 5473858 (11i.ATG_PF.H.RUP5)

5500118 Superceded by 5473858 (11i.ATG_PF.H.RUP5)

5335967

E-Business Suite Critical Patch January 2007

E-Business Suite Critical Patch April 2007

Patch 5021981 (FND)

Release 11.5.10 CU2

Oracle E-Business Suite 11i with 11i.ATG_PF.H RUP4 (4676589) installed

Any other patches listed for the Oracle E-Business Suite Release 11i base release on which 11i.ATG_PF.H RUP4 is installed.

Reference:

  • Note:402670.1 Oracle E-Business Suite Critical Patch Update Note January 2007
  • Note:420072.1 Oracle E-Business Suite Critical Patch Update Note April 2007