Thursday, July 30, 2009

ORA-01081: cannot start already-running ORACLE - shut it down first

Though we have latest database but still we are having some old legacy system. I was trying to stop 8.1.7.4.0 database instance and it failed with given below error.

Issue

While starting 8.1.7.4 database I got "ORA-01081: cannot start already-running ORACLE - shut it down first" error though database was not running.

# ps –ef | grep smon

# svrmgrl             

Oracle Server Manager Release 3.1.7.0.0 - Production

Copyright (c) 1997, 1999, Oracle Corporation.  All Rights Reserved.

Oracle8i Enterprise Edition Release 8.1.7.4.0 - Production

With the Partitioning and Parallel Server options

JServer Release 8.1.7.4.0 - Production

SVRMGR> connect internal

Connected.

SVRMGR> startup                            

ORA-01081: cannot start already-running ORACLE - shut it down first

SVRMGR> exit

Server Manager complete.

Solution

I performed as given below.

# ipcs -b | grep oracle

m 290455565   00000000 --rw-r-----   oracle      dba 249442304

m 290455566 0xbc71ed40 --rw-r-----   oracle      dba 2147483648

/opt/app/oracle/JAIdb/8.1.7/network/admin

# ipcrm -m 290455565

/opt/app/oracle/JAIdb/8.1.7/network/admin

# ipcrm -m 290455566

/opt/app/oracle/JAIdb/8.1.7/network/admin

# sqlplus sys

SQL*Plus: Release 8.1.7.0.0 - Production on Sun Feb 10 15:23:27 2008

(c) Copyright 2000 Oracle Corporation.  All rights reserved.

Enter password:

Connected to an idle instance.

SQL> startup

ORACLE instance started.

Total System Global Area 2396762004 bytes

Fixed Size                    73620 bytes

Variable Size            1677832192 bytes

Database Buffers          688128000 bytes

Redo Buffers               30728192 bytes

Database mounted.

Database opened.

SQL>

 

OracleBI Discoverer was unable to authenticate using the password provided

My day started with lots of technical issues and most important issue I was suppose to address first was Discoverer login issue using SSO enabled Ebusiness suite Application (11.5.9).

After production clone (11.5.9) we registered instance to SSO OID (101420) as given below.

Registered SSO

$ txkrun.pl -script=SetSSOReg \

> -registersso=Yes \

> -appspass=pass4jait \        ß Oracle Applications apps schema password

> -infradbhost=jai001.orbit.com\    ß OID server name

> -infradbport=1530 \            ß OID LDAP Port

> -infradbsid=SSOT \            ß SSO OID Database SID

> -orassopass=D9v01D75 \         ß It's OID orasso user's password.

> -systempass=pass4jait \        ß Oracle Application System Password

> -ssosdkpass=ssosdk            ß Oracle Applications SSOSDK schema password

*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS

*** Log File = /opt01/app/oracle/jaitcomn/rgf/JAIT_ios11301e/sso/txkSetSSOReg_Mon_Jul_27_16_01_54_2009.log

Program : /opt01/app/jaitappl/fnd/11.5.0/patch/115/bin/txkSetSSOReg.pl started @ Mon Jul 27 16:01:54 2009

*** Log File = /opt01/app/oracle/jaitcomn/rgf/JAIT_ios11301e/sso/txkSetSSOReg_Mon_Jul_27_16_01_54_2009.log

######################## WARNING ########################################

This application works with SSOSDK version 9.0.2 or higher. If lower version

(3.0.9) of SSOSDK was installed in your system and you have a registered

partner application, this process will remove the 3.0.9 version of the SSOSDK

schema and install the 9.0.2 version.

######################## WARNING ########################################

Beginning input parameter validation for SSO registration.

Beginning loading SSO SDK into database if necessary.

Loading of SSO SDK into database completed successfully.

Input parameter validation for SSO registration completed.

BEGIN SSO REGISTRATION:

Beginning to register partner application.

Partner application has been registered successfully.

Single Sign-On partner application registered successfully.

End of /opt01/app/jaitappl/fnd/11.5.0/patch/115/bin/txkSetSSOReg.pl : No Errors encountered

Registered OID

$ txkrun.pl -script=SetSSOReg \

> -provtmp=$FND_TOP/admin/template/ProvOIDToApps.tmp \

> -registeroid=Yes \

> -appspass=pass4jait \            ß Oracle Applications apps schema password

> -infradbhost=jai001.orbit.com\        ß OID server name

> -orcladminpass=ssotirisqa123 \        ß OID orcladmin user password

> -instpass=welcome123 \            ß New password to register application in SSO

> -ldapport=3030 \                ß OID LDAP Port

> -appname="JAIT" \                ß Application Name of Oracle applications

> -svcname="JAIT"                ß Service Name of Oracle Applications

*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS

*** Log File = /opt01/app/oracle/jaitcomn/rgf/JAIT_ios11301e/sso/txkSetSSOReg_Mon_Jul_27_16_04_37_2009.log

Program : /opt01/app/jaitappl/fnd/11.5.0/patch/115/bin/txkSetSSOReg.pl started @ Mon Jul 27 16:04:38 2009

*** Log File = /opt01/app/oracle/jaitcomn/rgf/JAIT_ios11301e/sso/txkSetSSOReg_Mon_Jul_27_16_04_37_2009.log

Beginning input parameter validation for OID registration.

Input parameters validation for OID registration completed.

BEGIN OID REGISTRATION:

Beginning to register Application and Service containers if necessary.

Application and Service containers were found and thus not created.

Beginning to register application in Oracle Internet Directory.

Registration of application in Oracle Internet Directory completed successfully.

Beginning to register instance password in Oracle Internet Directory.

Registration of instance password in Oracle Internet Directory completed successfully.

Beginning to test application registration in Oracle Internet Directory.

Testing of application registration in Oracle Internet Directory completed successfully.

Beginning to register provisioning profile in Oracle Internet Directory.

Registration of provisioning profile in Oracle Internet Directory completed successfully.

Application is now registered successfully with provisioning in Oracle Internet Directory.

End of /opt01/app/jaitappl/fnd/11.5.0/patch/115/bin/txkSetSSOReg.pl : No Errors encountered

[orajait@ios11301e] /home/orajait

After registration I made sure following SSO related profile value was set at site level

Application SSO LDAP Synchronization : Enabled

Applications Override SSO Server Language : Use SSO Server Language

Applications SSO Allow Multiple Accounts : Disabled

Applications SSO Allow Multiple Accounts : Disabled

Applications SSO Auto Link User : Enabled

Applications SSO Enable OID Identity Add Event : Enabled

Applications SSO Linking Source of Truth : Oracle Internet Directory

Applications SSO Login Types : SSO

Applications SSO Type : SSWA w/SSO

Applications SSO User Creation and Updation Allowed : Enabled

I logged in to jait (http://jait.orbit.com:8020) and login was though.

Discoverer Configuration

I performed following steps to configure already installed discoverer (10.1.2.0.2)

  1. Copy the dbc file from the application's $FND_TOP/secure to the discoverer $ORACLE_HOME/discoverer/secure
  2. Add the TNS entry for the database in the tnsnames.ora of the $ORACLE_HOME/network/admin (discoverer home)
  3. Add the variable TNS_ADMIN to the $ORACLE_HOME/bin/iasenv.sh of discoverer home
  4. Perform as given below to change plug-in from Sun Java to jinitiator for discoverer plus

    Logon to Enterprise Manager on the Oracle Business Intelligence 10g Release 2 ORACLE_HOME using the ias_admin account.

    - Select the "Oracle Business Intelligence 10g Release 2 Oracle Home" Instance

    - Select "Discoverer"

    - Select "Discoverer Plus"

    - Select "Configuration Java Plugin"

    - In the "Java Plugin" section switch the drop-down list from "Sun Java Plugin (1.4)" to "Oracle JInitiator".

    - Press "OK" to apply the changes.

  5. On the Oracle Business Intelligence 10g Release 2 filesystem, open the file $ORACLE_HOME/discoverer/config/configuration.xml

    - Find the line with the tag <jvm name="jinitiator"/>, for example
    <jvm name="jinitiator" classid="clsid:CAFECAFE-0013-0001-0022-ABCDEFABCDEF" plugin_setup="http://mybi.company.com:7780/jinitiator/jinit.exe" version="1.3.1.22" versionie="1,3,1,22" type="application/x-jinit-applet" plugin_page="/PlusPluginPage.uix" disco_archive="disco5i.jar" d4o_archive="d4o_double.jar"/>

    Update the line and specify usage of JInitiator 1.3.1.18, for example
    <jvm name="jinitiator" classid="clsid:CAFECAFE-0013-0001-0018-ABCDEFABCDEF" plugin_setup="http://ios0504e.orbit.com:8080/jinitiator/jinit13118.exe" version="1.3.1.18" versionie="1,3,1,18" type="application/x-jinit-applet" plugin_page="/PlusPluginPage.uix" disco_archive="disco5i.jar" d4o_archive="d4o_double.jar"/

  6. Save the configuration.xml file
  7. Copy the Jinitiator 1.3.1.18 executable from the application tier to the $ORACLE_HOME/jinit directory as jinit13118.exe.
  8. Change Oracle Applications Context file as given below

    <discoinstancename oa_var="s_discoinstance"><node>.orbit.com_8080</discoinstancename>

    <disco_ver_comma oa_var="s_disco_ver_comma">10,1,2</disco_ver_comma>

    <disco_standalone oa_var="s_disco_standalone">true</disco_standalone>

    <disco_protocol oa_var="s_disco_protocol">http</disco_protocol>

    <disco_machine oa_var="s_disco_machine"><node>.orbit.com</disco_machine>

    <disco_port oa_var="s_disco_port" oa_type="PORT">8080</disco_port>

  9. Save the changes.
  10. Modify the following profiles:

    ICX: Discoverer Default End User Layer Schema Prefix # IKNDWEUL ß Discoverer EUL schema

    ICX: Discoverer Launcher # http://<discoverer_host.domain:80/discoverer/plus?Connect=[APPS_SECURE]

    ICX: Discoverer Release # 10

    ICX: Discoverer Viewer Launcher # http://<discoverer_host.domain:80/discoverer/viewer?Connect=[APPS_SECURE]

    ICX: Discoverer use Viewer # Yes

  11. Copy the pref.txt under /staging/software/disc10gR2 to $ORACLE_HOME/discoverer/util. Run applypreferences.sh.
  12. Increase the timeout value in httpd.conf to 3600
  13. Bounce of discoverer will be needed after this change.
  14. Change $ORACLE_HOME/discoverer/discwb.sh

    # FND_TOP=$OH/discoverer

    FND_TOP=<fnd top of middle tier>

  15. Modify $ORACLE_HOME/bin/iasenv.sh

    if [ x${LD_LIBRARY_PATH} != x ] ; then

    LD_LIBRARY_PATH=$LD_LIBRARY_PATH

    else

    TO

    if [ x${LD_LIBRARY_PATH} != x ] ; then

    LD_LIBRARY_PATH=/opt01/disc10g/oracle/disc10g/discoverer/lib:$LD_LIBRARY_PATH

    else

  16. Change following value in Discoverer $OH/opmn/conf/opmn.xml file as below, this will drive how many max. connections can go to a discoverer installation. Make it to 200. Also make sure it's there in our discoverer installation.

    <process-set id="SessionServer" minprocs="0" maxprocs="100" restart-on-death="false" parallel-requests="true"/>

  17. Restart Discoverer Services.

After configuration I tried to click on discoverer report link using responsibility created for discoverer report. I got given below error.

A connection error has occurred.

- OracleBI Discoverer was unable to authenticate using the password provided.

This can happen due to an invalid password or because the password was lost while using back, forward, or refresh in your browser. Please enter the password again to continue.

- Unable to connect to Oracle Applications database: your password has expired.

Several time I checked discoverer profile values I also compared SSO profile and discoverer profile from other instance where it was working fine. Finally I decided to turn Discoverer debug on and checked log. I performed following steps to turn discoverer login trace on (mentioned in dark blue color) and restarted Discoverer services.

$ cp $ORACLE_HOME/opmn/conf/opmn.xml cp $ORACLE_HOME/opmn/conf/opmn.xml. 092409

$ vi opmn.xml

+185 <environment>

+186 <variable id="DISCOVERER_ALLOW_DB_CONNECT_STRING" value="YES"/>

+187 <variable id="PREFERENCE_PORT" value="1980"/>

+188 <variable id="DISCO_DIR" value="$ORACLE_HOME"/>

+189 <variable id="DCLOG_CONFIG_FILE" value="/opt01/disc10g/oracle/disc10g/discoverer/util/ServerLogParams.txt"/>

+190 <variable id="LIBPATH" value="/opt01/disc10g/oracle/disc10g/discoverer/lib:/opt01/disc10g/oracle/disc10g/lib32:/usr/lib" append="true"/>

+191 <variable id="DC10_REG" value="/opt01/disc10g/oracle/disc10g/discoverer/"/>

+192 <variable id="FND_TOP" value="/opt01/disc10g/oracle/disc10g/discoverer/"/>

+193 <variable id="FND_SECURE" value="/opt01/disc10g/oracle/disc10g/discoverer/secure"/>

+194 <variable id="NLS_LANG" value="AMERICAN_AMERICA.UTF8"/>

+195 <variable id="AFLOG_ENABLED" value="Y"/>

+196 <variable id="AFLOG_FILENAME" value="/opt01/disc10g/oracle/disc10g/discoverer/logs/apps_trace.txt"/>

+197 <variable id="AFLOG_LEVEL" value="STATEMENT"/>

+198 <variable id="AFLOG_MODULE" value="fnd.src.security.afscp fnd.src.osd.afenv"/>

+199 </environment>

$ $ORACLE_HOME/opmn/bin/opmnctl stopall

$$ORACLE_HOME/opmn/bin/opmnctl startall

I reproduced issue after restarting discoverer services and check $ORACLE_HOME/discoverer/logs/apps_trace.txt file.

=================STARTUP TIME= =================

PROCEDURE fnd.src.security.afscp.afscpcfg: Begin (config_file = /opt01/disc10g/oracle/disc10g/discoverer/secure/ios11301e_jait.dbc)

STATEMENT fnd.src.security.afscp.afscpinit: $Header: afscp.lc 115.25 2002/08/26 18:33:29 sdstratt ship $

PROCEDURE fnd.src.osd.afenv.usd_load_customfile: Begin(configname = nil)

PROCEDURE fnd.src.osd.afenv.usd_load_customfile: End: No config file specified

STATEMENT fnd.src.security.afscp.afscpcfg: About to load config file.

PROCEDURE fnd.src.osd.afenv.usd_load_customfile: Begin (configname = /opt01/disc10g/oracle/disc10g/discoverer/secure/ios11301e_jait.dbc)

STATEMENT fnd.src.osd.afenv.usd_load_customfile: File found in local directory

STATEMENT fnd.src.osd.afenv.usd_load_customfile: Loading config file contents

PROCEDURE fnd.src.osd.afenv.usd_load_customfile: End: Success

STATEMENT fnd.src.security.afscp.afscpcfg: Config file loaded.

STATEMENT fnd.src.security.afscp.afscpcfg: Config file params (fndnam = APPS, gwyuid = APPLSYSPUB/PUB, two_task = JAIT, appl_server_id = 6E91FD59C398E0EAE04

30A0B3EDBE0EA37940464221077978720915360229902)

PROCEDURE fnd.src.security.afscp.afscpcfg: End: Success

PROCEDURE fnd.src.security.afscp.afscpfree: Begin

PROCEDURE fnd.src.security.afscp.afscpfree: End

PROCEDURE fnd.src.security.afscp.afscpinit: Begin(fndnam = APPS, gwyuid = APPLSYSPUB/PUB@JAIT)

STATEMENT fnd.src.security.afscp.afscpinit: $Header: afscp.lc 115.25 2002/08/26 18:33:29 sdstratt ship $

PROCEDURE fnd.src.osd.afenv.usd_load_customfile: Begin(configname = nil)

PROCEDURE fnd.src.osd.afenv.usd_load_customfile: End: No config file specified

PROCEDURE fnd.src.security.afscp.afscpinit: End

PROCEDURE fnd.src.security.afscp.afscptik: Begin(ticket = sKXNm0vKEEVLxtU_0AAsD_x-:S)

PROCEDURE fnd.src.security.afscp.afscpsup: Begin

STATEMENT fnd.src.security.afscp.afscpsup: Username = GUEST

PROCEDURE fnd.src.security.afscp.afscpsup: End: Success

PROCEDURE fnd.src.security.afscp.afscptik: End: Success

PROCEDURE fnd.src.security.afscp.afscpsvr: Begin

PROCEDURE fnd.src.security.afscp.afscpsvr: End: Success

PROCEDURE fnd.src.security.afscp.afscpcon: Begin

STATEMENT fnd.src.security.afscp.afscpcon: Connecting to gwyuid = APPLSYSPUB/PUB@JAIT

STATEMENT fnd.src.security.afscp.afscpcon: New security pkg: Retrieving encrypted pwd

STATEMENT fnd.src.security.afscp.afscpcon: New security pkg success

STATEMENT fnd.src.security.afscp.afscpcon: Decrypting apps password

STATEMENT fnd.src.security.afscp.afscpcon: Closing gwyuid connection

STATEMENT fnd.src.security.afscp.afscpcon: Opening apps connection

STATEMENT fnd.src.security.afscp.afscpcon: Processing ticket

PROCEDURE fnd.src.security.afscp.get_icx_ticket: Begin

PROCEDURE fnd.src.security.afscp.get_icx_ticket: End: Not a one-time ticket.

STATEMENT fnd.src.security.afscp.afscpcon: Retrieve session values from ticket

STATEMENT fnd.src.security.afscp.afscpcon: Set icx_session_id profile

STATEMENT fnd.src.security.afscp.afscpcon: Get responsibility info

STATEMENT fnd.src.security.afscp.afscpcon: user_id = 45514, resp_type = I, resp_key = resp_id = 50380

STATEMENT fnd.src.security.afscp.afscpcon: appl_type = I, appsname = , appl_id = 20003, secgrp = STANDARD

STATEMENT fnd.src.security.afscp.afscpcon: Decrypting new oracle pwd for resp

STATEMENT fnd.src.security.afscp.afscpcon: Constructing connect string for resp

PROCEDURE fnd.src.security.afscp.afscpcon: End: Success

PROCEDURE fnd.src.security.afscp.afscpfree: Begin

PROCEDURE fnd.src.security.afscp.afscpfree: End

=================STARTUP TIME= =================

I did not noticed any issue in error log. Finally I turned on trace on discoverer services on instance where it was working and I compared working instance log with non working instance log. I noticed given below difference.

Non working discoverers header version was different from working discoverer header version

Working Discoverer Header Version

STATEMENT fnd.src.security.afscp.afscpinit: $Header: afscp.lc 115.25 2002/08/26 18:33:29 sdstratt ship $

Non Working Discoverer Header Version

STATEMENT fnd.src.security.afscp.afscpinit: $Header: afscp.lc 120.5.12000000.2 2007/05/25 20:33:57 mskees ship $

I mentioned this to other senior member and he recalled new installed discoverer was not upgraded. We asked junior team member to upgrade discoverer

Current version :

$cd $ORACLE_HOME/bin

$ strings -a dis51ws | grep "Discoverer Version:"

Discoverer Version:Server 10.1.2.48.18

New Expected Version (working version)

$  strings -a dis51ws | grep "Discoverer Version:"

Discoverer Version:Server 10.1.2.54.25

After discoverer upgrade problem was resolved. I will describe detail steps of discoverer upgrade in other post.

Wednesday, July 29, 2009

Download patch from Metalink and upload SR file to/from Unix Server

Steps to Upload trace/log file to Metalink Service Request on Unix Server

Login to unix server

$ftp ftp.oracle.com

User id : anonymous

Password : anonymous

ftp> cd support/incoming/

ftp> mkdir <SR number>

ftp> 6895234.994

ftp> bin

ftp> mput *.zip

Steps to Download patch from Metalink on Unix Server

$ftp updates.oracle.com

User Name : <MetalinkUserID>

Password : <password>

 

ftp> cd <patch number>

ftp> cd 5105622 (e.g.)

ftp> ls

ftp> bin

ftp> get <patch_full_name>

How to enable user’s Trace in EBusiness Suite?

  1. Login to Ebusiness Suite and Select System Administrator Responsibility
  2. Set profile for particular user i.d. JAISINGH Go to 'System Responsibility' à Profile à System
  3. Set profile 'Initialization SQL Statement – Custom' at user level to

    BEGIN FND_CTL.FND_SESS_CTL ('','','TRUE','TRUE','LOG','ALTER SESSION SET TRACEFILE_IDENTIFIER =''JHSINGH'' EVENTS=''10046 TRACE NAME CONTEXT FOREVER, LEVEL 12''');END;

  4. Exit application after change
  5. Ask user to produce issue
  6. Go to db parameter user_dump_dest location and search for *JHSINGH*.trc

Reference: Meatlink Note 787724.1