time out while waiting for a managed process to start

Issue

We deployed third party service in oracle 10giAS 10.1.3.0. While starting opmn services we noticed given below error on $ prompt.

opmnctl: starting opmn and all managed processes...

================================================================================

opmn id= eos123:4201

1 of 2 processes started.

 

ias-instance id=ias_1013.eos123.orbit.org

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

ias-component/process-type/process-set:

default_group/ORBIT/default_group/

 

Error

--> Process (index=1,uid=1201922234,pid=63455)

time out while waiting for a managed process to start

Log:

/usr01/oracle/10giAS1013/opmn/logs/default_group~ORBIT ~default_group~1.log

 

Solution

1. Make timeout change in $IAS_ORACLE_HOME/opmn/conf/opmn.xml file

   <start timeout="1200"/>

2. Reload opmn services

   $IAS_ORACLE_HOME/opmn/bin/opmnctl reload

3. Validate issue

ORA-00600: internal error code, arguments: [LibraryCacheNotEmptyOnClose]

I had to restart oracle applications R12 database (10.2.0.3) and application services. While stopping database services I noticed given below error

Error

SQL> shutdown immediate

Database closed.

Database dismounted.

ORA-00600: internal error code, arguments: [LibraryCacheNotEmptyOnClose], [], [], [], [], [], [], []

 

Resolution

    This error can be ignored. This problem is fixed in 11.1.0.6 release.

 

Reference

    Metalink Id : 365103.1

Unable to Start “Output Post Processor” Concurrent Manager

After starting concurrent manager I noticed "Output Post Processor" did not start. I tried to start "Output Post Processor" using "Administrator Concurrent Manager" screen but it failed with given error in manager log file

 

Error

Unable to initialize state monitor.

oracle.apps.fnd.cp.gsm.GenCartCommException: ORA-01403: no data found

ORA-06512: at "APPS.FND_CP_GSM_IPC", line 499

ORA-06512: at line 1

 

    at oracle.apps.fnd.cp.gsm.GenCartComm.initService(GenCartComm.java:233)

    at oracle.apps.fnd.cp.gsm.GenCartComm.<init>(GenCartComm.java:80)

    at oracle.apps.fnd.cp.gsf.GSMStateMonitor.init(GSMStateMonitor.java:74)

    at oracle.apps.fnd.cp.gsf.GSMStateMonitor.<init>(GSMStateMonitor.java:62)

    at oracle.apps.fnd.cp.gsf.GSMServiceController.init(GSMServiceController.java:111)

    at oracle.apps.fnd.cp.gsf.GSMServiceController.<init>(GSMServiceController.java:66)

    at oracle.apps.fnd.cp.gsf.GSMServiceController.main(GSMServiceController.java:428)

 

Resolution

I notice "Service Manager" was down. So I restarted "Service Manager" and then restarted "Output Post Processor". Concurrent manager started successfully.

 

Unable to Start Conflict Resolution Manager

I had to restart application services to resolve Self Service connection issue. Concurrent manager was not starting. Strange thing was $APPLCSF/$APPLLOG/$TWO_TASK_<mm><dd>.mgr (Internal Manager) log was not showing getting updated. After couple of hour of struggle we noticed file system was full by 100%. Some time we miss very small thing to solve such issues. After starting concurrent manager I noticed Concurrent Resolution Manager (FNDCRM) was not coming up.

I performed following check.

1. Lots of requests where getting assigned to "Concurrent Resolution Manager" so I had to clear queue
2. There was on incompatibility associated to either Standard concurrent manager or Conflict Resolution Manager'
3. I also executed cmclean.sql script (from oracle) and restarted concurrent but it did not helped.
4. Finally I relinked FNDCRM concurrent manager executable and restarted concurrent manger. It resolved issue.

   Source application environment

   adrelink.sh force=y "fnd FNDCRM"

    

    

    

The Distributed Caching System failed to initialize on port

Today was terrible day in office I had to face lots of issue for critical instances. All of sudden it was noticed that one of R12 instance was irresponsive. I was asked to jump to issue. I was asked not to restart concurrent manager. I was only suppose to bound applications (web server). In order to resolve issue I performed following.

1. Stopped Web services in following order

   Sourced application environment

   $ADMIN_SCRIPTS_HOME/adformsrvctl.sh stop

   $ADMIN_SCRIPTS_HOME /jtffmctl.sh stop

   $ADMIN_SCRIPTS_HOME /adoafmctl.sh stop

   $ADMIN_SCRIPTS_HOME /adoacorectl.sh stop

   $ADMIN_SCRIPTS_HOME /adapcctl.sh stop

   $ADMIN_SCRIPTS_HOME/adopmnctl.sh stop

   $ADMIN_SCRIPTS_HOME/mwactl.sh -login <sysadmin password> stop_force <MWAPort>

    

2. Started Services

   Sourced application enviroment

   $ADMIN_SCRIPTS_HOME/adopmnctl.sh start

   $ADMIN_SCRIPTS_HOME/adapcctl.sh start

   $ADMIN_SCRIPTS_HOME/adoacorectl.sh start

   $ADMIN_SCRIPTS_HOME/adoafmctl.sh start

   $ADMIN_SCRIPTS_HOME/jtffmctl.sh start

   ${SCRIPT_DIR}/mwactl.sh start

   $ADMIN_SCRIPTS_HOME/adopmnctl.sh status

   $ADMIN_SCRIPTS_HOME/adformsrvctl.sh start

Even after restarting services did not came up. I noticed given below errors for "s_java_object_cache_port" context variaable (from $CONTEXT_FILE) port.

$LOG_HOME/logs/ora/10.1.3/opmn/oafm_default_group_1/oafmstd.err

10/02/05 11:47:38 Error: <connector name="OracleASjms" path="OracleASjms.rar" /> will not be bootstrapped since corresponding module declaration was not found in application.xml.

10/02/05 11:47:38 Error: <connector name="OracleASjms" path="OracleASjms.rar" /> will not be bootstrapped since corresponding module declaration was not found

in application.xml.

10/02/05 11:48:06 Exception in static block of jtf.cache.CacheManager. Stack trace is: oracle.apps.jtf.base.resources.FrameworkException: IAS Cache initializa

tion failed. The Distributed Caching System failed to initialize on port: s_java_object_cache_port. The list of hosts in the distributed caching system is: <ipaddress>. The port <port> should be free on each host running the JVMs.

at oracle.apps.jtf.cache.IASCacheProvider.init(IASCacheProvider.java:335)

at oracle.apps.jtf.cache.CacheManager.activateCache(CacheManager.java:1451)

at oracle.apps.jtf.cache.CacheManager.initCache(CacheManager.java:768)

$LOG_HOME /logs/ora/10.1.3/opmn/oacore_default_group_2/oacorestd.err

Error: <connector name="OracleASjms" path="OracleASjms.rar" /> will not be bootstrapped since corresponding module declaration was not found

in application.xml.

10/02/05 11:47:32 Exception in static block of jtf.cache.CacheManager. Stack trace is: oracle.apps.jtf.base.resources.FrameworkException: IAS Cache initializa

tion failed. The Distributed Caching System failed to initialize on port: s_java_object_cache_port. The list of hosts in the distributed caching system is: <IPADDRESS>. The po

rt s_java_object_cache_port should be free on each host running the JVMs.

at oracle.apps.jtf.cache.IASCacheProvider.init(IASCacheProvider.java:335)

at oracle.apps.jtf.cache.CacheManager.activateCache(CacheManager.java:1451)

$LOG_HOME /logs/ora/10.1.3/opmn/oacore_default_group_3/oacorestd.err

Error: <connector name="OracleASjms" path="OracleASjms.rar" /> will not be bootstrapped since corresponding module declaration was not found

in application.xml.

10/02/05 11:46:49 Exception in static block of jtf.cache.CacheManager. Stack trace is: oracle.apps.jtf.base.resources.FrameworkException: IAS Cache initializa

tion failed. The Distributed Caching System failed to initialize on port: s_java_object_cache_port. The list of hosts in the distributed caching system is: <IPAddress> The po

rt <port> should be free on each host running the JVMs.

at oracle.apps.jtf.cache.IASCacheProvider.init(IASCacheProvider.java:335)

at oracle.apps.jtf.cache.CacheManager.activateCache(CacheManager.java:1451)

at oracle.apps.jtf.cache.CacheManager.initCache(CacheManager.java:768)

at oracle.apps.jtf.cache.CacheManager.<clinit>(CacheManager.java:390)

 

Solution

I had to restart oracle applications Rel 12 web services along with concurrent manager.

I performed following.

1. Source environmennt
2. Stoped all web services running on middle tier
3. Stopped concurrent manager
4. Started web services along with concurrent manager
5. I also checked $LOG_HOME /logs/ora/10.1.3/opmn/*-default_group_?/oa*std.err file and did not noticed above errors