Oracle EBS R12 – Concurrent request log and output files display with delay

We faced one issue in our Oracle EBS R12 where users reported that Concurrent request log and output files are not displaying instantly but are showing up after few seconds of delay.

Same situation was faced when trying to open any manager log files. Since log/out were opening up fine, albeit with some delay, so RRA/FNDWR setup was evidently fine.

Below is the usual application flow when user try to see any log/out file.

a) when user select request out or log file temporary URL like 
http://<server_name>:<port>/OA_CGI/FNDWRR.exe?temp_id=350331520 is generated.

b) The file name and nodename are selected from the database next (fnd_concurrent_requests for .req or .out 
AND fnd_concurrent_processes for manager log files

c) Node_name in our case was showing up value like  "NODE_NAME : FNDFS_APPLTOP_<server>

d) The client takes the above string and attempts to use it as a SQL*Net connect descriptor. 
SQL*Net will attempt to resolve this descriptor  into a host and SID using a local tnsnames.ora file 
or Oracle Names. This is the $TNS_ADMIN directory

e) The FNDFS (or RRA) executable runs next. It sends the requested file to the browser.

We executed the standard concurrent program ‘FS Diagnostic Test Program’ which helped us in getting us to the right direction for this issue. The concurrent program ‘FS Diagnostic Test Program’ is useful for tracing the RRA(Report Review Agent) including FNDWRR, FNDFS. You can find What happens when you click on “View Output” or click on the “View Log” . It points which FNDFS entry is used actually by the concurrent request.

In our case at step (d) the delay was happening. Entry in application tnsnames was having entry of one bad server which was decommissioned recently. Since server was not reachable so timeout was happening causing couple seconds of delay.

We had multi node apps tier and one of the entry in the tnsnames.ora fort the FNDFS was pointing to wrong/nonexistent server.

FNDFS_APPLTOP_<server>=
        (DESCRIPTION=
            (ADDRESS_LIST=
                (ADDRESS=(PROTOCOL=tcp)(HOST=<badserver>.domain.com)(PORT=1636))
                (ADDRESS=(PROTOCOL=tcp)(HOST=<server1>.domain.com)(PORT=1636))
                (ADDRESS=(PROTOCOL=tcp)(HOST=<server2>.domain.com)(PORT=1636))
                (ADDRESS=(PROTOCOL=tcp)(HOST=<server3>.domain.com)(PORT=1636))
            )
            (CONNECT_DATA=
                (SID=FNDFS)
            )
        )

If a node is removed from the cluster then cluster must be altered to to eliminate unnecessary waits for TCP Timeout while creating new connections.

This erroneous <badserver> entry was present in FND_NODES which means that when this node was removed from the application tier, FND_NODES was not cleaned/modified to remove the entry for this server

For the time being, we backed up and modified the tnsnmaes.ora to remove all FNDFS erroneous entries.

The correct solution is to cleanup the actual source of error in FND_NODES, by cleaning using fnd_conc_clone.setup_clean and then running autoconfig on all database and application nodes.

Author
Recent Posts

Brijesh Gogia

I’m an experienced Cloud/Oracle Applications/DBA Architect with more than 15 years of full-time DBA/Architect experience. I have gained wide knowledge on Oracle and Non-Oracle software stack running on-prem and on Cloud and have worked on several big projects for multi-national companies. I enjoy working with leading-edge technology and have a passion for Cloud architecture, automation, database performance, and stability. Thankfully my work allows me time for researching new technologies (and to write about them).

Oracle EBS R12 – Concurrent request log and output files display with delay

Related posts: