Recently, while working on our Oracle E-Business Suite 12.2 environment, I encountered an interesting issue that I thought was worth sharing.
The Problem
A few of our custom concurrent programs were not completing. They were running forever, with no errors in the log files and no obvious clues in the usual diagnostic areas.
Like most DBAs would do, I checked:
-
Concurrent request log and output – nothing unusual.
-
Database session waits – nothing conclusive.
-
Trace files – no errors.
Eventually, I raised an Oracle SR to get assistance, but even after multiple attempts, we couldn’t pinpoint the issue.
The Breakthrough
Finally, I decided to take a different approach. I used the strace command on the Linux OS to trace the system calls of the concurrent program's process ID (PID).
What I saw was eye-opening — the process was generating a large number of file-related system calls, and one particular line caught my attention:
It was trying to write temporary files to a production directory path that did not exist in the cloned test server!
Root Cause
On further investigation, I discovered that the $APPLLKOR environment variable was pointing to a non-existing directory:
This was a leftover from the production environment after the test instance was cloned. Since the directory did not exist on the test server, the concurrent programs were unable to write the required temporary files and got stuck.
The Fix
I corrected the $APPLLKOR variable in the environment file to point to a valid directory on the test server. After making this change, I restarted the EBS application services.
Immediately, the concurrent programs that were stuck started completing successfully!
Key Takeaways
-
Think beyond EBS and DB logs – sometimes the issue is at the OS or filesystem level.
-
straceis a powerful tool – it can reveal exactly what the process is trying to do at the system call level. -
Post-clone checks are critical – always verify environment variables and directory structures after cloning from production.
This experience reinforced for me that troubleshooting sometimes requires going one level deeper and thinking outside the usual EBS framework.