Oracle E-Business Suite Maintenance Guide Release 12.2（Patching Procedures）

Preparing for Patching

For patches that have manual steps, the patch readme file instructs you to use Oracle Patch Application Assistant (PAA) to create customized instructions for your system. PAA consolidates and displays only the relevant manual steps for all the patches you want to apply, including steps that you have completed. It also automatically merges the contents of individual patch readme files for a merged patch.

Create Checklist of Product Functionality Disabled in Online Patching Cycle

During an online patching cycle, the following product restrictions will apply. Before you commence patching, you should therefore ensure there will be no requirement for any these actions or features until the cycle is complete.

• Payroll

  • Users will not be able to define Fast Formulas or use the Fast Formula Assistant.

  • Users will not be able to perform dynamic trigger maintenance.

  • Users will not be able to create, update, or delete US Cities.

  • Data Pump meta-mapper generator will be disabled.

  • The Japanese Balance Dimensions concurrent program will be deferred to after the cutover phase is complete.

  • Pension Calculation Setup cannot be used.

  • US localization earnings and deduction setup cannot be used.

  • Tax Withholding Rules Setup cannot be used.

  • Wage Attachment Earnings Rules Setup cannot be used.

  • Garnishment Rules Setup cannot be used.

  • Quick Paint Reports cannot be used.

  • Quantum Program Update Installer execution is unavailable.

• Order Management:

  • Creation of a new Defaulting Condition in the Attribute Defaulting Rules form is disabled, unless the same seeded condition already exists for a given attribute.

• Warehouse Management:

  • WMS Rule creation is restricted.

• Inventory:

  • Concurrent program “Generate Stock Locator Flexfield Definition for Mobile Transactions” will be disabled.

• Public Sector Financials International:

  • Users will not be able to run the following concurrent programs:

    • Subledger Security: Apply Security

    • Subledger Security: Import/Export Data Fix

• Subledger Accounting:

  • Users will not be able to Validate the Application Accounting definitions.

• Accounts Receivable:

  • Users will not be able to create new Transaction Sources.

• Incentive Compensation:

  • Transaction collection process for new mappings will not be available and any changed mapping will continue to use previous mapping rules.

  • Users will not be able to run the “Synchronize Classification Rulesets” program.

  • Users will not be able to use the “Formula Generation” feature.

  • Users will not be able to specify new formulas or changes to compensation rules.

• Oracle Demand Planning:

  • Demand plans will not be available for users.

Global Inventory Requirements

• A global (central) inventory is required for all Oracle E-Business Suite Release 12.2 application tier nodes.

• The central inventory location must be identified by the /oracle/oraInventory.loc file.

• On a shared file system, the global inventory location must be shared and used by all participating nodes.

• The use of a local inventory per Oracle E-Business Suite installation is not currently supported.

If you are using a UNIX platform, you should verify the existence and contents of the oraInst.loc file, which specifies the location of the oraInventory.loc file global inventory file.

1. Check that oraInst.loc exists in the correct directory for your platform:

   Platform	oraInst.loc Location
   Oracle Solaris SPARC (64-bit)	/var/opt/oracle
   Linux x86-64	/etc
   IBM AIX on Power Systems (64-bit)	/etc
   HP-UX-Itanium	/var/opt/oracle

2. Confirm that the contents of oraInst.loc look like this:

   inventory_loc=/oracle/oraInventory

   where /oracle/oraInventory points to the directory where the central inventory is located. This location must be writable by the user account that is to run Rapid Install.

   Incorrect permissions on oraInventory may cause issues not only with online patching (fs_clone phase), but also when installing a system with Rapid Install or cloning a system with Rapid Clone.

   Note: If your system has separate installation user accounts for the database and the applications, both users must be in the same install group (inst_group) in oraInst.loc, which will need to contain a line such as inst_group=oracle.

If the oraInst.loc file does not exist, create it in the correct directory with contents as shown above.

Set Up Secure Shell on Application Tier Nodes

In a multi-node environment, adop commands are invoked by a user on the primary node. Internally, adop uses Secure Shell (ssh) to automatically execute required patching actions on all secondary nodes. You must set up passwordless ssh connectivity from the primary node to all secondary nodes.

Note: Rapid Install and Rapid Clone set up the ssh key infrastructure.

Principles

The ssh-keygen command is used to generate a private/public key pair. The private key is for the node from where all the remote nodes will subsequently be accessible by an ssh login that requires no password. The public key must be copied to each remote node’s <User Home Dir>/.ssh directory.

In essence, the sequence is as follows:

1. The following command initiates creation of the key pair:

   $ ssh-keygen -t rsa

   Note: The <Enter> key should be pressed instead of a passphrase being entered.

2. The private key is saved in <User Home Dir>/.ssh/id_rsa.

   Important: As this read-only file is used to decrypt all correspondence encrypted with the public key, its contents must not be shared with anyone.

3. The public key is saved in <User Home Dir>/.ssh/id_rsa.pub.

4. The contents of the public key are then copied to the <User Home Dir>/.ssh /authorized_keys file on the systems you subsequently wish to ssh to without being prompted for a password.

The following example demonstrates the steps:

1. $ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/u01/user2/.ssh/id_rsa):<Enter>  Enter passphrase:<Enter> Enter same passphrase again:<Enter> Your identification has been saved in /u01/user2/.ssh/id_rsa. Your public key has been saved in /u01/user2/.ssh/id_rsa.pub. The key fingerprint is: 16:d0:e2:dd:37:2f:8e:d5:59:3e:12:9d:2f:12:1e:5a

2. $ scp -pr /u01/user2/.ssh/id_rsa.pub / user2@system1:/u01/user2/.ssh/authorized_keys user2@system1's password:<password> id_rsa.pub 100% 398 0.4KB/s 00:00

3. $ ssh user2@system1

   Note: If you receive this message, it can safely be ignored: Warning: untrusted X11 forwarding setup failed: xauth key data not generated Warning: No xauth data; using fake authentication data for X11 forwarding.

Once this has been done for the relevant operating system account on all nodes – that is, ssh can log in from the primary node to each secondary node without entering a password – so you are ready to run adop on multiple application tier nodes. It must be run on at least the master (admin) node: from there, it will attempt to contact all the other application tier nodes that are part of the same Oracle E-Business Suite instance, and will run the required steps remotely on those nodes.

Important: If you change the password for the relevant operating system account on one or more nodes, you must regenerate the ssh credentials either using the $AD_TOP>/patch/115/bin/txkRunSSHSetup.pl script, or your own native solution if you prefer.

The txkRunSSHSetup.pl script has a -help option that shows relevant usage options.

For example, a basic command to enable ssh would be:

$ perl <AD_TOP>/bin/txkRunSSHSetup.pl enablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3$ 

To verify ssh operation:

$ perl <AD_TOP>/bin/txkRunSSHSetup.pl verifyssh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3 / -invalidnodefile=<filename to report ssh verification failures> @ nodes list> 

To disable ssh:

perl <AD_TOP>/bin/txkRunSSHSetup.pl disablessh

Creating Customized Instructions for Patching Using PAA

Requirement: How do I know which manual steps associated with a patch apply to my system?

Sorting through the manual steps in a patch readme file to determine which ones apply to your system can be time-consuming. The Patch Application Assistant allows you to create a customized set of steps to that apply to your unique instance. Using the information on this list reduces the possibility of performing steps that are not necessary or that have been completed previously during the application of another patch.

When you download and unzip a patch, it delivers a static README.html file that advises you if the patch requires manual steps. If manual steps are required, you can generate a list of the steps by running a Perl script (admsi.pl) to initiate PAA. Once you have generated the list, use the PAA interface to see a full list of steps, or only those steps that apply to your system.

After successfully performing each manual step, you can record that it was completed. When applying patches in the future, this information is displayed in the PAA interface so that you can see which manual steps you have already performed.

To run PAA

1. Download the patch that you want to apply and set (source) your environment. On UNIX systems, you must also set the environment variable DISPLAY to an active and authorized display.

   For instructions on setting your environment, see: Running AD Utilities in this book.

2. Run the admsi.pl script to generate customized installation instructions.

   $ admsi.pl 

   The Oracle Patch Application Assistant welcome page appears:

   You can select:

   • View instance-specific instructions for a new patch.

   • View generic instructions as shipped by Oracle for a new patch – to view all the generic manual steps for a particular patch, including the completed steps.

   • Look at all incomplete tasks from previous patches – to view all the manual steps that have not been completed from previous patches.

3. Select View instance-specific instructions for a new patch. Enter the APPS password, and select the location where the patch is staged. Click Next.

   The Summary of Installation Instructions page appears:

   This page summarizes all the manual steps for the patch, grouped into the following categories: Preparation Tasks, Pre-Install Tasks, Apply the Patch, Post-Install Tasks, Finishing Tasks, and Additional Information. This page displays only those categories in which there are manual steps.

4. Click the plus-sign icon in each category for more detailed information. For example, if you click the plus sign icon next to Best Practices, the Preparation Tasks screen appears with the tasks suggested for preparing your system for patching.

5. After you have completed all the manual steps in a category, check the Completed box to record the completion status in the database, then click Next. If a patch that you apply in the future contains any of the same manual steps, it will be marked as completed to inform you that you do not have to perform that task again.

   After you have completed all manual steps in all categories, the system returns you to the Summary of Installation Instructions page.

   Note the column of Completed boxes that corresponds to each task in a category. Check marks appear in the boxes for which you have completed manual steps.

6. Click Save to record tasks completed in the database. Click Cancel to exit PAA.

The Online Patching Cycle

This section describes the online patching cycle from beginning to end, illustrating the actions taken in the different phases and putting into context the more detailed description of online patching in the following sections.

Important: This section is designed to be read in conjunction with the important background material provided in the Patching and Management Tools chapter of Oracle E-Business Suite Concepts.

Applying Oracle E-Business Suite patches without a significant system downtime is referred to as online patching, and a new utility, adop, is used to apply patches.

Online patching is supported by the capability of storing multiple application editions in the database, and the provision of a dual application tier file system. At any given point in time, one of these file systems is designated as ‘run’ (part of the running system) and the other is the ‘patch’ (either being patched or awaiting the start of the next patching cycle). Whichever is the current run file system appears to the user in exactly the same way as the single application tier file system did in Oracle E-Business Suite releases prior to 12.2.

Important: The existence of the dual file system has implications for patches that change the system configuration. The adop utility is required for applying software patches to the patch file system, but is not required for applying patches that change the configuration: such patches can be applied to the run file system, or optionally to the patch file system (both options are supported, with automatic synchronization taking place).

There are also implications for where general (non-patching) maintenance activities are carried out. For important information on choosing the appropriate file system to run AD tools from, refer to: Choosing the Correct File System For Maintenance Tasks in Chapter 7 of this book.

A new environment variable, $FILE_EDITION, shows the current designation of a given dual file system member. Three other new environment variables designate the root directories of the run ($RUN_BASE), patch ($PATCH_BASE), and non-editioned ($NE_BASE) file systems.

For example:

• $FILE_EDITION = patch

• $RUN_BASE = /u01/R122_EBS/fs1

• $PATCH_BASE = /u01/R122_EBS/fs2

• $NE_BASE = /u01/R122_EBS/fs_ne

When a patch is being applied, the Oracle E-Business Suite system is running in normal production mode (full functionality, with some documented exceptions) in the run edition of the file system and database. Full application functionality is retained as patch execution proceeds, until the cutover phase is reached (as described later in this section).

Important: Do not attempt to clone an Oracle E-Business Suite system while an online patching cycle is in progress.

The online patching cycle consists of a number of high level phases:

1. prepare

2. apply

3. finalize

4. cutover

5. cleanup

A high level overview of an online patching cycle would, programmatically, look like this:

# Prepare for patching: $ adop phase=prepare  # Apply patches: $ adop phase=apply patches=<patch number>  # Apply any customizations to patch edition (optional): $ . <EBS_ROOT>/EBSapps.env patch $ sqlplus apps/apps @my_custom_script_01 $ sqlplus apps/apps @my_custom_script_02 ...  # Finalize patch application: $ adop phase=finalize  # Perform cutover: $ adop phase=cutover $ . <EBS_ROOT>/EBSapps.env run  # Perform user acceptance testing via application UI  # Perform cleanup: $ adop phase=cleanup

Important Additional Points

• After an online patching cycle is started, you should not perform any configuration changes in the run edition file system. Any that are made will not be propagated, and will therefore be lost after cutover is complete.

• The prepare, apply, and fs_clone phases all require at least 10GB of free disk space. All other phases require 1GB of free space. A warning message will be displayed if less than the needed amount is available.

• Customizations are applied to the patch edition during the apply phase, normally after any Oracle E-Business Suite patches have been applied.

Special Phases

Two additional phases are provided for specialized use. Neither can be run in conjunction with any other phase. Further details of these phases are described in later sections.

• The abort phase is used to terminate a patching cycle before it is complete, and roll back any changes that have been made.

• The fs_clone phase is a separate command used to synchronize the patch file system with the run file system. Use of fs_clone is normally not required. Situations that do require fs_clone are will explicitly document that requirement.

  Note: Standard cloning (using adcfgclone.pl) cannot be used to synchronize the run and patch file systems. It can only be used for traditional cloning scenarios, for example cloning a development system to a test system. Conversely, fs_clone is only for use with adop.

The Online Patching Cycle

adop will automatically set its environment as required, but it is the user’s responsibility to set the environment correctly for any other commands that may be run. Set the run edition environment whenever executing commands that you intend to affect the run edition.

For example:

$ ssh-keygen -t rsa

0

Set the patch edition environment whenever you intend to execute commands that affect the patch edition.

For example:

$ ssh-keygen -t rsa

1

The adop tool executes non-interactively, executing the specified phase or phases in order. In a multi-node deployment, adop is only executed by the user on the master node: internally, adop will use ssh remote execution to run required actions on all secondary nodes automatically. In addition, adop can be used to generate reports about patching operations in the environment.

adop is typically run as follows:

$ ssh-keygen -t rsa

2

The phase parameter accepts the following phase names. These names can be specified individually, or (except where otherwise noted) with other phase names in a comma-separated list:

• prepare – Prepares the environment for patching.

• apply – Applies the specified patches to the environment.

• finalize – Performs any final steps required to make the system ready for cutover.

• cutover – Shuts down application tier services, makes the patch edition the new run edition, and then restarts application tier services. This is the only phase the involves a brief downtime.

• cleanup – Removes obsolete code and data from old editions.

• abort – Aborts the online patching cycle by dropping the database patch edition. This phase cannot be specified with any other phase.

• fs_clone – Recreates the patch edition file system as an exact copy of the run edition file system. This phase cannot be specified with any other phase. Use of fs_clone is normally not required. Situations that do require fs_clone are will explicitly document that requirement. If running this phase, ensure that your current working directory is not within the patch edition file system.

Online Patching Cycle Steps – Prepare Phase

This section describes the principles of adop operation in the prepare phase, followed by the steps you take to run this phase.

Note: The exact actions taken during the prepare phase are context-dependent: for example, the first time it is ever run on a system; when it is run after an apply phase has been aborted; and when it has been run after cutover.

Principles

In the prepare phase, adop:

1. Checks whether to perform a cleanup, which will be needed if the user failed to invoke cleanup after the cutover phase of a previous online patching cycle.

2. Validates system configuration to ensure that the system is ready to start an online patching cycle.

3. Checks to see if the database is prepared for online patching:

   • Checks if the database user is edition-enabled (at least one user, the PL/SQL API, will return ‘true’). If no edition-enabled users exist, adop displays a message to that effect.

   • Checks to see if the patch service has been created. adop requires that a special database service exists for the purpose of connecting to the patch edition. This service is created automatically, but its continued existence is validated on each prepare.

   • Checks to see if logon trigger exists and is enabled. If the logon trigger is missing or the patch service has not been created, adop will automatically try to fix the issue so that it can proceed. If it cannot do so, it will exit with an error message.

4. Checks the file system, using the TXK script $AD_TOP/patch/115/bin/txkADOPPreparePhaseSanityCheck.pl. This script checks for the file system space, database connections, and so on.

5. Produces a report showing information about the most important tablespaces is generated. This report is created in $APPL_TOP/admin/$TWO_TASK/out.

6. Checks for the existence of the "Online Patching In Progress" (ADZDPATCH) concurrent program. This program prevents certain predefined concurrent programs from being started, and as such needs to be active while a patching cycle is in progress (that is, while a database patch edition exists). The flow of control is as follows.

   1. If the ADZDPATCH program exists, a message to this effect is displayed to the user. If it does not exist, it is started.

   2. The status of ADZDPATCH is determined. If it is pending, it may be waiting for an incompatible program to finish. At that point, its status will change to running, and it will allow the prepare phase to proceed. A message to this effect is displayed to the user.

   3. The next stage depends on whether the concurrent managers are running:

      1. If the concurrent managers are all down, the prepare phase continues, with ADZDPATCH entering a status of pending (with the highest priority) until the managers are started.

      2. If the concurrent managers are partially up, but there is no manager defined that can run ADZDPATCH, then the prepare phase will exit with an error.

      3. If the concurrent managers are up, and there is one defined that can run ADZDPATCH, processing will loop until ADZDPATCH changes status from pending to running (that is to say, as noted in Step 2, no incompatible programs are found). The prepare phase then continues.

   Note: ADZDPATCH is cancelled when the cutover phase is complete.

7. Invokes the TXK script $AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl to synchronize the patches which have been applied to the run appltop, but not the patch APPL_TOP. The script depends on the adop repository for patches that have been applied on the run APPL_TOP but not the patch APPL_TOP.

8. Checks the database for the existence of a patch edition, and creates one if it does not find one.

9. Calls the $AD_TOP/patch/115/bin/txkADOPPreparePhaseSanityCheck.pl script again to confirm that the database connection to the patch edition is working.

If any of these checks fail, adop will exit with an error message.

Steps

You perform the following steps in the prepare phase.

Important: Before you run the prepare phase, you should ensure that the following space requirements are met:

• SYSTEM tablespace has a minimum of 25 GB of free space

• APPS_TS_SEED tablespace has a minimum of 5 GB of free space

You can check the available space by running the $AD_TOP/sql/ADZDSHOWTS.sql script. For information about increasing the size of a tablespace, refer to the Altering and Maintaining Tablespaces section in the Managing Tablespaces chapter of Oracle Database Administrator’s Guide.

1. Set the environment by executing (sourcing) the run file system environment file:

   $ ssh-keygen -t rsa

   3

   For more information, see Setting the Environment in Running AD Utilities

   You can confirm that the environment is properly set by examining the relevant environment variables:

   $ ssh-keygen -t rsa

   4

   If you had sourced the incorrect environment file (i.e. from the patch file system), the environment variables would show as:

   $ ssh-keygen -t rsa

   5

2. Download patches to be applied and place then in the $PATCH_TOP directory of your system. This directory is pre-created by the install in the non-editioned file system (fs_ne) and should not be changed.

   Important: On a multi-node system with non-shared file systems, you must copy the patch files to each separate $PATCH_TOP directory, so that the patch files are available from the same location on all nodes.

3. Unzip the patch:

   $ ssh-keygen -t rsa

   6

4. Prepare the system for patching by running the following command to start a new patching cycle:

   $ ssh-keygen -t rsa

   7

   In this step, the patch file system APPL_TOP is synchronized with the run file system APPL_TOP. This can be done by either of two methods, the first of which is the default:

   • Method 1 – Identify the patches that were applied to the run APPL_TOP and apply them to the patch APPL_TOP. The following steps are performed automatically:

     1. The list of patches that need to be applied to the patch APPL_TOP are identified from the database.

     2. The merged patches are applied using the adop utility.

     This lightweight process is the preferred approach. The adop tool identifies the delta patches to be applied, and applies them silently to the current patch APPL_TOP. As this procedure only requires the application of delta patches, it is relatively fast compared to Method 2 (below).

   • Method 2 – Create a new patch file system by cloning the run file system.

     This method is useful if the APPL_TOPs have become very unsynchronized (meaning that there would be a large number of delta patches to apply). It is a heavyweight process, taking a backup of the entire current patch APPL_TOP and then cloning the run APPL_TOP to create a new patch APPL_TOP. A total of least 75 GB of free disk space is required. Also, you will need at least 25 GB of free space in your temporary directory (typically /tmp). This method is more resource intensive, and will only be used when the state of the patch file system is unknown.

     The steps are as follows:

     1. The fs_clone phase is run using the following command:

        $ ssh-keygen -t rsa

        8

        This command must be invoked from the run file system, before the next prepare phase is run.

     Note: If an fs_clone operation fails, you can rerun it with the option force=yes to restart it from the beginning (with the same session ID), or force=no to restart it from the point where it failed.

   Windows users should refer to their platform-specific release notes for restrictions that currently apply when running fs_clone.

   Using the skipsyncerror parameter

   In some circumstances, the delta-style (incremental) synchronization method may fail when applying a series of patches to the patch edition. This can happen if the previous patching cycle included patches that failed to apply correctly, and was followed by subsequent patches that corrected the issue.

   The skipsyncerror parameter enables you to specify that you expect any synchronization errors in the prepare phase to be fixed automatically in the synchronization that takes place with subsequent patches.

   If the value of the parameter is passed as ‘yes’, the first patch to be synchronized will be done with the ‘autoskip’ flag set.

   Important: It is your responsibility to check the log files and correct any errors in the subsequent apply phase, or to confirm that synchronization with subsequent patches resolved the issue.

   An example of using this parameter would be as follows.

   1. You run adop phase=prepare.

   2. The phase fails with an error when trying to synchronize the run and patch file systems. That is, the attempt to synchronize a patch fails, but it is known that a subsequent patch will correct the problem.

   3. You examine the log files and conclude that the synchronization errors will be fixed automatically in the synchronization that takes place with subsequent patches.

   4. You run the command adop phase=prepare skipsyncerror=yes to restart the prepare phase. This time, application of the patch that failed in the previous prepare will be retried with the ‘autoskip’ flag set.

   Alternatively, if you are not confident that the error will be fixed (for example, you cannot identify the cause from examination of the log files), you should:

   1. Run the command adop phase=abort

   2. Run the command adop phase=fs_clone

   3. Rerun the command adop phase=prepare

   Synchronizing Customizations

   The default delta-style (incremental) method of file system synchronization handles official patches but will not synchronize any manually applied customizations. Examples of patching actions that are not synchronized by default include:

   • Compiling user-defined JSPs

   • Copying some third-party libraries

   • Copying and compiling user-defined concurrent programs

   • Copying and generating user-defined forms

   To include custom patching actions in the default file system synchronization, you must include the required commands in the Custom Synchronization Driver, $APPL_TOP_NE/ad/custom/adop_sync.drv. You will add your customizations to the following section of the file:

   $ ssh-keygen -t rsa

   9

   All the actions defined in this file will be performed by adop automatically during the prepare phase. Be aware that there are two categories of custom command in adop_sync.drv: those that are run one time only, and those that are run at each file system synchronization (during the adop prepare phase).

   Important: The adop_sync.drv file is not currently reset to its template file at any point. Consequently, after cutover (and before the next prepare phase), you should review the contents of adop_sync.drv and ensure the requirementns for your custom commands continue to be met.

   This is only an outline of the steps you need to take to preserve customizations. For full details, refer to Oracle E-Business Suite Developer’s Guide

   Prepare Phase in Multi-Node Environments

   In a multi-node environment, one application tier node will be designated as the primary node. This is the node where the Admin Server is located, and will usually also be the node that runs Oracle HTTP Server. All other application tier nodes are designated as secondary nodes.

   In a multi-node environment, you must enable ssh from the primary node to all secondary nodes to permit adop remote invocation. For ssh setup steps, refer to Set Up Secure Shell on Application Tier Nodes. In a multi-node environment, adop commands are always run from the primary node only. adop executes required patching actions on the secondary nodes automatically via remote invocation.

   If a node unexpectedly becomes inaccessible via ssh, it will be abandoned by adop, and the appropriate further actions taken. Consider a scenario where the adop phase=prepare command is run in a system with ten application tier nodes. The command is successful on nine nodes, but fails on the tenth. In such a case, adop will identify the services enabled on nodes 1-9. If they are sufficient for Oracle E-Business Suite to continue to run normally, adop will mark node 10 as abandoned and then proceed with its patching actions. If they are not sufficient, adop will proceed no further.

Online Patching Cycle Steps – Apply Phase

This section describes the principles of adop operation in the apply phase, followed by the steps you take to run this phase.

Principles

In the apply phase, adop applies the specified patches to the system. In an online patching cycle, patches are applied to the patch edition of the database and file system.

Steps

In this phase, you will apply the patches that you designated for inclusion in this patching cycle. You can apply as many patches as you want per patching cycle. By default, a list of patches is applied one at a time, in the specified order. If you specify the merge option "merge=yes", the listed patches will automatically be merged and the resulting merged patch will be applied.

The following example will illustrate the options.

$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/u01/user2/.ssh/id_rsa):<Enter>  Enter passphrase:<Enter> Enter same passphrase again:<Enter> Your identification has been saved in /u01/user2/.ssh/id_rsa. Your public key has been saved in /u01/user2/.ssh/id_rsa.pub. The key fingerprint is: 16:d0:e2:dd:37:2f:8e:d5:59:3e:12:9d:2f:12:1e:5a

0

This uses the input_file that was mentioned earlier in this section.

An example input_filemight liook like this:

$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/u01/user2/.ssh/id_rsa):<Enter>  Enter passphrase:<Enter> Enter same passphrase again:<Enter> Your identification has been saved in /u01/user2/.ssh/id_rsa. Your public key has been saved in /u01/user2/.ssh/id_rsa.pub. The key fingerprint is: 16:d0:e2:dd:37:2f:8e:d5:59:3e:12:9d:2f:12:1e:5a

1

Reports under the $APPL_TOP/admin/<SID>/out directory can help you identify and diagnose problems that may occur in the online patching cycle. These reports list the proposed changes to database objects, both new and modified.

The key files to examine are:

• $APPL_TOP/admin/<SID>/out/adzdcmped.out

• $APPL_TOP/admin/<SID>/log/u<patch_number>.log

Note: For merged patches, the log file name will be derived from the timestamp when merging was performed.

Using the analytics parameter in apply

If you want to use the analytics parameter (see adop Parameters) with the apply phase, enter the command:

$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/u01/user2/.ssh/id_rsa):<Enter>  Enter passphrase:<Enter> Enter same passphrase again:<Enter> Your identification has been saved in /u01/user2/.ssh/id_rsa. Your public key has been saved in /u01/user2/.ssh/id_rsa.pub. The key fingerprint is: 16:d0:e2:dd:37:2f:8e:d5:59:3e:12:9d:2f:12:1e:5a

2

Specifying this option will cause adop to run the following scripts and generate the associated output files (reports):

• ADZDCMPED.sql – This script is used to display the differences between the run and patch editions, including new and changed objects. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>/adzdcmped.out.

• ADZDSHOWED.sql – This script is used to display the editions in the system. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowed.out.

• ADZDSHOWOBJS.sql – This script is used to display the summary of editioned objects per edition. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowobjs.out

• ADZDSHOWSM.sql – This script is used to display the status report for the seed data manager. The output file location is: /u01/R122_EBS/fs_ne/EBSapps/log/adop/<adop_sessionID>/<apply_directory>/<context_name>adzdshowsm.out

Note: The analytics parameter should only be used when required, because of the extra processing needed.

Online Patching Cycle Steps – Finalize Phase

The finalize phase is used to perform any remaining processing that is needed to ensure the system is ready for the fastest possible cutover.

The key actions of the finalize phase are:

1. Pre-compute DDL that needs to be run at cutover.

2. Compile all invalid objects.

3. Validate that the system is ready for cutover.

4. If finalize_mode=full, compute statistics for key data dictionary tables for improved performance.

Run the finalize phase as follows:

$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/u01/user2/.ssh/id_rsa):<Enter>  Enter passphrase:<Enter> Enter same passphrase again:<Enter> Your identification has been saved in /u01/user2/.ssh/id_rsa. Your public key has been saved in /u01/user2/.ssh/id_rsa.pub. The key fingerprint is: 16:d0:e2:dd:37:2f:8e:d5:59:3e:12:9d:2f:12:1e:5a

3

Online Patching Cycle Steps – Cutover Phase

This section describes the principles of adop operation in the cutover phase, followed by the manual steps you can optionally execute to run this phase.

Important: You must ensure that no users remain on the system during cutover, as there will be a short downtime period while the application tier services are automatically shut down and restarted. In addition, any third-party processes connected to the old run edition of the database should be shut down, otherwise they will be terminated automatically. You can, however, defer running cutover until a time which will cause minimal disruption to users.

Principles

The key actions performed in the cutover phase are:

1. Shut down internal concurrent manager: adop will signal the internal concurrent manager to shut down. If any concurrent requests are in progress, this step will wait for those requests to finish before proceeding. The system is still available to users during this waiting period. If you do not wish to wait indefinitely for concurrent requests to finish, specify the cm_wait parameter "cm_wait=<maximum_minutes_to_wait>".

2. Shut down application tier services: All application tier services are brought down. During this period, the system is unavailable to users.

3. Cutover database: Promote patch database edition to become the new run database edition, using adzdpmgr.pl script.

4. Cutover file system: Promote patch file system to become the new run file system, switching the $FILE_EDITION values in the patch and run enviroments. The current patch APPL_TOP becomes the new run APPL_TOP, and the current run APPL_TOP becomes the new patch APPL_TOP.

5. Retire old editions: Disable access to old database editions.

6. Terminate old database sessions: Terminate any database connections to the old run edition of the database.

7. Start application tier services: Application tier services are restarted, on the new run edition. The system is now available again to users.

Note: The adop utility invokes the TXK script txkADOPCutOverPhaseCtrlScript.pl to perform tasks 1, 2, 3, 5, and 6. Task 4 is performed by AutoConfig.

1. Before running the cutover command, ensure you are ready to commit to application of the selected patches. Once cutover is complete, it is not possible to revert to the previous edition.

   Note: Cutover will take longer if it has to wait for long-running concurrent processes to complete. In such a case, you can expect to see an informational message of the form:

   $ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/u01/user2/.ssh/id_rsa):<Enter>  Enter passphrase:<Enter> Enter same passphrase again:<Enter> Your identification has been saved in /u01/user2/.ssh/id_rsa. Your public key has been saved in /u01/user2/.ssh/id_rsa.pub. The key fingerprint is: 16:d0:e2:dd:37:2f:8e:d5:59:3e:12:9d:2f:12:1e:5a

   4

   If you do not want to wait for in-progress concurrent requests to finish normally, you can terminate the internal concurrent manager by executing the adcmctl.sh abort command from a different shell.

2. In most cases (but see below for the important exception of analytics), you then proceed to execute cutover with the command:

   $ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/u01/user2/.ssh/id_rsa):<Enter>  Enter passphrase:<Enter> Enter same passphrase again:<Enter> Your identification has been saved in /u01/user2/.ssh/id_rsa. Your public key has been saved in /u01/user2/.ssh/id_rsa.pub. The key fingerprint is: 16:d0:e2:dd:37:2f:8e:d5:59:3e:12:9d:2f:12:1e:5a

   5

   This will promote the patch edition to be the new run edition, as well as switching the patch and run labels on the file systems (and thereby, as noted above, changing the patch file system to be the new run file system and the run file system to be the new patch file system).

   Important: In the event of problems with the cutover phase, refer to My Oracle Support Knowledge Document 1584097.1, Oracle E-Business Suite Release 12.2: Backup and Recovery Guidelines For Online Patching Cutover.

   Deferring Application Tier Restart at Cutover

   In some cases, you may need to perform additional manual steps after cutover but before restarting the application tier services. If this is the case, you can supply an additional parameter to the cutover command that causes the application services to remain shut down:

   $ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/u01/user2/.ssh/id_rsa):<Enter>  Enter passphrase:<Enter> Enter same passphrase again:<Enter> Your identification has been saved in /u01/user2/.ssh/id_rsa. Your public key has been saved in /u01/user2/.ssh/id_rsa.pub. The key fingerprint is: 16:d0:e2:dd:37:2f:8e:d5:59:3e:12:9d:2f:12:1e:5a

   6

   With this parameter, cutover will complete without restarting the application tier services. You can perform any additional steps that require the services to be shut down, and then start the application tier services manually using the adstrtal.sh script.

   You must then also run the steps in the following section, Patching the Database Tier:

   Patching the Database Tier

   These steps are performed post-cutover.

   1. On the application tier, as the applmgr user:

      1. Change directory to the run file system $APPL_TOP and source your environment file.

      2. Run the following command:

         $ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/u01/user2/.ssh/id_rsa):<Enter>  Enter passphrase:<Enter> Enter same passphrase again:<Enter> Your identification has been saved in /u01/user2/.ssh/id_rsa. Your public key has been saved in /u01/user2/.ssh/id_rsa.pub. The key fingerprint is: 16:d0:e2:dd:37:2f:8e:d5:59:3e:12:9d:2f:12:1e:5a

         7

         This will create the appsutil.zip file in <INST_TOP>/admin/out.

   2. On the database tier, as the oracle user:

      Copy or ftp the appsutil.zip file to the RDBMS_ORACLE_HOME, then run the following commands:

      $ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/u01/user2/.ssh/id_rsa):<Enter>  Enter passphrase:<Enter> Enter same passphrase again:<Enter> Your identification has been saved in /u01/user2/.ssh/id_rsa. Your public key has been saved in /u01/user2/.ssh/id_rsa.pub. The key fingerprint is: 16:d0:e2:dd:37:2f:8e:d5:59:3e:12:9d:2f:12:1e:5a

      8

   3. Run AutoConfig on the database tier.

   4. Run AutoConfig on the run file system of each application tier node.

   5. Start the application tier services.

   JAR Files and Cutover

   In an online patching cycle, the requisite JAR files are initially stored in the $APPL_TOP/admin/<SID>/out directory, and then uploaded into the database during the cutover phase. Therefore, the out directory must not be deleted at least until cutover is complete.

Online Patching Cycle Steps – Cleanup Phase

This section describes the principles of adop operation in the cleanup phase, followed by the steps performed in this phase.

Important: If you fail to run the cleanup phase explicitly, it will be run automatically on the next prepare cycle, but this will cause a delay in starting your next online patching cycle.

Principles

Various actions are performed during cleanup, including dropping (removing) obsolete:

• Crossedition triggers

• Seed data

• Editioned code objects (covered objects)

• Indexes

• Columns

• Editions

Steps

1. Cleanup is performed with the command:

   $ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/u01/user2/.ssh/id_rsa):<Enter>  Enter passphrase:<Enter> Enter same passphrase again:<Enter> Your identification has been saved in /u01/user2/.ssh/id_rsa. Your public key has been saved in /u01/user2/.ssh/id_rsa.pub. The key fingerprint is: 16:d0:e2:dd:37:2f:8e:d5:59:3e:12:9d:2f:12:1e:5a

   9

   The adop parameter cleanup_mode provides control of cleanup processing:

   • cleanup_mode=quick – Performs minimum cleanup, including removal of obsolete crossedition triggers and seed data.

   • cleanup_mode=standard – Does the same as quick mode, and also drops (removes) obsolete editioned code objects (covered objects). This is the default mode , so does not need to be specified.

   • cleanup_mode=full – Performs maximum cleanup, which drops all obsolete code and data from earlier editions

   Choosing the Cleanup Mode

   Generally, you can use standard mode (the default). You might want to use the other modes in the following circumstances:

   • Use quick cleanup when you need to start the next patching cycle as soon as possible. For example, if you want to start a new patching cycle right away, but have not yet run cleanup from the previous patching cycle, you can use quick cleanup mode to complete the essential cleanup tasks as fast as possible.

   • Use full cleanup when you want to recover the maximum amount of space in the database. If you have run a large number of patching cycles, or applied a very large patch such as a rollup, significant space may be consumed by obsolete table columns and recovered by running a full cleanup. A full cleanup should only be performed when there is no immediate need to start a new patching cycle.

   Using the analytics parameter in cleanup

   If you want to use the analytics parameter (see adop Parameters) with the cleanup phase, enter the command:

   $ scp -pr /u01/user2/.ssh/id_rsa.pub / user2@system1:/u01/user2/.ssh/authorized_keys user2@system1's password:<password> id_rsa.pub 100% 398 0.4KB/s 00:00

   0

   Specifying this option will cause adop to run the following script and generate the associated output file (report):

   • ADZDCLEANUPRP.sql – This script is used to display the display the cleanup status. The output file location is: $NE_BASE/EBSapps/log/adop/<adop_sessionID>/<cleanup_directory>/<context_name>/adzdcleanuprp.out.

   Note: The analytics parameter should only be used when required, because of the extra processing needed.

Online Patching Cycle Steps – Abort Phase

If for some reason either the prepare or apply phase failed or gave problems, you can abort the patching cycle at either of these points by running the following command:

$ scp -pr /u01/user2/.ssh/id_rsa.pub / user2@system1:/u01/user2/.ssh/authorized_keys user2@system1's password:<password> id_rsa.pub 100% 398 0.4KB/s 00:00

1

In the abort phase, adop does the following:

1. Checks that there is an in-progress online patching cycle, i.e. whether the abort call is valid.

2. Checks for the existence of a patch edition and drops one if it exists.

3. Cancels the ADZDPATCH concurrent program, if it is running.

4. Deletes the rows inserted for the pending session ID from the ad_adop_sessions and ad_adop_session_patches tables.

Be aware of the following important points:

• After running abort, you must always run a full cleanup (with the command adop phase=cleanup cleanup_mode=full). This will remove any columns that were added by the patch but are no longer needed because of the abort. If they are not removed, they may cause problems in a later patching cycle.

• If any attempt was made to apply patches to the patch edition, after abort you must run the fs_clone phase (adop phase=fs_clone) to recreate the patch file system.

This section covers a variety of tasks that may apply either to individual online patching operations, or to your system setup as a whole. Diagnostic, troubleshooting, and reporting features are also described.

Manual Post-Installation Tasks

Traditionally, some patches have associated post-installation tasks, including recompilation of invalid packages, regenerating JAR files, and running AutoConfig. In an online patching environment such as Release 12.2 such tasks will normally be performed automatically in the apply phase.

If a post-installation patch step mentions any tasks that need to be performed explicitly, where they are run from depends on the type of patching:

• In a normal online patching cycle, the steps should be executed from the patch file system after the apply phase.

• If the patch is being applied in hotpatch mode or downtime mode, the steps should be executed from the run file system after the apply phase.

Considerations When Applying Patches That Have Already Been Applied

As mentioned in the adop Options section of Chapter 2, if you try to apply a patch that has already been applied and do not specify the forceapply parameter, adop will display an error like this:

$ scp -pr /u01/user2/.ssh/id_rsa.pub / user2@system1:/u01/user2/.ssh/authorized_keys user2@system1's password:<password> id_rsa.pub 100% 398 0.4KB/s 00:00

2

There are two more scenarios that may apply in this kind of situation.

• When a failed patch session is restarted with abandon=no, restart=yes, the patches applied in current adop session will not be applied even if options=forceapply is specified. For example, you run the command adop phase=apply options=forceapply patches=1111,2222 , and application of patch 1111 is successful but patch 2222 fails. After correcting the problem, you try to rerun adop with the command adop phase=apply options=forceapply patches=1111,2222 abandon=no, restart=yes. In this example, patch 1111 would be skipped as it had successfully been applied, and application of patch 2222 would resume. If you wanted to apply patch 1111 again, you would need to specify abandon=yes, restart=no.

• If you apply multiple patches with merge=yes , and you do not specify options=forceapply, the patches will be applied only if at least one of the patches has not been successfully applied before.

  Note: This check will be performed for AD and non-AD patches separately, as adop applies these two categories of patch in different sessions.

Configuration Management and Online Patching

The following guidelines apply to making configuration changes to Oracle E-Business Suite in the context of online patching. They particularly apply to the technology stack and application components that reside in the file system.

Note: For specific instructions on how to patch technology stack components, refer to My Oracle Support Knowledge Document 1355068.1, Oracle E-Business Suite 12.2 Patching Technology Components Guide.

The two basic scenarios are online and offline configuration changes. Each will be considered in turn.

Online configuration changes are performed within the context of an online patching cycle. This is the recommended strategy.

First, you prepare your system by running the adop phase=prepare command. You then make the desired configuration changes to the patch file system. They may include:

• Oracle WebLogic Server configuration changes

• HTTP Server configuration changes

• File system changes performed by the AD utilities

After making the configuration changes, you must run the command adop phase=cutover to promote them.

You must also run the command adop phase=fs_clone to propagate the configuration changes to the secondary file system.

Offline configuration changes are applied directly to the run file system, outside an online patching cycle.You can use the adop -status command to verify that no patching cycle is currently active. After making the desired configuration changes, you must explicitly run the adop phase=fs_clone command to propagate the changes to the patch file system.

Important: This offline scenario will require a period of downtime for users.

Customizing Patch File System Backup Count

A new AutoConfig context variable, s_fs_backup_count, is used to specify the ‘Patch File System Backup Count’. This variable denotes the number of backups of the patch file system that are to be preserved by adop. The variable is used during the fs_clone phase, where the existing patch file system is backed up before it is recreated from the run file system.

Valid values for the s_fs_backup_count variable are 0-9. A value of 0 (the default) will not preserve any patch file system backups. A value of 1 will preserve the latest patch file system backup, a value of 2 will preserve the latest two backups, and so on. You can normally leave the setting at the default value of 0 (no backup), as the patch edition file system can be fully recreated from the run edition file system by fs_clone.

Note: For more information about AutoConfig, see Technical Configuration chapter of Oracle E-Business Suite Setup Guide.

Requirements When Running Oracle HTTP Server on a Privileged Port

On a UNIX system the TCP/IP port numbers below 1024 are special in that only processes with root privileges are allowed to listen on those ports. If you have configured Oracle HTTP Server to run on a privileged port, you must perform the following additional steps when running an online patching cycle. These steps are required for both SSL and non-SSL privileged ports.

• Before running the prepare phase or the fs_clone phase, you must run the following commands as the root user on both the run file system and the patch file system:

  $ scp -pr /u01/user2/.ssh/id_rsa.pub / user2@system1:/u01/user2/.ssh/authorized_keys user2@system1's password:<password> id_rsa.pub 100% 398 0.4KB/s 00:00

  3

• After running the prepare phase or the fs_clone phase, you must run the following commands as the root user on the patch file system:

  $ scp -pr /u01/user2/.ssh/id_rsa.pub / user2@system1:/u01/user2/.ssh/authorized_keys user2@system1's password:<password> id_rsa.pub 100% 398 0.4KB/s 00:00

  4

For more information, see: Starting Oracle HTTP Server on a Privileged Port, Oracle Fusion Middleware Administrator’s Guide for Oracle HTTP Server and Running Oracle HTTP Server on a Privileged Port, Oracle E-Business Suite Setup Guide.

Integrating Your Custom Tasks Into the Online Patching Cycle

You may have business-specific tasks specific that need to be performed before, during or after a patching cycle. Support for such tasks is currently provided by callout points at the begining and end of the cutover phase of the online patching cycle. This support will be extended in future releases of Oracle E-Business Suite.

Diagnostics and Troubleshooting

This section describes a tool that helps you locate errors and warnings in your adop log files, and goes on to list some common issues that can occur.

The Online Patching Log Analyzer Utility

This utility analyzes adop log directories for errors and warnings, and displays messages to help the user quickly identify any problems that may have occurred during an adop run. It thereby offers an alternative to reviewing log files manually.

The Log Analyzer utility can be run without options, to scan all log directories of the latest adop session for errors:

$ scp -pr /u01/user2/.ssh/id_rsa.pub / user2@system1:/u01/user2/.ssh/authorized_keys user2@system1's password:<password> id_rsa.pub 100% 398 0.4KB/s 00:00

5

The utility can also be run with various options, as shown in the following list.

• To scan log directories relating to the latest run of adop in the latest session:

  $ scp -pr /u01/user2/.ssh/id_rsa.pub / user2@system1:/u01/user2/.ssh/authorized_keys user2@system1's password:<password> id_rsa.pub 100% 398 0.4KB/s 00:00

  6

• To scan log directories relating to the latest run of the specified phase, in the latest session:

  $ scp -pr /u01/user2/.ssh/id_rsa.pub / user2@system1:/u01/user2/.ssh/authorized_keys user2@system1's password:<password> id_rsa.pub 100% 398 0.4KB/s 00:00

  7

• To scan all log directories of a given session (represented by a session_id) for errors:

  $ scp -pr /u01/user2/.ssh/id_rsa.pub / user2@system1:/u01/user2/.ssh/authorized_keys user2@system1's password:<password> id_rsa.pub 100% 398 0.4KB/s 00:00

  8

• To scan a specific log directory for errors:

  $ scp -pr /u01/user2/.ssh/id_rsa.pub / user2@system1:/u01/user2/.ssh/authorized_keys user2@system1's password:<password> id_rsa.pub 100% 398 0.4KB/s 00:00

  9

• To see a complete list of supported parameters:

  $ ssh user2@system1

  0

Common Issues

This section describes known issues and common problems with online patch application.

• Defaults File Becomes Corrupt

  The system-generated defaults file is located here:

  $ ssh user2@system1

  1

  If this file becomes corrupt, running AutoConfig will automatically instantiate a new copy.

• Cutover Fails

  If you need general assistance with recovering from a cutover failure, refer to My Oracle Support Knowledge Document 1584097.1, Oracle E-Business Suite Release 12.2: Backup and Recovery Guidelines For Online Patching Cutover

  If you attempt to resume a failed session after cutover exits with cutover_status=3, you may receive an ‘Invalid Credentials’ error. This will be because the database patch edition has already been promoted to be the new run edition. To resume and complete cutover successfully, run the command:

  $ ssh user2@system1

  2

  The cutover operation is the most critical phase of an online patching cycle. Although other adop operations can be left to run unattended, you should carefully monitor the progress of cutover, so that you can respond quickly in case of any issues. If cutover fails to complete, check log messages for any problems that may require correction. Then try executing the cutover command again. When cutover is re-executed after a previous failure, adop will restart cutover processing at the failure point for any nodes that did not complete, and the processing may be successful this time.

• Concurrent Processing Log Files Cannot Be Created

  If you have chosen to set the $APPLLDM environment variable to ‘product’ (so the concurrent processing log and out files will be written to product-specific directories under $APPLCSF), you may experience a known issue where concurrent processing log files cannot be created. This will be fixed in a future release. In Release 12.2.0, the workaround is to create the following directories under $APPLCSF: sqlap/log, sqlap/out, sqlgl/log, sqlgl/out, ofa/log, and ofa/out.

  Note: For more information on options for setting $APPLLDM, refer to the Patching and Management Tools chapter of Oracle E-Business Suite Concepts.

• Nodes Are Abandoned

  In a multi-node environment, the user executes adop commands only on the primary node. The primary instance of adop then uses ssh remote invocation to execute patching actions on all secondary nodes. At the end of each primary node adop command, a summary report is produced showing the status of the command on each secondary node.

  For example:

  $ ssh user2@system1

  3

  If a secondary node fails to complete a phase, you should investigate the related log messages. You may be able to retry running the phase by invoking adop again, this time from the primary node. When you retry a partially failed command, adop will determine which secondary nodes have not yet completed the phase, and only retry the command on the failed nodes.

  If a secondary node fails and the issue cannot be easily resolved, you may be able to continue adop processing by invoking adop from the primary node in the next phase. When this happens, adop will display a confirmation warning:

  $ ssh user2@system1

  4

  If you respond ‘y’ to the above prompt, and all the essential services are available on completed nodes, adop will ignore any failed nodes and continue processing with the remaining available nodes. If an essential service is not available on any of the remaining available nodes, adop cannot continue and will display the following error:

  $ ssh user2@system1

  5

  Once the cutover phase completes on the primary node, adop will mark all failed nodes as ‘ABANDONED’.

  Note: A primary node can never be a abandoned node.

  Abandoned nodes must either be removed from the system, or recloned (copied) from an available node. If you start a new adop session (prepare, apply, fs_clone) while abandoned nodes are present in the system, adop will display an error:

  $ ssh user2@system1

  6

  For further information, refer to the above-mentioned My Oracle Support Knowledge Document 1677498.1, How to Restore An Abandoned Node in Oracle E-Business Suite Release 12.2.

Reporting

To help diagnose issues, or simply gain knowledge about the status of your system, you can run the Online Patching Diagnostic Reports utility, $AD_TOP/bin/adopreports.sh.

The adopreports utility is invoked by entering the command:

$ ssh user2@system1

7

This displays the adopreports Main Menu:

$ ssh user2@system1

8

Choosing option 1 from the Main Menu displays the Run Edition Reports Sub Menu:

$ ssh user2@system1

9

Choosing option 2 from the Main Menu displays the Patch Edition Reports Sub Menu:

$ perl <AD_TOP>/bin/txkRunSSHSetup.pl enablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3$ 

0

Choosing option 3 from the Main Menu displays the Other Generic Reports Sub Menu:

$ perl <AD_TOP>/bin/txkRunSSHSetup.pl enablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3$ 

1

Patching HRMS Legislative Data

Special instructions apply to installation of the HRMS Legislative Data patch. These include adop options that will:

• Prevent attempts to synchronize the run and patch file systems when applying a patch file (in this case hrglobal.drv) that includes database operations only.

• Ensure that adop will proceed with the relevant patching request, instead of deeming the upgrade to have been installed already.

Important: Before applying the HRMS Legislative Data patch, refer to the full instructions provided in My Oracle Support Knowledge Document 1469456.1, Datainstall and HR Global Application: 12.2 Specifics.

Patching NLS Systems

These patching procedures apply regardless of whether you are running American English (US) and one additional language, or American English (US) and several additional languages. If your system uses multiple languages, you can use AD Merge Patch to create merged patches in whichever of the following ways suits you best:

• A single, merged patch that contains all languages (including US English).

• One merged patch for US English and a second merged patch for all other languages.

• A separate merged patch for each language.

Before the introduction of online patching, the choice of which strategy to follow largely depended on the downtime that was acceptable: for example, the first option was straightforward but required the greatest downtime, and the third was the most complex but allowed users of a particular language to resume their work as soon as the relevant patch was applied. The second option often provided the best compromise between easy application and minimum downtime. Now, however, use of online patching means that downtime is greatly reduced as a factor when determining the strategy that most closely suits the organization’s needs

When merging multiple language patches, AD Merge Patch converts the character set according to the NLS_LANG variable in the Oracle E-Business Suite environment file. If you changed your character set since the initial installation, you might need to update the NLS_LANG variable. If this variable is not set properly, run AutoConfig from Oracle Applications Manager to update the Oracle E-Business Suite context with the correct character set information, then run the appropriate AutoConfig command to recreate the Oracle E-Business Suite environment file. Reset the environment using the new environment file before merging patches.

For more information, see: AD Merge Patch.

Applying a Single Patch to an NLS Installation

Requirement: I need to apply a single patch to an Oracle E-Business Suite NLS installation.

If an Oracle E-Business Suite system contains languages other than American English (US), the recommended method is to apply the US patch first and then apply the translation patch for each installed language. If you have installed more than one additional language, you can merge all the translation patches and apply them as a single, merged NLS patch.

You can also merge US patches with the additional language patches. However, depending on your system topology, it may be necessary to keep the US and non-US patches separate.

To apply a single patch to an NLS installation

This procedure assumes that you will apply US and language patches separately.

1. Use adop to start a new patching cycle (adop phase=prepare).

2. Use adop to apply the patch driver of the US patch.

3. Use adop to apply the patch drivers of each NLS patch. If you have merged the individual NLS patches for a system that runs multiple languages, apply the driver for the merged NLS patch. See: Applying a Patch Interactively,Oracle E-Business Suite Maintenance Guide.

4. Use adop to complete the patching cycle (finalize, cutover, cleanup phases).

Applying Multiple Patches to an NLS Installation

Requirement: I need to apply several patches to an Oracle E-Business Suite NLS installation.

If an Oracle E-Business Suite system contains multiple languages other than American English (US) and you are applying multiple patches for each language, the recommended method is to merge all US patches into a single patch and all patches for every non-US language into a single patch. Then apply the merged US patch, followed by the merged language patch.

You can also merge US patches with the additional language patches or merge each language in separate language-specific patches. Depending on your system topology, it may be necessary to keep the US and non-US patches separate. This procedure assumes that you will apply US and language patches separately.

To apply multiple patches to an NLS installation

This example assumes the system has American English, French, and German installed.

1. Use adop to start a new patching cycle (adop phase=prepare).

2. Use AD Merge Patch to merge the US (American English) patches into a single patch.

3. Use AD Merge Patch to merge the French and German patches into a single NLS patch.

4. Use adop to apply all drivers of the merged US patch.

5. Use adop to apply all drivers of the merged NLS patch.

6. Use adop to complete the patching cycle (finalize, cutover, cleanup phases).

Keeping Your System Current

Each time you apply a patch, AutoPatch stores the associated information in the Oracle Applications Manager (OAM) patch history database. The OAM Patch Wizard and Applied Patches tools provide graphical user interfaces that you can use to query the database for a complete history of patches applied to your system, to search for the patches you have already applied, and to determine existing patches that should be applied to keep your system current. Patch Wizard determines which recommended patches you should apply to your system, and the impact of applying these patches.

Before running Patch Wizard, you must set up My Oracle Support credentials. You must also set up preferences and filters that govern the way you download patches. To see how to complete these one-time tasks, as well as learn about navigating the Patch Wizard pages and submitting requests, see: Patch Wizard.

Creating a List of Recommended Patches

Requirement: How do I determine if there are patches that I have not yet applied?

Patch Wizard creates a list of patches by comparing the patches in the patch history database against a list of recommended patches in a Patch Information Bundle file downloaded from My Oracle Support. It then determines which of the recommended patches you should apply to your system and reports the contents of the patch and the files that it will update when applied.

It does not report on all available patches, but only patches at the current codeline, such as high-priority patches, and those that update your system to a new codeline (pre-upgrade patches).

To see a list of patches recommended for your system

1. Access Oracle Applications Manager.

   Follow the instructions in Accessing Patch Wizard to access OAM. All procedures in this section begin with the Site Map.

2. Access the Patch Wizard home page.

   From the Site Map (Maintenance tab), click Patch Wizard under the Patching and Utilities heading.

   Site Map Page

   

   The Patch Wizard home page appears.

3. Submit a request for recommended patches.

   From the Recommend Patches page, select a patch filter. Use the magnifying glass icon to see a list of available patch filters.

   Recommend Patches Page – Top

   

   After you have entered the requested information, click OK. The results of your request are shown in the Results section of the Patch Wizard main page. You can also schedule the request for a future date.

4. Track the status of your request.

   From the main page, you can track the status of your recommended patch request. Click the Job Status icon for the Recommend/Analyze Patches task.

   Job Status Page

   

   The Job Status page displays summary information. If you click the Show/Hide icon corresponding to your request ID, the page displays more details. For more information about the fields and functions on this page, see: Patch Wizard.

Downloading Recommended Patches

Requirement: How do I use Patch Wizard to download patches?

Patch Wizard can download patches based on either the list created by the "recommend patches" request or any list of patches entered in the Download Patches page.

The Download Patches page prompts you for information about the patches to download, then downloads them directly from My Oracle Support. The Merge Options section of this page defines how patches should be merged after they are downloaded.

To download patches using Patch Wizard

1. Access Oracle Applications Manager.

   Follow the instructions in Accessing Patch Wizard to access OAM. All procedures in this section begin with the Site Map.

2. Access the Patch Wizard home page.

   From the Site Map (Maintenance tab), click Patch Wizard under the Patching and Utilities heading.

   Site Map Page

   

   The Patch Wizard home page appears.

   Patch Wizard Home Page

   

3. Submit a request to download patches.

   Click the Tasks icon for Download Patches. The Download Patches page appears.

   Download Patches Page – Top

   

   On this page, list the patches you want to download in the Patch List field.

   Another option is to click the Details icon for a recommended patch request in the Results section of the Patch Wizard home page.

   Patch Wizard Home Page – Recommended Patches Results

   

   The Recommended Patches Results page for the recommended patch request appears.

   Recommended Patches Results Page

   

   Select any number of recommended patches on this page and click the Download button. This populates the Patch List field in the Download Patches page with the selected patch numbers.

4. Set download options.

   On the Download Patches page, set Merge options and indicate information about languages and platforms. If you choose to automatically merge patches while downloading, specify the merged patch name and the merging strategy. You can select the languages and platform of the patches to download. When you provide information in this section of the page, Patch Wizard only downloads patches that match the selected languages and platform. You can also schedule the download for a future date.

5. Submit request.

   After you have entered the patch information, click OK.

6. Track the status of your request.

   From the main page, you can track the status of your patch request. Click the Job Status icon for Download Patches.

   Job Status Page

   

   The Job Status page displays. If you click the Show/Hide icon corresponding to you request ID, the page displays more details. For more information about the fields and functions on this page, see: Patch Wizard.

Determining Patch Impact on System Files

Requirement: Before I apply a patch, can I see which system files will be affected?

Patch Wizard provides a Patch Impact Summary page that shows the impact of a specific patch if applied to your system. It contains the following information: Patch Impact Analysis, Direct Impact Summary, and Indirect Impact Summary. By reviewing these results, you can see detailed information about files included in a patch, as well as the effect a specific patch will have on your existing system files. For example, you can see information about total files in the patch, the number and type of files that will be installed, and which existing files will be changed. See: Patch Wizard.

To view the information on the Patch Impact Summary page

1. Access Oracle Applications Manager.

   Follow the instructions in Accessing Patch Wizard to access OAM. All procedures in this section begin with the Site Map.

2. Access the Patch Wizard home page.

   From the Site Map (Maintenance tab), click Patch Wizard under the Patching and Utilities heading.

   Site Map Page

   

3. View recommended patches results.

   From the home page, click the Details icon for an item in the Results section.

   Recommended Patches Results Page

   

   The Recommended Patches Results page for the recommended patch request appears.

   Recommended Patches Results Page

   

4. Access the Patch Impact Analysis page.

   Clicking the Impact icon in the Recommended Patches Results page opens the Patch Impact Analysis page for the selected patch.

   Patch Impact Analysis Page

   

   Many of the line items on this page are links to detailed information about the impact of the patch on the system. For example, the File Types Installed value is a link to a page that lists the file types and the number of unchanged, changed, and new files in the file system as a result of applying the selected patch.

Creating Patch Recommendations Without an Internet Connection

Requirement: How do I use the features of creating patch recommendations if I do not have access to an Internet connection?

You can run Patch Wizard without access to an Internet connection, if necessary, by downloading the Patch Information Bundle to a system which has Internet access. Once the download is complete, copy the Patch Information Bundle file to the Patch Wizard’s staging directory. Then run Patch Wizard as you normally would based on the files you copied to the staging directory.

To create recommendations without using an Internet connection

1. Download the Patch Information Bundle to a system which has Internet access.

2. Set up a staging directory on a system that does not have Internet access. Patch Wizard must be able to read from and write to this staging directory.

3. Copy the Patch Information Bundle zip file to the staging directory. The zip file must be copied to a system that can access the Patch Wizard staging directory. If the staging directory is on a local disk, the zip file must be copied to the system where you run Patch Wizard. If the staging directory is on a shared (network) disk, it can be copied to any system with access to the shared disk.

4. Run Create Recommendations as you normally would from this point.

Analyzing Applied Patches

As you apply patches, AutoPatch records the actions in the Oracle E-Business Suite patch history database. You can query this database using the Oracle Applications Manager (OAM) Applied Patches feature, which provides easy access to reports based on your search criteria.

Note: Patch information s is not stored in the database if the patch is applied in pre-install mode or test mode. Also, if patch application does not run successfully to completion, the associated information is neither uploaded to the patch history database nor available in the Applied Patches feature.

You enter search criteria on a search patches page, either Simple Search or Advanced Search. A summary report is displayed at the bottom of the search page.

Several detailed reports are also available, including Timing Details, Files Copied, Bug Fixes, and Action Summary. Most of these detailed reports have a standard layout. The top portion displays the criteria that were used for the search, and the bottom portion displays the results of the search.

See also Applied Patches in the OAM Help system.

Determining If a Patch Was Applied

Requirement: Can I determine if a specific patch has been applied to my Oracle E-Business Suite system?

To determine which patches were applied, enter a patch ID in the Applied Patch Check area of the Software Updates page. You can perform a simple search by entering an ID or a series of IDs separated by commas.

To determine if a patch was applied

1. Access Oracle Applications Manager.

   Follow the instructions in Accessing Patch Wizard to access OAM.

2. Access the Software Updates page.

   From the Applications Dashboard, click the Software Updates tab. The Software Updates page appears.

   Software Updates Page

   

3. Enter a patch ID.

   In the Applied Patch Check area of the Software Updates page, enter a patch ID or a series of IDs separated by commas. Your queried ID appears in the corresponding column depending on whether it has been applied.

Searching for Patch Details

Requirement: What information is available on the Patch Details report? How do I create the report?

From any Patch Summary report, you can click the Details icon for a selected row to open the Patch Details report, which displays summary information carried over from the Results portion of either the Simple Search or Advanced Search page.

This report also contains more specific information about the patch, including:

• Name of the driver file and the date and time it was applied

• Command line options used to run the file

• Platform of the driver file

• Location where the driver was run

• Report on whether a codelevel was introduced, and if so, which one

From the Patch Details page, you can also access additional information about a patch, including timing details, files copied, bug fixes, and a summary of actions performed.

See: Applied Patches. See also Applied Patches in the OAM Help.

To review patch details

1. Access Oracle Applications Manager.

   Follow the instructions in Accessing Patch Wizard to access OAM.

2. Create a Patch Summary report.

   From the Site Map (Maintenance tab), click Applied Patches under the Patching and Utilities heading. From either the Simple Search or Advanced Search page, enter a patch number or a date range to create a Patch Summary report. Click Go.

3. Select the patch.

   Click the Details icon in any selected row of the Patch Summary report. The Patch Details report appears.

   Patch Details Report

   

   The report displays patch details such as driver files, start and end dates, and platform. It also provides access to other patch details related to the driver files, such as files copied and bug fixes. You can select a driver from the list, and click one of the additional detail buttons to see other reports.

4. View additional details.

   As an example of the details that are available for a selected driver, click Files Copied.

   Files Copied Report

   

   For each file, the Files Copied report shows the product short name, the directory where the file was copied, the name of the file, and the version number. To view other information associated with the driver file, click the Patch Details link at the top of the page to return to the previous page.

   As another example, click the Bug Fixes button.

   Bug Fixes Report

   

   The Bug Fixes report lists all bug fixes included in the selected driver file. It contains the bug number, the associated product, and whether the bug fix was applied. If the fix was not applied, the Remarks column explains why.

5. View the Action Summary report.

   You can create a report that summarizes the actions of a selected driver file. Click the Patch Details link at the top of the page to return to the Patch Details page. (You can also access the Action Summary report by clicking the bug fix number on the Bug Fixes report.)

   From the Patch Details page, select a driver and click the Action Summary button.

   Action Summary Report

   

   The Action Summary report shows more information about the driver and its actions. For definitions of the column headings, see Action Summary.

   If the driver selected contains a database portion, the Patch Summary report shows the driver actions, such as sql and exec. If the driver performed actions on the database, the Details icon is active. Click it to see the Action Details report.

Searching for Translation Patches

Requirement: My Oracle E-Business Suite system operates in multiple languages. I want to make sure translation patches have been applied successfully.

If a patch has an associated translation patch, you apply the translation patch separately. AutoPatch stores information in the patch history database about all translation patches you apply.

To search for translation patches

1. Access the Oracle Applications Manager.

   Follow the instructions in Accessing Patch Wizard to access OAM.

2. Access the Simple Search page.

   Site Map Page

   

   From the Site Map (Maintenance tab), click Applied Patches under the Patching and Utilities heading. The Simple Search page appears.

   Enter the search criteria. For details about using the Simple Search page, see: Determining If a Patch Was Applied, or click the OAM Help button.

3. Specify the patch.

   On the Simple Search page, enter the ID of the translation patch in the Patch field. Click Go.

4. Review the Patch Summary report

   All applications of the patch are displayed. If multiple translations were applied, there will be multiple rows. The Language column shows the languages applied.

   Applied Patches Page – Simple Search

   

Viewing Applied Patches in a Report Format

Requirement: Can I review applied patches information without the OAM screens?

There may be times when you want to view applied patch history without running the Oracle Applications Manager. For example, you may need to view large amounts of data, or you may just need a list of patches without the detail provided in the OAM Patch History reports. In these cases, you can run command line scripts that list all patches applied in each AutoPatch session, all files affected by a patch, or all patches applied within a certain date range. The scripts, and a description of the reports they produce, are listed in the following table.

The XML reports produced by adphrept.sql and adfhrept.sql can either be processed as XML or viewed as HTML.

To run a report that provides a listing of applied patches, follow the appropriate instructions in this section.

To see a list of all completed AutoPatch sessions with patch details

Run the adphrept.sql script (located in <AD_TOP>/patch/115/sql). This script produces an XML report showing individual AutoPatch sessions. If a patch was applied more than once, this report lists each application of the patch. If a merged patch was applied, it lists the merged patch by patch name. It does not list the individual patches within the merged patch.

To run adphrept.sql, use the following parameters:

$ perl <AD_TOP>/bin/txkRunSSHSetup.pl enablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3$ 

2

For <query_depth>, specify 1 (details of patches only), 2 (details of patches and their bug fixes only), or 3 (details of patches, bug fixes, and bug actions).

At the command prompt, enter the report command and enter values for the parameters and prompts. For example, to see complete patch details for AutoPatch sessions that were run during January 2009, enter the following, using the mm/dd/yyyy date format:

UNIX:

$ perl <AD_TOP>/bin/txkRunSSHSetup.pl enablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3$ 

3

Windows:

$ perl <AD_TOP>/bin/txkRunSSHSetup.pl enablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3$ 

4

The <AD_TOP>/html directory contains the adpchrep.xsl style sheet for displaying the XML output file in HTML format. To view the XML file as HTML, copy both the adpchrep.xsl style sheet and XML output report to a directory accessible by a browser. Open the directory in your browser and click the XML filename.

To display information about files changed by patches

Run the adfhrept.sql script (located in <AD_TOP>/patch/115/sql) to produce an XML report named adfilerep.xml. Use the following parameters:

$ perl <AD_TOP>/bin/txkRunSSHSetup.pl enablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3$ 

5

At the command prompt, enter the report command and enter values for the parameters and prompts. For example, to see the complete file version history for admorgb.pls considering only patches applied in January 2008, enter the following, using mm/dd/yyyy format:

UNIX:

$ perl <AD_TOP>/bin/txkRunSSHSetup.pl enablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3$ 

6

Windows:

$ perl <AD_TOP>/bin/txkRunSSHSetup.pl enablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3$ 

7

The <AD_TOP>/html directory contains the adfilerep.xsl style sheet for displaying the XML output file in HTML format. To view the XML file as HTML, copy both the adfilerep.xsl style sheet and XML output report to a directory accessible by a browser. Open the directory in your browser and click on the XML filename.

To see a list of all patches in a given date range

The adpchlst.sql report (located in <AD_TOP>/patch/115/sql) produces a list (adpchlst.lst) of all patches in a date range, without patch detail. It differs from adphrept.sql in two ways: it lists a patch only once regardless of how many times it was applied, and it lists individual patches included within a merged patch. For example, if you combine patches 123, 124, and 125 in a merged patch called merged1, the report lists patches 123, 124, and 125, but not merged1.

At the command prompt, enter the report command and enter the date parameters in mm/dd/yyyy format. For example, to see a list of patches applied in July 2014, enter the following:

UNIX:

$ perl <AD_TOP>/bin/txkRunSSHSetup.pl enablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3$ 

8

Windows:

$ perl <AD_TOP>/bin/txkRunSSHSetup.pl enablessh -contextfile=<CONTEXT_FILE> -hosts=h1,h2,h3$ 

9

Monitoring Patches in Progress

Requirement: Can I monitor the progress of a patch while it is being applied?

Depending on the size and complexity of a patch, it may take from several minutes to several hours to completely apply it to your system. It is useful to know what a patch is currently doing and how long individual steps are taking.

The Timing Reports feature of Oracle Applications Manager allows administrators to view live information about in-progress patches, and the time taken by each patch action. To use this feature:

1. Run adop to start the patching session.

2. In Oracle Applications Manager, navigate to Timing Reports (Navigation: Sitemap > Maintenance > Patching and Utilities > Timing Reports).

3. Click on the refresh icon at any time to obtain the latest status.

It is also possible to monitor progress of a patching cycle by reviewing:

• adop messages – As adop runs, it displays messages on the screen about the status and progress of the patching process.

• Patch log files – adop creates log files containing information about the patching actions performed in an online patching cycle.

• Worker status – For jobs run in parallel, use AD Controller to view the status of the concurrent manager and workers assigned to process jobs. See AD Controller examples in Managing Worker Processes.

Analyzing Patches Without an Internet Connection

Requirement: How do I analyze specific patches if I do not have access to an Internet connection?

You can run Patch Wizard to analyze specific patches without access to an Internet connection, if necessary, by downloading the patches to a system which has Internet access. Once the download is complete, copy the patches to the Patch Wizard’s staging directory. Then run Patch Wizard as you normally would based on the files you copied to the staging directory.

To analyze specific patches without using an Internet connection

1. Download the patch zip file(s) to a system which has Internet access.

2. Set up a staging directory on a system that does not have Internet access. Patch Wizard must be able to read from and write to this staging directory.

3. Copy the patch zip file(s) to the <staging directory>/ad directory, if the downloaded patch is an AD product patch. Otherwise, copy the patch zip file(s) to <staging directory>/nonad directory. The zip file(s) must be copied to a system that can access the Patch Wizard staging directory. If the staging directory is on a local disk, the zip file(s) must be copied to the system where you run Patch Wizard. If the staging directory is on a shared (network) disk, it can be copied to any system with access to the shared disk.

4. Run Analyze Specific Patches as you normally would from this point.