Oracle Upgrade from 10.2.0.2 to 10.2.0.3

Prior to installing the Rome muon calibration DB for replication we needed to update our Oracle installation. I received an email from Gancho on January 30, 2007 about this:

Hi Shawn,


Just for your information I am forwarding Eva's mail.
The IT advices the all Tier1 sites to upgrade their Oracle servers to version 10.2.0.3, because of fixing some bugs related to the Streams.
When you have time please consider that. 
Here it is applied on INTR (in the MUON's case it is the destination DB) 
 
Cheers, Gancho

Installing the Upgrade

First I used the Oracle Enterprise Manager at http://umors.grid.umich.edu:1158/em to copy over the needed patch 5337014 for Linux x86_64.

The EM system unpacked the downloaded zip file into /app/oracle/patchsets/EMStagedPatches/5337014. Starting from section 5.5 of the README for 5337014 I did:

export ORACLE_HOME=/app/oracle/oracle/product/10.2.0/asm
export ORACLE_SID=muoncal

Then I needed to stop all processes. I used the EM to setup a 1 hour blackout for the upgrade so the monitoring would be correct. Next I logged in as the 'oracle' user on umors.grid.umich.edu and did:

muoncal on umors.grid.umich.edu: emctl stop dbconsole
TZ set to US/Michigan
Oracle Enterprise Manager 10g Database Control Release 10.2.0.2.0  
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
http://umors.grid.umich.edu:1158/em/console/aboutApplication
Stopping Oracle Enterprise Manager 10g Database Control ... 
 ...  Stopped. 

Then:

muoncal on umors.grid.umich.edu: isqlplusctl stop
iSQL*Plus 10.2.0.2.0
Copyright (c) 2003, 2005, Oracle.  All Rights Reserved.
iSQL*Plus instance on port 5560 is not running ...

Then:

muoncal on umors.grid.umich.edu: isqlplusctl stop
iSQL*Plus 10.2.0.2.0
Copyright (c) 2003, 2005, Oracle.  All Rights Reserved.
iSQL*Plus instance on port 5560 is not running ...
muoncal on umors.grid.umich.edu: sqlplus /NOLOG

SQL*Plus: Release 10.2.0.2.0 - Production on Wed Jan 31 11:04:15 2007

Copyright (c) 1982, 2005, Oracle.  All Rights Reserved.

SQL> CONNECT SYS/xxxxxxxx AS SYSDBA
Connected.
SQL> SHUTDOWN IMMEDIATE
Database closed.
Database dismounted.
ORACLE instance shut down.

Then I shutdown (as 'root') the oracleasm service:

[umors:~]# service oracleasm stop
Unmounting ASMlib driver filesystem:                       [  OK  ]
Unloading module "oracleasm":                              [  OK  ]

Now I continued with the "6 Installation Tasks" in the README.

I made sure X11 was working as the 'oracle' user to my remote system and then did:

cd /app/oracle/patchsets/EMStagedPatches/5337014/Disk1
./runInstaller

I got an error screen about another process (still running) that needed to be shutdown: ocssd.bin.

I needed to also shutdown init.cssd (as 'root'):

[umors:~]# service init.cssd stop
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
Shutdown has begun. The daemons should exit soon.

I then clicked 'retry' on the Error window. After some more checks a window popped up showing me what it would upgrade. I clicked the "Install" button.

At 50% of the install a pop-up window reported an _Error in writing file '/app/oracle/oracle/product/10.2.0/asm/oc4j/j2ee/isqlplus/applicati on-deployments/default/defaultWebApp/orion-web.xml'_. I checked (as 'root') and this file was owned by root. I changed it:

chmod oracle.oinstall orion-web.xml

and clicked 'Retry' and the installation proceeded.

At 100% a pop-up box requested I run /app/oracle/oracle/product/10.2.0/asm/root.sh. In a 'root' window I ran this script:

[umors:asm]# ./root.sh
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configuration for local CSS has been initialized

Adding to inittab 
Startup will be queued to init within 30 seconds.
Checking the status of new Oracle init process...
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
        umors
CSS is active on all nodes.
Oracle CSS service is installed and running under init(1M)

I then returned to the pop-up window and clicked "OK". It then started running a "Oneoff Patch Application".

The installation finished successfully and I clicked 'EXIT'.

I then followed the instructions in section 7.2.1 of the README. As the 'oracle' user I made sure the ORACLE_HOME and ORACLE_SID values were correct. As root I started up the Oracle ASM single instance:

[umors:asm]# service oracleasm start
Loading module "oracleasm":                                [  OK  ]
Mounting ASMlib driver filesystem:                         [  OK  ]
Scanning system for ASM disks:                             [  OK  ]

Then I started the listener:

muoncal on umors.grid.umich.edu: lsnrctl start

LSNRCTL for Linux: Version 10.2.0.3.0 - Production on 31-JAN-2007 11:32:01

Copyright (c) 1991, 2006, Oracle.  All rights reserved.

Starting /app/oracle/oracle/product/10.2.0/asm/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 10.2.0.3.0 - Production
System parameter file is /app/oracle/oracle/product/10.2.0/asm/network/admin/listener.ora
Log messages written to /app/oracle/oracle/product/10.2.0/asm/network/log/listener.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=umors.grid.umich.edu)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 10.2.0.3.0 - Production
Start Date                31-JAN-2007 11:32:01
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      ON
Listener Parameter File   /app/oracle/oracle/product/10.2.0/asm/network/admin/listener.ora
Listener Log File         /app/oracle/oracle/product/10.2.0/asm/network/log/listener.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=umors.grid.umich.edu)(PORT=1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

Then I ran the Oracle DB Upgrade Assistant:

dbua

After the Welcome screen I selected "Upgrade a Database", clicked "Next", used the selected /app/oracle/oracle/product/10.2.0/asm, clicked "Next". At this point I see two pop-up windows: "Database Upgrade Assistant" showing "Getting database Information. Please wait..." and Initialization Parameter Filename" showing that the Upgrade Assistant failed in bringing up the database muoncal...Oracle needs the pfile. Provide the full path name of the init file for the database muoncal. I selected /app/oracle/oracle/product/10.2.0/asm/admin/muoncal/pfile/initora.6172006132225

This failed. I don't know what the 'dbua' wants at this point. See the attached PNG files below for the initial pop-up box:

and the resulting error:

I then cancelled out of the 'dbua' window and tried following section 7.2.2 of the README for a manual upgrade.

When I got to the "STARTUP NOMOUNT" it failed:

SQL> startup nomount
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATAFLASH/muoncal/spfilemuoncal.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATAFLASH/muoncal/spfilemuoncal.ora
ORA-15077: could not locate ASM instance serving a required diskgroup
SQL> 

So this must be the pfile the 'dbua' system wanted. I am not sure why this isn't available.

I then figured out I need to set the ORACLE_SID to '+ASM' and do the startup for this:

export ORACLE_SID=+ASM
sqplus "SYS/xxxxxx AS SYSDBA"

SQL> startup
ASM instance started

Total System Global Area  130023424 bytes
Fixed Size                  2071000 bytes
Variable Size             102786600 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
SQL> SELECT NAME,TYPE,TOTAL_MB,FREE_MB FROM V$ASM_DISKGROUP;

NAME                           TYPE     TOTAL_MB    FREE_MB
------------------------------ ------ ---------- ----------
DATAFLASH                      NORMAL     476944     336912

SQL> 

Now I can retry the previous attempt:

export ORACLE_SID=muoncal
dbua

This now seems to work (since ASM is up and mounted).

The dbua completed successfully:

I am issueing (as 'root') a 'service dbora start/stop'.

-- ShawnMcKee - 31 Jan 2007