Reinstallation of Oracle for the Muon Calibration Center

On May 2nd, 2009 our primary Oracle server (umors.grid.umich.edu) was compromised because we had forgotten to propagate system reconfigurations after a prior breakin on other boxes in this subnet. This forced us to remove the umors.grid.umich.edu system from the network at 9:58 AM on May 2nd 2009.

Steps to Remediate

  • The original system was isolated from the network (primary network was administratively disabled at it's switch)
  • A separate NIC was connected to our private network using 10.1.1.120 (non-routed) for access.
  • Access to an NFS mounted area of 13TB on umfs14.aglt2.org was setup (mount point /atlas/data18a)
  • An "oracle" directory was created at /atlas/data18a/oracle for use as a backup target
  • The Enterprise Manager was used to create a "Whole Database" backup to this disk area.
    • There were many problems. Backup wouldn't run. Showed "Waiting for Output" even after 12 hours.
    • Jobs were stopped, redefined and retried. Oracle MetaLink was consulted for the various problems.
    • Resolution was to repair the "agent" access. The 'agent' was setup for secure access that was apparently broken. Took approximately 8 hours to resolve!
  • Backup was finished successfully on May 3, 17:58 Eastern time.
  • Old system is being backed up to disk and will be reinstalled as a "worker-node"

A replacement system was identified: head03.aglt2.org, Dell PE2950 with 32GB of RAM, two 250GB drivers in hardware RAID-1 configuration, 2 1GE network ports, 1 10GE Myricom NIC, DRAC5 (remote access card), dual quad-core Intel 5440 processors and redundant power supplies.

This system was installed with 64bit RHEL5.3. The network was configured to use both the 1GE NICs in active-standby (mode=1) bonded mode. The bonded NIC bond0 was setup as a VLAN trunk supporting 4 VLANS: 6 (grid private network), 808 (.grid.umich.edu), 4001 (.aglt2.org) and 4010 (AGLT2 private network). Some additional software was installed:

  • OpenAFS 1.4.10 x86_64
  • Syslog-ng (configured to report to our central syslog-ng host)
  • SNMPD (net-snmp) setup for our monitoring system

The following URLs were consulted for further needed configurations:

http://www.oracle.com/technology/pub/articles/smiley_10gdb_install.html https://hep.pa.msu.edu/twiki/bin/view/AGLT2/OracleInstall

New Host Preparations

After the new umors.grid.umich.edu was ready (see above) we began the pre-install setup documented for Oracle Installation.

NOTE: See Metalink document [[https://metalink.oracle.com/CSP/main/article?cmd=show&type=NOT&id=456634.1][456634.1] for installation work-around on RHEL5.

Refer to the 10gR2 library at Oracle_docs From the x86_64 release notes for Linux:

3 Preinstallation Requirements

You must review the following sections before installing Oracle Database 10g release 2:

Note: When installing SUSE Linux Enterprise Server 10, if you choose Oracle Server Base and C/C++ Compiler and Tools options in the Software Selection and System Tasks window, then the following prerequisites are automatically available in the operating system.

  • Install libaio Before Installing or Upgrading (OK on RHEL5...already exists)
  • Install oracleasm-support to use ASMLib
    • oracleasm-2.6.18-128.1.6.el5-2.0.5-1.el5.x86_64.rpm
    • oracleasm-2.6.18-128.1.6.el5debug-2.0.5-1.el5.x86_64.rpm
    • oracleasmlib-2.0.4-1.el5.x86_64.rpm
    • oracleasm-support-2.1.3-1.el5.x86_64.rpm
    • Did 'rpm -ivh asm.rpm' on the above (Required install of kernel-debug as well)
  • Install binutils on Oracle Enterprise Linux 4.0 and Red Hat Enterprise Linux 4.0 (Not applicable on RHEL5)
  • Install libxm Before Installing Oracle Lite (Not applicable for our install)
  • Configuring Kernel Parameters
    • None list from this page were applicable. However the other URL noted some needed changes.
  • Oracle HTTP Server on Oracle Enterprise Linux 4.0 and Red Hat Enterprise Linux 4.0
  • Oracle HTTP Server on Oracle Enterprise Linux 5.0 and Red Hat Enterprise Linux 5.0
    • Did _ ln -s /usr/lib/libgdbm.so.2.0.0 /usr/lib/libdb.so.2_

We next needed to setup the needed users. I wanted to make sure and use the same uid/gid as on the prior server (so we checked the old /etc/group and /etc/passwd):

[root@umors asmlib]# groupadd -g 9059 oinstall
[root@umors asmlib]# groupadd -g 9060 dba
[root@umors asmlib]# grep oracle /atlas/data18a/umors/passwd
oracle:x:55621:9059::/app/oracle:/bin/bash
[root@umors asmlib]# useradd -u 55621 -g oinstall -G dba oracle
[root@umors asmlib]# id oracle
uid=55621(oracle) gid=9059(oinstall) groups=9060(dba),9059(oinstall)

Then we setup the 'oracle' user pw (used the 'SN' one):

[root@umors asmlib]# passwd oracle
Changing password for user oracle.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

Next let's setup the kernel parameters:

[root@umors asmlib]# cat >> /etc/sysctl.conf <<EOF
> kernel.shmall = 2097152
> kernel.shmmax = 536870912
> kernel.shmmni = 4096
> kernel.sem = 250 32000 100 128
> fs.file-max = 65536
> net.ipv4.ip_local_port_range = 1024 65000
> net.core.rmem_default=262144
> net.core.wmem_default=262144
> net.core.rmem_max=262144
> net.core.wmem_max=262144
> EOF
[root@umors asmlib]# /sbin/sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 262144
net.core.wmem_max = 262144

Additional some more settings need to be configured:

limits.conf:

# Changes for Oracle 
oracle           soft    nproc   2047
oracle           hard    nproc   16384
oracle           soft    nofile  1024
oracle           hard    nofile  65536

login:

# Added for Oracle
session    required     pam_limits.so

profile:

# Added for Oracle
if [ $USER = "oracle" ]; then
        if [ $SHELL = "/bin/ksh" ]; then
              ulimit -p 16384
              ulimit -n 65536
        else
              ulimit -u 16384 -n 65536
        fi
fi

Setup the directories. On the original system we had used /app/oracle. On this system we already have a large /opt partition on the RAID-1 system disk that we will use for the install:

[root@umors asmlib]# mkdir -p /opt/app/oracle
[root@umors asmlib]# chown  -R oracle:oinstall /opt/app/oracle
[root@umors asmlib]# chmod -R 775 /opt/app/oracle

Note: this is a change from the prior install in that the Oracle base install location is /opt/app/oracle now.

At this point I changed the HOME area for the 'oracle' user to be /opt/app/oracle by editing the /etc/passwd entry.

I then ran the ASM configure:

[root@umors asmlib]# service oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]

Later we will need to add the RAID-10 array at /dev/sdb

I created a disk partition /dev/sdb1 on this RAID-10 (4x1TB) array. This can be used as input to create the ASM device:
[root@umors ~]# service oracleasm createdisk DISK1 /dev/sdb1
Marking disk "DISK1" as an ASM disk:                       [  OK  ]

Before beginning an install, an important issue to address is covered by:

2.4.3 Installing on Multihomed Computers

You can install Oracle Database on a multihomed computer. A multihomed computer is associated with multiple IP addresses. This is typically achieved by having multiple network cards on the computer. Each IP address is associated with a host name. In addition, you can set up aliases for the host name. By default, Oracle Universal Installer uses the ORACLE_HOSTNAME environment variable setting to find the host name. If ORACLE_HOSTNAME is not set and you are installing on a computer that has multiple network cards, then Oracle Universal Installer determines the host name by using the first entry in the /etc/hosts file.

Clients must be able to access the computer either by using this host name or by using aliases for this host name. To verify this, ping the host name from the client computers using the short name (hostname only) and the full name (hostname and domain name). Both tests must be successful.

Setting the ORACLE_HOSTNAME Environment Variable

Use the following procedure to set the ORACLE_HOSTNAME environment variable.

For example, if the fully qualified host name is somehost.us.acme.com, then enter one of the following commands:

Bourne, Bash, or Korn shell:

$ ORACLE_HOSTNAME=somehost.us.acme.com
$ export ORACLE_HOSTNAME

C shell:

% setenv ORACLE_HOSTNAME somehost.us.acme.com

We need to setup
ORACLE_HOSTNAME=umors.grid.umich.edu

Installing Oracle

Now we are ready to install Oracle on the new umors.grid.umich.edu.

Move the /root/database and /root/companion (unpacked install distributions) to a subdirectory of the 'oracle' account:

[root@umors ~]# mkdir /opt/app/oracle/10gR2_db
[root@umors ~]# mv database/ /opt/app/oracle/10gR2_db/
[root@umors ~]# mv companion/ /opt/app/oracle/10gR2_db/

Change ownership so this is all owned by 'oracle': chmod -R 775 /opt/app/oracle

Now we can login as 'oracle' to do the install.

[root@umors ~]# su - oracle [oracle@umors ~]$ cd [oracle@umors ~]$ ls 10gR2_db [oracle@umors ~]$ cd 10gR2_db/database [oracle@umors 10gR2_db]$ export ORACLE_HOSTNAME=umors.grid.umich.edu

Note we defined the ORACLE_HOSTNAME is this is a multihomed host. We want to use the correct name.

Now we can start the install ( as 'oracle') remembering we have to account for the RHEL5 glitch:

./runinstaller -ignoreSysPrereqs

This failed though with the following:

[oracle@umors database]$ ./runInstaller -ignoreSysPrereqs
Starting Oracle Universal Installer...

Checking installer requirements...

Checking operating system version: must be redhat-3, SuSE-9, redhat-4, UnitedLinux-1.0, asianux-1 or asianux-2
                                      Passed


All installer requirements met.

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2009-05-04_01-01-59PM. Please wait ...[oracle@umors database]$ Oracle Universal Installer, Version 10.2.0.1.0 Production
Copyright (C) 1999, 2005, Oracle. All rights reserved.

Exception java.lang.UnsatisfiedLinkError: /tmp/OraInstall2009-05-04_01-01-59PM/jre/1.4.2/lib/i386/libawt.so: libXp.so.6: cannot open shared object file: No such file or directory occurred..
java.lang.UnsatisfiedLinkError: /tmp/OraInstall2009-05-04_01-01-59PM/jre/1.4.2/lib/i386/libawt.so: libXp.so.6: cannot open shared object file: No such file or directory

As 'root' I needed to install 'libXp'. Then I needed to make sure that the 'oracle' user could open an X-window to my system.

I went through a few screens (clicking "next") until I got to "Select Configuration Option". Near the beginning I changed the "OraDB10g_home1" to "OraDB10g_ASM" and the path from '/opt/app/oracle/product/10.2.0/db_1' to end at '.../asm' to match the orginal install. All the product-specific tests passed. Then I selected "Configure Automatic Storage Management (ASM)" I used the 'QN' pw. On the next screen I changed the disk group name from "DATA" to "DATAFLASH" and selected the 'ORCL:DISK1' disk and set the 'Redundancy' to "External". Then on the next screen I clicked "Install".

It then installed...during the middle I had to run two scripts as 'root'. No errors occurred and it seemed to finish:

The following J2EE Applications have been deployed and are accessible at the URLs listed below.

iSQL*Plus URL:
http://umors.grid.umich.edu:5560/isqlplus

iSQL*Plus DBA URL:
http://umors.grid.umich.edu:5560/isqlplus/dba

Enterprise Manager 10g Database Control URL:

Separate Restore Attempt

I found and tried the info at http://youngcow.net/doc/oracle10g/backup.102/b14191/rcmrecov002.htm

The original dbid is 3255750732

First we need to verify that NFS (where the backup is) is mounted correctly. We need the following NFS options:

rsize=32768,wsize=32768,noac,hard

Add these to /etc/fstab and remount.

Use RMAN

export ORACLE_SID=muoncal

rman target / nocatalog

RMAN> set dbid 3255750732; RMAN> startup nomount

RMAN> run {
2>  allocate channel oem_restore type disk;
3>  restore spfile to pfile '/opt/app/oracle/oracle/product/10.2.0/asm/dbs/initmuoncal.ora' from  autobackup recovery area='/atlas/data18a/umors/slash/app/oracle/flash_rec_area' db_name='muoncal';
4> shutdown abort;
5> }

Edit resulting /opt/app/oracle/oracle/product/10.2.0/asm/dbs/initmuoncal.ora file, substituting original path /app/oracle... to /opt/app/oracle... in each relevant entry.

Create missing directories from original file system: $ORACLE_BASE/admin/muoncal/... and $ORACLE_BASE/flash_rec_area so we can startup the DB: (ORACLE_BASE=/opt/app/oracle)

RMAN> startup force nomount pfile='/opt/app/oracle/oracle/product/10.2.0/asm/dbs/initmuoncal.ora';

Oracle instance started

Total System Global Area    2147483648 bytes

Fixed Size                     2022144 bytes
Variable Size                838862080 bytes
Database Buffers            1291845632 bytes
Redo Buffers                  14753792 bytes

Try to restore control file:

RMAN> run {
2>  allocate channel oem_restore type disk;
3> restore controlfile from  autobackup recovery area='/atlas/data18a/umors/slash/app/oracle/flash_rec_area' db_name='muoncal';
4> alter database mount;
5> }

allocated channel: oem_restore
channel oem_restore: sid=156 devtype=DISK

Starting restore at 04-MAY-09

recovery area destination: /atlas/data18a/umors/slash/app/oracle/flash_rec_area
database name (or database unique name) used for search: MUONCAL
channel oem_restore: autobackup found in the recovery area
channel oem_restore: autobackup found: /atlas/data18a/umors/slash/app/oracle/flash_rec_area/MUONCAL/autobackup/2009_05_03/o1_mf_s_685907783_4zw4r887_.bkp
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00601: fatal error in recovery manager
RMAN-03004: fatal error during execution of command
ORA-01092: ORACLE instance terminated. Disconnection forced

The same failure happens if I remove the May 3, 2009 backup and let it find the next one:

muoncal on umors.grid.umich.edu: rman target / nocatalog

Recovery Manager: Release 10.2.0.1.0 - Production on Mon May 4 21:46:57 2009

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database (not started)

RMAN> set dbid 3255750732;

executing command: SET DBID

RMAN>

RMAN> startup force nomount pfile='/opt/app/oracle/oracle/product/10.2.0/asm/dbs/initmuoncal.ora';

Oracle instance started

Total System Global Area    2147483648 bytes

Fixed Size                     2022144 bytes
Variable Size                838862080 bytes
Database Buffers            1291845632 bytes
Redo Buffers                  14753792 bytes

RMAN> run {
2> allocate channel oem_restore type disk;
3> restore controlfile from  autobackup recovery area='/atlas/data18a/umors/slash/app/oracle/flash_rec_area' db_name='muoncal';
4> alter database mount;
5> }

allocated channel: oem_restore
channel oem_restore: sid=156 devtype=DISK

Starting restore at 04-MAY-09

recovery area destination: /atlas/data18a/umors/slash/app/oracle/flash_rec_area
database name (or database unique name) used for search: MUONCAL
channel oem_restore: autobackup found in the recovery area
channel oem_restore: autobackup found: /atlas/data18a/umors/slash/app/oracle/flash_rec_area/MUONCAL/autobackup/2009_05_02/o1_mf_s_685809869_4zs54gxw_.bkp
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-00601: fatal error in recovery manager
RMAN-03004: fatal error during execution of command
ORA-01092: ORACLE instance terminated. Disconnection forced

Problem above was because I hadn't upgraded the +ASM instance to 10.2.0.4! After doing that I got:

RMAN> run {
2> allocate channel oem_restore type disk;
3> restore controlfile from  autobackup recovery area='/atlas/data18a/umors/slash/app/oracle/flash_rec_area' db_name='muoncal';
4> alter database mount;
5> }

allocated channel: oem_restore
channel oem_restore: sid=156 devtype=DISK

Starting restore at 05-MAY-09

recovery area destination: /atlas/data18a/umors/slash/app/oracle/flash_rec_area
database name (or database unique name) used for search: MUONCAL
channel oem_restore: autobackup found in the recovery area
channel oem_restore: autobackup found: /atlas/data18a/umors/slash/app/oracle/flash_rec_area/MUONCAL/autobackup/2009_05_03/o1_mf_s_685907783_4zw4r887_.bkp
channel oem_restore: control file restore from autobackup complete
output filename=+DATAFLASH/muoncal/controlfile/current.256.686019077
output filename=/opt/app/oracle/flash_rec_area/MUONCAL/controlfile/o1_mf_4zzkg795_.ctl
Finished restore at 05-MAY-09

database mounted
released channel: oem_restore

Lets try to restore the database now:

Find the SCN to save to:


SQL> SET NUMFORMAT 999999999999999
SQL> select MIN(SCN) FROM (SELECT MAX(NEXT_CHANGE#) SCN FROM V$ARCHIVED_LOG GROUP BY THREAD#);

        MIN(SCN)
----------------
   6057815325809

Now there is another issue to address. We have block level change tracking enabled. It needs to be disabled before we proceed:

muoncal on umors.grid.umich.edu: sqlplus "SYS/xxxxxxx AS SYSDBA"

SQL*Plus: Release 10.2.0.4.0 - Production on Tue May 5 10:56:29 2009

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> alter database disable block change tracking;

Database altered.

Now lets proceed with the restore. We need to use the SCN determined above to limit the restore:

RMAN> run {
2> set until SCN 6057815325809;
3> restore database;
4> recover database;
5> }

executing command: SET until clause

Starting restore at 05-MAY-09
Starting implicit crosscheck backup at 05-MAY-09
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=159 devtype=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: sid=152 devtype=DISK
allocated channel: ORA_DISK_3
channel ORA_DISK_3: sid=150 devtype=DISK
allocated channel: ORA_DISK_4
channel ORA_DISK_4: sid=149 devtype=DISK
Crosschecked 21 objects
Crosschecked 756 objects
Finished implicit crosscheck backup at 05-MAY-09

Starting implicit crosscheck copy at 05-MAY-09
using channel ORA_DISK_1
using channel ORA_DISK_2
using channel ORA_DISK_3
using channel ORA_DISK_4
Crosschecked 12 objects
Finished implicit crosscheck copy at 05-MAY-09

searching for all files in the recovery area
cataloging files...
no files cataloged

using channel ORA_DISK_1
using channel ORA_DISK_2
using channel ORA_DISK_3
using channel ORA_DISK_4

channel ORA_DISK_1: starting datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
restoring datafile 00004 to +DATAFLASH/muoncal/datafile/users.259.596035211
restoring datafile 00006 to +DATAFLASH/muoncal/datafile/atlas_muon_calib_data01.280.596107405
restoring datafile 00010 to +DATAFLASH/muoncal/datafile/atlas_muon_calib_cosmic01.2132.624467291
channel ORA_DISK_1: reading from backup piece /atlas/data18a/oracle/o2ke45rs_1_1
channel ORA_DISK_2: starting datafile backupset restore
channel ORA_DISK_2: specifying datafile(s) to restore from backup set
restoring datafile 00001 to +DATAFLASH/muoncal/datafile/system.256.596035211
restoring datafile 00002 to +DATAFLASH/muoncal/datafile/undotbs1.258.596035211
restoring datafile 00011 to +DATAFLASH/muoncal/datafile/streams_data01.3117.654176245
channel ORA_DISK_2: reading from backup piece /atlas/data18a/oracle/o4ke45rt_1_1
channel ORA_DISK_3: starting datafile backupset restore
channel ORA_DISK_3: specifying datafile(s) to restore from backup set
restoring datafile 00003 to +DATAFLASH/muoncal/datafile/sysaux.257.596035211
restoring datafile 00005 to +DATAFLASH/streams_tbs
restoring datafile 00007 to +DATAFLASH/muoncal/datafile/logminer_ts.343.596279813
channel ORA_DISK_3: reading from backup piece /atlas/data18a/oracle/o5ke45rt_1_1
channel ORA_DISK_4: starting datafile backupset restore
channel ORA_DISK_4: specifying datafile(s) to restore from backup set
restoring datafile 00008 to +DATAFLASH/muoncal/datafile/atlas_muon_calib_ed01.2130.624465707
restoring datafile 00009 to +DATAFLASH/muoncal/datafile/atlas_muon_calib_cdc01.2131.624467143
restoring datafile 00012 to +DATAFLASH/muoncal/datafile/atlas_muoncalib_mich_data01.3118.654176623
channel ORA_DISK_4: reading from backup piece /atlas/data18a/oracle/o3ke45rs_1_1
channel ORA_DISK_2: restored backup piece 1
piece handle=/atlas/data18a/oracle/o4ke45rt_1_1 tag=BACKUP_MUONCAL.GRI_050309052247
channel ORA_DISK_2: restore complete, elapsed time: 00:01:46
channel ORA_DISK_3: restored backup piece 1
piece handle=/atlas/data18a/oracle/o5ke45rt_1_1 tag=BACKUP_MUONCAL.GRI_050309052247
channel ORA_DISK_3: restore complete, elapsed time: 00:01:46
channel ORA_DISK_4: restored backup piece 1
piece handle=/atlas/data18a/oracle/o3ke45rs_1_1 tag=BACKUP_MUONCAL.GRI_050309052247
channel ORA_DISK_4: restore complete, elapsed time: 00:02:21
channel ORA_DISK_1: restored backup piece 1
piece handle=/atlas/data18a/oracle/o2ke45rs_1_1 tag=BACKUP_MUONCAL.GRI_050309052247
channel ORA_DISK_1: restore complete, elapsed time: 00:02:46
Finished restore at 05-MAY-09

Starting recover at 05-MAY-09
using channel ORA_DISK_1
using channel ORA_DISK_2
using channel ORA_DISK_3
using channel ORA_DISK_4

starting media recovery

archive log thread 1 sequence 5758 is already on disk as file +DATAFLASH/muoncal/1_5758_596035279.dbf
archive log filename=+DATAFLASH/muoncal/1_5758_596035279.dbf thread=1 sequence=5758
media recovery complete, elapsed time: 00:00:01
Finished recover at 05-MAY-09

It worked! However I don't have any dbconsole operational at this point. I need to recreate it I guess.

Try connecting to sqlplus as sysdba:


SQL> alter database open resetlogs;

Database altered.

Success! (Thanks to Jacek Wojcieszuk and Carlos Fenando Gamboa!)

Related info. There is a possible helpful reference at http://www.dbapool.com/printarticle.php?articles=0604200802 but this link mentions having the same OS and the same path on the old and new system.

I do have sqplus access as SYSDBA from the new system to the old system via the private network. I am trying to work out the best method to get the old database operational on the new system. It seems there should be a straightforward way (via RMAN?) to dump from the old running database to the new host. Both are running Oracle 10.2.0.4 and ASM (same diskgroup "DATAFLASH" on old and new systems). Anyway the above sequence seems to have worked.

Now lets try to recreate the DB console: (NOTE: First I needed to reset the dbsnmp user password via 'alter user...')

muoncal on umors.grid.umich.edu: emca -config dbcontrol db -repos create

STARTED EMCA at May 5, 2009 11:16:04 AM
EM Configuration Assistant, Version 10.2.0.1.0 Production
Copyright (c) 2003, 2005, Oracle.  All rights reserved.

Enter the following information:
Database SID: muoncal
Listener port number: 1526
Password for SYS user:
Password for DBSNMP user:
Password for SYSMAN user:
Email address for notifications (optional): smckee@umich.edu
Outgoing Mail (SMTP) server for notifications (optional): smtp.itd.umich.edu
ASM ORACLE_HOME [ /opt/app/oracle/oracle/product/10.2.0/asm ]:
ASM SID [ +ASM ]:
ASM port [ 1526 ]:
ASM user role [ SYSDBA ]:
ASM username [ SYS ]:
ASM user password:
-----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ /opt/app/oracle/oracle/product/10.2.0/asm

Database hostname ................ umors.grid.umich.edu
Listener port number ................ 1526
Database SID ................ muoncal
Email address for notifications ............... smckee@umich.edu
Outgoing Mail (SMTP) server for notifications ............... smtp.itd.umich.edu
ASM ORACLE_HOME ................ /opt/app/oracle/oracle/product/10.2.0/asm
ASM SID ................ +ASM
ASM port ................ 1526
ASM user role ................ SYSDBA
ASM username ................ SYS

-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: y
May 5, 2009 11:16:50 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /opt/app/oracle/oracle/product/10.2.0/asm/cfgtoollogs/emca/muoncal/emca_2009-05-05_11-16-04-AM.log.
May 5, 2009 11:16:50 AM oracle.sysman.emcp.EMReposConfig createRepository
INFO: Creating the EM repository (this may take a while) ...
May 5, 2009 11:16:50 AM oracle.sysman.emcp.EMReposConfig invoke
SEVERE: Error creating the repository
May 5, 2009 11:16:50 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Refer to the log file at /opt/app/oracle/oracle/product/10.2.0/asm/cfgtoollogs/emca/muoncal/emca_repos_create_<date>.log for more details.
May 5, 2009 11:16:50 AM oracle.sysman.emcp.EMConfig perform
SEVERE: Error creating the repository
Refer to the log file at /opt/app/oracle/oracle/product/10.2.0/asm/cfgtoollogs/emca/muoncal/emca_2009-05-05_11-16-04-AM.log for more details.
Could not complete the configuration. Refer to the log file at /opt/app/oracle/oracle/product/10.2.0/asm/cfgtoollogs/emca/muoncal/emca_2009-05-05_11-16-04-AM.log for more details.

The problem was an existing SYSMAN setup. I followed this Metalink article to fix it: https://metalink.oracle.com/metalink/plsql/showdoc?db=NOT&id=278100.1&blackframe=1 (manual removal in 4).

Then I retried:
muoncal on umors.grid.umich.edu: emca -config dbcontrol db -repos create

STARTED EMCA at May 5, 2009 11:37:26 AM
EM Configuration Assistant, Version 10.2.0.1.0 Production
Copyright (c) 2003, 2005, Oracle.  All rights reserved.

Enter the following information:
Database SID: muoncal
Listener port number: 1526
Password for SYS user:
Password for DBSNMP user:
Password for SYSMAN user:
Email address for notifications (optional): smckee@umich.edu
Outgoing Mail (SMTP) server for notifications (optional): smtp.itd.umich.edu
ASM ORACLE_HOME [ /opt/app/oracle/oracle/product/10.2.0/asm ]:
ASM SID [ +ASM ]:
ASM port [ 1526 ]:
ASM user role [ SYSDBA ]:
ASM username [ SYS ]:
ASM user password: ssh(2765525) debug2: channel 0: window 32750 sent adjust 32786
ASM user password:
-----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ /opt/app/oracle/oracle/product/10.2.0/asm

Database hostname ................ umors.grid.umich.edu
Listener port number ................ 1526
Database SID ................ muoncal
Email address for notifications ............... smckee@umich.edu
Outgoing Mail (SMTP) server for notifications ............... smtp.itd.umich.edu
ASM ORACLE_HOME ................ /opt/app/oracle/oracle/product/10.2.0/asm
ASM SID ................ +ASM
ASM port ................ 1526
ASM user role ................ SYSDBA
ASM username ................ SYS

-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: y
May 5, 2009 11:38:07 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /opt/app/oracle/oracle/product/10.2.0/asm/cfgtoollogs/emca/muoncal/emca_2009-05-05_11-37-26-AM.log.
May 5, 2009 11:38:08 AM oracle.sysman.emcp.EMReposConfig createRepository
INFO: Creating the EM repository (this may take a while) ...
May 5, 2009 11:40:05 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully created
May 5, 2009 11:40:22 AM oracle.sysman.emcp.ParamsManager getLocalListener
WARNING: Error retrieving listener for umors.grid.umich.edu
May 5, 2009 11:40:23 AM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
INFO: Securing Database Control (this may take a while) ...
May 5, 2009 11:40:25 AM oracle.sysman.emcp.util.PlatformInterface executeCommand
WARNING: Error executing /opt/app/oracle/oracle/product/10.2.0/asm/bin/emctl config emkey -repos
May 5, 2009 11:40:25 AM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
WARNING: Configuring EM-Key failed.
May 5, 2009 11:40:25 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration
WARNING: Error securing Database control.
May 5, 2009 11:40:25 AM oracle.sysman.emcp.EMDBPostConfig setWarnMsg
INFO: Error securing Database Control, Database Control has been brought up in non-secure mode. To secure the Database Control execute the following command(s):

 1) Set the environment variable ORACLE_SID to muoncal
 2) /opt/app/oracle/oracle/product/10.2.0/asm/bin/emctl stop dbconsole
 3) /opt/app/oracle/oracle/product/10.2.0/asm/bin/emctl config emkey -repos -sysman_pwd < Password for SYSMAN user >
 4) /opt/app/oracle/oracle/product/10.2.0/asm/bin/emctl secure dbconsole -sysman_pwd < Password for SYSMAN user >
 5) /opt/app/oracle/oracle/product/10.2.0/asm/bin/emctl start dbconsole

 To secure Em Key, run /opt/app/oracle/oracle/product/10.2.0/asm/bin/emctl config emkey -remove_from_repos -sysman_pwd < Password for SYSMAN user >
May 5, 2009 11:40:25 AM oracle.sysman.emcp.util.DBControlUtil startOMS
INFO: Starting Database Control (this may take a while) ...
May 5, 2009 11:41:59 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: Database Control started successfully
May 5, 2009 11:41:59 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: >>>>>>>>>>> The Database Control URL is http://umors.grid.umich.edu:1158/em <<<<<<<<<<<


Error securing Database Control, Database Control has been brought up in non-secure mode. To secure the Database Control execute the following command(s):

 1) Set the environment variable ORACLE_SID to muoncal
 2) /opt/app/oracle/oracle/product/10.2.0/asm/bin/emctl stop dbconsole
 3) /opt/app/oracle/oracle/product/10.2.0/asm/bin/emctl config emkey -repos -sysman_pwd < Password for SYSMAN user >
 4) /opt/app/oracle/oracle/product/10.2.0/asm/bin/emctl secure dbconsole -sysman_pwd < Password for SYSMAN user >
 5) /opt/app/oracle/oracle/product/10.2.0/asm/bin/emctl start dbconsole

 To secure Em Key, run /opt/app/oracle/oracle/product/10.2.0/asm/bin/emctl config emkey -remove_from_repos -sysman_pwd < Password for SYSMAN user >

Next step is to secure as is noted above.

Actually removing and recreating as above worked to create a secure connection. The problem above was that the /etc/oratab didn't include the muoncal instance.

I also need to fix the missing orapwmuoncal file by following: Doc ID: 274115.1

Then I could login to https://umors.grid.umich.edu:1158/em

Next was to restore the block change tracking:

SQL> alter database enable block change tracking;

Database altered.

-- ShawnMcKee - 04 May 2009
Topic revision: r7 - 05 May 2009, ShawnMcKee
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback