Oracle 19c DBA Portal

Oracle 19c Enterprise Edition (19.24) • Complete Instructor Notes & Definitions

Oracle DBA Complete Knowledge Portal

Your complete, uncondensed Oracle 19c DBA Learning Portal—faithful to your instructor's exact explanations, syntax, definitions, and classroom examples from Day 1 (Linux, Users, Profiles, Tablespaces, Architecture) through RMAN, Data Pump, Data Guard, all 6 DR Restorations, all 6 DC–DR Out-of-Sync labs, Control File/ORL/PFILE recovery, Patching, and the Verbatim Interview Definitions Bank.

19
Full Modules
13
Action Labs
8
Wisdom Sets

1. Oracle DBA Overview, Version History & Linux Essentials

Foundation • RHEL/OL 7.9
Oracle Database Release History (Instructor Notes)
  • 9i / 10g: Legacy Grid/Internet versions (do not claim in 4+ yr resume).
  • 11g (11.2.0.4): Stable classic production release; Snapshot Standby required manual Guaranteed Restore Point (GRP).
  • 12c (12.1 / 12.2): Introduced Multitenant (CDB/PDB), online `move datafile`, Service-based RMAN restore, automatic GRP on Snapshot Standby.
  • 18c: Short-lived innovation release ("disaster version, almost no enterprise uses it").
  • 19c (19.3 base → 19.24 RU): Current Long-Term Support (LTS) production standard across banking & enterprise.
  • 21c / 23c / 26ai: Newer upgrade targets; upgrades always follow Oracle's Compatibility Matrix.
Daily DBA Ownership: DB & Listener uptime, Tablespace monitoring (>70% alert), Archive mount monitoring, RMAN L0/L1 backup verification, DC–DR Sync (`Difference = 0`), Alert Log review, User/Tablespace tickets, UAT Data Pump refreshes, Quarterly DR Switchover & Patching.
Essential Linux Commands & Environment Bootstrapping
# 1) VMware Linux Network Bootstrap (when ifconfig has no IP):
su - root
dhclient
ifconfig -a                  # Get IP (DC: 192.168.44.129 | DR: 192.168.44.128)

# 2) Server & Oracle Discovery (First 60 seconds on any new server):
uname -a                     # Verify OS kernel (Linux x86_64)
df -h                        # Mount points & free space (/oracle, /data1, /data2, /redo1, /archives)
ls -ld /archives             # Verify mount ownership/permissions (oracle:oinstall) before mkdir!
du -sh *                     # Size of each file/folder in current directory
ps -ef | grep pmon           # Check running Oracle DB instances
ps -ef | grep tns            # Check running Listeners (also reveals $ORACLE_HOME path!)
ps -ef | grep mrp            # Check if DR MRP0 recovery process is running
pwd && ls -lrt               # Present directory & timestamp-sorted files
cd ../network/admin          # Quick 1-step jump from $ORACLE_HOME/dbs to network/admin
mkdir -p /data2/oradata/ICICI/redo
cp -p source_file target_path
scp -pr * oracle@192.168.44.128:/data2/RMANBACKUP/RMANBACKUP_12NOV

# 4) Export Oracle Environment (Required on every fresh terminal):
export ORACLE_SID=ICICI
export ORACLE_HOME=/oracle/app/product/19.3/dbhome_1
export PATH=$ORACLE_HOME/bin:$PATH
env | grep ORA

2. Oracle Database Architecture (Instance, Memory, Processes & Storage)

Instance + Database
Memory Structures (SGA & PGA)

Instance = SGA + Background Processes (in RAM; lost on shutdown).

  • Shared Pool: Library Cache (parsed SQL/PLSQL execution plans) + Data Dictionary Cache (metadata).
  • Database Buffer Cache: Holds copies of data blocks read from disk (`Dirty`, `Clean`, `Pinned`, `Free`).
  • Redo Log Buffer: Circular memory buffer storing all DML/DDL change vectors before LGWR flushes to disk.
  • Large Pool: Used by RMAN I/O slaves, parallel queries, and shared server.
  • Java & Streams Pool: JVM code and replication/AQ buffering.
  • PGA (Program Global Area): Private non-shared memory per server process for sorting, hash joins, and session cursors.
Key Background Processes
  • PMON (Process Monitor): Cleans up failed user sessions, releases locks/resources.
  • SMON (System Monitor): Performs automatic instance/crash recovery on startup & cleans temporary segments.
  • DBWn (Database Writer): Writes modified (dirty) buffers from DB Buffer Cache to physical Datafiles on checkpoint or threshold.
  • LGWR (Log Writer): Flushes Redo Log Buffer to Online Redo Log files on `COMMIT`, every 3 sec, 1/3 full, or before DBWn writes.
  • CKPT (Checkpoint): Signals DBWn to flush dirty blocks and updates Control File & Datafile headers with the latest SCN.
  • ARCn (Archiver): Copies full Online Redo Logs to Archive Log Destination (`log_archive_dest_1`) before overwrite.
  • LREG (Listener Registration): Dynamically registers DB service/instance with the Listener (formerly PMON in 11g).
  • RECO (Recoverer): Resolves in-doubt distributed transactions.
Physical vs. Logical Architecture

Database = Physical Files on Disk (persistent).

  • Core Physical Files: Datafiles (`.dbf`), Control Files (`.ctl` — heart/brain storing DB name, SCN, file paths), Online Redo Logs (`.log`).
  • Supporting Physical Files: Parameter File (`spfile`/`init.ora`), Password File (`orapw<SID>`), Archive Logs (`.arc`), Alert & Trace Logs.
  • Logical Hierarchy (Top → Down):
    Database → Tablespace → Segment (Table/Index) → Extent → Oracle Block (8 KB)
  • Physical Mapping: 1 Tablespace → 1 or more Datafiles; 1 Datafile belongs to ONLY 1 Tablespace; 1 Oracle Block → multiple OS blocks.

3. Database Startup Stages, Shutdown Modes & Alert Log Navigation

NOMOUNT → MOUNT → OPEN
Stage 1: NOMOUNTReads PFILE/SPFILE

Allocates SGA & starts background processes. Instance is started, database is NOT mounted (`ORA-01507` if you query `v$database`). Used for: Manual DB Creation, Restoring Control File, checking `show parameter control;`.

Stage 2: MOUNTReads Control Files

Opens Control Files and verifies identical versions across multiplexed copies. Applications CANNOT connect. Used for: Enabling Archivelog mode, RMAN Restore/Recovery, Physical Standby DR (`MRP0`), Renaming/Moving files, Dropping DB (`exclusive restrict`).

Stage 3: OPENOpens DFs & ORLs

Verifies SCN sync between Control File and Datafile headers, opens Datafiles & Online Redo Logs. Primary opens `READ WRITE`; Active Data Guard opens `READ ONLY WITH APPLY`.

4 Shutdown Modes Comparison
ModeBehaviorProd Usage
NORMALWaits for ALL users to disconnect manually.Avoid (hangs forever)
TRANSACTIONALWaits for active txns to commit/rollback.Rarely used
IMMEDIATERolls back uncommitted txns, disconnects users, clean checkpoint.Standard Default
ABORTInstant crash stop (no checkpoint); SMON recovers on next start.Emergency / Live CF deleted
Startup, Status & Alert Log Lookup Commands
-- Always start with MOUNT first to isolate PFILE/CF vs Datafile errors:
sqlplus / as sysdba
startup mount;
alter database open;
select inst_id,name,db_unique_name,open_mode,log_mode,database_role,switchover_status from gv$database;

-- Find Alert Log inside SQL*Plus:
select value from v$diag_info where name='Diag Trace';
-- Find Alert Log from OS even when DB is DOWN:
adrci                        # Type: show home  -> exit
cd /oracle/app/diag/diag/rdbms/icici/ICICI/trace
tail -100f alert_ICICI.log

4. User Management, Roles, Privileges & Profiles (Complete Syntax)

DBA_USERS • DBA_PROFILES
User Creation, Modification, Locking & Dropping

Never leave application users on the `SYSTEM` tablespace—always assign a dedicated default tablespace and quota.

-- 1) Basic User Creation (lands in SYSTEM initially) + Reassign Default TS:
create user teja identified by teja123;
alter user teja default tablespace TEJA;

-- 2) Full Production User Creation Syntax:
create user app_user identified by "App#2026"
  default tablespace TEJA
  temporary tablespace TEMP1
  quota unlimited on TEJA
  profile APP_PROFILE;

-- 3) Account & Password Maintenance:
alter user teja identified by newpass123;
alter user teja account unlock;
alter user teja account lock;
alter user teja password expire;
alter user teja quota 500m on TEJA;
drop user teja cascade;          -- Drops user AND all owned tables/objects!

-- 4) System Privileges, Object Privileges & Roles:
grant connect, resource to teja;
grant create session, create table, create view to teja;
grant select, insert, update on hr.employees to teja;
revoke create table from teja;
create role app_ro_role;
grant select on hr.employees to app_ro_role;
grant app_ro_role to teja;
Security Profiles & Dictionary Views (`u.sql`)

Profiles enforce password aging (e.g., 90-day banking rotation), failed-login lockout, and idle session timeouts.

-- Create & Assign Custom Security Profile:
create profile APP_PROFILE limit
  failed_login_attempts 3
  password_lock_time 1
  password_life_time 90
  password_grace_time 7
  password_reuse_time 365
  password_reuse_max 5
  idle_time 15
  sessions_per_user 10;

alter user teja profile APP_PROFILE;
-- To prevent service account password expiry:
alter profile SERVICE_PROF limit password_life_time unlimited;

-- Inspect Users, Profiles & Privileges:
select username, account_status, default_tablespace, temporary_tablespace, profile, expiry_date
from dba_users where username='TEJA';
select resource_name, limit from dba_profiles where profile='APP_PROFILE';
select * from dba_role_privs where grantee='TEJA';
select * from dba_sys_privs where grantee='TEJA';

-- Reusable u.sql Script (/home/oracle/u.sql):
set lines 199 pages 1999
col USERNAME for a45
select USERNAME,ACCOUNT_STATUS,DEFAULT_TABLESPACE from dba_users;
select USERNAME,ACCOUNT_STATUS,DEFAULT_TABLESPACE from dba_users where USERNAME='&USER';

5. Tablespace & Datafile Management, Sizing Queries & `ts.sql`

70% Alert • 1024 DF Max
Tablespace Types, Limits & Creation/Resize Commands
  • 3 Tablespace Types: Permanent (SYSTEM, SYSAUX, User data), Undo (rollback, read consistency, flashback), Temporary (disk sorting, hash joins).
  • Smallfile (Default): Max 1024 datafiles × 32 GB (at 8K block size) = 32 TB. When it hits 1024 DFs, create a new tablespace & change user's default TS.
  • Bigfile: Single giant datafile up to 32 TB (cannot add a 2nd datafile).
  • 70% Threshold Rule: Add a datafile or resize as soon as usage exceeds 70%. Always verify `df -h` first!
-- Create Permanent, Undo & Temporary Tablespaces:
create tablespace TEJA datafile '/data1/oradata/ICICI/data/teja01.dbf' size 100m;
create tablespace APP_TS datafile '/data1/oradata/ICICI/data/app01.dbf'
  size 1g autoextend on next 100m maxsize 30g;
create undo tablespace UNDOTBS2 datafile '/data1/oradata/ICICI/data/undo02.dbf' size 2g;
create temporary tablespace TEMP2 tempfile '/data1/oradata/ICICI/data/temp02.dbf' size 2g;

-- Add Datafiles, Resize, Enable Autoextend & Online Move (12c/19c):
alter tablespace TEJA add datafile '/data1/oradata/ICICI/data/teja02.dbf' size 100m;
alter database datafile '/data1/oradata/ICICI/data/teja01.dbf' resize 500m;
alter database datafile '/data1/oradata/ICICI/data/teja01.dbf' autoextend on next 100m maxsize 30g;
alter database move datafile '/data1/oradata/ICICI/data/teja01.dbf' to '/data2/oradata/ICICI/data/teja01.dbf';

-- Drop Tablespace (and physical OS files):
drop tablespace TEJA including contents and datafiles;
Physical vs. Logical Size, TEMP Usage & `ts.sql`
-- Physical Allocated DB Size (GB):
select sum(bytes)/1024/1024/1024 "Physical_GB" from dba_data_files;
-- Logical Actual Data Size (GB):
select sum(bytes)/1024/1024/1024 "Logical_GB" from dba_segments;
-- Free Space = Physical_GB - Logical_GB

-- Datafiles for a specific Tablespace (Part 2 of ts.sql):
set lines 230 pages 200
col file_name for a80
col tablespace_name for a20
select d.file_id, d.tablespace_name, d.file_name, v.creation_time, v.bytes/1024/1024/1024 GB
from dba_data_files d, v$datafile v
where tablespace_name=upper('&tablespace_name') and file_id=file#
order by v.creation_time;

-- TEMP Tablespace Usage Query:
SELECT A.tablespace_name tablespace, D.mb_total,
       SUM(A.used_blocks * D.block_size)/1024/1024 mb_used,
       D.mb_total - SUM(A.used_blocks * D.block_size)/1024/1024 mb_free
FROM v$sort_segment A,
     (SELECT B.name, C.block_size, SUM(C.bytes)/1024/1024 mb_total
      FROM v$tablespace B, v$tempfile C WHERE B.ts#=C.ts# GROUP BY B.name, C.block_size) D
WHERE A.tablespace_name = D.name GROUP BY A.tablespace_name, D.mb_total;

6. Parameter Files (PFILE vs. SPFILE) & Static vs. Dynamic Parameters

$ORACLE_HOME/dbs
PFILE vs. SPFILE & Startup Search Order
  • PFILE (`init<SID>.ora`): Plain text, editable with `vi`, client-side; changes made via `alter system` are NOT persisted across restarts.
  • SPFILE (`spfile<SID>.ora`): Binary server-side parameter file (NEVER edit with `vi`!). Persists `alter system` changes across restarts. Always run Production on SPFILE!
  • Startup Precedence in `$ORACLE_HOME/dbs`: 1) `spfile<SID>.ora` → 2) `spfile.ora` → 3) `init<SID>.ora`.
  • Static vs. Dynamic (`v$parameter`): `ISSYS_MODIFIABLE='FALSE'` = Static (requires `scope=spfile` + DB bounce; omitting `scope=spfile` throws `ORA-02095`). `IMMEDIATE` = Dynamic (`scope=memory | spfile | both`).
  • 3 Scope Values: `scope=both` (writes to memory + SPFILE — default), `scope=spfile` (writes only to SPFILE — needs bounce), `scope=memory` ("up to down" — active in RAM only, lost on shutdown).
  • Archive Format Tokens (`ICICI_%t_%s_%r.arc`): `%t` = thread/timestamp, `%s` = log sequence number (increments sequentially), `%r` = resetlogs ID / DBID.
  • Explicit Path Rule: If `dbs/initICICI.ora` is missing and you booted from `/tmp/...`, plain `create spfile from pfile;` FAILS (`LRM-00109`)—you MUST specify `create spfile from pfile='/tmp/...';`!
Exact `initICICI.ora` Template & Parameter Commands
# Exact initICICI.ora (On DR, set *.db_unique_name='ICICI_DR'!)
*.audit_file_dest='/oracle/app/audit'
*.audit_trail='db'
*.control_files='/oracle/app/control/ICICI/control01.ctl','/data1/control/ICICI/control02.ctl'
*.db_block_size=8192
*.db_name='ICICI'
*.db_unique_name='ICICI'
*.diagnostic_dest='/oracle/app/diag'
*.log_archive_dest_1='LOCATION=/archives/ICICIARCH'
*.log_archive_format='ICICI_%t_%s_%r.arc'
*.processes=100
*.undo_tablespace='UNDOTBS'

-- SQL*Plus Parameter Checks & Backups:
show parameter pfile;             -- Blank value = running on PFILE!
select name, value, issys_modifiable from v$parameter where name in ('processes','open_cursors','control_files');
create pfile='/tmp/initICICI_bkp_12112026.ora' from spfile;
create spfile from pfile='/tmp/initICICI_bkp_12112026.ora';

7. Manual Database Creation & Archive Log Mode Management

dbcreation.sql • ARCHIVELOG
Manual Database Creation Steps (CLI Method)
  1. Create OS directories (`/data1/oradata/ICICI/data`, `/data1/control/ICICI`, `/redo1/oradata/ICICI/redo`, `/oracle/app/audit`, `/oracle/app/diag`, `/archives/ICICIARCH`).
  2. Create `initICICI.ora` in `$ORACLE_HOME/dbs` and password file via `orapwd`.
  3. Export `ORACLE_SID=ICICI`, connect `sqlplus / as sysdba`, run `startup nomount;`.
  4. Execute `@dbcreation.sql` (`CREATE DATABASE ICICI ... DATAFILE ... SYSAUX ... UNDO ... DEFAULT TEMPORARY ... LOGFILE GROUP 1/2/3 ...`).
  5. Run Data Dictionary scripts: `@?/rdbms/admin/catalog.sql`, `@?/rdbms/admin/catproc.sql`, and as `SYSTEM`: `@?/sqlplus/admin/pupbld.sql`.
  6. Run `create spfile from pfile;` and bounce to verify SPFILE startup.
Classroom Gotcha (`SBI` Lab): If querying `dba_data_files` or `dba_segments` throws `ORA-00942: table or view does not exist` on a newly created DB, `catalog.sql` / `catproc.sql` did not finish running cleanly!
Mutual `tnsnames.ora`, `tnsping` & Password File (`orapwd`)
# Both DC & DR must contain BOTH entries in tnsnames.ora:
ICICI_DC =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.44.129)(PORT = 1595))
    (CONNECT_DATA = (SERVER = DEDICATED)(SERVICE_NAME = ICICI))
  )
ICICI_DR1 =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.44.128)(PORT = 1595))
    (CONNECT_DATA = (SERVER = DEDICATED)(SERVICE_NAME = ICICI))
  )

# Connectivity & Firewall Verification:
tnsping ICICI_DC             # Must return OK (0-10 msec)
tnsping ICICI_DR1
telnet 192.168.44.128 1595   # Tests firewall port reachability

# Password File Creation & 90-Day SYS Password Rotation Sync:
orapwd file=$ORACLE_HOME/dbs/orapwICICI password=oracle entries=10 force=y
# Whenever SYS password changes on DC, MUST copy to DR immediately:
scp -pr $ORACLE_HOME/dbs/orapwICICI oracle@192.168.44.128:$ORACLE_HOME/dbs/

9. RMAN Backups, Incremental L0/L1, BCT & `nohup` Execution

Space = DB/8 • Channels = CPU/2
RMAN Concepts, Sizing Math & BCT
  • Physical vs. Logical: RMAN = Physical (block-level, supports point-in-time recovery). Data Pump = Logical (object-level).
  • Full vs. Level 0 vs. Level 1: `FULL` cannot be a parent for incremental backups. Weekly Level 0 is the base; daily Level 1 backs up only changed blocks (Differential = since last L0/L1; Cumulative = since last L0). If L1 runs without L0, Oracle auto-promotes it to L0.
  • Production Sizing Formulas:
    • Compressed Backupset Size ≈ DB Size / 8 (6 GB → ~800 MB; 100 TB → ~12.5 TB).
    • Parallel Channels = CPU_COUNT / 2 (`show parameter cpu;`: 4 CPUs → 2 channels). RMAN splits datafiles evenly across channels (e.g., 50 DFs → 25 on `c1`, 25 on `c2`).
  • Piece Size Audit: Always verify backup piece sizes in the target directory with `du -sh *` (confirm datafiles, archives, current CF & standby CF all landed).
-- Enable & Verify Block Change Tracking (BCT):
alter database enable block change tracking using file '/data1/oradata/ICICI/bct_icici.chg';
select status, filename from v$block_change_tracking;

-- Interview Command (Short Form):
backup as compressed backupset database plus archivelog;
backup as compressed backupset incremental level 0 database plus archivelog;
backup as compressed backupset incremental level 1 database plus archivelog;

-- RMAN Maintenance Commands:
list backup summary;
crosscheck backup;
crosscheck archivelog all;
delete noprompt obsolete;
delete noprompt expired archivelog all;
Practical Production Script (`backup.cmd`) + `nohup` Background Execution

Always put `.cmd` one folder *above* the dated backup folder so logs & pieces stay clean. Take BOTH `current controlfile` (UAT clone) and `standby controlfile` (DR build) in one run.

# 1) Prepare dated directory & step one level up:
mkdir -p /data2/RMANBACKUP/RMAN_BACKUP_12NOV2026
cd /data2/RMANBACKUP

# 2) vi backup.cmd
run {
  allocate channel c1 type disk;
  allocate channel c2 type disk;
  backup database format '/data2/RMANBACKUP/RMAN_BACKUP_12NOV2026/datafiles_%d_%T_%U';
  backup archivelog all format '/data2/RMANBACKUP/RMAN_BACKUP_12NOV2026/archives_%d_%T_%U';
  backup current controlfile format '/data2/RMANBACKUP/RMAN_BACKUP_12NOV2026/currentcontrol_%d_%T_%U';
  backup current controlfile for standby format '/data2/RMANBACKUP/RMAN_BACKUP_12NOV2026/standbycontrol_%d_%T_%U';
  release channel c1;
  release channel c2;
}

# 3) Launch in background (survives PuTTY socket drops!):
nohup rman target / cmdfile=backup.cmd log=backup_12NOV2026.log &
jobs
tail -100f backup_12NOV2026.log
# Done when "released channel: c1/c2" & "Recovery Manager complete." appear.

10. Data Pump (`expdp` / `impdp`) — Logical Backups & UAT Refreshes

Logical Export / Import
Key Parameters, 4 Speed Boosters & UAT Pre-Checks
  • 4 Levels: `full=y`, `schemas=TEJA`, `tables=TEJA.EMP`, `tablespaces=TEJA`.
  • 4 Speed-Up Options: `parallel=4`, `compression=all`, `dumpfile=exp_%U.dmp` (splits across parallel workers), and `access_method=direct_path` (`direct=Y` bypasses SQL/Buffer Cache).
  • `table_exists_action`: `SKIP` (default), `APPEND`, `TRUNCATE`, `REPLACE` (drops & recreates — needs App Team approval!).
  • UAT Pre-Check: Always verify the target tablespace exists on UAT before `impdp` (or use `remap_tablespace=PROD_TS:UAT_TS`).
Complete `expdp`, `impdp` & Live Pause/Resume Commands
-- 1) Create Oracle Directory Object:
!mkdir -p /data2/EXPDP
create or replace directory EXP_DIR as '/data2/EXPDP';
grant read, write on directory EXP_DIR to system;
select directory_name, directory_path from dba_directories;

-- 2) Parallel Compressed Schema Export (in nohup):
nohup expdp \"/ as sysdba\" directory=EXP_DIR dumpfile=teja_%U.dmp logfile=exp_teja.log \
  schemas=TEJA parallel=4 compression=all job_name=EXP_TEJA_JOB &

-- 3) Schema Import with Remap on UAT:
nohup impdp \"/ as sysdba\" directory=IMP_DIR dumpfile=teja_%U.dmp logfile=imp_teja.log \
  remap_schema=TEJA:TEJA_UAT remap_tablespace=TEJA:USERS \
  parallel=4 table_exists_action=replace &

-- 4) Pause/Resume Live Job on High CPU Alert:
expdp \"/ as sysdba\" attach=EXP_TEJA_JOB
Export> status
Export> stop_job=immediate    # Later resume with: start_job

11. Oracle Data Guard Architecture, Protection Modes & Live Sync Verification

LNS/ARCH → RFS → MRP0
1. Maximum Performance (Default)

`ASYNC` redo shipping. Zero performance hit on Primary; potential 1–2 min data loss on sudden DC crash. Best for long-distance DR (e.g., Hyd ↔ Mumbai).

2. Maximum Availability (Zero Loss)

`SYNC` (`NOAFFIRM` ack from SRL; `AFFIRM` waits for DB apply — too slow) + SRLs (`ORL + 1` on BOTH DC & DR). On network glitch, waits `NET_TIMEOUT` (30s) × `RETRIES` (5) = 150 sec, then auto-enters `RESYNCHRONIZATION` (Max Performance behavior) so Production stays UP.

3. Maximum Protection (Avoid!)

`SYNC AFFIRM`. Strict zero loss, but if DR or network fails, Primary Production shuts down/hangs! Never used in banking.

Real-Time Apply vs. Delay Apply: In Real-Time Apply, MRP reads redo directly from active SRLs as RFS writes it. In Delay Apply, SRLs archive to remote archive files first and MRP applies from those archive files later. Also note: create both ORLs & SRLs on BOTH DC & DR so Switchover works seamlessly in either direction!
Data Guard Parameters & Monitoring Queries
-- DC Dest_2 (Points to DR | ENABLE):
ALTER SYSTEM SET log_archive_dest_2='service=ICICI_DR1 nodelay db_unique_name=ICICI_DR';
ALTER SYSTEM SET log_archive_dest_state_2=enable sid='*' scope=both;

-- DR Dest_2 (Points to DC | MUST stay DEFER while Standby!):
ALTER SYSTEM SET log_archive_dest_2='service=ICICI_DC nodelay db_unique_name=ICICI';
ALTER SYSTEM SET log_archive_dest_state_2=defer sid='*' scope=both;
ALTER SYSTEM SET standby_file_management=AUTO;

-- Production Hardening (Eliminates alert log warnings & enables auto-FAL):
ALTER SYSTEM SET log_archive_config='dg_config=(ICICI,ICICI_DR)';
ALTER SYSTEM SET fal_server='ICICI_DC';       -- On DR (set 'ICICI_DR1' on DC)

-- Live Process & Sync Gap Verification on DR:
SELECT PROCESS, PID, STATUS, THREAD#, BLOCK#, SEQUENCE# FROM V$MANAGED_STANDBY;
SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Received", APPL.SEQUENCE# "Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Diff"
FROM (SELECT THREAD#, SEQUENCE# FROM GV$ARCHIVED_LOG WHERE (THREAD#, FIRST_TIME) IN (SELECT THREAD#, MAX(FIRST_TIME) FROM GV$ARCHIVED_LOG GROUP BY THREAD#)) ARCH,
     (SELECT THREAD#, SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#, FIRST_TIME) IN (SELECT THREAD#, MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL
WHERE ARCH.THREAD# = APPL.THREAD#;
Reading Live DR Alert Log Sync Proof
  • rfs: Archived Log entry added for T-1.S-277 → Archive 277 received on DR.
  • rfs: Opened log for T-1.S-278 → RFS is open and ready to receive next log 278.
  • PR00: Media Recovery Log .../ICICI_1_277...arc → MRP applied sequence 277.
  • PR00: Media Recovery Waiting for T-1.S-278 (in transit) → MRP is 100% caught up and waiting for DC's current redo!
  • DC Switchover Status Meanings: `TO STANDBY` = Synced; `RESOLVABLE GAP` = Lagging/RFS idle; `FAILED DESTINATION` = Dest error/Listener down.

12. All 6 DR Creation & Restoration Methods (Backup, Service & Active Duplicate)

Same vs. Different Mounts
# Restoration Method When to Use in Production Key RMAN Command / Difference
1 Backup-Based (Same Mounts) Large DBs (100 TB) or zero live CPU/network load allowed on DC. `restore standby controlfile from '...'` → `catalog start with` → `restore database;`
2 Backup-Based (Diff Mounts) DC uses `/data1`, DR only has `/data2`. `set newname for database to '/data2/.../%b';` → `restore database;` → `switch database to copy;`
3 Service-Based (Same Mounts) DB ≤ 1 TB, no extra backup disk space, run after 2:00 AM. `restore standby controlfile from service ICICI_DC;` → `restore database from service ICICI_DC using compressed backupset;`
4 Service-Based (Diff Mounts) On-the-fly network build with different DR mount paths. `set newname for database to '/data2/.../%b';` → `restore database from service ICICI_DC ...;` → `switch database to copy;`
5 Active Duplicate (Same Mounts) Single-command DR clone over network (`target` + `auxiliary`). `duplicate target database for standby from active database dorecover nofilenamecheck;`
6 Active Duplicate (Diff Mounts) Single-command DR clone to different mount points. Set `db_file_name_convert` & `log_file_name_convert` and OMIT `nofilenamecheck`!
Mandatory Pre-Drop & Post-Restore DR Checklist:

1) Before dropping old DR: take PFILE/CF backup & `report schema`, then `shut immediate;` → `startup mount exclusive restrict;` → `drop database;`.
2) After restoring DR: create SPFILE (`create spfile from pfile;` + bounce), drop & recreate DR Online Redo Logs (RMAN restores datafiles, NOT redo logs — prevents `ORA-00313`/`ORA-19527`), set `standby_file_management=AUTO`, keep `dest_state_2=DEFER` on DR, and start `MRP0`.

13. All 6 DC–DR Out-of-Sync Scenarios & Live Troubleshooting Notes

Complete Incident Matrix
# Out-of-Sync Scenario Symptom / Alert Log Clues Exact DBA Resolution
1 RFS Stuck / Transport Stalled DC shows `RESOLVABLE GAP` or `FAILED DESTINATION`; `gv$archive_dest_status` shows no error; no RFS on DR. On DC toggle `dest_state_2=defer` → `enable` + `alter system switch all logfile;`. RFS immediately wakes up on DR.
2 `standby_file_management=MANUAL` + New DC Datafile MRP terminates within 1 second of starting; `v$datafile` shows `$ORACLE_HOME/dbs/UNNAMED00004` (`Size 0` in `report schema`). Run `alter database create datafile '.../UNNAMED00004' as '/data1/.../teja01.dbf' size 100m;` → set `standby_file_management=AUTO` BEFORE starting MRP!
3 DR Mount Point 100% Full (Resize / New DF) MRP stops with `ORA-19502` / `ORA-27072` (no space on `/data1`) when DC adds or resizes a datafile. Move a datafile to free mount (`/data2`) using `alter database move datafile '/data1/...' to '/data2/...';` (or create new DF on `/data2`) → restart MRP.
4 `orapwd` Mismatch (SYS Password Changed) DC `gv$archive_dest_status` shows `ORA-01017: invalid username/password` / `ORA-16191` after 90-day rotation. `scp -pr $ORACLE_HOME/dbs/orapwICICI oracle@192.168.44.128:$ORACLE_HOME/dbs/` → toggle DC `dest_state_2` `DEFER → ENABLE`.
5 DR Listener Down / Firewall Port Blocked `tnsping ICICI_DR1` fails with `TNS-12541: no listener` or timeout. Start listener on DR (`lsnrctl start ICICI_DR1`); verify port reachability (`telnet 192.168.44.128 1595`); involve Network team if blocked.
6A DC Archives Deleted (RMAN Backup Exists) MRP stuck at `WAIT_FOR_GAP`; missing sequences exist in DC RMAN archivelog backup. On DC: `restore archivelog from sequence 281 until sequence 290;` → `scp -pr` to DR `/archives/ICICIARCH` → catalog on DR → bounce MRP.
6B DC Archives Deleted (Zero Backup) + New DFs (`7,8,9`) `FAL: Error fetching gap sequence 281 to 287`; MRP stuck at `WAIT_FOR_GAP`. Incremental Service Recovery (`restore standby controlfile from service ICICI_DC` → restore missing DFs `7,8,9` BEFORE recover → `recover database from service ICICI_DC noredo using compressed backupset;` via `nohup` → toggle `MANUAL` to recreate DR ORLs avoiding `ORA-01275` → set `AUTO` → open ADG & start MRP).

14. Planned Switchover, Switchback & Emergency Failover

Role Transitions
Planned Switchover (Reversible • Zero Data Loss)
  1. Pre-checks: App Team confirmation, Sync Gap = `0`, DC shows `SWITCHOVER_STATUS = TO STANDBY`.
  2. On DC: `alter database commit to switchover to physical standby with session shutdown;` (auto-shuts down DC → `ORA-01034`).
  3. On DR: Verify `TO PRIMARY` → `alter database commit to switchover to primary with session shutdown;` → `alter database open;` (auto-recreates `temp01.dbf`).
  4. On Old DC (New Standby): Reconnect → `startup mount;` → `alter system set log_archive_dest_state_2=defer sid='*' scope=both;` (prevents `ORA-16009`).
  5. On New Primary (DR): Toggle `dest_state_2` `DEFER → ENABLE` + `switch all logfile;` → start `MRP0` on New Standby (DC) → verify reverse sync!
Emergency Failover (Irreversible • DC Dead)

Executed ONLY when Production DC is completely lost and management approves DR activation:

-- On DR:
recover managed standby database finish;
alter database activate standby database;
alter database open;
-- Hand over DR IP/TNS to App Team!

Banking 2-Min Rule & Post-Failover Rule: In Max Performance, App Manager coordinates branches to replay the last 1–2 mins of transactions if DC crashed abruptly. When old DC comes back online, NEVER open it as Primary—drop and rebuild old DC as Standby!

15. Snapshot Standby & Active Data Guard (ADG)

FRA • Guaranteed Restore Point
Snapshot Standby Concept & FRA Rules
  • Purpose: Temporarily opens Physical Standby DR in `READ WRITE` mode for QA/App testing without touching Production.
  • How Rollback Works: Uses Flash Recovery Area (FRA) + Guaranteed Restore Point (GRP) (auto-created in 12c/19c; manual in 11g). All test writes are cleanly discarded upon revert.
  • Manual Restore Point Cleanup: If you manually create a GRP (`create guaranteed restore point teja_rp flashback database;`), you MUST run `drop restore point teja_rp;` once done—otherwise FRA grows endlessly until DB hangs! (Auto-created 12c+ Snapshot GRPs drop automatically on revert.)
  • Background Protection: While in Snapshot mode, RFS continues receiving DC archives (apply pauses until revert).
  • Strict FRA Order: Always set `db_recovery_file_dest_size` BEFORE `db_recovery_file_dest`. Ensure DR is on SPFILE & `dest_state_2=DEFER`.
Convert → Test → Revert to ADG (`READ ONLY WITH APPLY`)
-- 1) Configure FRA & Convert to Snapshot Standby:
!mkdir -p /archives/RECO
alter system set db_recovery_file_dest_size=2g;
alter system set db_recovery_file_dest='/archives/RECO';
recover managed standby database cancel;
alter database convert to snapshot standby;
alter database open;                -- READ WRITE / SNAPSHOT STANDBY

-- 2) Revert to Physical Standby & Open Active Data Guard (ADG):
shut immediate;
startup mount;
alter database convert to physical standby;
alter database open read only;
recover managed standby database disconnect from session;
-- Confirm READ ONLY WITH APPLY; if DC shows FAILED DESTINATION, run DEFER -> ENABLE on DC!

16. Control File Multiplexing, Redo Log (ORL/SRL) Resizing & PFILE/SPFILE Recovery

Safe-Side Redundancy
Control File Multiplex & Crash Recovery

Add 3rd Copy: `alter system set control_files='...','...','...' scope=spfile;` → `shut immediate;` BEFORE `cp` (prevents `ORA-00214`) → `startup mount;` → `open;`.
Deleted While Running (`ORA-00210`): `shut abort;` or `kill -9 <PMON_PID>` → `startup nomount;` → `cp` surviving CF → `mount` → `open`.
Deleted While Down (`ORA-00205`): Stays in `NOMOUNT` → `cp` surviving CF → `alter database mount;`.

ORL Resizing & SRL Matching

Lifecycle: `UNUSED → CURRENT → ACTIVE → INACTIVE`.
Resize (100M → 200M): ALWAYS ADD new groups (30–32) FIRST → `switch logfile` → `alter system checkpoint;` (clears `ACTIVE`/`ORA-01624`) → DROP `INACTIVE` old groups → `rm` OS files.
Member Rules: Can ADD member to `CURRENT`; can DROP member from `ACTIVE`/`INACTIVE` (`CURRENT` = `ORA-01609`).
SRLs: Always `ORL + 1` groups of matching size (200M).

4 Missing Parameter File Scenarios

1) Only SPFILE lost: DB silently boots on PFILE (`create spfile from pfile;` + bounce).
2) Only PFILE lost: DB boots on SPFILE (`create pfile from spfile;`).
3) Both lost + `/tmp` backup exists: `startup nomount pfile='/tmp/...';` → `create spfile from pfile='/tmp/...';` → bounce.
4) Both lost + ZERO backup: `adrci` → `show home` → copy non-default params from `alert_ICICI.log` into `initICICI.ora` → boot → create SPFILE.

17. Oracle Patching (`OPatch`, PSU / RU & `datapatch`)

n-1 Quarter Rule • DR First
Patch Types & Enterprise Strategy
  • PSU / RU (Release Update): Quarterly (Jan, Apr, Jul, Oct — mid-month) cumulative security + bug fixes for Linux/AIX. Applying Oct automatically includes a missed Jul patch!
  • Other Patch Types: CPU (deprecated — skips missed quarters), Bundle Patch (Windows ONLY), Interim / Bug Patch, OJVM, One-off, Timezone, Combo, and Hot Patch.
  • `n-1` Quarter Rule: Stay 1 quarter behind the newest PSU in Production so newly discovered bugs are already ironed out (e.g., if Oct 19.34 releases, apply Jul 19.33).
  • Order: Patch Standby DR first → verify → patch Production DC → run `datapatch -verbose` on Primary (SQL changes replicate to DR via redo!).
OPatch Execution Flow (Including OPatch Upgrade & 50 GB Space Check)
# 1) Read MOS README.txt & check OPatch version (Upgrade if lower than required):
cd $ORACLE_HOME/OPatch && ./opatch version
./opatch lsinventory
# If OPatch upgrade needed: mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch_bkp
# unzip new OPatch.zip in /data2 -> mv OPatch $ORACLE_HOME/ -> re-verify ./opatch version

# 2) Unzip PSU patch & run Conflict Check + System Space Check (min 50 GB free in $ORACLE_HOME):
cd /data2/PATCH/36582781
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./
$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -ph ./

# 3) Stop DB & Listener -> Apply Binary Patch -> Start DB & Listener:
$ORACLE_HOME/OPatch/opatch apply

# 4) Apply SQL Dictionary Changes (Primary OPEN READ WRITE):
$ORACLE_HOME/OPatch/datapatch -verbose
# Verify in SQL*Plus:
select patch_id, action, status, description from dba_registry_sqlpatch;

18. Verbatim Instructor Definitions & 4+ Yr DBA Interview Bank

Exact Classroom Phrasing
Core Concept Definitions (As Taught in Class)
  • Control File Multiplexing: "Safe side — maintaining 2 or 3 identical copies across different mount points (`/data1`, `/redo1`, `/data2`) so even if one copy is lost, the surviving copy restores the database immediately."
  • Why `alter system checkpoint;` during ORL Resize? "When a switched redo log stays `ACTIVE` (`ORA-01624`), invoking a checkpoint flushes all dirty buffers from memory to datafiles, turning `ACTIVE` into `INACTIVE` so we can safely drop the group."
  • What is `nohup`? "No Hangup — runs RMAN backup/restore in the Linux background so even if our PuTTY/VPN session disconnects, the job continues uninterrupted until `released channel`."
  • RFS vs. MRP0: "RFS (Remote File Server) receives redo/archives from Production into DR (`Opened log` = waiting for next sequence); MRP0 (Managed Recovery Process) reads those archives and applies them to DR datafiles (`Media Recovery Waiting ... in transit` = 100% synced)."
  • What is Snapshot Standby? "Converting a Physical Standby DR temporarily into `READ WRITE` mode for application testing using FRA + Guaranteed Restore Point, while RFS still receives DC archives; once testing ends, we revert it back to Physical Standby and all test data is cleanly wiped."
4+ Years Experienced DBA Self-Introduction & Interview Delivery
"Hi, I'm Vamsi, having 4+ years of hands-on experience as an Oracle Database Administrator supporting production 11g, 12c, and 19c databases on Linux. My day-to-day responsibilities include database health & alert log monitoring, tablespace management (acting at 70% threshold), RMAN Level 0/1 & `nohup` backups with BCT, Data Pump UAT refreshes, User/Profile administration, Control File & Redo Log multiplexing/resizing, building and maintaining Data Guard DR setups (both Backup-based and Service-based), troubleshooting DC–DR out-of-sync gaps (including `UNNAMED` datafiles and Incremental Service Recovery), executing quarterly DR Switchover & Snapshot Standby drills, and applying quarterly `n-1` PSU patches via `OPatch` and `datapatch`."
  • Delivery Tip 1: Say "restore database from service" — NEVER say "restore datafiles from service".
  • Delivery Tip 2: Never read quotes/brackets aloud—explain the flow fluently and conclude with "This is how I handle it in production."
  • Delivery Tip 3: Skip family background/school marks—keep personal intro to 10 seconds and spend 90 seconds on production DBA work!

19. Complete ORA- Error & Alert Log Forensics Dictionary (From Live DC/DR Logs)

Instant Root-Cause Lookup
Error / Alert Signature Exact Meaning (Why It Happened in Lab/Prod) Immediate Fix / Action
ORA-00214 Control file version mismatch (e.g., `control01` v1305 vs `control03` v1302) from copying a live CF. `shut immediate;` → delete stale copy → `cp` surviving CF while DB is DOWN → `startup mount;`.
ORA-00210 / ORA-27041 Control file deleted while DB is OPEN (`shut immediate` fails!). `shut abort;` (or OS `kill -9 <PMON_PID>`) → `startup nomount;` → `cp` surviving CF → `alter database mount; open;`.
ORA-00205 Control file missing while starting DB (`startup mount` halts in `NOMOUNT`). No shutdown needed! `show parameter control;` → `cp` surviving CF to missing path → `alter database mount;`.
ORA-01078 / LRM-00109 Both `spfile<SID>.ora` & `init<SID>.ora` missing (or plain `create spfile from pfile;` used without explicit path). `startup nomount pfile='/tmp/...';` → `create spfile from pfile='/tmp/...';` (or rebuild from `alert_<SID>.log`).
ORA-01507 / ORA-01109 `ORA-01507`: DB is only in `NOMOUNT`. `ORA-01109`: DB is `MOUNTED` (not open — cannot switch logfiles). Run `alter database mount;` (for 01507) or `alter database open;` (for 01109).
ORA-01081 / ORA-01219 `01081`: DB already running during `startup`. `01219`: Non-fixed view (or misspelled `v$datbase`) queried in `MOUNT`. `shut immediate;` first (for 01081); fix spelling to `v$database` or open DB (for 01219).
ORA-02095 Attempted to modify a Static parameter (`control_files`) in memory. Append `scope=spfile;` and bounce the database.
ORA-01184 `logfile group 20 already exists` — used `add logfile group` instead of `add logfile member`. Use `alter database add logfile member '<path>' to group 20;`.
ORA-01609 / 01623 / 01624 `01609`: Cannot drop member of `CURRENT` log. `01623`: Cannot drop `CURRENT` group. `01624`: Cannot drop `ACTIVE` group. Run `alter system switch logfile;` + `alter system checkpoint;` until `STATUS='INACTIVE'` (or `ACTIVE` for member drop).
ORA-01275 `DROP/ADD LOGFILE not allowed if standby file management is automatic` on DR. Temporarily set `standby_file_management=MANUAL`, drop/add DR ORLs, then immediately restore `AUTO`!
ORA-00313 / ORA-19527 Post-restore DR alert log flood: restored CF points to DC's ORL/SRL paths that don't exist on DR. Drop logical Prod ORLs/SRLs on DR and recreate fresh DR ORLs (toggle `MANUAL` → `AUTO` if needed).
ORA-19505 (RMAN Failover) During DR restore, RMAN tries DC backup folder first, fails, and auto-failovers to cataloged DR folder. Benign / Expected! Verify next line says `failover to piece handle=/data2/RMANBACKUP/RMANBACKUP_12NOV/...`.
ORA-16009 `invalid redo transport destination` on newly mounted Standby right after Switchover. Run `alter system set log_archive_dest_state_2=defer sid='*' scope=both;` on the new Standby.
ORA-01157 / ORA-16037 `01157` on `temp01.dbf`: Oracle auto-recreates missing tempfile on Switchover `open`. `16037`: Normal MRP cancel on Switchover. Benign / Self-Healing! Confirm `Re-creating tempfile` & `Completed: alter database open` in alert log.
ORA-01034 / ORA-00274 `01034`: Instance auto-stopped after Switchover to Standby. `00274`: Typo in `recover managed standby` (`disconect`). Reconnect `sqlplus / as sysdba` → `startup mount;` (for 01034); fix spelling to `disconnect` (for 00274).
Checkpoint not complete / No SRLs `Checkpoint not complete`: Redo switches faster than DBWR flush. `No SRLs`: Real-Time Apply ignored (`PR00` waits for switches). Resize/add ORL groups if frequent in Prod; add `ORL + 1` matching SRLs on DR to enable Real-Time Apply.