HADR monitoring

High Availability Disaster Recovery (HADR) pairs get their own page and metric groups beyond the summary table on Home: role, state, sync mode, connect status and heartbeats, log positions, standby lag, and a takeover-readiness composite you can alert on directly, so “is the standby actually ready to take over” is answerable from Enterprise Manager rather than from a manual check on the standby. A standby can be monitored as a target in its own right, and a target can follow whichever side is primary through a takeover.

Prerequisites for this page

• A database participating in an HADR pair — the HADR metric groups populate only on databases in a pair, and read empty otherwise.
• The target’s licence status is Active — see Troubleshooting.

Where to find it: the target’s HADR page; the summary table on the Home page; the full column set under All Metrics → HADR_Status and All Metrics → HADR_Readiness.

In this page: The HADR page · HADR_Status metric group · HADR_Readiness and takeover readiness · Default alerts · Monitoring a standby · Following the primary through a takeover · Associations

The HADR page

The target’s HADR page shows, per standby, Takeover Readiness (whether it could take over now, a 0–100 readiness score, and the reason in plain words) and Standby Lag (log gap, log received but not yet replayed, replay lag, missed heartbeats, time since the last receive, and whether reads on standby and the replay-only window are on). Bars show the log gap and readiness score per standby live, a Trend section shows log gap and replay lag over a chosen window, and the target’s incidents sit at the bottom.

The HADR Status table on Home is the quick-glance view: role, state, sync mode, connect status and time, missed heartbeats, timeout, and the primary and standby log positions.

HADR_Status metric group

HADR_Status collects every 15 minutes, keyed by database name. Beyond what Home shows, it carries local and remote host, service, and instance identity, the HADR timeout in seconds, the primary’s log file and page, and the primary and standby log positions in bytes (HadrPrimaryLogLsn, HadrStandbyLogPos) — the detail you need to reason about replay lag rather than just connection health.

Shipped alert: HadrConnectStatus — Warning when CONGESTED, Critical when DISCONNECTED (1 occurrence), so it fires within 15 minutes of a pair breaking.

HADR_Readiness and takeover readiness

HADR_Readiness collects every 5 minutes, one row per standby. It adds the composite this page exists for: TakeoverReady, a single column that answers whether the standby could actually take over right now, plus log-gap, replay-gap, and replay-lag columns that explain why when it cannot.

• TakeoverReady = 1 — the standby is caught up enough to take over.
• TakeoverReady = 0 — it is not; check the gap and lag columns for how far behind.
• TakeoverReady = -1 — the database is an auxiliary standby in a multi-standby configuration, not the one that would be promoted. This value never alerts, because it does not describe a readiness problem.

Three more columns help you read it:

• ValuesLive is YES while the pair is connected. When it reads NO, the log gap, received-not-replayed gap and replay lag are last-known values: Db2 stops updating them while the pair is disconnected.
• StateAlert is OK when the standby is in the state it should be in — PEER, or REMOTE_CATCHUP under SUPERASYNC; an auxiliary standby is always OK. Otherwise it holds the unexpected state, which its alert names.
• ReplayOnlyWindow reads Y while the standby is in a replay-only window (DDL or a maintenance operation replaying from the primary), and the readiness detail names the window. Neither the readiness score nor TakeoverReady is affected, because a replay-only window does not stop a takeover.

TimeSinceLastRecv is in milliseconds. A 0 means a message arrived within the last second; it is also what Db2 reports while the pair is not connected.

Default alerts

Metric	Column	Warning	Critical	Occurrences
HADR_Status	HadrConnectStatus	CONGESTED	DISCONNECTED	1
HADR_Readiness	TakeoverReady	Not set	Equal to 0	1
HADR_Readiness	HadrLogGap	10 MB	100 MB	3
HADR_Readiness	RecvReplayGap	10 MB	100 MB	3
HADR_Readiness	ReplayLagSec	60 seconds	300 seconds	3
HADR_Readiness	StateAlert	Anything but OK	Not set	2

A healthy PEER pair measures exactly 0 on every gap and on replay lag, so “not zero” is already abnormal; the byte and second values are the tunable part. All of these are ordinary Enterprise Manager metric thresholds — edit them per target under Metric and Collection Settings, the same way as any other metric. See Alerts and thresholds.

Monitoring a standby

A standby database of an HADR pair can be added as a target in its own right: same type, its own host and port. A standby is never reported Down for being a standby. What it reports depends on whether reads on standby (DB2_HADR_ROS=ON on the standby instance) is enabled:

Standby	Availability (Response)	Other groups	CollectorStatus
Reads on standby on	Up (DB_STATUS is STANDBY or ACTIVE_STANDBY)	Collected, read with Uncommitted Read (UR)	OK
Reads on standby off	Up	Not collected: a standby without reads on standby accepts no connections (SQL1776N reason 1)	DISABLED, with the reason and what to change
In its replay-only window (reads on standby on)	Up	Not collected this cycle: connections are blocked while the window lasts (SQL1776N reason 4)	DISABLED until the window closes

• No false alerts from a standby. The groups a standby cannot serve write nothing, so they raise no metric collection errors and no threshold evaluates them.
• Roles are followed live. After a takeover, the new primary collects everything on its next collection, and the new standby switches to the behaviour above. The plug-in decides per collection, so there is nothing to refresh.

• Set the status threshold on a standby target. DetailedResponse : db_status warns on anything but ACTIVE. A standby reports STANDBY, or ACTIVE_STANDBY with reads on standby, so on a standby target set that threshold to its own healthy value:

  emcli modify_threshold -target_name=<standby target> -target_type=ip_db2_database \
                         -metric=DetailedResponse -column=db_status -warning_threshold=ACTIVE_STANDBY
  

• A standby added while it refuses connections shows its version as “unknown (HADR standby …)”, because the plug-in cannot read it. Once the database is readable (after a takeover, or with reads on standby on), refresh the version with emctl reload agent dynamicproperties <target>:ip_db2_database or an agent restart. Until then the version check cannot catch an unsupported Db2 level, so add a standby as a target only if it runs 11.5 or 12.1, as the primary does.

Following the primary through a takeover

By default a target monitors the one database at its Host and Port, so after an HADR takeover it is watching the new standby. To have a target follow whichever side is primary, give it the other side of the pair as an alternate. For a target on the primary db2host:50000 with its standby on db2host:25011:

Property	Value
HADR Alternate Servers	db2host
HADR Alternate Ports	25011
Client Affinities List	YES (optional; see below)

Every collection opens a fresh connection. When the configured server cannot serve it — including an HADR standby, which refuses connections with SQL1776N — the driver tries the alternates. So after a takeover the target reads the new primary on its next collection, and after a failback it reads the original one again. This has been verified against a live pair through a takeover and a failback, with every group collected on each side.

• Client Affinities List = YES tries the servers in a fixed order: this target’s own Host and Port first, then the alternates. The plug-in puts the target’s own server at the head of the list for you. With Client Affinities List unset, the driver tries Host and Port first and the alternates only when that fails.
• Retries default to 2, 1 second apart, so a pair that is entirely down does not hold every collection.
• Servers and ports pair by position. A list that does not pair up (a different count, or a port that is not a number from 1 to 65535) leaves client reroute off, with a warning in the collector log, rather than breaking a target that connects fine without it.
• Backup age follows the primary too. Each HADR instance keeps its own recovery history, so after a takeover DB_Backup and DB_History report the new primary’s history. Back up both sides, or expect the backup-age alert after a takeover.
• Keep a separate standby target if you also want the standby watched. A target that follows the primary always shows the primary; the standby’s own view needs a target on the standby’s host and port without alternates.

Associations

Enterprise Manager links each Db2 target to related targets, which places it in topology views and in the Related Targets region on Home:

• runs_on — the database to the host target it runs on, matched from the server name the database reports. No matching host target means no runs_on.
• depends_on — a target with HADR Alternate Servers / Ports set (normally the HADR primary) to each Db2 target those alternates name, matched by database name, port and host. These are the same properties that let it follow the primary through a takeover.

Both are derived from the daily configuration collection, so a change to the alternates, or a standby target added later, appears after the primary’s next configuration collection.

Related

• Monitoring pages — the HADR page, and the HADR Status table on Home
• Targets and properties — the client-reroute properties alongside every other target property
• Alerts and thresholds — tuning the default thresholds above, and the HADR runbook
• Trial setup — the guided evaluation checklist, including the HADR step

↑ IBM DB2 Plug-in documentation