Integration Plumbers // Docs

Documentation for Integration Plumbers products.

MySQL plug-in for Oracle Enterprise Manager: AI troubleshooting guide

This file is written to be loaded into an AI assistant, and to be read by a database administrator. It is self-contained: the assistant needs no other document to use it.

How to use this file

What the plug-in is. The MySQL plug-in for Oracle Enterprise Manager, made by Integration Plumbers, monitors MySQL from an Enterprise Manager management agent. The agent connects to MySQL over JDBC with an ordinary read-only account and runs every collection. Nothing is installed on the database host.

The three target types.

Target type What it represents
MySQL Database One MySQL server instance, standalone or a member of a cluster
MySQL Cluster One InnoDB Cluster, or the Group Replication group behind it, as a whole
MySQL ClusterSet One InnoDB ClusterSet: a primary cluster, its replica clusters and the replication between them

The Monitoring Readiness page. The console has a Monitoring Readiness page for MySQL Database targets and for MySQL ClusterSet targets. It runs read-only checks under the target’s monitoring credentials when it opens and each time you choose Refresh. For each check it shows a status, the current value, the required value and a detail text. The detail text can include a SQL statement to run. The page never runs a statement itself.

To use this file with the page.

  1. Open the target’s Monitoring Readiness page and choose Refresh.
  2. Copy every check that is not OK: its panel name, label, status, current value, required value and detail text.
  3. Give this file and those results to your assistant, with the MySQL server version and the Enterprise Manager version.
  4. Find each check below by its section id and item id, which are in the headings.

Every statement in this file is for an administrator to review and run. The plug-in changes nothing on the server. The monitoring account has no write privilege, so a fix statement fails if you run it as that account. Run it as an account with the privilege the statement needs, after checking it against your own change process.

Remove secrets before you share anything. Passwords and licence keys must not appear in text you send to an assistant or to support. The plug-in removes passwords it knows about from the page text, but check the text yourself.

Status words

The collector emits one of four statuses per check. The page shows them with these words.

Collector status Page word Meaning
ok OK The check ran and the requirement is met.
warn Attention The feature works but something is missing or limited, or an optional check is not met.
fail Not functional A required check ran and failed. The pages that depend on it are empty or wrong.
unknown Unknown The check did not run or could not decide. Unknown means not checked. It is never a pass.

Each check is either required or optional. An optional check that fails counts as Attention, never Not functional. A panel shows the worst status of its checks, in the order Not functional, Attention, Unknown, OK. The line above the panels says all features are ready only when every panel is OK.

When the page shows nothing was verified

The page shows one of three Unknown states for the whole page when it has no check results. In each, nothing was verified, and no check is reported as ready.

What the page says Meaning What to do
“The readiness check could not be read, so nothing was verified. Check that the target is up and its agent is reachable, then press Refresh.” The read failed. Check that the target is Up and that its agent is running and reachable from the OMS (emctl status agent on the agent host). Choose Refresh.
“The readiness check did not answer within 30 seconds, so nothing was verified. The agent may be busy or unreachable. Press Refresh to try again.” The read timed out after 30 seconds. Check that the agent is not overloaded. See the sizing rule in the Prerequisites chapter of the user guide. Choose Refresh.
“The readiness check returned no results, so nothing was verified. The usual cause is that the MySQL Connector/J driver is not installed on the agent host; see the prerequisites in the user guide. Press Refresh to try again.” The check ran and returned no rows. Check that exactly one mysql-connector-j-*.jar is in the agent’s plug-in driver directory. See “MySQL Connector/J not found” below.

Readiness checks

The checks below are the complete set the collector emits. A MySQL Database target emits sections connection, performance_schema, sys_schema, backup and license. A MySQL ClusterSet target emits sections connection and mysql_shell. A MySQL Cluster target has no Monitoring Readiness page.

Each check is a row with a section id and an item id. The privilege checks are in section connection, shown in the panel Connection and privileges.

Panel (section id) Check label (item id) Required Applies to
Connection and privileges (connection) Database connection (database_connection) yes Database, ClusterSet
Connection and privileges (connection) PROCESS privilege (process_priv) yes Database
Connection and privileges (connection) REPLICATION CLIENT privilege (replication_client_priv) yes Database
Connection and privileges (connection) Read access to performance_schema (select_perf_schema) yes Database
performance_schema (performance_schema) performance_schema enabled (enabled) yes Database
performance_schema (performance_schema) Statement digest consumers (statements_digest_consumer) yes Database
performance_schema (performance_schema) Statement instruments (statement_instruments) yes Database
performance_schema (performance_schema) Wait instruments (wait_instruments) optional Database
performance_schema (performance_schema) Memory instruments (memory_instruments) optional Database
sys schema (sys_schema) sys schema installed (installed) yes Database
Backup tool (backup) Backup tool history (backup_tool) optional Database
Plug-in license (license) Plug-in licence (license_active) yes Database
MySQL Shell (mysql_shell) MySQL Shell (mysqlsh) on the agent host (mysqlsh_installed) optional ClusterSet
MySQL Shell (mysql_shell) ClusterSet AdminAPI connection shape (adminapi_connection_shape) optional ClusterSet

How to read an Unknown reason

An Unknown check puts its reason in the Current column. These are the reasons the collector uses.

Current value Meaning
not checked: connection failed The database connection failed, so every check that needs it was not run. Fix the connection first.
not checked: performance_schema is OFF performance_schema is off, so the consumer and instrument checks cannot be read. Fix performance_schema enabled first.
not checked: probe did not run The collector did not run that probe. Choose Refresh. If it repeats, send the page results to support.
not checked: <reason> A privilege probe was not run and gave its reason.
undetermined: <message> The probe ran and failed for a reason other than a missing privilege or object. The message is the server’s, on one line, cut to 200 characters. Read it as the evidence.
undetermined: ... followed by a fixed sentence The probe ran and returned something the check cannot decide on. Each such sentence is listed under the check it belongs to.

Section connection, item database_connection: Database connection

SELECT user, host FROM mysql.user ORDER BY user, host;

Section connection, item process_priv: PROCESS privilege

GRANT PROCESS ON *.* TO '<monitoring user>'@'<host>';

Section connection, item replication_client_priv: REPLICATION CLIENT privilege

GRANT REPLICATION CLIENT ON *.* TO '<monitoring user>'@'<host>';

Section connection, item select_perf_schema: Read access to performance_schema

GRANT SELECT ON performance_schema.* TO '<monitoring user>'@'<host>';

The documented monitoring user has a global SELECT, PROCESS, REPLICATION CLIENT grant on *.*, which covers all three privilege checks. A missing privilege here usually means the account was created with a narrower grant than the documented one:

CREATE USER 'em_monitoring'@'%' IDENTIFIED BY '<strong password>';
GRANT SELECT, PROCESS, REPLICATION CLIENT ON *.* TO 'em_monitoring'@'%';

No SUPER and no write privilege is needed. The account’s host clause has to match the agent’s address as MySQL sees it. A socket connection arrives as localhost.

In every grant above, <monitoring user> and <host> stand for the account shown in the Current column of the connection row. The plug-in writes the account into the statement for you, so copy the statement from the page when you can.

Section performance_schema, item enabled: performance_schema enabled

SHOW VARIABLES LIKE 'performance_schema';

Section performance_schema, item statements_digest_consumer: Statement digest consumers

SELECT NAME, ENABLED FROM performance_schema.setup_consumers
WHERE NAME IN ('global_instrumentation', 'statements_digest');
UPDATE performance_schema.setup_consumers SET ENABLED = 'YES' WHERE NAME IN ('global_instrumentation', 'statements_digest');

This change is lost at restart. To keep it, put the matching performance_schema_consumer_* option in the server configuration. On a managed service, put it in the parameter group.

Section performance_schema, item statement_instruments: Statement instruments

SELECT COUNT(*) AS total, SUM(ENABLED = 'YES' AND TIMED = 'YES') AS on_and_timed
FROM performance_schema.setup_instruments WHERE NAME LIKE 'statement/%';
UPDATE performance_schema.setup_instruments SET ENABLED = 'YES', TIMED = 'YES' WHERE NAME LIKE 'statement/%';

This change is lost at restart. To keep it, put the matching performance_schema_instrument option in the server configuration. On a managed service, put it in the parameter group.

Section performance_schema, item wait_instruments: Wait instruments

UPDATE performance_schema.setup_instruments SET ENABLED = 'YES', TIMED = 'YES' WHERE NAME LIKE 'wait/io/file/%' OR NAME LIKE 'wait/io/table/%';

It is lost at restart; the persistence note under statement_instruments applies.

Section performance_schema, item memory_instruments: Memory instruments

UPDATE performance_schema.setup_instruments SET ENABLED = 'YES' WHERE NAME LIKE 'memory/%';

It is lost at restart; the persistence note under statement_instruments applies.

Section sys_schema, item installed: sys schema installed

SELECT sys_version FROM sys.version;
GRANT SELECT ON sys.* TO '<monitoring user>'@'<host>';

Section backup, item backup_tool: Backup tool history

SELECT COUNT(*) FROM mysql.backup_history;
SELECT COUNT(*) FROM PERCONA_SCHEMA.xtrabackup_history;

Section license, item license_active: Plug-in licence

Status Meaning Fix
License Required No key has been entered. Enter the key in the target’s License Key property.
Invalid Signature The key text was altered. Paste the key exactly as issued, as one line, with no wrapping and no trailing spaces.
Wrong Plug-in The key was issued for a different plug-in, for example a general-availability key on a beta install. Use the key issued for the plug-in you installed.
Expired The expiry date has passed. Request a new key.

Section mysql_shell, item mysqlsh_installed: MySQL Shell (mysqlsh) on the agent host

mysqlsh --version

Section mysql_shell, item adminapi_connection_shape: ClusterSet AdminAPI connection shape

Current value Cause Fix
Unix socket connection A socket target cannot use the AdminAPI, which needs direct TCP sessions to every member. Configure the target with TCP endpoints.
TLS mode verify_ca or TLS mode verify_identity mysqlsh refuses these modes without a CA certificate, and this release does not provide the truststore support they need. The collection reports TLS_TRUSTSTORE_REQUIRED. Set the target’s TLS Mode to required (or disabled).
Kerberos configuration file not readable: <path> The agent’s operating-system user cannot read the file. mysqlsh would silently use /etc/krb5.conf, so the plug-in refuses to run it. Check the path in the target’s Kerberos Configuration File property and the file’s permissions.

Collection failures

These are failures that are not on the readiness page, or that stop the page from running.

MySQL Connector/J not found

The driver is not inside the plug-in. Every collection of every MySQL target on an agent host, and every Run EXPLAIN job, stops with one message until the driver is in place. The target does not show a MySQL problem. When it is missing, the Monitoring Readiness page returns no results.

Connection failures

SELECT user, host FROM mysql.user WHERE user = '';

ClusterSet health

The ClusterSet Health metric reports health_status and fallback_reason. When MySQL Shell cannot be used, health_status reads UNKNOWN and dr_promotion_ready reads -1: promotion readiness was not assessed. The DR Promotion Ready alert fires only on 0.

fallback_reason or status Meaning Fix
MYSQLSH_NOT_FOUND mysqlsh is not on the agent user’s PATH. A missing prerequisite, not a sick cluster. See check mysqlsh_installed.
TLS_TRUSTSTORE_REQUIRED The target’s TLS Mode is verify_ca or verify_identity. Use required or disabled.
MEMBER_UNREACHABLE mysqlsh connected through one listed endpoint, then could not reach a different member. The AdminAPI opens its own session to every member that Group Replication considers online. Open a network path from the agent host, on the MySQL port, to every member of every cluster. The agent host must also resolve each member’s host name.
KERBEROS_TICKET The Kerberos exchange failed before any member saw a credential. mysqlsh reports every such failure as Unknown MySQL error. As the agent’s operating-system user, run klist. If there is no ticket or it is expired, restore the unattended refresh (k5start, or a scheduled kinit -kt <keytab> <principal>). Other causes: an unreachable KDC, clock skew, a member without a mysql/<host> service principal.
KERBEROS_CLIENT_UNAVAILABLE mysqlsh could not load authentication_kerberos_client. Install the MySQL Shell package that ships its client plug-ins and the Kerberos client libraries.
KERBEROS_CONFIG_UNREADABLE The target’s Kerberos Configuration File is not readable by the agent’s operating-system user. See check adminapi_connection_shape.
KERBEROS_NOT_SUPPORTED The endpoint refused the Kerberos client plug-in. A MySQL Router port does this: Kerberos does not pass through Router. Configure the target with member endpoints instead of a Router port, or use password authentication.
dr_promotion_ready = 0, health_status NOT_READY mysqlsh assessed the ClusterSet and a readiness check failed. A ClusterSet with no replica clusters also reads 0. The ClusterSet DR Health page shows which check failed.

Run EXPLAIN job failures

Metrics that look wrong

Versions and platforms

Where to look

The plug-in’s own log is on the agent host that monitors the target. The plug-in writes its warnings and errors there. The file is mysql_oem.log, in the logging folder of the plug-in’s directory under the agent’s plugins directory:

plugins/ip.em.xmyb.agent.plugin_<version>/logging/mysql_oem.log

<version> is the deployed plug-in version. emctl status agent on the agent host prints Agent Home. If you cannot find the plugins directory from there, search for the file by name:

find <agent base directory> -name mysql_oem.log

The log rolls over at 15 MB into files named mysql_oem-<date>-<n>.log in the same folder, and the plug-in keeps up to 10 of them. One log serves every MySQL target that the agent monitors, and a line does not always name its target, so match lines to a target by time.

Other places that the user guide names:

What to send support

Send these, with passwords and licence keys removed:

Do not send passwords, licence keys, keytabs or Kerberos tickets. If a statement or log line contains one, replace it with <removed> first.

More help

The human-oriented version of this guide is the published troubleshooting page: https://docs.integrationplumbers.io/mysql/troubleshooting.html

Prerequisites, including the monitoring user, TLS, Unix sockets and Connector/J: https://docs.integrationplumbers.io/mysql/prerequisites.html

The monitoring pages, including the Monitoring Readiness page: https://docs.integrationplumbers.io/mysql/monitoring-pages.html

↑ MySQL Plug-in documentation