diff --git a/docs/en/solutions/Configure_MySQL_Audit_Log_Filter_Plugin.md b/docs/en/solutions/Configure_MySQL_Audit_Log_Filter_Plugin.md new file mode 100644 index 00000000..2e9418d9 --- /dev/null +++ b/docs/en/solutions/Configure_MySQL_Audit_Log_Filter_Plugin.md @@ -0,0 +1,160 @@ +--- +kind: + - How To +products: + - Alauda Application Services +ProductsVersion: + - 4.0,4.1,4.2,4.3 +id: KB260515006 +--- + +# Configure MySQL Audit Log Filter Plugin + +## Issue + +You need to capture an audit trail of connection events, DDL, and DML on a MySQL 8.0 instance. MySQL Community Server does not ship the proprietary `audit_log.so` plugin from MySQL Enterprise, but Percona Server provides a drop-in replacement, `audit_log_filter.so`, that emits per-event JSON records and supports event-class filtering. This how-to enables the plugin on a `Mysql` CR-managed instance, defines a sensible default filter set, and verifies the output. + +## Environment + +- Alauda Application Services for MySQL 4.0 and later +- MySQL Server 8.0.36 or later (the plugin requires this minimum patch level) +- Cluster access via `kubectl` + +> The plugin is shipped as a preview feature in MySQL 8.0 and is generally available in MySQL 8.4. Treat it as feature-stable for the audit pipeline but expect minor variable renames between point releases. + +## Resolution + +### 1. Load the plugin via the `Mysql` CR + +Set the plugin and its configuration variables under `spec.params.mysql.mysqld`. Use `loose_` prefixed variables so the server does not refuse to start if the plugin is not yet loaded: + +```yaml +spec: + params: + mysql: + mysqld: + plugin_load_add: "audit_log_filter.so" + loose_audit_log_filter_format: "JSON" + loose_audit_log_filter_rotate_on_size: "104857600" +``` + +Variable meaning: + +- `plugin_load_add` — additionally load the named plugin at startup. +- `loose_audit_log_filter_format` — output format. `JSON` is recommended for downstream parsing. +- `loose_audit_log_filter_rotate_on_size` — file-rotation size in bytes (here, 100 MiB). The plugin keeps up to 1 GiB of rotated history by default. + +Full variable reference: [Percona audit log filter variables](https://docs.percona.com/percona-server/8.0/audit-log-filter-variables.html). + +Apply the change. The operator performs a rolling restart of the MySQL pods. + +### 2. Verify the plugin is loaded + +After the rolling restart completes, connect to the read-write service and check the plugin status plus its variables: + +```sql +SELECT PLUGIN_NAME, PLUGIN_STATUS +FROM information_schema.PLUGINS +WHERE PLUGIN_NAME LIKE '%audit%'\G + +SHOW GLOBAL VARIABLES LIKE 'audit_log_filter%'; +``` + +Confirm that `AUDIT_LOG_FILTER` is `ACTIVE` and that the variable values match the YAML you applied. + +### 3. Initialize the plugin + +The plugin ships an initialization script that creates its filter tables in the `mysql` system schema. Run it once against the read-write endpoint: + +```bash +mysql -h -read-write -uroot -p \ + < /usr/share/percona-server/audit_log_filter_linux_install.sql +``` + +Enter the root password when prompted. + +### 4. Define and attach filters + +Connect to the read-write endpoint and create two filters: + +- `quiet` for the operator's health-check and management users, to avoid flooding the log with no-op queries. +- `default` for every other user, recording connect/disconnect events, general statements, and write-side table access. + +Run the following against the read-write endpoint: + +```sql +SET @quiet = ' +{ + "filter": { + "class": [ + { + "name": "table_access", + "event": [ + { "name": "insert" }, + { "name": "delete" }, + { "name": "update" } + ] + } + ] + } +}'; + +SELECT audit_log_filter_set_filter('quiet', @quiet); +SELECT audit_log_filter_set_user('exporter@localhost', 'quiet'); +SELECT audit_log_filter_set_user('manage@localhost', 'quiet'); +SELECT audit_log_filter_set_user('healthchecker@localhost', 'quiet'); + +SET @default = ' +{ + "filter": { + "class": [ + { + "name": "connection", + "event": [ + { "name": "connect" }, + { "name": "disconnect" } + ] + }, + { "name": "general" }, + { + "name": "table_access", + "event": [ + { "name": "insert" }, + { "name": "delete" }, + { "name": "update" } + ] + } + ] + } +}'; + +SELECT audit_log_filter_set_filter('default', @default); +SELECT audit_log_filter_set_user('%', 'default'); +``` + +Adjust the filter JSON to match the auditing scope you actually need. The full schema is documented at [audit log filter definitions](https://dev.mysql.com/doc/refman/8.4/en/audit-log-filter-definitions.html). + +### 5. Detach or remove filters + +Detach a filter from a user (`%` matches all otherwise-unconfigured users; substitute an actual `user@host` to detach for a single account): + +```sql +SELECT audit_log_filter_remove_user('%'); +``` + +Delete a filter definition: + +```sql +SELECT audit_log_filter_remove_filter('default'); +``` + +### 6. Inspect the audit log files + +The audit log files live alongside the MySQL data directory inside every MySQL pod. List them on a given pod: + +```bash +kubectl -n exec -it -c mysql -- \ + ls -lh /var/lib/mysql/audit_filter* +``` + +Each rotated file is JSON-line formatted and ready to ship to a log aggregator. diff --git a/docs/en/solutions/Configure_MySQL_Connection_Control_Plugin_for_Failed_Login_Lockout.md b/docs/en/solutions/Configure_MySQL_Connection_Control_Plugin_for_Failed_Login_Lockout.md new file mode 100644 index 00000000..cd74b991 --- /dev/null +++ b/docs/en/solutions/Configure_MySQL_Connection_Control_Plugin_for_Failed_Login_Lockout.md @@ -0,0 +1,114 @@ +--- +kind: + - How To +products: + - Alauda Application Services +ProductsVersion: + - 4.0,4.1,4.2,4.3 +id: KB260515003 +--- + +# Configure MySQL Connection Control Plugin for Failed Login Lockout + +## Issue + +By default, MySQL accepts an unlimited number of password retries against any account. To mitigate brute-force attacks, operators often need to introduce a progressive delay after a configurable number of consecutive failed login attempts. This how-to enables the upstream `CONNECTION_CONTROL` plugin on a MySQL 8.0 instance managed by Alauda Application Services for MySQL (Group Replication topology) and verifies the throttling behavior. + +> The plugin works for MGR because the Router performs transparent L4 routing and preserves the original MySQL handshake. The same plugin is not effective behind a ProxySQL fronted Percona XtraDB Cluster (PXC) deployment because ProxySQL terminates and rewraps the authentication exchange. + +## Environment + +- Alauda Application Services for MySQL 4.0 and later +- A MySQL Group Replication (MGR) instance backed by the `Mysql` CR +- Cluster access via `kubectl` + +## Resolution + +### 1. Plugin variables + +The plugin exposes three variables that govern the delay applied after consecutive failed connection attempts: + +| Variable | Default | Description | +| --- | --- | --- | +| `connection_control_failed_connections_threshold` | `3` | Number of consecutive failed attempts before the delay kicks in. `0` disables the feature. Range: `0`–`2147483647`. | +| `connection_control_min_connection_delay` | `1000` ms | Minimum delay added before the server responds to a failed attempt once the threshold is reached. Range: `1000`–`2147483647`. | +| `connection_control_max_connection_delay` | `2147483647` ms | Upper bound of the delay. Range: `1000`–`2147483647`. | + +### 2. Enable the plugin via the `Mysql` CR + +Edit the MGR instance YAML and add the plugin entries under `spec.paras.mysqld`. The following example locks out an account after 5 failed attempts and holds each subsequent failure open for at least 5 minutes (300 000 ms): + +```yaml +spec: + paras: + mysqld: + plugin-load: connection_control.so + connection-control-failed-connections-threshold: "5" + connection-control-min-connection-delay: "300000" +``` + +Apply the change. The operator performs a rolling restart of the MGR pods to load the plugin. + +### 3. Verify the plugin is active + +After the rolling restart completes, exec into the Router pod and connect to the read-only service: + +```bash +kubectl -n get svc | grep -read +# Note the cluster-ip of -read-only and use it as the connection host. + +kubectl -n exec -it -router- -c router -- bash +mysql -uroot -h -P 3306 -p"$MYSQL_PASSWORD" +``` + +Check the plugin status and the current variables: + +```sql +SELECT PLUGIN_NAME, PLUGIN_STATUS +FROM information_schema.plugins +WHERE PLUGIN_NAME LIKE 'CONNECTION%'; + +SHOW VARIABLES LIKE 'connection_control%'; +``` + +Expected output: + +``` ++------------------------------------------+---------------+ +| PLUGIN_NAME | PLUGIN_STATUS | ++------------------------------------------+---------------+ +| CONNECTION_CONTROL | ACTIVE | +| CONNECTION_CONTROL_FAILED_LOGIN_ATTEMPTS | ACTIVE | ++------------------------------------------+---------------+ + ++-------------------------------------------------+------------+ +| Variable_name | Value | ++-------------------------------------------------+------------+ +| connection_control_failed_connections_threshold | 5 | +| connection_control_max_connection_delay | 2147483647 | +| connection_control_min_connection_delay | 300000 | ++-------------------------------------------------+------------+ +``` + +### 4. Validate the lockout behavior + +Trigger several failed logins with a deliberately wrong password and time each attempt. The connection time should grow once the threshold is exceeded: + +```bash +for i in $(seq 1 8); do + time mysql -uroot -h -P 3306 -p"wrongpass" -e "SELECT 1" || true +done +``` + +Inspect the per-account failed-attempt counter from any session: + +```sql +SELECT * +FROM performance_schema.connection_control_failed_login_attempts; +``` + +Each row shows the user/host pair and the number of consecutive failures currently held against it. A successful login resets the counter for that pair. + +### 5. Roll back + +To disable the lockout, remove the `plugin-load` and `connection-control-*` entries from `spec.paras.mysqld` and re-apply the CR. The operator restarts the pods and the plugin is no longer loaded. diff --git a/docs/en/solutions/How_to_Deploy_CloudBeaver_for_MySQL_Management.md b/docs/en/solutions/How_to_Deploy_CloudBeaver_for_MySQL_Management.md new file mode 100644 index 00000000..8b7452ee --- /dev/null +++ b/docs/en/solutions/How_to_Deploy_CloudBeaver_for_MySQL_Management.md @@ -0,0 +1,151 @@ +--- +kind: + - How To +products: + - Alauda Application Services +ProductsVersion: + - 4.0,4.1,4.2,4.3 +id: KB260515005 +--- + +# How to Deploy CloudBeaver for MySQL Management + +## Issue + +You need a web-based SQL client for browsing and querying MySQL instances managed by Alauda Application Services, without installing a desktop tool on every operator's workstation. CloudBeaver is the open-source web edition of DBeaver and runs as a single-pod Deployment on Kubernetes. This how-to deploys CloudBeaver and connects it to a MySQL Group Replication (MGR) instance. + +## Environment + +- Any Kubernetes cluster reachable to operators (NodePort exposure is used below; Ingress works equally well) +- A `StorageClass` capable of provisioning ReadWriteOnce PVCs — used to persist CloudBeaver workspace state across pod restarts +- Network reachability from the CloudBeaver pod to the target MySQL Router service + +## Resolution + +### 1. Prepare the manifest + +Save the following to `cloudbeaver.yaml`. Adjust `storageClassName`, image registry, and resource requests to match your environment: + +```yaml +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: cloudbeaver +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: sc-topolvm # replace with any RWO StorageClass available in the cluster + volumeMode: Filesystem +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: cloudbeaver +spec: + replicas: 1 + selector: + matchLabels: + app: cloudbeaver + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 1 + template: + metadata: + labels: + app: cloudbeaver + spec: + containers: + - name: cloudbeaver + image: docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest + imagePullPolicy: Always + ports: + - name: web + containerPort: 8978 + protocol: TCP + resources: + limits: + cpu: "1" + memory: 1Gi + requests: + cpu: 100m + memory: 256Mi + volumeMounts: + - name: cloudbeaver-data + mountPath: /opt/cloudbeaver/workspace + volumes: + - name: cloudbeaver-data + persistentVolumeClaim: + claimName: cloudbeaver +--- +apiVersion: v1 +kind: Service +metadata: + name: cloudbeaver +spec: + type: NodePort + selector: + app: cloudbeaver + ports: + - name: web + port: 8978 + targetPort: 8978 + protocol: TCP +``` + +> CloudBeaver stores its administrator password, saved connections, and query history under `/opt/cloudbeaver/workspace`. Without a persistent volume, everything is lost on every pod restart. Confirm the chosen `storageClassName` exists in the target cluster before applying. + +### 2. Deploy + +```bash +kubectl -n apply -f cloudbeaver.yaml +kubectl -n rollout status deploy/cloudbeaver +``` + +### 3. Discover the access URL + +The Service uses NodePort, so any node IP plus the allocated NodePort exposes the UI: + +```bash +namespace= +HOST=$(kubectl -n "$namespace" get pod -l app=cloudbeaver \ + -o jsonpath='{.items[0].status.hostIP}') +PORT=$(kubectl -n "$namespace" get svc cloudbeaver \ + -o jsonpath='{.spec.ports[0].nodePort}') +echo "http://$HOST:$PORT" +``` + +Open the URL in a browser. + +### 4. Initial setup + +1. On first launch CloudBeaver prompts you to set the administrator password. Choose a strong password and store it safely — this account governs all subsequent server-side configuration. +2. Log in with the administrator user. +3. (Optional) Switch the UI language under the user menu in the top-right. + +### 5. Connect to a MySQL instance + +1. Click **New Connection** and choose **MySQL**. +2. Fill **Host** with the Router service of the target MGR instance and **Port** with the read-write port. From outside the cluster, retrieve the NodePort: + + ```bash + kubectl -n get svc -router + ``` + +3. Enter the application database user and password. +4. Under **Driver Properties**, set `allowPublicKeyRetrieval` to `TRUE` so the MySQL 8 driver can complete the `caching_sha2_password` handshake against a non-TLS endpoint. Do not enable this against an untrusted network — it lets the client retrieve the server's public key over an unencrypted channel. +5. Under **Access Management**, grant the current CloudBeaver user permission to use the new connection. +6. Save and test the connection. SQL editors can now be opened from the browser and queries executed against the MGR cluster. + +### 6. Uninstall + +```bash +kubectl -n delete -f cloudbeaver.yaml +``` + +The PVC is removed alongside the rest of the manifest. To preserve CloudBeaver state for a future redeploy, delete only the Deployment and Service and re-attach the existing PVC during the next install. diff --git a/docs/en/solutions/How_to_Import_and_Export_MySQL_Data.md b/docs/en/solutions/How_to_Import_and_Export_MySQL_Data.md new file mode 100644 index 00000000..514d43a2 --- /dev/null +++ b/docs/en/solutions/How_to_Import_and_Export_MySQL_Data.md @@ -0,0 +1,101 @@ +--- +kind: + - How To +products: + - Alauda Application Services +ProductsVersion: + - 4.0,4.1,4.2,4.3 +id: KB260515002 +--- + +# How to Import and Export MySQL Data + +## Issue + +You need to move business data between two MySQL databases — for example, migrating data from a self-hosted MySQL instance into a managed MySQL cluster on the ACP platform. The procedure must preserve referential integrity (triggers, routines, events) without polluting the destination with the source's system tables. + +## Environment + +- Source: any MySQL 5.7 or 8.0 database +- Destination: a MySQL 8.0 database, including an MGR-based MySQL cluster running under Alauda Application Services 4.x +- `mysqldump` utility from a MySQL release greater than or equal to the destination version + +## Resolution + +### 1. Plan the migration + +1. **Provision the destination cluster first.** Reserve enough storage to hold the logical dump in addition to the imported data set. +2. **Decide on a consistent cutover.** If the application cannot tolerate inconsistencies between the source and destination, stop application writes to the source before taking the dump. Otherwise rely on `--single-transaction` for a transactionally consistent snapshot of InnoDB tables. +3. **Do not dump system databases.** Restoring `mysql`, `information_schema`, `performance_schema`, or `sys` from another MySQL instance can corrupt the destination's privilege and metadata catalogues. Only dump the business schemas. +4. **Recreate application users on the destination.** The destination starts with its own privilege catalogue, so application-facing accounts must be created explicitly. Do not reuse `root` for application traffic; create dedicated accounts with the minimum required grants. + +### 2. Choose a `mysqldump` version + +The `mysqldump` client must be at least the version of the destination server. Two common arrangements: + +- Run `mysqldump` from inside a destination pod / host, which already ships a compatible binary; or +- Install a standalone `mysqldump` of an appropriate version on a jump host with network access to the source. + +Confirm both server and client versions before starting: + +```bash +mysql --version +mysqldump --version +``` + +### 3. Export the source business schemas + +Use a single transactional dump that captures triggers, routines, and events, and writes the source GTID set so the destination can replay binlog positions if needed: + +```bash +mysqldump \ + --host= \ + --user=root \ + --password='' \ + --single-transaction \ + --source-data=1 \ + --set-gtid-purged=AUTO \ + --triggers \ + --routines \ + --events \ + --databases ... \ + > _fullbackup.sql +``` + +Flag notes: + +- `--single-transaction` — consistent snapshot of InnoDB tables without table locks. +- `--source-data=1` — embed `CHANGE MASTER` / binlog position metadata in the dump. +- `--set-gtid-purged=AUTO` — preserve GTID information when the source uses GTIDs. +- `--triggers --routines --events` — include stored programs and event scheduler entries. +- `--databases` — dump only the listed schemas; never use `--all-databases`. + +For MySQL 5.7 sources, replace `--source-data=1` with `--master-data=1`. + +### 4. Import into the destination + +Run `mysql` against the destination using the same credentials and database list. The dump file is self-contained, so a single invocation re-creates schemas, tables, and stored programs: + +```bash +mysql \ + --host= \ + --user=root \ + --password='' \ + < _fullbackup.sql +``` + +For very large dumps, prefer running the import close to the destination (same VPC / same node) to reduce network round-trip overhead. + +### 5. Post-import verification + +1. Count rows in a sample of business tables on both sides: + ```sql + SELECT COUNT(*) FROM .; + ``` +2. Re-create application users on the destination with the minimum required grants: + ```sql + CREATE USER 'app'@'%' IDENTIFIED BY ''; + GRANT SELECT, INSERT, UPDATE, DELETE ON .* TO 'app'@'%'; + FLUSH PRIVILEGES; + ``` +3. Point the application at the destination and confirm read/write traffic before decommissioning the source. diff --git a/docs/en/solutions/How_to_Validate_MySQL_InnoDB_Cluster_Data_Import_Export.md b/docs/en/solutions/How_to_Validate_MySQL_InnoDB_Cluster_Data_Import_Export.md new file mode 100644 index 00000000..d3dd455f --- /dev/null +++ b/docs/en/solutions/How_to_Validate_MySQL_InnoDB_Cluster_Data_Import_Export.md @@ -0,0 +1,186 @@ +--- +kind: + - How To +products: + - Alauda Application Services +ProductsVersion: + - 4.0,4.1,4.2,4.3 +id: KB260515004 +--- + +# How to Validate MySQL InnoDB Cluster Data Import and Export + +## Issue + +You want a repeatable, end-to-end exercise to validate logical import and export against a MySQL InnoDB Cluster (Group Replication) instance: loading a sample dataset onto the PRIMARY, enabling `secure_file_priv` and `local_infile` so client tools can read and write files, and round-tripping data with a workbench-style client or `mysqldump`. This how-to walks through the procedure using the public `employees` sample database. + +## Environment + +- Alauda Application Services for MySQL 4.0 and later +- A running MGR instance (`Mysql` CR) with at least one ONLINE PRIMARY and two ONLINE SECONDARY members +- Cluster access via `kubectl` +- A workstation with the `mysql` client, `tar`, and optionally MySQL Workbench + +## Resolution + +### 1. Stage the `employees` sample database + +Download and extract the sample dataset on any host that can reach the cluster: + +```bash +wget https://launchpadlibrarian.net/24493586/employees_db-full-1.0.6.tar.bz2 +tar -xjvf employees_db-full-1.0.6.tar.bz2 +``` + +Edit `employees_db/employees.sql` and comment out the legacy `storage_engine` directives that no longer exist in MySQL 8.0; otherwise the loader fails with an unknown-variable error: + +```sql +-- set storage_engine = InnoDB; +-- set storage_engine = MyISAM; +-- set storage_engine = Falcon; +-- set storage_engine = PBXT; +-- set storage_engine = Maria; +-- select CONCAT('storage engine: ', @@storage_engine) as INFO; +``` + +### 2. Identify the PRIMARY member + +The `employees.sql` loader executes DDL and DML, so it must run against the PRIMARY: + +```bash +kubectl -n get pod -owide + +kubectl -n exec -it -0 -c mysql -- \ + mysql -uroot -p"$MYSQL_PASSWORD" -e \ + "SELECT MEMBER_HOST, MEMBER_PORT, MEMBER_STATE, MEMBER_ROLE + FROM performance_schema.replication_group_members;" +``` + +Expected output: + +``` ++---------------------+-------------+--------------+-------------+ +| MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | ++---------------------+-------------+--------------+-------------+ +| -0. | 3306 | ONLINE | PRIMARY | +| -1. | 3306 | ONLINE | SECONDARY | +| -2. | 3306 | ONLINE | SECONDARY | ++---------------------+-------------+--------------+-------------+ +``` + +Note the pod that hosts the PRIMARY (``) for the next step. + +### 3. Copy the sample dataset to the PRIMARY pod + +Only `/var/lib/mysql` inside the container is writable by the `mysql` user. + +```bash +kubectl cp -n -c mysql ./employees_db :/var/lib/mysql/ +kubectl -n exec -it -c mysql -- ls -lh /var/lib/mysql/employees_db +``` + +### 4. Load the sample database on the PRIMARY + +```bash +kubectl -n exec -it -c mysql -- bash -c \ + 'cd /var/lib/mysql/employees_db && mysql -uroot -p"$MYSQL_PASSWORD" < employees.sql' +``` + +Successful output includes `CREATING DATABASE STRUCTURE` followed by repeated `LOADING
` lines. + +After the load succeeds, remove the staging directory so MySQL does not treat it as a database directory: + +```bash +kubectl -n exec -it -c mysql -- rm -rf /var/lib/mysql/employees_db +``` + +### 5. Enable file import and export + +By default the server restricts both `secure_file_priv` and `local_infile`: + +```sql +SELECT @@secure_file_priv, @@local_infile; +-- +-----------------------+----------------+ +-- | /var/lib/mysql-files/ | 0 | +-- +-----------------------+----------------+ +``` + +Edit the `Mysql` CR and set the values under `spec.paras.mysqld`: + +```yaml +spec: + paras: + mysqld: + secure_file_priv: "" # empty string = unrestricted; or set to a directory to scope LOAD DATA / SELECT ... INTO OUTFILE + local_infile: "1" # enable LOAD DATA LOCAL INFILE +``` + +Re-apply the CR. The operator performs a rolling restart. Re-check: + +```sql +SELECT @@secure_file_priv, @@local_infile; +-- +--------------------+----------------+ +-- | | 1 | +-- +--------------------+----------------+ +``` + +> Setting `secure_file_priv` to an empty string removes all server-side restrictions on file paths used by `LOAD DATA`, `SELECT ... INTO OUTFILE`, and `LOAD_FILE()`. In production, restrict it to a dedicated directory rather than leaving it unrestricted. + +### 6. Connect a workbench client through the Router + +Retrieve the Router service (NodePort or LoadBalancer) of the instance: + +```bash +kubectl -n get svc -router +``` + +In MySQL Workbench (or any compatible GUI client), create a connection to the Router host and the read-write port using the root credentials. + +### 7. Export a table + +Sanity-check the source data first: + +```sql +SELECT COUNT(*) FROM employees.departments; +``` + +The `employees` schema is heavily foreign-key constrained; suspend constraints when dropping tables for round-trip tests and re-enable them afterwards: + +```sql +SET FOREIGN_KEY_CHECKS = 0; +-- ... drop / import operations ... +SET FOREIGN_KEY_CHECKS = 1; +``` + +Equivalent CLI export with `mysqldump`: + +```bash +mysqldump -h -P -uroot -p"$MYSQL_PASSWORD" \ + --set-gtid-purged=OFF \ + employees departments > employees_departments.sql +``` + +Workbench equivalent: **Server → Data Export**, select `employees.departments`, and write to `employees_departments.sql`. The resulting dump begins with the standard `mysqldump` header and contains `CREATE TABLE` plus `INSERT` statements. + +### 8. Import the dump back + +Drop the table on the cluster, then re-import: + +```sql +SET FOREIGN_KEY_CHECKS = 0; +DROP TABLE employees.departments; +SET FOREIGN_KEY_CHECKS = 1; +``` + +```bash +mysql -h -P -uroot -p"$MYSQL_PASSWORD" \ + employees < employees_departments.sql +``` + +Workbench equivalent: **Server → Data Import → Import from Self-Contained File**, choose `employees_departments.sql`, target schema `employees`, then **Start Import**. + +Verify the row count matches the pre-export value: + +```sql +SELECT COUNT(*) FROM employees.departments; +``` diff --git a/docs/en/solutions/Troubleshooting_MySQL_Initialization_Failure_due_to_aio_max_nr.md b/docs/en/solutions/Troubleshooting_MySQL_Initialization_Failure_due_to_aio_max_nr.md new file mode 100644 index 00000000..839f4b50 --- /dev/null +++ b/docs/en/solutions/Troubleshooting_MySQL_Initialization_Failure_due_to_aio_max_nr.md @@ -0,0 +1,74 @@ +--- +kind: + - How To +products: + - Alauda Application Services +ProductsVersion: + - 4.0,4.1,4.2,4.3 +id: KB260515001 +--- + +# Troubleshooting MySQL Initialization Failure Caused by Exhausted Async I/O Slots + +## Issue + +A MySQL Pod (single-instance, PXC, or MGR) fails to initialize and the container log shows `io_setup() failed`. Inspecting the host shows that `/proc/sys/fs/aio-nr` is at or close to `/proc/sys/fs/aio-max-nr`, meaning the kernel's pool of async I/O contexts is exhausted and MySQL cannot register the AIO contexts it needs to open InnoDB tablespaces. + +This typically appears on hosts that already serve many AIO-heavy workloads (multiple NFS or distributed-storage mounts, other database containers, virtualized I/O paths), where the default `fs.aio-max-nr = 65536` is no longer sufficient. + +## Environment + +- Alauda Application Services for MySQL on ACP (any topology: standalone MySQL, MySQL-PXC, MySQL-MGR) +- Linux kernel with libaio (any supported distribution) +- Host has reached or is close to the kernel default `fs.aio-max-nr = 65536` + +## Resolution + +### 1. Confirm the symptom + +On the node hosting the failing Pod, check the current AIO usage: + +```bash +cat /proc/sys/fs/aio-nr +cat /proc/sys/fs/aio-max-nr +``` + +If `aio-nr` is at or near `aio-max-nr`, the host has run out of AIO contexts and any new MySQL container scheduled on it will fail with `io_setup() failed`. + +### 2. Raise `fs.aio-max-nr` temporarily + +This unblocks initialization without rebooting the node: + +```bash +echo 1048576 > /proc/sys/fs/aio-max-nr +cat /proc/sys/fs/aio-max-nr +``` + +After the value is raised, delete the failing MySQL Pod so the operator schedules a fresh one; initialization should now succeed. + +### 3. Persist the change across reboots + +Add the setting to `/etc/sysctl.conf` (or a drop-in under `/etc/sysctl.d/`): + +```bash +echo 'fs.aio-max-nr = 1048576' >> /etc/sysctl.conf +sysctl -p +cat /proc/sys/fs/aio-max-nr +``` + +Apply this on every node that may host a MySQL Pod — if scheduling moves the Pod onto a node where the value is still the default, the failure recurs. + +### 4. Choose an appropriate value + +`fs.aio-max-nr` caps the number of outstanding async I/O requests the kernel will accept system-wide. The default `65536` is sized for a light desktop workload, not a database host. + +| Host profile | Suggested value | +| --- | --- | +| Dedicated database host with fast storage (NVMe/flash) | `1048576` or higher | +| General-purpose Kubernetes node also running databases | `262144` | +| Hosts with multiple NFS / distributed-storage mounts | `1048576` (each backend consumes contexts) | + +Notes: + +- Each AIO context costs a small amount of kernel memory; on nodes with less than 16 GB of RAM, prefer the lower value to avoid wasting memory. +- High-performance storage benefits most from the larger value — without it, the storage layer will be unable to drive enough concurrent requests to saturate the device.