The 2 AM PagerDuty AlertYou’re paged at 2 AM. Your monitoring dashboard shows a 100% drop in traffic, and the database is down. When you try to restart MySQL, it crashes instantly. A quick look at the error logs (usually found at /var/log/mysql/error.log or via journalctl -xeu mysql) reveals a frustrating message:
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
Linux systems use error code 11 to signal EAGAIN (Resource temporarily unavailable). In this context, the InnoDB engine is attempting to place an exclusive fcntl lock on the ibdata1 file. It fails because another process—or a hung thread—already owns that lock. MySQL shuts down immediately to prevent two processes from writing to the same file, which would cause irreversible data corruption within minutes.
Common Triggers- A previous MySQL instance crashed but left a 'zombie' process behind.- Systemd attempted to restart MySQL before the old process finished flushing its 10GB buffer pool to disk.- Incorrect ownership (e.g., root owns a file instead of mysql) prevents the service from managing its own locks.- In Docker environments, a container restart might fail to release the file handle on the host volume.- Two different MySQL instances are accidentally configured to use the same datadir.## The Debugging Workflow### 1. Hunt for Zombie ProcessesDon't trust systemctl status. Sometimes the service manager thinks MySQL is stopped, but the Linux kernel is still tracking a live process. Use ps to find any lingering database engines.
ps aux | grep -iE 'mysql|mariadb'
If you see a process ID (PID) like 1234 running, attempt a graceful shutdown first. If it doesn't respond within 30 seconds, use a stronger signal:
# Try the polite way first
kill 1234
# Wait 10 seconds. If it's still there, force it:
kill -9 1234
2. Identify the File HolderWhen ps doesn't show anything obvious, the lock might be held by a process you don't recognize. The fuser utility tells you exactly who is touching your data file. Run this as root:
fuser -v /var/lib/mysql/ibdata1
If a PID appears in the output, that is your culprit. This is common in setups where backup scripts or automated snapshots might be locking the database files.
3. Check for Stale Lock FilesMySQL creates small .lock files to coordinate access. If the server loses power, these files might persist on disk. Check your data directory for these remnants:
ls -la /var/lib/mysql/*.lock /var/lib/mysql/mysql.sock.lock
If you have confirmed that no MySQL processes are running, move these files to a temporary directory like /tmp/ and try starting the service again.
Proven Solutions### Solution A: Reset Directory PermissionsPermission issues often masquerade as lock errors. If the mysql user cannot write to the directory, it cannot create the lock. This frequently happens after a manual data migration or an rsync operation.
# Reset ownership to the mysql user and group
chown -R mysql:mysql /var/lib/mysql
# Set standard directory permissions (750)
find /var/lib/mysql -type d -exec chmod 750 {} +
# Set standard file permissions (640)
find /var/lib/mysql -type f -exec chmod 640 {} +
Solution B: Resolving Docker Mount ConflictsIn Docker or Kubernetes, this error usually means the host OS hasn't released the file handle from a previous container. Simply restarting the container often isn't enough. You may need to restart the Docker daemon itself to clear the kernel-level file locks.
docker-compose down
systemctl restart docker
docker-compose up -d
Solution C: Emergency Recovery ModeIf there are no other processes and permissions are perfect, your ibdata1 file might be stuck in a 'dirty' state. You can bypass the initial lock check by entering recovery mode. Add this line to your my.cnf or /etc/mysql/mysql.conf.d/mysqld.cnf under the [mysqld] section:
innodb_force_recovery = 1
Start the service. If it boots, immediately perform a full backup using mysqldump. Once your data is safe, remove the recovery line and attempt a normal restart. Never run a production site with recovery mode enabled, as it disables many data integrity checks.
Verifying the FixAfter clearing the lock, start the service and watch the logs in real-time:
systemctl start mysql
tail -f /var/log/mysql/error.log
Look for the message: [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. If you see that, your database is back online and healthy.