Browse Source

Hopeful fix for "Performing repository CHECKPOINT..." deadlock.

This is probably our number one reliability issue at the moment, and has been a problem for a very long time.

The existing CHECKPOINT_LOCK would prevent new connections being created when we are checkpointing or about to checkpoint. However, in many cases we obtain the db connection early on and then don't perform any queries until later. An example would be in synchronization, where the connection is obtained at the start of the process and then retained throughout the sync round. My suspicion is that we were encountering this series of events:
1. Open connection to database
2. Call maybeCheckpoint() and confirm there are no active transactions
3. An existing connection starts a new transaction
4. Checkpointing is performed, but deadlocks due to the in-progress transaction

This potential fix includes preparedStatement.execute() in the CHECKPOINT_LOCK, to block any new transactions being started when we are locked for checkpointing. It is fairly high risk so we need to build some confidence in this before releasing it.
AT-sleep-until-message
CalDescent 3 years ago
parent
commit
ee95a00ce2
  1. 9
      src/main/java/org/qortal/repository/hsqldb/HSQLDBRepository.java

9
src/main/java/org/qortal/repository/hsqldb/HSQLDBRepository.java

@ -703,8 +703,11 @@ public class HSQLDBRepository implements Repository {
private ResultSet checkedExecuteResultSet(PreparedStatement preparedStatement, Object... objects) throws SQLException {
bindStatementParams(preparedStatement, objects);
if (!preparedStatement.execute())
throw new SQLException("Fetching from database produced no results");
// synchronize to block new executions if checkpointing in progress
synchronized (CHECKPOINT_LOCK) {
if (!preparedStatement.execute())
throw new SQLException("Fetching from database produced no results");
}
ResultSet resultSet = preparedStatement.getResultSet();
if (resultSet == null)
@ -1056,4 +1059,4 @@ public class HSQLDBRepository implements Repository {
return DEADLOCK_ERROR_CODE.equals(e.getErrorCode());
}
}
}

Loading…
Cancel
Save