Implementing Automatic Retry for Database Deadlocks in Spring Boot using Spring Retry
Overview
- Lack of experience in handling deadlocks in databases can lead to panic during the most critical moments when a production-level system faces a high influx of customers. No matter how clearly you write business logic or how thoroughly you conduct unit tests, you can still encounter numerous unexpected deadlocks at the production level, depending on the traffic. This article aims to explain how to minimize deadlocks in a Spring Boot and Spring Data JPA environment, focusing on the
READ-COMMITTEDtransaction isolation level and the usage of@Retryable.
Transactions and Isolation Levels
- The
InnoDBstorage engine of MySQL/MariaDB supports transactional functionality. Transactions are a method to ensure data consistency among queries executed within a single client connection. Clients can choose different levels of isolation for each transaction. The stricter the isolation level, the less it is affected by other transactions but this results in longer shared lock times and reduced concurrency. Conversely, a relaxed isolation level leads to more influence from other transactions, decreasing consistency but shortening shared lock times and improving concurrency. The two most commonly used isolation levels provided by InnoDB are REPEATABLE-READ and READ-COMMITTED.
Deadlocks and Isolation Levels
- A deadlock occurs when a transaction A encounters a shared lock on a row and transaction B attempts to write to that row. Deadlocks are neither bugs nor errors; they are phenomena observed in environments requiring concurrency. MySQL authority Bill Karwin recommends using the READ-COMMITTED transaction isolation level to prevent and mitigate deadlocks, and to write source code that retries the logic in case of a deadlock rather than throwing an exception. [Related Link]
Strategies to Minimize Deadlocks
Minimize the transaction scope as much as possible (to reduce shared lock time). In particular, @Transactional annotations at the @Controller level, which are very likely to cause deadlocks, should be removed.
Execute all queries at the READ-COMMITTED transaction isolation level (to minimize shared lock time).
In case of a deadlock, use @Retryable to retry execution after a random duration within a specified time range (to cope with temporarily inevitable deadlocks).
build.gradle.kts
- In the root of the project, add the following to your build.gradle.kts file.
dependencies {
implementation("org.springframework.boot:spring-boot-starter-aop")
implementation("org.springframework.retry:spring-retry:2.0.5")
}
@EnableRetry
To activate
@Retryable, it's necessary to write a @Configuration bean and specify@EnableRetryat the class level.The following example shows a custom bean created without specifying @EnableRetry. This is done to consider the operational order between @Transactional and @Retryable when both are annotated on the same method. To ensure that @Retryable always operates first, the priority is set to the highest as shown below.
import org.springframework.context.annotation.Configuration
import org.springframework.core.Ordered
import org.springframework.retry.annotation.RetryConfiguration
@Configuration
class RetryConfig : RetryConfiguration() {
override fun getOrder(): Int {
return Ordered.HIGHEST_PRECEDENCE
}
}
@Retryable
@Retryablecan be specified at all class and method levels of Spring beans. One of the significant uses of @Retryable is for automatic retry handling in case of deadlocks. In anticipation of deadlocks, it can be written in @Repository like below. (It's not necessary to apply only to @Repository; it can be written as needed in a higher class calling @Repository like @Service.)
import org.springframework.dao.DataIntegrityViolationException
import org.springframework.data.jpa.repository.JpaRepository
import org.springframework.retry.annotation.Backoff
import org.springframework.retry.annotation.Retryable
import org.springframework.stereotype.Repository
import org.springframework.transaction.annotation.Isolation
import org.springframework.transaction.annotation.Transactional
@Repository
// Minimize shared locks by relaxing transaction isolation level to READ-COMMITTED
@Transactional(readOnly = true, isolation = Isolation.READ_COMMITTED)
// Activate automatic retry for up to 3 times with a random interval of 1-3 seconds in case of an exception
@Retryable(
maxAttempts = 3,
backoff = Backoff(random = true, delay = 1000, maxDelay = 3000),
// Exclude retries for exceptions due to data integrity violations
exclude = [DataIntegrityViolationException::class]
)
interface FooRepository : JpaRepository<Foo, Long> {
// For write operations, set the default readOnly = false
@Transactional(isolation = Isolation.READ_COMMITTED)
fun deleteById(id: Long) { ... }
}
@Repository and @Transactional
All @Repository in Spring Data JPA are by default specified with
@Transactional(readOnly = true)at the class level, and every delete, save, and flush method is specified with@Transactional(readOnly = false). This means that a transaction at the minimum query level is executed even if no @Transactional is specified at the higher calling level.In the code introduced earlier, understanding this feature,
@Transactional(readOnly = true, isolation = Isolation.READ_COMMITTED),@Transactional(isolation = Isolation.READ_COMMITTED)are specified at the class level to ensure that the READ_COMMITTED transaction isolation level operates in any situation.There might be concerns that declaring @Transactional on each method would ignore the transaction declared at a higher level and create individual transactions. However, if you look inside the annotation, it has a default Propagation.REQUIRED propagation level, which naturally continues the transaction of the method calling side, operating as a single transaction.