Have you ever encountered a situation where a customer was charged twice for the same order? Or the system created two identical records just because the user accidentally clicked the "Submit" button twice or because the request timed out and was retried? 🤯 These are painful problems, especially in the Microservices world, where services communicate with each other over the network - an inherently unreliable environment.
Today, we'll explore an extremely important concept that helps solve this problem: Idempotency , and how to handle it effectively even when you're using a Load Balancer .
1. What is Idempotency? It sounds "dangerous" but it's very simple! 🤔
In simple terms, an operation is called idempotent if performing it multiple times gives the same result as performing it just once .
In API:
2. Why Does Idempotency Survive in Microservices? 🏥
Microservices architecture is based on network communication (HTTP, message queue...). Network is unreliable (errors, timeouts, latency). When service A calls service B:
=> Idempotency ensures that retry requests are SAFE and do not cause unwanted side effects.
3. Popular Idempotency Assurance Strategies 🛡️
The most popular and powerful approach is to use an Idempotency Key :
4. New Challenges: Idempotency and Load Balancer ⚖️ Problems and Solutions
Everything seems fine until you deploy your service with multiple instances (replicas) behind a Load Balancer for increased load capacity and High Availability.
What is the problem?
Solution: Shared State!
To solve this problem, information about the processed Idempotency-Keys (and corresponding responses) cannot reside in the local memory of each instance. It needs to be stored in a central, shared state that is accessible to all instances.
There are two common ways to create this shared state:
Choosing between Cache and Database depends on:
5. Idempotency Example with Spring Boot (Running Behind Load Balancer) ☕
The Controller and Service code basically doesn't change much from the previous example, but the key point lies in the implementation of ProcessedRequestStore.
@RestController
@RequestMapping("/payments")
public class PaymentController {
@Autowired
private PaymentService paymentService;
// Implementation use shared state (Redis/DB)
@Autowired
private ProcessedRequestStore processedRequestStore;
@PostMapping
public ResponseEntity processPayment(
@RequestHeader("Idempotency-Key") String idempotencyKey,
@RequestBody PaymentRequest request) {
// 1. Check shared store (Redis/DB) xem key đã xử lý chưa
Optional<PaymentResponse> cachedResponse = processedRequestStore.findResponse(idempotencyKey);
if (cachedResponse.isPresent()) {
System.out.println("Idempotency key found, returning cached response for key: " + idempotencyKey);
return ResponseEntity.ok(cachedResponse.get());
}
// 2. Mark as processing in shared store (atomic operation!)
if (!processedRequestStore.markAsProcessing(idempotencyKey)) {
System.out.println("Conflict: Idempotency key is currently being processed: " + idempotencyKey);
return ResponseEntity.status(HttpStatus.CONFLICT).body(null);
}
PaymentResponse response;
try {
// 3. Logic
System.out.println("Processing new request for key: " + idempotencyKey);
response = paymentService.executePayment(request);
// 4. Save to shared store
processedRequestStore.saveResponse(idempotencyKey, response);
System.out.println("Saved response for key: " + idempotencyKey);
} catch (Exception e) {
processedRequestStore.clearProcessingMark(idempotencyKey); // Clear mark if error
System.err.println("Error processing key " + idempotencyKey + ": " + e.getMessage());
throw e;
} finally {
// processedRequestStore.clearProcessingMark(idempotencyKey);
}
return ResponseEntity.status(HttpStatus.CREATED).body(response);
}
}
// --- Interface ProcessedRequestStore ---
interface ProcessedRequestStore {
Optional findResponse(String idempotencyKey);
void saveResponse(String idempotencyKey, PaymentResponse response);
boolean markAsProcessing(String idempotencyKey);
void clearProcessingMark(String idempotencyKey);
}
// --- (Spring Data Redis) ---
@Component("redisProcessedRequestStore")
public class RedisProcessedRequestStore implements ProcessedRequestStore {
@Autowired
private StringRedisTemplate redisTemplate;
private static final String RESPONSE_KEY_PREFIX = "idempotency:response:";
private static final String PROCESSING_MARK_PREFIX = "idempotency:processing:";
private static final Duration RESPONSE_TTL = Duration.ofHours(24);
private static final Duration PROCESSING_TTL = Duration.ofSeconds(30);
private final ObjectMapper objectMapper = new ObjectMapper(); // serialize/deserialize response
@Override
public Optional<PaymentResponse> findResponse(String idempotencyKey) {
String jsonResponse = redisTemplate.opsForValue().get(RESPONSE_KEY_PREFIX + idempotencyKey);
if (jsonResponse != null) {
try {
return Optional.of(objectMapper.readValue(jsonResponse, PaymentResponse.class));
} catch (JsonProcessingException e) {
// Handle error deserializing
System.err.println("Error deserializing cached response for key " + idempotencyKey);
return Optional.empty();
}
}
return Optional.empty();
}
@Override
public void saveResponse(String idempotencyKey, PaymentResponse response) {
try {
String jsonResponse = objectMapper.writeValueAsString(response);
redisTemplate.opsForValue().set(RESPONSE_KEY_PREFIX + idempotencyKey, jsonResponse, RESPONSE_TTL);
redisTemplate.delete(PROCESSING_MARK_PREFIX + idempotencyKey);
} catch (JsonProcessingException e) {
// Handle error serializing
System.err.println("Error serializing response for key " + idempotencyKey);
}
}
@Override
public boolean markAsProcessing(String idempotencyKey) {
// SET key value EX seconds NX (Atomic operation)
Boolean success = redisTemplate.opsForValue().setIfAbsent(
PROCESSING_MARK_PREFIX + idempotencyKey,
"processing",
PROCESSING_TTL
);
return Boolean.TRUE.equals(success);
}
@Override
public void clearProcessingMark(String idempotencyKey) {
redisTemplate.delete(PROCESSING_MARK_PREFIX + idempotencyKey);
}
}
// --- PaymentService và DTOs (Giữ nguyên) ---Key points in the Redis example:
6. Other Notes (Remain the same):
Conclude:
Idempotency is the foundation for reliable microservice systems. When you have a Load Balancer, using shared state (via Distributed Cache or Shared Database) to manage Idempotency Keys is a must. This way, you can confidently retry requests without fear of causing duplicate data, making the system more stable and consistent. Happy coding!