Fan Out Pattern
In computer science and networking, “fan-out” refers to spreading data, requests, or tasks from one origin to numerous destinations. The “fan-out pattern” illustrates systems wherein a sole component or service dispatches data or requests to several other components or services.
Here’s a concise overview of the fan-out concept:
- Single Producer: A single entity, usually a component or service, generates or dispatches data, requests, or tasks.
- Multiple Consumers: These data or tasks from the producer are then taken up by various other components or services.
- Parallel Processing: A key feature of the fan-out approach is enabling parallel processing. This means data or tasks aren’t processed one after the other; instead, they’re handed over to many consumers for simultaneous handling.
- Load Distribution: By employing the fan-out method, the workload gets spread out over many components or services, preventing any one component from being overwhelmed.
- Scalability: This approach shines in systems aiming for scalability. As demand grows, more consumers can be integrated to manage the rising workload.
- Push once in SNS, receive in all SQS queues that are subscribers
- Fully decoupled, no data loss
- SQS allows for data persistence, delayed processing, and retries of work
- Ability to add more SQS subscribers over time
- Make sure your SQS queue access policy allows for SNS to write
- Cross-Region Delivery: works with SQS Queues in other regions
SNS
Amazon SNS is a managed messaging service that lets you decouple publishers from subscribers. This is useful for application-to-application messaging for microservices, distributed systems, and serverless applications.
SQS
Amazon SQS provides queues for high-throughput, system-to-system messaging. You can use queues to decouple heavyweight processes and to buffer and batch work. Amazon SQS stores messages until microservices and serverless applications process them.
Amazon SQS allows producers to send messages to a queue. Messages are then stored in an SQS Queue. When consumers are ready to process new messages they poll them from the queue. Applications, microservices, and multiple AWS services can take the role of producers or consumers.
Benefits and features
- Highly scalable Standard and FIFO queues: Queues scale elastically with your application. Nearly unlimited throughput and no limit to the number of messages per queue in Standard queues. First-in-first-out delivery and exactly once processing in FIFO queues.
- Durability and availability: Your queues are distributed on multiple servers. Redundant infrastructure provides highly concurrent access to messages.
- Security: Protection in transit and at rest. Transmit sensitive data in encrypted queues. Send messages in a Virtual Private Cloud.
- Batching: Send, receive, or delete messages in batches of up to 10 messages or 256KB to save costs.
Create SQS Queue And SNS Topic
Step1: Create SNS Queue
- Access the AWS Console and navigate to SNS.
- Select “Create Topic” and designate a name for it.
- Opt for the “Standard” type at this moment.
- Maintain the default settings for the remaining options.
- Confirm and save your settings.
Step2: Create SQS Queue
- Log into the AWS Console -> Locate SQS.
- Select “Create Queue”. Opt for the “Standard” option.
- Name the queue.
- Turn off encryption.
- Keep the access policy unchanged.
- Finalize and create the queue.
Step3: Update Policy to access SQS, SNS and attach to the IAM role
Step4: Update Access Policy of SQS to Allow SNS publish Message
{
"Version": "2012-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__owner_statement",
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": "SQS:*",
"Resource": "arn:aws:sqs:us-east-2:261183892025:tradesman",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:sns:us-east-2:261183892025:tradesman"
}
}
}
]
}
Step5: Add SQS subscription to SNS Topic
- Goto to SNS -> Topic -> Subscriptions
- Select the Protocol as “Amazon SQS.” This will display a dropdown containing the SQS queues from the same region and account. Pick the ARN associated with the specified queue.
- Make sure to select “Enable raw message delivery.”
Step6: Test by Sending Message from SNS to SQS
Create an Spring Boot Application
Step1: Add Dependencies
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-sns</artifactId>
<version>1.12.623</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-sqs</artifactId>
<version>1.12.623</version>
</dependency>
Step2: Create code for SNS
Create SNS Client
@Configuration
public class SNSConfig {
@Value("${cloud.aws.region.static}")
private String region;
@Bean
public AmazonSNS amazonSNS() {
return AmazonSNSClientBuilder.standard().withRegion(region)
.build();
}
}
Use SNS client
@Service
@Slf4j
@RequiredArgsConstructor
public class SNSService {
private final AmazonSNS amazonSns;
@Value("${topic.name}")
private String topicArn;
public void sendSNSEvent(JobModel jobModel) {
try {
String message = new Gson().toJson(jobModel);
PublishRequest request = new PublishRequest().withMessage(message).withTopicArn(topicArn);
amazonSns.publish(request);
} catch (Exception e) {
throw e;
}
}
}
Step3: Create code for SQS
Create SQS Client
@Configuration
public class SQSConfig {
@Value("${cloud.aws.region.static}")
private String region;
@Bean
public AmazonSQS amazonSQS() {
return AmazonSQSClientBuilder.standard().withRegion(region)
.build();
}
}
Use SQS client
@Service
@Slf4j
@RequiredArgsConstructor
public class SQSListener {
private final AmazonSQS amazonSQSClient;
@Value("${queue.name}")
private String queueName;
public String retrieveSQSEvent() {
String queueUrl = amazonSQSClient.getQueueUrl(queueName).getQueueUrl();
log.info("Reading SQS Queue done: URL {}", queueUrl);
ReceiveMessageResult receiveMessageResult = amazonSQSClient.receiveMessage(queueUrl);
List<JobModel> jobModels = new ArrayList<>();
for (Message message : receiveMessageResult.getMessages()) {
JobModel jobModel = new Gson().fromJson(message.getBody(), JobModel.class);
jobModels.add(jobModel);
}
return new Gson().toJson(jobModels);
}
}
Step4: Test
Send Message
Receive Message
