1 00:00:00,240 --> 00:00:01,073 (narrator) Now let's talk 2 00:00:01,073 --> 00:00:05,330 about the SNS plus SQS Fan Out pattern. 3 00:00:05,330 --> 00:00:07,730 The idea is that you want a message to be sent 4 00:00:07,730 --> 00:00:09,720 to multiple SQS queues, 5 00:00:09,720 --> 00:00:11,530 but if you send them individually 6 00:00:11,530 --> 00:00:14,960 to every SQS queue, they can be problems associated with it. 7 00:00:14,960 --> 00:00:17,350 For example, if your application crashes in between, 8 00:00:17,350 --> 00:00:19,020 if they're delivery failures, 9 00:00:19,020 --> 00:00:21,960 or if you add more SQS queues down the road. 10 00:00:21,960 --> 00:00:23,910 Therefore, we want to use the Fan Out Pattern. 11 00:00:23,910 --> 00:00:27,190 The idea is that you're going to push once in SNS topic 12 00:00:27,190 --> 00:00:28,560 and then you're going to subscribe 13 00:00:28,560 --> 00:00:31,960 as many SQS queues as you want to the SNS topic. 14 00:00:31,960 --> 00:00:33,400 These queues are subscribers 15 00:00:33,400 --> 00:00:36,920 and they will all receive the messages sent into SNS. 16 00:00:36,920 --> 00:00:38,870 So for example, we have a buying service 17 00:00:38,870 --> 00:00:39,950 that wants to send messages 18 00:00:39,950 --> 00:00:41,770 into two SQS queues. 19 00:00:41,770 --> 00:00:43,830 What it will do instead, it will send one message 20 00:00:43,830 --> 00:00:46,710 into an SNS topic, and the queues are subscribers 21 00:00:46,710 --> 00:00:49,220 of that SNS topic so that the fraud service 22 00:00:49,220 --> 00:00:51,650 and the shipping service can read all messages 23 00:00:51,650 --> 00:00:53,780 from their own SQS queues. 24 00:00:53,780 --> 00:00:54,920 The idea with this as well is 25 00:00:54,920 --> 00:00:57,100 that it's a fully decoupled model 26 00:00:57,100 --> 00:00:58,990 and there is no data loss. 27 00:00:58,990 --> 00:01:02,200 SQS will give you data persistence, delayed processing, 28 00:01:02,200 --> 00:01:03,890 as well as reties of work. 29 00:01:03,890 --> 00:01:05,800 And with this pattern, we can add more 30 00:01:05,800 --> 00:01:10,310 SQS queues as a subscriber of the SNS topic over time. 31 00:01:10,310 --> 00:01:11,890 For this, we need to make sure 32 00:01:11,890 --> 00:01:15,350 that you SQS queue access policy, as we've seen from before 33 00:01:15,350 --> 00:01:17,630 allows your SNS topic to write 34 00:01:17,630 --> 00:01:20,110 to your SNS, to your SQS queue. 35 00:01:20,110 --> 00:01:22,610 So again yet, another use case 36 00:01:22,610 --> 00:01:25,233 for using Q Access Policies. 37 00:01:26,100 --> 00:01:27,960 Next. So how do we use this pattern 38 00:01:27,960 --> 00:01:29,160 for other purposes? 39 00:01:29,160 --> 00:01:32,150 For example, S3 events into multiple queues. 40 00:01:32,150 --> 00:01:34,860 So there is a limitation of S3 event rules is 41 00:01:34,860 --> 00:01:37,680 that for a combination of event type, for example 42 00:01:37,680 --> 00:01:40,610 an object is being created and a prefix, for example 43 00:01:40,610 --> 00:01:44,780 images/ you can only have one S3 events rule. 44 00:01:44,780 --> 00:01:48,910 But what if you want to send the same S3 event notification 45 00:01:48,910 --> 00:01:51,430 to multiple SQS queues. In that case, 46 00:01:51,430 --> 00:01:52,860 you would use the Fan Out Pattern. 47 00:01:52,860 --> 00:01:55,530 So for example, we have S3 object created 48 00:01:55,530 --> 00:01:58,010 as an event appearing in S3 buckets 49 00:01:58,010 --> 00:02:01,240 and we will send this event into an SNS topic 50 00:02:01,240 --> 00:02:03,880 and we will subscribe many SQS queues 51 00:02:03,880 --> 00:02:05,920 to the SNS topic as a fan out pattern, 52 00:02:05,920 --> 00:02:09,410 but we could also subscribe other type of application, 53 00:02:09,410 --> 00:02:11,770 email, Lambda functions, et cetera, et cetera. 54 00:02:11,770 --> 00:02:14,360 And then what we get from this is that the message 55 00:02:14,360 --> 00:02:15,920 of the events happening 56 00:02:15,920 --> 00:02:17,550 in Amazon S3 is going to go 57 00:02:17,550 --> 00:02:19,490 to many different destinations 58 00:02:19,490 --> 00:02:21,313 thanks to this fan out pattern. 59 00:02:22,170 --> 00:02:23,370 Another architecture is 60 00:02:23,370 --> 00:02:25,110 that you can send data directly 61 00:02:25,110 --> 00:02:28,990 from SNS to Amazon S3 Through Kinesis Data Firehose. 62 00:02:28,990 --> 00:02:32,410 So because SNS has a direct integration with KDF 63 00:02:32,410 --> 00:02:34,030 then your buying service can send data 64 00:02:34,030 --> 00:02:35,650 into an SNS topic, 65 00:02:35,650 --> 00:02:38,010 then Kinesis Data Firehose, KDF, 66 00:02:38,010 --> 00:02:40,310 will receive that information. 67 00:02:40,310 --> 00:02:41,640 And then from Kinesis Data Firehose, 68 00:02:41,640 --> 00:02:44,017 you can send it into your Amazon S3 bucket. 69 00:02:44,017 --> 00:02:45,760 Or for of that matter, 70 00:02:45,760 --> 00:02:49,150 any supported KDF Specific destination 71 00:02:49,150 --> 00:02:51,210 which allows you to be really extensible 72 00:02:51,210 --> 00:02:54,020 in the way you want to maybe persist your messages 73 00:02:54,020 --> 00:02:55,870 from your SNS topic. 74 00:02:55,870 --> 00:02:57,600 So we can apply the final pattern 75 00:02:57,600 --> 00:02:58,840 to FIFO topics as well. 76 00:02:58,840 --> 00:03:03,290 So Amazon SNS has a FIFO or FIFO capability 77 00:03:03,290 --> 00:03:04,570 which is first in, first out 78 00:03:04,570 --> 00:03:06,930 which gives ordering messages in the topic. 79 00:03:06,930 --> 00:03:10,200 So the producers send the messages 1, 2, 3, 4 80 00:03:10,200 --> 00:03:14,140 and the subscribers can only be an SQS FIFO queue for now 81 00:03:14,140 --> 00:03:17,470 which is receiving messages 1, 2, 3, 4 in order. 82 00:03:17,470 --> 00:03:19,540 So the idea is that with SNS FIFO 83 00:03:19,540 --> 00:03:21,460 we get the same features as SQS FIFO. 84 00:03:21,460 --> 00:03:24,030 We get Ordering Message Group ID. 85 00:03:24,030 --> 00:03:27,050 We get Deduplication using a Deduplication ID 86 00:03:27,050 --> 00:03:29,030 or Content Based Deduplication, 87 00:03:29,030 --> 00:03:31,200 and we can have SQS FIFO queues 88 00:03:31,200 --> 00:03:34,180 as subscribers of the FIFO SNS topic. 89 00:03:34,180 --> 00:03:35,930 In terms of throughputs, you're limited, 90 00:03:35,930 --> 00:03:38,800 you get the same throughputs as the SQS FIFO queue 91 00:03:38,800 --> 00:03:41,180 because right now only SQS FIFO queues can read 92 00:03:41,180 --> 00:03:42,830 from SNS FIFO topics. 93 00:03:42,830 --> 00:03:44,260 So why do we need this? 94 00:03:44,260 --> 00:03:47,160 Well, in case you wanna do a fan out using SQS FIFO. 95 00:03:47,160 --> 00:03:49,990 So you need fan out, ordering, and deduplication. 96 00:03:49,990 --> 00:03:51,570 So the bank service will send the data 97 00:03:51,570 --> 00:03:54,580 into an SNS FIFO topic, and then it will fan out 98 00:03:54,580 --> 00:03:57,840 to two SQS FIFO queue which can have the fraud service 99 00:03:57,840 --> 00:04:01,210 and the shipping service read from FIFO queues as well. 100 00:04:01,210 --> 00:04:02,860 One last feature of SNS 101 00:04:02,860 --> 00:04:06,070 that can be really handy regarding the fan out pattern is 102 00:04:06,070 --> 00:04:08,950 that you can do message filtering in SNS. 103 00:04:08,950 --> 00:04:10,840 So what is message filtering? 104 00:04:10,840 --> 00:04:12,920 Well it's JSON policy that is used 105 00:04:12,920 --> 00:04:14,600 to filter messages sent 106 00:04:14,600 --> 00:04:17,140 to the SNS topics as subscription. 107 00:04:17,140 --> 00:04:19,399 So if a subscription doesn't have a filter policy 108 00:04:19,399 --> 00:04:20,709 it will receive a every message 109 00:04:20,709 --> 00:04:22,170 and that's the default behavior, 110 00:04:22,170 --> 00:04:23,720 but let's take an example of what happens 111 00:04:23,720 --> 00:04:26,610 when we set up a message filtering policy. 112 00:04:26,610 --> 00:04:28,130 So we have a buying service, 113 00:04:28,130 --> 00:04:30,910 and it sends transactions into an SNS topic. 114 00:04:30,910 --> 00:04:32,330 For example, the transaction look 115 00:04:32,330 --> 00:04:33,790 like do there's an order number. 116 00:04:33,790 --> 00:04:35,510 There's a product, for example, the pencil 117 00:04:35,510 --> 00:04:39,550 the quantity for, and the states, which is placed. 118 00:04:39,550 --> 00:04:43,640 Now, we want to rate an SQS queue just for placed orders 119 00:04:43,640 --> 00:04:45,910 not all the orders, but just for place orders. 120 00:04:45,910 --> 00:04:48,250 So for this, we will subscribe the SQS queue 121 00:04:48,250 --> 00:04:52,370 into the SNS topic and we'll apply a filter policy in JSON. 122 00:04:52,370 --> 00:04:54,010 And we will specify in a policy 123 00:04:54,010 --> 00:04:56,680 that we want to have state equals placed. 124 00:04:56,680 --> 00:04:59,480 And so only the message is matching the policy will go 125 00:04:59,480 --> 00:05:00,930 into the SQS queue. 126 00:05:00,930 --> 00:05:03,880 But then we can have an SQS queue for canceled orders. 127 00:05:03,880 --> 00:05:06,100 And so we can create our own filter policy 128 00:05:06,100 --> 00:05:08,480 for canceled orders and have those coming 129 00:05:08,480 --> 00:05:11,210 from the same SNS topic into the SQS queue. 130 00:05:11,210 --> 00:05:12,950 And so the placed orders and the cancel orders 131 00:05:12,950 --> 00:05:15,830 in SQS queue will not have the same messages. 132 00:05:15,830 --> 00:05:17,620 We can also use the same filter policy 133 00:05:17,620 --> 00:05:20,500 the canceled one to create an email subscription 134 00:05:20,500 --> 00:05:22,230 for canceled orders. 135 00:05:22,230 --> 00:05:25,160 We could have a filter policy for declined orders 136 00:05:25,160 --> 00:05:27,920 for example, and as another SQS queue 137 00:05:27,920 --> 00:05:29,480 or we can create an SQS queue 138 00:05:29,480 --> 00:05:32,520 without a filter policy to have all messages 139 00:05:32,520 --> 00:05:34,120 from that SNS topic. 140 00:05:34,120 --> 00:05:36,660 So using all these fan out patterns 141 00:05:36,660 --> 00:05:39,550 and message filtering FIFO queues, and FIFO topics, 142 00:05:39,550 --> 00:05:41,290 we get lots of different possibilities 143 00:05:41,290 --> 00:05:44,180 and the exam will try to test you on all of those. 144 00:05:44,180 --> 00:05:45,350 So that's it for this lecture. 145 00:05:45,350 --> 00:05:46,183 I hope you liked it. 146 00:05:46,183 --> 00:05:47,960 And I will see you in the next lecture.