1 00:00:00,280 --> 00:00:01,113 ‫So let's have a look at some 2 00:00:01,113 --> 00:00:03,270 ‫advanced concepts for SQS FIFO. 3 00:00:03,270 --> 00:00:05,230 ‫The first one is deduplication. 4 00:00:05,230 --> 00:00:08,316 ‫So there is a deduplication interval that is five minutes. 5 00:00:08,316 --> 00:00:11,809 ‫And that means that if you send the same message twice 6 00:00:11,809 --> 00:00:12,944 ‫within that five minutes, 7 00:00:12,944 --> 00:00:15,117 ‫then the second message will be refused. 8 00:00:15,117 --> 00:00:18,022 ‫And there are two deduplication methods. 9 00:00:18,022 --> 00:00:22,160 ‫The first one is called content-based deduplication. 10 00:00:22,160 --> 00:00:23,967 ‫What will happen is as soon as you send a message 11 00:00:23,967 --> 00:00:27,360 ‫into SQS there will be a has calculated 12 00:00:27,360 --> 00:00:32,360 ‫using the SHA-256 method algorithm of the message body. 13 00:00:32,584 --> 00:00:35,696 ‫And if the same message body is encountered twice, 14 00:00:35,696 --> 00:00:37,757 ‫the same hash will be the same twice. 15 00:00:37,757 --> 00:00:40,081 ‫And so therefore, the second message will be refused. 16 00:00:40,081 --> 00:00:44,037 ‫Or you an explicitly provide a message deduplication ID 17 00:00:44,037 --> 00:00:46,270 ‫directly while sending a message. 18 00:00:46,270 --> 00:00:49,001 ‫And if the same deduplication ID is encountered twice 19 00:00:49,001 --> 00:00:50,958 ‫then the message will be gone. 20 00:00:50,958 --> 00:00:52,480 ‫So let's have a look. 21 00:00:52,480 --> 00:00:53,369 ‫Our SQS FIFO is here. 22 00:00:53,369 --> 00:00:55,580 ‫And we produce a hello world message. 23 00:00:55,580 --> 00:00:59,460 ‫And say we have enabled content-based deduplication. 24 00:00:59,460 --> 00:01:02,600 ‫In that case, the SQS FIFO queue will generate 25 00:01:02,600 --> 00:01:04,597 ‫a SHA-256 has of that message. 26 00:01:04,597 --> 00:01:07,509 ‫Which may look like something like this in my screen. 27 00:01:07,509 --> 00:01:11,005 ‫And so if the producer sends the exact same message 28 00:01:11,005 --> 00:01:13,397 ‫it will be hashed to the exact same hash 29 00:01:13,397 --> 00:01:15,369 ‫and SQS FIFO will know it. 30 00:01:15,369 --> 00:01:19,263 ‫And therefore, the second message will be refused. 31 00:01:19,263 --> 00:01:20,534 ‫The second advanced concept we need 32 00:01:20,534 --> 00:01:22,531 ‫to look at is message grouping. 33 00:01:22,531 --> 00:01:25,180 ‫So the idea is, if you specify the same value 34 00:01:25,180 --> 00:01:27,820 ‫for the message group ID in an SQS FIFO queue, 35 00:01:27,820 --> 00:01:30,550 ‫which is a mandatory parameter when 36 00:01:30,550 --> 00:01:32,378 ‫you send the message to a FIFO queue. 37 00:01:32,378 --> 00:01:34,441 ‫Then you will only have one consumer. 38 00:01:34,441 --> 00:01:38,761 ‫And all the messages will be in order for that one consumer. 39 00:01:38,761 --> 00:01:41,002 ‫But if you just need ordering 40 00:01:41,002 --> 00:01:44,127 ‫at the level of a subset of messages, 41 00:01:44,127 --> 00:01:46,352 ‫then you should specify different values 42 00:01:46,352 --> 00:01:48,080 ‫for message group ID. 43 00:01:48,080 --> 00:01:50,880 ‫The idea is that the messages that will share 44 00:01:50,880 --> 00:01:54,000 ‫the common message group ID in order within the group. 45 00:01:54,000 --> 00:01:56,410 ‫And each group ID will have a different consumer. 46 00:01:56,410 --> 00:01:58,178 ‫So you can enable parallel processing 47 00:01:58,178 --> 00:01:59,828 ‫on your SQS FIFO queue. 48 00:01:59,828 --> 00:02:03,830 ‫But ordering across groups is not guaranteed. 49 00:02:03,830 --> 00:02:06,910 ‫So for example, let's say we have a FIFO queue. 50 00:02:06,910 --> 00:02:09,679 ‫And we're grouping messages into three groups: A, B, and C. 51 00:02:09,679 --> 00:02:12,154 ‫Say we have message A1, A2, A3. 52 00:02:12,154 --> 00:02:14,970 ‫Then we can have a consumer for the group A. 53 00:02:14,970 --> 00:02:18,018 ‫Then we have another group of messages: B1, B2, B3, B4. 54 00:02:18,018 --> 00:02:19,937 ‫We can have another consumer for that group. 55 00:02:19,937 --> 00:02:22,160 ‫And another set of message. 56 00:02:22,160 --> 00:02:24,876 ‫For example, for consumer group, we have C1 and C2. 57 00:02:24,876 --> 00:02:26,700 ‫The idea is that, for example, 58 00:02:26,700 --> 00:02:28,671 ‫maybe sometimes you don't need the total ordering 59 00:02:28,671 --> 00:02:29,504 ‫of all the messages. 60 00:02:29,504 --> 00:02:33,192 ‫But you want ordering of all the messages 61 00:02:33,192 --> 00:02:35,953 ‫for a specific customer ID. 62 00:02:35,953 --> 00:02:38,875 ‫And for example, for that one specific customer ID, 63 00:02:38,875 --> 00:02:40,574 ‫you can use this as your message group ID. 64 00:02:40,574 --> 00:02:41,929 ‫And so that means that you can have 65 00:02:41,929 --> 00:02:46,929 ‫as many consumers as users you have in your application. 66 00:02:48,380 --> 00:02:51,130 ‫And for each user, the messages will be in order, 67 00:02:51,130 --> 00:02:55,337 ‫thanks to the guarantee of the SQS FIFO queue. 68 00:02:55,337 --> 00:02:57,290 ‫So back into our FIFO queue. 69 00:02:57,290 --> 00:02:59,230 ‫Let's edit it, and I'm going 70 00:02:59,230 --> 00:03:02,197 ‫to enable content-based deduplication. 71 00:03:02,197 --> 00:03:05,730 ‫Where the deduplication ID is going to be computed 72 00:03:05,730 --> 00:03:08,541 ‫as a SHA-256 of my messages. 73 00:03:08,541 --> 00:03:11,150 ‫So I click on save, and here we go. 74 00:03:11,150 --> 00:03:14,130 ‫Now let's go into send and receive messages. 75 00:03:14,130 --> 00:03:17,030 ‫And I will send the message hello world. 76 00:03:17,030 --> 00:03:17,863 ‫For the message group ID, 77 00:03:17,863 --> 00:03:19,660 ‫I will just keep it as demo for now. 78 00:03:19,660 --> 00:03:20,911 ‫And for now, as you can see, 79 00:03:20,911 --> 00:03:22,863 ‫the message deduplication ID is optional 80 00:03:22,863 --> 00:03:26,577 ‫because we have enabled content-based deduplication. 81 00:03:26,577 --> 00:03:28,457 ‫So let's send that message. 82 00:03:28,457 --> 00:03:31,100 ‫As yo can see, the message is sent and ready to be received. 83 00:03:31,100 --> 00:03:33,060 ‫And the message available is one. 84 00:03:33,060 --> 00:03:36,170 ‫But if I send again that message, 85 00:03:36,170 --> 00:03:38,071 ‫and again, and again, and again, 86 00:03:38,071 --> 00:03:41,068 ‫the number of messages available will keep on being one. 87 00:03:41,068 --> 00:03:44,140 ‫Because that message has already been seen SQS, 88 00:03:44,140 --> 00:03:46,780 ‫and so there is some deduplication happening. 89 00:03:46,780 --> 00:03:48,510 ‫But if I send a different message. 90 00:03:48,510 --> 00:03:49,841 ‫For example, hello world two. 91 00:03:49,841 --> 00:03:52,690 ‫Then a second message will be available 92 00:03:52,690 --> 00:03:54,520 ‫right here in SQS FIFO. 93 00:03:54,520 --> 00:03:58,030 ‫So as we can see, this is the deduplication happening here. 94 00:03:58,030 --> 00:04:00,934 ‫But for example, if I wrote an other message 95 00:04:00,934 --> 00:04:03,605 ‫and I used your own deduplication ID. 96 00:04:03,605 --> 00:04:06,433 ‫So for example, your own deduplication token. 97 00:04:06,433 --> 00:04:09,776 ‫So 1-2-3, and then you send that message again. 98 00:04:09,776 --> 00:04:12,250 ‫Thanks to the fact that we specified 99 00:04:12,250 --> 00:04:13,676 ‫the same deduplication ID. 100 00:04:13,676 --> 00:04:15,641 ‫Then you will only see one of 101 00:04:15,641 --> 00:04:18,499 ‫these messages in two SQS FIFO, okay? 102 00:04:18,499 --> 00:04:21,990 ‫The other think I want to show you is the group ID. 103 00:04:21,990 --> 00:04:24,300 ‫So it's not something that's very easy to show. 104 00:04:24,300 --> 00:04:25,850 ‫But the idea is that, for example, 105 00:04:25,850 --> 00:04:30,060 ‫if we have a user bought an apple. 106 00:04:30,060 --> 00:04:33,085 ‫And the message group ID is user123. 107 00:04:33,085 --> 00:04:36,910 ‫And I will just remove this message deduplication ID. 108 00:04:36,910 --> 00:04:40,290 ‫And then the user bought a banana. 109 00:04:40,290 --> 00:04:43,993 ‫And then the user bought strawberries, right? 110 00:04:43,993 --> 00:04:46,200 ‫Because all these messages share 111 00:04:46,200 --> 00:04:48,150 ‫the same message group ID. 112 00:04:48,150 --> 00:04:51,009 ‫Then they will be in order for that user123. 113 00:04:51,009 --> 00:04:55,636 ‫But if I now send a user bought an apple. 114 00:04:55,636 --> 00:05:00,636 ‫And an apple that's going to be green, a green apple. 115 00:05:01,060 --> 00:05:04,704 ‫And we specify a different message group ID: So user234. 116 00:05:04,704 --> 00:05:06,921 ‫Now, these messages for that user234 117 00:05:06,921 --> 00:05:07,754 ‫will be in order for that user. 118 00:05:09,190 --> 00:05:11,230 ‫And so now, in my SQS FIFO queue, 119 00:05:11,230 --> 00:05:14,250 ‫I can have multiple consumers running at the same time. 120 00:05:14,250 --> 00:05:15,113 ‫And each of the consumers will be consuming 121 00:05:15,113 --> 00:05:18,325 ‫from a different message group ID, okay? 122 00:05:18,325 --> 00:05:19,246 ‫So that's it. 123 00:05:19,246 --> 00:05:20,250 ‫Now when you're done, you can just 124 00:05:20,250 --> 00:05:21,248 ‫pull the messages and have a look. 125 00:05:21,248 --> 00:05:24,753 ‫But I will go ahead and just simply delete them. 126 00:05:24,753 --> 00:05:26,520 ‫So that's it, I hope you liked it 127 00:05:26,520 --> 00:05:28,470 ‫and I will see you in the next lecture.