1 00:00:00,660 --> 00:00:02,740 So let's about SQS, 2 00:00:02,740 --> 00:00:05,020 and at the core of SQS is a queue 3 00:00:05,020 --> 00:00:07,330 because SQS is a simple queuing service. 4 00:00:07,330 --> 00:00:08,970 So we have an SQS queue, 5 00:00:08,970 --> 00:00:11,150 and it is going to contain messages. 6 00:00:11,150 --> 00:00:12,610 And to contain messages, 7 00:00:12,610 --> 00:00:16,070 well, something needs to send messages into our SQS queue 8 00:00:16,070 --> 00:00:19,140 and whatever sends a message into our SQS queue 9 00:00:19,140 --> 00:00:21,110 is called a producer. 10 00:00:21,110 --> 00:00:23,490 So it's possible for us to have one producer, 11 00:00:23,490 --> 00:00:24,580 but also have more. 12 00:00:24,580 --> 00:00:26,380 You can have multiple producers 13 00:00:26,380 --> 00:00:29,370 sending many messages into an SQS queue. 14 00:00:29,370 --> 00:00:31,120 And the message could be whatever he wants. 15 00:00:31,120 --> 00:00:33,670 For example, it could be process this order, 16 00:00:33,670 --> 00:00:35,180 or process this video. 17 00:00:35,180 --> 00:00:38,350 Whatever message you create goes into the queue. 18 00:00:38,350 --> 00:00:41,680 Then something needs to process the messages from the queue 19 00:00:41,680 --> 00:00:44,610 and receive them, and it's called consumers. 20 00:00:44,610 --> 00:00:47,420 So consumers will poll the messages from the queue, 21 00:00:47,420 --> 00:00:49,620 so that means that they will ask the queue, 22 00:00:49,620 --> 00:00:50,970 do you have any message for me? 23 00:00:50,970 --> 00:00:52,880 And the queue says, yes, here it is. 24 00:00:52,880 --> 00:00:54,600 And the consumer will poll these messages 25 00:00:54,600 --> 00:00:56,210 and get some information. 26 00:00:56,210 --> 00:00:57,540 And then with that message, 27 00:00:57,540 --> 00:01:00,640 it will process it and then delete it back from the queue. 28 00:01:00,640 --> 00:01:02,240 And you may have multiple consumers 29 00:01:02,240 --> 00:01:04,670 consuming messages from an SQS queue. 30 00:01:04,670 --> 00:01:07,770 So a queuing service is here to be a buffer 31 00:01:07,770 --> 00:01:11,980 to decouple between your producers and your consumers. 32 00:01:11,980 --> 00:01:14,190 Now, SQS is a complicated service 33 00:01:14,190 --> 00:01:15,320 and we'll see that in-depth, 34 00:01:15,320 --> 00:01:18,670 but the very first offering is called Amazon SQS 35 00:01:18,670 --> 00:01:20,430 for standard queues. 36 00:01:20,430 --> 00:01:24,590 And SQS, historically on AWS, it is the oldest offering. 37 00:01:24,590 --> 00:01:27,470 It was one of the first services on AWS. 38 00:01:27,470 --> 00:01:29,540 It's over 10 years old so it's quite, 39 00:01:29,540 --> 00:01:31,670 quite set in stone about how it works. 40 00:01:31,670 --> 00:01:33,020 It's a fully managed service, 41 00:01:33,020 --> 00:01:35,510 and it will be used to decouple applications. 42 00:01:35,510 --> 00:01:39,310 So anytime you see application decoupling in your exam, 43 00:01:39,310 --> 00:01:41,850 think about Amazon SQS. 44 00:01:41,850 --> 00:01:45,250 Now, why is SQS so special? 45 00:01:45,250 --> 00:01:47,380 Well, we get an unlimited throughputs. 46 00:01:47,380 --> 00:01:48,213 That means you can send 47 00:01:48,213 --> 00:01:50,620 as many messages per second as you want, 48 00:01:50,620 --> 00:01:53,670 and the queue can have as many messages as you want as well. 49 00:01:53,670 --> 00:01:55,390 So there's no limit on throughputs 50 00:01:55,390 --> 00:01:58,710 and no limits in the number of messages in the queue. 51 00:01:58,710 --> 00:02:00,920 Now, each message is short-lived. 52 00:02:00,920 --> 00:02:01,770 What does that mean? 53 00:02:01,770 --> 00:02:03,310 That means that by default, 54 00:02:03,310 --> 00:02:06,860 the message will stay in the queue for four days 55 00:02:06,860 --> 00:02:08,530 and the maximum amount of time 56 00:02:08,530 --> 00:02:11,860 that a message can be in a queue is 14 days. 57 00:02:11,860 --> 00:02:14,380 That means that as soon as you send a message to a queue, 58 00:02:14,380 --> 00:02:18,250 it has to be read by consumer and deleted from the queue 59 00:02:18,250 --> 00:02:21,410 after being processed within that retention period, 60 00:02:21,410 --> 00:02:23,340 otherwise, it will be lost. 61 00:02:23,340 --> 00:02:24,990 Then we have low-latency. 62 00:02:24,990 --> 00:02:27,560 So SQS means that, for them, 63 00:02:27,560 --> 00:02:28,930 is that whenever you send a message 64 00:02:28,930 --> 00:02:30,330 or read a message from SQS, 65 00:02:30,330 --> 00:02:32,010 you will get a response very quickly, 66 00:02:32,010 --> 00:02:34,900 less than 10 milliseconds on publish and receive. 67 00:02:34,900 --> 00:02:37,330 And the messages in SQS have to be small. 68 00:02:37,330 --> 00:02:41,880 They have to be less than 256 kilobytes per messages sent. 69 00:02:41,880 --> 00:02:43,030 Now SQS is a queuing service, 70 00:02:43,030 --> 00:02:45,820 so you can see high throughput, high volume, and so on, 71 00:02:45,820 --> 00:02:49,450 and so, it is possible to have duplicates messages. 72 00:02:49,450 --> 00:02:50,760 That means that, for example, 73 00:02:50,760 --> 00:02:53,250 sometimes a message will be delivered twice, 74 00:02:53,250 --> 00:02:56,070 and so this is why it's called at least once delivery. 75 00:02:56,070 --> 00:02:58,860 And if you go ahead with writing an application, 76 00:02:58,860 --> 00:03:00,870 you need to take that into account. 77 00:03:00,870 --> 00:03:03,690 It can also have out of the order messages, 78 00:03:03,690 --> 00:03:06,450 which means it's best effort ordering, 79 00:03:06,450 --> 00:03:09,640 and we'll see there is another type of offering from SQS 80 00:03:09,640 --> 00:03:11,450 that can deal with that limitation, 81 00:03:11,450 --> 00:03:13,740 but we'll see this later on in this section. 82 00:03:13,740 --> 00:03:16,810 So, let's go back to our message producers. 83 00:03:16,810 --> 00:03:20,410 So, messages that are up to 256 kilobytes 84 00:03:20,410 --> 00:03:23,900 are sent into SQS by producers. 85 00:03:23,900 --> 00:03:25,950 And how does it happen? 86 00:03:25,950 --> 00:03:28,290 Well, the producers will send the messages to SQS 87 00:03:28,290 --> 00:03:31,310 using an SDK, software development kits. 88 00:03:31,310 --> 00:03:35,710 And the API to send a message to SQS is called SendMessage. 89 00:03:35,710 --> 00:03:36,840 Very simple. 90 00:03:36,840 --> 00:03:38,790 Now the message, it will be written, 91 00:03:38,790 --> 00:03:41,270 it will be persisted into the SQS queue 92 00:03:41,270 --> 00:03:44,920 until a consumer reads that message and deletes it, 93 00:03:44,920 --> 00:03:47,940 which signifies that the message has been processed. 94 00:03:47,940 --> 00:03:49,110 We know the retention. 95 00:03:49,110 --> 00:03:51,220 And so, what's a use case for producing messages? 96 00:03:51,220 --> 00:03:53,760 For example, you want to process an order, 97 00:03:53,760 --> 00:03:56,920 for example, packets, and then ship it to the center. 98 00:03:56,920 --> 00:03:58,780 And so you wanna do this in your own time 99 00:03:58,780 --> 00:04:01,530 so you will send a message into the SQS queue 100 00:04:01,530 --> 00:04:02,890 with maybe some information, 101 00:04:02,890 --> 00:04:05,870 such as the order ID, the customer ID, 102 00:04:05,870 --> 00:04:07,180 and any attributes you may want. 103 00:04:07,180 --> 00:04:08,840 For example, the address, and so on. 104 00:04:08,840 --> 00:04:11,230 And then your consumer, that is in application rights, 105 00:04:11,230 --> 00:04:14,260 will have to deal with that message itself. 106 00:04:14,260 --> 00:04:15,490 So, again, to confirm, 107 00:04:15,490 --> 00:04:19,329 SQS standard has unlimited throughputs. 108 00:04:19,329 --> 00:04:21,610 So we've seen about producers. Very simple. 109 00:04:21,610 --> 00:04:23,750 Now let's look about consumers. 110 00:04:23,750 --> 00:04:25,663 So consumers, they are applications 111 00:04:25,663 --> 00:04:28,150 that we have to write with some code 112 00:04:28,150 --> 00:04:31,910 and these applications can be running on EC2 instances, 113 00:04:31,910 --> 00:04:34,550 so your virtual servers on AWS, 114 00:04:34,550 --> 00:04:37,930 but also they can be running on your own on-premises servers 115 00:04:37,930 --> 00:04:39,350 if you wanted to, 116 00:04:39,350 --> 00:04:41,000 or, we haven't seen it yet, 117 00:04:41,000 --> 00:04:42,930 but they can also be running 118 00:04:42,930 --> 00:04:45,620 on Lambda functions on AWS Lambda. 119 00:04:45,620 --> 00:04:46,453 We'll see in this course, 120 00:04:46,453 --> 00:04:49,220 it is a serverless compute type of service. 121 00:04:49,220 --> 00:04:51,550 What this means is that you can just read messages 122 00:04:51,550 --> 00:04:52,660 directly from them as well. 123 00:04:52,660 --> 00:04:55,040 We'll see this later on this course, do not worry. 124 00:04:55,040 --> 00:04:59,700 So, going back to our simple use case about EC2 instances, 125 00:04:59,700 --> 00:05:02,200 our queue has a consumer 126 00:05:02,200 --> 00:05:05,767 and the consumer polls for SQS for messages. 127 00:05:05,767 --> 00:05:08,280 And that means that the consumer will ask the queue, 128 00:05:08,280 --> 00:05:10,360 do you have messages for me? 129 00:05:10,360 --> 00:05:13,630 And the consumer may receive up to 10 messages at a time. 130 00:05:13,630 --> 00:05:16,420 So if there are messages in the SQS queue, 131 00:05:16,420 --> 00:05:17,940 it will receive a valid response saying 132 00:05:17,940 --> 00:05:21,140 here are the messages that are waiting for you. 133 00:05:21,140 --> 00:05:23,140 Then the consumer, it's your code, 134 00:05:23,140 --> 00:05:25,620 has a responsibility to process these messages. 135 00:05:25,620 --> 00:05:29,110 For example, insert some orders into an RDS database. 136 00:05:29,110 --> 00:05:31,670 So you will go ahead and for each order, 137 00:05:31,670 --> 00:05:33,910 you will insert it into your RDS database, 138 00:05:33,910 --> 00:05:36,350 something that you have to write, obviously with your code, 139 00:05:36,350 --> 00:05:39,210 and then because these messages have been processed 140 00:05:39,210 --> 00:05:40,500 because they have been received 141 00:05:40,500 --> 00:05:43,320 and inserted into an Amazon RDS database, 142 00:05:43,320 --> 00:05:44,660 your consumer will go ahead 143 00:05:44,660 --> 00:05:47,030 and delete these messages from the queue 144 00:05:47,030 --> 00:05:49,330 using the DeleteMessage API. 145 00:05:49,330 --> 00:05:51,990 And this will guarantee that no other consumer 146 00:05:51,990 --> 00:05:53,590 will be able to see these messages 147 00:05:53,590 --> 00:05:56,973 and therefore, the message processing is complete. 148 00:05:57,870 --> 00:05:59,830 So now we can scale this up. 149 00:05:59,830 --> 00:06:02,000 We can have multiple consumers at a time. 150 00:06:02,000 --> 00:06:04,293 So our SQS queue can have multiple consumers 151 00:06:04,293 --> 00:06:08,270 that will receive and process these messages in parallel. 152 00:06:08,270 --> 00:06:10,660 So here we have three on EC2 instances, 153 00:06:10,660 --> 00:06:12,920 and so each consumer will receive 154 00:06:12,920 --> 00:06:16,560 a different set of messages by calling the poll function. 155 00:06:16,560 --> 00:06:18,560 And so, if somehow a message 156 00:06:18,560 --> 00:06:20,260 is not processed fast enough by a consumer, 157 00:06:20,260 --> 00:06:22,660 it will be received by other consumers, 158 00:06:22,660 --> 00:06:26,080 and so this is why we have at least once delivery. 159 00:06:26,080 --> 00:06:29,520 And this is also why we have best-effort message ordering. 160 00:06:29,520 --> 00:06:31,030 Now, as I said, when the consumers 161 00:06:31,030 --> 00:06:33,040 are done with the messages, they will have to delete them, 162 00:06:33,040 --> 00:06:36,090 otherwise, other consumers will see these messages. 163 00:06:36,090 --> 00:06:38,300 And so what this means is that with SQS queues, 164 00:06:38,300 --> 00:06:41,480 if we need to increase the throughputs 165 00:06:41,480 --> 00:06:43,000 because we have more messages, 166 00:06:43,000 --> 00:06:46,500 then we can add consumers and do horizontal scaling 167 00:06:46,500 --> 00:06:49,410 to improve the throughput of processing. 168 00:06:49,410 --> 00:06:51,820 And so, if you remember what we said, 169 00:06:51,820 --> 00:06:54,530 this is a perfect use case for using SQS 170 00:06:54,530 --> 00:06:56,860 with your Auto Scaling groups, or ASG. 171 00:06:56,860 --> 00:06:58,170 So, what does that mean? 172 00:06:58,170 --> 00:07:00,310 Well, that means that your consumers 173 00:07:00,310 --> 00:07:02,190 will be running on EC2 instances 174 00:07:02,190 --> 00:07:04,170 inside of an Auto Scaling group 175 00:07:04,170 --> 00:07:07,530 and they will be polling for messages from the SQS queue. 176 00:07:07,530 --> 00:07:10,110 But now your Auto Scaling group has to be scaling 177 00:07:10,110 --> 00:07:11,850 on some kind of metric, 178 00:07:11,850 --> 00:07:15,240 and a metric that is available to us is the Queue Length. 179 00:07:15,240 --> 00:07:17,850 It's called ApproximateNumberOfMessages. 180 00:07:17,850 --> 00:07:21,650 It is a CloudWatch Metric that's available in any SQS queue. 181 00:07:21,650 --> 00:07:23,190 And we could set up an alarm, 182 00:07:23,190 --> 00:07:28,110 such as whenever the queue length go over a certain level, 183 00:07:28,110 --> 00:07:30,310 then please set up a CloudWatch Alarm, 184 00:07:30,310 --> 00:07:33,670 and this alarm should increase the capacity 185 00:07:33,670 --> 00:07:36,750 of my Auto Scaling group by X amount. 186 00:07:36,750 --> 00:07:38,330 And this will guarantee that 187 00:07:38,330 --> 00:07:40,220 the more messages you have in your SQS queue, 188 00:07:40,220 --> 00:07:43,260 maybe because there's a surge of orders on your websites, 189 00:07:43,260 --> 00:07:45,670 the more EC2 instances will be provided 190 00:07:45,670 --> 00:07:47,140 by your Auto Scaling group, 191 00:07:47,140 --> 00:07:49,520 and you will accordingly process these messages 192 00:07:49,520 --> 00:07:51,160 at a higher throughputs. 193 00:07:51,160 --> 00:07:52,880 So this is a very common integration 194 00:07:52,880 --> 00:07:55,190 that you will see in the exam. 195 00:07:55,190 --> 00:07:57,240 Now SQS, again, the use case 196 00:07:57,240 --> 00:08:02,040 is to decouple between applications, so application tiers. 197 00:08:02,040 --> 00:08:04,660 So, for example, let's take an example of an application 198 00:08:04,660 --> 00:08:06,520 that processes videos. 199 00:08:06,520 --> 00:08:09,560 We could have just one big application 200 00:08:09,560 --> 00:08:12,220 that's called a front-end that will take the request 201 00:08:12,220 --> 00:08:14,370 and whenever a video needs to be processed, 202 00:08:14,370 --> 00:08:15,370 it will do the processing 203 00:08:15,370 --> 00:08:17,850 and then insert that into an S3 bucket. 204 00:08:17,850 --> 00:08:19,150 But the problem is that processing 205 00:08:19,150 --> 00:08:21,210 may be very, very long to do 206 00:08:21,210 --> 00:08:23,680 and it may just slow down your websites 207 00:08:23,680 --> 00:08:26,330 if you do this in the front-end here. 208 00:08:26,330 --> 00:08:29,210 So instead, you can decouple your application here and say, 209 00:08:29,210 --> 00:08:32,309 wait a minute, the request of processing a file 210 00:08:32,309 --> 00:08:34,120 and the actual processing of a file 211 00:08:34,120 --> 00:08:36,230 can happen in two different applications. 212 00:08:36,230 --> 00:08:38,200 And therefore, whenever you take a request 213 00:08:38,200 --> 00:08:39,419 to process a file, 214 00:08:39,419 --> 00:08:42,659 you will send a message into an SQS queue. 215 00:08:42,659 --> 00:08:45,210 Now, when you do the request to process, 216 00:08:45,210 --> 00:08:46,950 that file will be in the SQS queue 217 00:08:46,950 --> 00:08:50,100 and you can create a second processing tier 218 00:08:50,100 --> 00:08:52,160 called the back-end processing application 219 00:08:52,160 --> 00:08:54,260 that will be in its own Auto-Scaling group 220 00:08:54,260 --> 00:08:57,180 to receive these messages, process these videos, 221 00:08:57,180 --> 00:08:59,770 and insert them into an S3 bucket. 222 00:08:59,770 --> 00:09:01,990 So as we can see here with this architecture, 223 00:09:01,990 --> 00:09:04,590 we can scale the front-end accordingly, 224 00:09:04,590 --> 00:09:07,010 and we can scale the back-end accordingly as well, 225 00:09:07,010 --> 00:09:08,410 but independently. 226 00:09:08,410 --> 00:09:11,430 And because the SQS queue has unlimited throughputs 227 00:09:11,430 --> 00:09:13,490 and it has unlimited number of messages 228 00:09:13,490 --> 00:09:16,370 in terms of the queue, then you are really safe, 229 00:09:16,370 --> 00:09:20,310 and this is a robust and scalable type of architecture. 230 00:09:20,310 --> 00:09:22,030 And also, for your front-end, 231 00:09:22,030 --> 00:09:25,030 you can use the optimal type of EC2 instances 232 00:09:25,030 --> 00:09:26,520 or architecture for your front-end. 233 00:09:26,520 --> 00:09:27,530 And for the back-end, 234 00:09:27,530 --> 00:09:29,120 maybe if you're doing some video processing, 235 00:09:29,120 --> 00:09:31,150 you can use some EC2 instances 236 00:09:31,150 --> 00:09:34,350 that have a GPU, a graphical processing unit, 237 00:09:34,350 --> 00:09:36,170 because you know that these type of instances 238 00:09:36,170 --> 00:09:38,490 will be optimal for doing this kind of workload. 239 00:09:38,490 --> 00:09:40,680 So this is the kind of architecture 240 00:09:40,680 --> 00:09:41,830 that will come up in the exam 241 00:09:41,830 --> 00:09:42,920 and that you're expected to know, 242 00:09:42,920 --> 00:09:45,170 and this is an amazing and tremendous use case 243 00:09:45,170 --> 00:09:47,110 for SQS queues. 244 00:09:47,110 --> 00:09:48,980 Finally, SQS security. 245 00:09:48,980 --> 00:09:50,910 So we have encryption in-flight 246 00:09:50,910 --> 00:09:54,320 by sending and producing messages using the HTTPS API, 247 00:09:54,320 --> 00:09:56,920 we get at-rest encryption using KMS keys, 248 00:09:56,920 --> 00:09:59,440 and if we wanted to, we can do client-side encryption, 249 00:09:59,440 --> 00:10:01,160 but that means that the client has to perform 250 00:10:01,160 --> 00:10:02,950 the encryption and the decryption itself. 251 00:10:02,950 --> 00:10:06,270 It's not something that's supported by SQS out of the box. 252 00:10:06,270 --> 00:10:09,420 For access controls, IAM policies are going to be able to 253 00:10:09,420 --> 00:10:11,860 regulate access to the SQS API, 254 00:10:11,860 --> 00:10:14,800 but we also have SQS access policies, 255 00:10:14,800 --> 00:10:17,160 which are similar to S3 bucket policies, 256 00:10:17,160 --> 00:10:19,150 and they're very helpful when you want to do 257 00:10:19,150 --> 00:10:21,290 cross-account access to SQS queue, 258 00:10:21,290 --> 00:10:23,150 or when you want you to allow other services, 259 00:10:23,150 --> 00:10:26,240 such as SNS that we'll see very soon, or Amazon S3, 260 00:10:26,240 --> 00:10:27,870 to write to an SQS queue, 261 00:10:27,870 --> 00:10:29,980 for example, with S3 events. 262 00:10:29,980 --> 00:10:31,840 So that's it for SQS for an overview. 263 00:10:31,840 --> 00:10:32,673 I hope you liked it, 264 00:10:32,673 --> 00:10:35,480 and I will see you in the next lecture for some practice.