1 00:00:00,070 --> 00:00:01,327 Though we have to 2 00:00:01,327 --> 00:00:03,530 talk about the S3 Baseline Performance. 3 00:00:03,530 --> 00:00:05,878 So by default, Amazon S3 automatically scales 4 00:00:05,878 --> 00:00:08,727 to a very, very high number of requests and 5 00:00:08,727 --> 00:00:10,954 has a very very low latency between 6 00:00:10,954 --> 00:00:15,180 100 and 200 milliseconds to get the first bites out of S3 7 00:00:15,180 --> 00:00:17,460 So this is quite fast and in terms of 8 00:00:17,460 --> 00:00:19,460 how many requests per second you can get 9 00:00:19,460 --> 00:00:23,240 you can get 3,500 PUT/COPY/POST/DELETE per second, 10 00:00:23,240 --> 00:00:27,410 per prefix and 5,500 GET/HEAD requests 11 00:00:27,410 --> 00:00:29,310 per second per prefix in a bucket. 12 00:00:29,310 --> 00:00:30,880 So this is something you can get on the website and 13 00:00:30,880 --> 00:00:32,740 I think it's not very clear so I'll explain to you 14 00:00:32,740 --> 00:00:34,980 what per second per prefix means. 15 00:00:34,980 --> 00:00:37,010 But what that means (indistinct) is that it's 16 00:00:37,010 --> 00:00:38,920 really, really high-performance. 17 00:00:38,920 --> 00:00:40,270 And there's no limits to the 18 00:00:40,270 --> 00:00:42,475 number of prefixes in your bucket. 19 00:00:42,475 --> 00:00:46,980 So let's take an example of four objects named file and 20 00:00:46,980 --> 00:00:49,277 let's analyze the prefix for that object. 21 00:00:49,277 --> 00:00:51,277 The first one is in your bucket 22 00:00:51,277 --> 00:00:55,040 in folder one sub folder one slash file. 23 00:00:55,040 --> 00:00:56,988 In this case, the prefix is going to be 24 00:00:56,988 --> 00:01:00,300 anything between the bucket and the file. 25 00:01:00,300 --> 00:01:03,860 So in this case, it is slash folder one slash sub one. 26 00:01:03,860 --> 00:01:06,342 So that means that for this file in this prefix 27 00:01:06,342 --> 00:01:11,342 you can get 3,500 PUTs and 5,500 GETs per second. 28 00:01:11,800 --> 00:01:15,260 Now, if we have another folder one and then sub two, 29 00:01:15,260 --> 00:01:18,850 the prefix is any anything between bucket and file. 30 00:01:18,850 --> 00:01:20,970 So slash folder one slash sub two. 31 00:01:20,970 --> 00:01:24,580 And so we get also 3,500 PUTs and 5,500 GETs 32 00:01:24,580 --> 00:01:26,720 for that one prefix and so on. 33 00:01:26,720 --> 00:01:30,320 So if I have one and two, we have different prefixes. 34 00:01:30,320 --> 00:01:33,070 And so it's easy now to understand what a prefix is. 35 00:01:33,070 --> 00:01:34,743 And so it's easy to understand the rule of 36 00:01:34,743 --> 00:01:37,312 3,500 PUTs and 5,500 GETs per second 37 00:01:37,312 --> 00:01:39,360 per prefix in a bucket. 38 00:01:39,360 --> 00:01:40,930 So that means that if you spread reads 39 00:01:40,930 --> 00:01:43,511 across all the four prefixes above evenly 40 00:01:43,511 --> 00:01:46,480 you can achieve 22,000 requests per second 41 00:01:46,480 --> 00:01:48,438 for HEAD and GETs, so that's perfect. 42 00:01:48,438 --> 00:01:49,930 Next, let's talk about 43 00:01:49,930 --> 00:01:52,620 KMS as a Limitation to S3 performance. 44 00:01:52,620 --> 00:01:54,900 So if you have KMS encryption on your objects 45 00:01:54,900 --> 00:01:58,399 using SSE-KMS, then you may be impacted by the KMS limits. 46 00:01:58,399 --> 00:02:02,420 When you upload a file, it will call S3 on your behalf 47 00:02:02,420 --> 00:02:04,630 The GenerateDataKey KMS API. 48 00:02:04,630 --> 00:02:06,090 And when you download a file from S3 49 00:02:06,090 --> 00:02:08,710 using SSE-KMS (indistinct) itself 50 00:02:08,710 --> 00:02:10,713 will call the Decrypt KMS-API. 51 00:02:10,713 --> 00:02:12,500 And so these two requests will 52 00:02:12,500 --> 00:02:14,051 count towards the KMS quota. 53 00:02:14,051 --> 00:02:17,880 So to set an example, our users connect their S3 bucket and 54 00:02:17,880 --> 00:02:19,970 they want to upload or download a file 55 00:02:19,970 --> 00:02:22,060 using SSE-KMS encryption. 56 00:02:22,060 --> 00:02:24,720 And so S3 bucket will perform an API call 57 00:02:24,720 --> 00:02:26,690 either generated a key or decrypt 58 00:02:26,690 --> 00:02:29,800 to a KMS key and get the result from it. 59 00:02:29,800 --> 00:02:32,152 And so by default KMS has a quota 60 00:02:32,152 --> 00:02:34,550 of number of requests per second and 61 00:02:34,550 --> 00:02:36,130 based on the region you're in, it could be 62 00:02:36,130 --> 00:02:38,360 5,500 per second, 10,000 per second, or 63 00:02:38,360 --> 00:02:40,360 30,000 per second request. 64 00:02:40,360 --> 00:02:42,160 And you can request a quota increase 65 00:02:42,160 --> 00:02:45,424 Using the Service Quotas console if you need to go over that limit. 66 00:02:45,424 --> 00:02:46,436 So what this means is that 67 00:02:46,436 --> 00:02:50,208 if you have more than 10,000 requests per second 68 00:02:50,208 --> 00:02:53,230 in a specific region that only supports 5,500 requests 69 00:02:53,230 --> 00:02:56,090 per second for KMS, then you will be throttled. 70 00:02:56,090 --> 00:02:58,060 So you need to ensure that KMS 71 00:02:58,060 --> 00:02:59,874 doesn't block your performance on S3. 72 00:02:59,874 --> 00:03:02,013 Now these quotas are pretty big for normal usage 73 00:03:02,013 --> 00:03:04,610 but still good to know if you have many, many files and 74 00:03:04,610 --> 00:03:06,707 a high usage of your S3 bucket. 75 00:03:06,707 --> 00:03:08,931 Now let's talk about S3 performance. 76 00:03:08,931 --> 00:03:10,618 How we can optimize it. 77 00:03:10,618 --> 00:03:13,100 The first one is multi-part upload. 78 00:03:13,100 --> 00:03:14,985 So it is recommended to use a multi-part upload 79 00:03:14,985 --> 00:03:17,930 for files that are over 100 megabytes and 80 00:03:17,930 --> 00:03:21,100 it must be used for files that are over five gigabytes. 81 00:03:21,100 --> 00:03:22,690 And what multi-part upload does 82 00:03:22,690 --> 00:03:24,860 is that it parallelize uploads and 83 00:03:24,860 --> 00:03:26,640 that will help us speed up the transfers 84 00:03:26,640 --> 00:03:28,570 to maximize the bandwidth. 85 00:03:28,570 --> 00:03:31,064 So as a diagram that always makes more sense. 86 00:03:31,064 --> 00:03:33,130 So we have a big file and we want to 87 00:03:33,130 --> 00:03:35,020 upload that file into Amazon S3. 88 00:03:35,020 --> 00:03:37,490 We will divide it in parts, so smaller chunks 89 00:03:37,490 --> 00:03:40,320 of that files and each of these parts will be uploaded 90 00:03:40,320 --> 00:03:42,580 in parallel to Amazon S3. 91 00:03:42,580 --> 00:03:45,370 In Amazon S3, once all the parts have been uploaded 92 00:03:45,370 --> 00:03:47,333 it's smart enough to put them together 93 00:03:47,333 --> 00:03:49,930 back into the big file. 94 00:03:49,930 --> 00:03:51,339 Okay, very important. 95 00:03:51,339 --> 00:03:53,769 Now we have S3 Transfer Acceleration 96 00:03:53,769 --> 00:03:56,550 which is for upload and download and 97 00:03:56,550 --> 00:03:58,740 it is to increase the transfer speed by 98 00:03:58,740 --> 00:04:01,363 transferring a file to an AWS edge location 99 00:04:01,363 --> 00:04:03,130 which will forward then the data 100 00:04:03,130 --> 00:04:05,350 to the S3 bucket in the target region. 101 00:04:05,350 --> 00:04:07,160 So edge locations that are more than regions, 102 00:04:07,160 --> 00:04:08,820 there are about 200 over 200 103 00:04:08,820 --> 00:04:10,601 edge locations today and it's growing. 104 00:04:10,601 --> 00:04:13,050 And let me show you in the graph what that means. 105 00:04:13,050 --> 00:04:14,787 And that's of Transfer Acceleration 106 00:04:14,787 --> 00:04:17,390 is compatible with multi-part upload. 107 00:04:17,390 --> 00:04:18,298 So let's have a look. 108 00:04:18,298 --> 00:04:20,711 We have a file in the United States of America, 109 00:04:20,711 --> 00:04:23,930 and we want to upload it to an S3 bucket in Australia. 110 00:04:23,930 --> 00:04:27,179 So what this will do is that we will upload that file 111 00:04:27,179 --> 00:04:29,920 through an edge location in the United States. 112 00:04:29,920 --> 00:04:31,440 We should be very, very quick and 113 00:04:31,440 --> 00:04:33,170 there we'll be using the public internet. 114 00:04:33,170 --> 00:04:34,743 And then from that edge location 115 00:04:34,743 --> 00:04:37,100 to the Amazon S3 bucket in Australia 116 00:04:37,100 --> 00:04:39,040 the edge location will transfer it 117 00:04:39,040 --> 00:04:41,710 over the fast private AWS network. 118 00:04:41,710 --> 00:04:42,996 So this is called Transfer Acceleration 119 00:04:42,996 --> 00:04:45,220 because we minimized the amount 120 00:04:45,220 --> 00:04:47,540 of public internet that we go through. 121 00:04:47,540 --> 00:04:48,970 And we maximized the amount of 122 00:04:48,970 --> 00:04:51,220 private AWS network that we go through. 123 00:04:51,220 --> 00:04:53,200 So Transfer Acceleration is a great way 124 00:04:53,200 --> 00:04:54,830 to speed up transfers. 125 00:04:54,830 --> 00:04:56,960 Okay. Now how about getting files? 126 00:04:56,960 --> 00:04:58,971 How about reading a file in the most efficient way? 127 00:04:58,971 --> 00:05:03,050 We have something called an S3 Byte-Range Fetches. 128 00:05:03,050 --> 00:05:04,438 And so it is to paralyze GETs by 129 00:05:04,438 --> 00:05:07,356 getting specific bite ranges for your files. 130 00:05:07,356 --> 00:05:10,190 So it's also in case you have a failure 131 00:05:10,190 --> 00:05:12,710 to get a specific byte-range then you can retry 132 00:05:12,710 --> 00:05:14,010 a smaller byte-range and you have 133 00:05:14,010 --> 00:05:15,810 better resilience in case of failures. 134 00:05:15,810 --> 00:05:18,530 So it can be used to speed up downloads this time. 135 00:05:18,530 --> 00:05:20,060 So let's, I have a file S3 it's 136 00:05:20,060 --> 00:05:22,180 really, really big and this is the file. 137 00:05:22,180 --> 00:05:23,710 Maybe you want to request a first part 138 00:05:23,710 --> 00:05:25,187 which is the first few bites of the file. 139 00:05:25,187 --> 00:05:28,600 And then the second part, and then the end parts. 140 00:05:28,600 --> 00:05:30,015 So we request all these parts 141 00:05:30,015 --> 00:05:32,640 as specific Byte-Range Fetches. 142 00:05:32,640 --> 00:05:33,473 That's why it's called Byte-Range 143 00:05:33,473 --> 00:05:35,300 because we only request a specific range 144 00:05:35,300 --> 00:05:38,394 of the file and all these requests can be made in parallel. 145 00:05:38,394 --> 00:05:39,860 So the idea is that we can 146 00:05:39,860 --> 00:05:43,422 parallelize the GETs and speed up the downloads. 147 00:05:43,422 --> 00:05:45,920 The second use case is to only retrieve 148 00:05:45,920 --> 00:05:47,830 a partial amount of the file. 149 00:05:47,830 --> 00:05:50,132 For example, if you know that the first 50 bytes 150 00:05:50,132 --> 00:05:52,810 of the file in S3 are a header and 151 00:05:52,810 --> 00:05:54,780 give you some information about the file, 152 00:05:54,780 --> 00:05:56,910 then you can just issue a header request, 153 00:05:56,910 --> 00:05:59,700 a Byte-Range request for the headers using the first, say, 154 00:05:59,700 --> 00:06:03,210 50 bytes and you would get that information very quickly. 155 00:06:03,210 --> 00:06:04,043 All right. 156 00:06:04,043 --> 00:06:04,876 So that's it for S3 Performance. 157 00:06:04,876 --> 00:06:06,313 We've seen how to speed up uploads, downloads. 158 00:06:06,313 --> 00:06:08,220 We've seen the baseline performance 159 00:06:08,220 --> 00:06:09,520 and we've seen the KMS limits. 160 00:06:09,520 --> 00:06:11,640 So make sure you know those into going into the exam. 161 00:06:11,640 --> 00:06:13,590 And I will see you in the next lecture.