1
00:00:00,170 --> 00:00:01,960
‫Now let's talk about CloudWatch Logs.

2
00:00:01,960 --> 00:00:05,160
‫So when you want to store your logs in AWS,

3
00:00:05,160 --> 00:00:06,420
‫the best place is CloudWatch Logs.

4
00:00:06,420 --> 00:00:07,253
‫So the idea is that you're going

5
00:00:07,253 --> 00:00:09,240
‫to group these logs into log groups,

6
00:00:09,240 --> 00:00:11,250
‫and this is a name that you want to give it,

7
00:00:11,250 --> 00:00:13,530
‫but usually it's representing an application.

8
00:00:13,530 --> 00:00:15,900
‫And within each log group you have log streams.

9
00:00:15,900 --> 00:00:18,450
‫And they represents instances within the application

10
00:00:18,450 --> 00:00:19,680
‫or different log file names

11
00:00:19,680 --> 00:00:21,460
‫or different containers, and so on,

12
00:00:21,460 --> 00:00:23,480
‫then you define log expiration policy.

13
00:00:23,480 --> 00:00:25,760
‫For example, you may never want the log to expire,

14
00:00:25,760 --> 00:00:27,810
‫or do you want it to be deleted after 30 days and so on,

15
00:00:27,810 --> 00:00:30,700
‫because you are paying for storage on CloudWatch Logs,

16
00:00:30,700 --> 00:00:31,533
‫then from Cloudwatch Logs,

17
00:00:31,533 --> 00:00:33,280
‫you can export the logs through multiple places,

18
00:00:33,280 --> 00:00:35,630
‫such as Amazon S3, Kinesis Data streams,

19
00:00:35,630 --> 00:00:39,190
‫Kinesis Data Firehose, Lambda, and OpenSearch.

20
00:00:39,190 --> 00:00:42,120
‫Now what types of logs can go into CloudWatch logs.

21
00:00:42,120 --> 00:00:44,162
‫Well we can send the logs using the SDK

22
00:00:44,162 --> 00:00:46,390
‫or the CloudWatch Logs Agent

23
00:00:46,390 --> 00:00:48,430
‫or the CloudWatch Unified Agent.

24
00:00:48,430 --> 00:00:51,220
‫Now the CloudWatch Unified Agent send logs to Cloudwatch

25
00:00:51,220 --> 00:00:54,260
‫into the CloudWatch Logs Agent is now sort of deprecated.

26
00:00:54,260 --> 00:00:56,580
‫You have Elastic Beanstalk, which is used to collect logs

27
00:00:56,580 --> 00:00:58,114
‫from the application directly into CloudWatch,

28
00:00:58,114 --> 00:01:00,010
‫ECS will send the logs directly

29
00:01:00,010 --> 00:01:01,867
‫from the containers into CloudWatch.

30
00:01:01,867 --> 00:01:05,490
‫Lambda will send logs from the functions themselves,

31
00:01:05,490 --> 00:01:07,090
‫VPC Flow Logs will send logs

32
00:01:07,090 --> 00:01:09,492
‫specific to your VPC metadata network traffic,

33
00:01:09,492 --> 00:01:12,550
‫API gateway will send all the requests

34
00:01:12,550 --> 00:01:14,710
‫made to the API gateway into CloudWatch logs,

35
00:01:14,710 --> 00:01:16,230
‫CloudTrail, you can send the logs there

36
00:01:16,230 --> 00:01:17,580
‫based on the filter

37
00:01:17,580 --> 00:01:19,690
‫and Route53 will log all the DNS queries

38
00:01:19,690 --> 00:01:22,040
‫made to its service.

39
00:01:22,040 --> 00:01:23,720
‫So another thing you can do that's very important

40
00:01:23,720 --> 00:01:26,060
‫is to define metric filter and insights.

41
00:01:26,060 --> 00:01:27,690
‫So the idea is that you have your Cloudwatch Logs

42
00:01:27,690 --> 00:01:29,400
‫and you can use your filter expressions, for example,

43
00:01:29,400 --> 00:01:32,310
‫to find all the specific IP within, within logs.

44
00:01:32,310 --> 00:01:34,590
‫So find the log lines where that IP appears

45
00:01:34,590 --> 00:01:36,850
‫or find every log line in your logs

46
00:01:36,850 --> 00:01:38,480
‫that contains the word errors.

47
00:01:38,480 --> 00:01:40,140
‫And then thanks to this metric filter,

48
00:01:40,140 --> 00:01:43,370
‫you can start counting these occurrences and then these

49
00:01:43,370 --> 00:01:44,980
‫becomes a metric, okay?

50
00:01:44,980 --> 00:01:48,670
‫And this metric can be linked into a CloudWatch alarm.

51
00:01:48,670 --> 00:01:50,800
‫Then the other feature that's really cool to discuss

52
00:01:50,800 --> 00:01:52,510
‫is CloudWatch Logs Insights.

53
00:01:52,510 --> 00:01:54,140
‫So the idea is that the CloudWatch Logs Insights,

54
00:01:54,140 --> 00:01:56,350
‫You can query logs and add these queries

55
00:01:56,350 --> 00:01:58,300
‫into CloudWatch Dashboards directly.

56
00:01:58,300 --> 00:02:00,570
‫And some common queries are added directly by AWS,

57
00:02:00,570 --> 00:02:03,100
‫and this is a very easy language to use.

58
00:02:03,100 --> 00:02:06,330
‫So first let's talk about the S3 export.

59
00:02:06,330 --> 00:02:08,656
‫So you send from CloudWatch into Amazon S3,

60
00:02:08,656 --> 00:02:11,130
‫and this can take up to 12 hours

61
00:02:11,130 --> 00:02:13,420
‫to become available for export.

62
00:02:13,420 --> 00:02:16,026
‫And the API call is CreateExportTask,

63
00:02:16,026 --> 00:02:18,056
‫but this is going to be done in its own time.

64
00:02:18,056 --> 00:02:20,356
‫And so it's not near real time or real time.

65
00:02:20,356 --> 00:02:23,130
‫Instead, if you want to stream logs from CloudWatch logs,

66
00:02:23,130 --> 00:02:25,100
‫you need to use subscriptions,

67
00:02:25,100 --> 00:02:26,047
‫now subscriptions are what?

68
00:02:26,047 --> 00:02:28,090
‫Well subscriptions are a filter

69
00:02:28,090 --> 00:02:29,910
‫that you apply on top of your Cloudwatch logs,

70
00:02:29,910 --> 00:02:31,930
‫and then you can send it to a destination.

71
00:02:31,930 --> 00:02:33,560
‫So it could be a Lambda function.

72
00:02:33,560 --> 00:02:37,380
‫For example, you define a custom or one that is used by AWS.

73
00:02:37,380 --> 00:02:40,400
‫If you want to send data into Amazon OpenSearch directly,

74
00:02:40,400 --> 00:02:41,818
‫or it could be Kinesis data Firehose,

75
00:02:41,818 --> 00:02:43,540
‫if you want to send it for example,

76
00:02:43,540 --> 00:02:45,380
‫to Amazon S3 in near real time.

77
00:02:45,380 --> 00:02:47,370
‫So this is a faster alternative

78
00:02:47,370 --> 00:02:48,203
‫to the one I just showed you

79
00:02:48,203 --> 00:02:51,710
‫from before using the exports from CloudWatch to S3,

80
00:02:51,710 --> 00:02:54,190
‫or you could use Kineses Data Streams, for example,

81
00:02:54,190 --> 00:02:56,658
‫to send the data into Kinesis Data Firehose

82
00:02:56,658 --> 00:03:00,793
‫Kinesis Data Analytics, Amazon EC2 or Lambda, and so on.

83
00:03:01,690 --> 00:03:02,523
‫Finally, with Cloudwatch Logs,

84
00:03:02,523 --> 00:03:04,960
‫you can do some log aggregation across accounts

85
00:03:04,960 --> 00:03:06,160
‫and across regions.

86
00:03:06,160 --> 00:03:09,050
‫So you may have like multiple accounts with,

87
00:03:09,050 --> 00:03:11,330
‫for example, a region that has a subscription filter.

88
00:03:11,330 --> 00:03:15,110
‫That's sent into Kinesis Data Streams in a common accounts,

89
00:03:15,110 --> 00:03:15,950
‫same for account B.

90
00:03:15,950 --> 00:03:18,050
‫So region two, same architecture and so on.

91
00:03:18,050 --> 00:03:20,780
‫And so we can centralize all these logs together into

92
00:03:20,780 --> 00:03:23,100
‫Kinesis Data Streams, for example, and then Data Firehose.

93
00:03:23,100 --> 00:03:25,036
‫And then for example, Amazon S3.

94
00:03:25,036 --> 00:03:26,390
‫Okay.

95
00:03:26,390 --> 00:03:28,950
‫So that's it for the overview of how Cloudwatch Logs work.

96
00:03:28,950 --> 00:03:31,783
‫I hope you liked it, and I will see you in the next lecture.