AWS let customer to generate and integrate some services logs. However, each log source has its own configuration making undertstanding quite difficult for new customers.
This post lists all main AWS services logging sources with a summary table, format, example and a Grok regex to parse log and ingest into a tool like Elastic Stack (ELK).
Summary table description
For each Service, the following summry will give you a clear overview of the log, its capabilities and limits.
| Log | Log source name (with doc link) |
|---|---|
| Content | Nature of event |
| Format | File format, value separator, possible custom format |
| Delivery | Frequency of sending logs |
| Output | Where we can stream log to another service/target |
| Filter | Possibility to filter log content |
| Scope | AWS context of regarding logs |
| Regional | AWS Region linking of the log source |
| ID | Amazon Resource Name (ARN) or ID format of the log resource (if exists) |
| Sharing | Native sharing capabilities with other AWS Accounts |
| Cost | Direct and indirect cost (without price) |
| Availability | Launch date |
Services logs
Native logging of most important AWS services.
CloudTrail logs
CloudTrail logs are splited into 3 parts, dependings of the input, output and sometimes event type.
| Log | CloudTrail Event History |
|---|---|
| Content | Last 90 days of AWS API call details. It's enabled by default and cannot be disabled. Included events types: Read and Write Management event (control plane). Formerly known as API Activity History. |
| Format | JSON |
| Delivery | 15 min |
| Output | Must be queried (API or Console) |
| Custom/Filter | apply only one filter at a time |
| Scope | Account |
| ID | None |
| Sharing | No |
| Regional | Yes. IAM Service events in us-east-1 only |
| Cost | Free |
| Availability | API Activity History: Mars 2015 (last 7 days on limited services and regions). Event History: August 2017 (GA, all regions), June 2018 (All Management events and last 90 days) |
-
| Log | CloudTrail trail |
|---|---|
| Content | AWS API call details. Included events types: Read/Write Management event (control plane), Data events (data plane) of S3 bucket and/or Lambda functions, Insights (unusual activity of write management events volume) |
| Format | JSON |
| Delivery | 15 min |
| Output | at least to a S3 bucket and optionnaly to a CloudWatch Logs log group. |
| Custom/Filter | Event type (Management, Data, Insight), Action type (Read, Write) and region (Current, All). KMS events can be excluded. A Organization trail can be created on the Master Account applied on all Accounts and regions . |
| Scope | Trail per Account or per Organization (current and future Accounts) |
| Regional | Yes. IAM Service events in us-east-1 only |
| ID | ARN: arn:aws:cloudtrail:<region>:<accountID>:trail/<trailName> |
| Sharing | No |
| Cost | The first trail of Management events is free, the following are paying as well as other events types: Data and Insights. Indirect charges: CloudWatch/S3 charges |
| Availability | November 2013 (GA) November 2016 (S3 Data Events) November 2019 (Insights) |
https://aws.amazon.com/blogs/mt/analyzing-cloudtrail-in-cloudwatch
-
| Log | EventBridge default bus |
|---|---|
| Content | AWS API call details (recorded by CloudTrail) can be catch in the EventBridge default bus of the Account. Only Write Management event (control plane) are available. Formerly known as 'CloudWatch Events' (same API). |
| Format | JSON |
| Delivery | near-real time (1 sec) |
| Output | EventBridge Rule |
| Custom/Filter | Service or/and action |
| Scope | Account |
| ID | ARN: arn:aws:events:<region>:<accountID>:rule/<ruleName> |
| Sharing | Yes by using another Account bus as Rule target |
| Regional | Yes. IAM Service events in us-east-1 only |
| Cost | Free. Indirect: EventBridge rule target charges |
| Availability | January 2016 |
Format
All Cloudtrail logs are generated in JSON. Content could be but complex to parse because JSON structure depends of each service and action details (What). However details about the Principal (Who) and Request Context (When and How) have the same structure.
Samples
I wrote a dedicated post. with some Cloudtrail samples.
AWS documentation lists some examples too.
Grok Regex
Not applicable to JSON. You should use json and split filter plugins in Logstash (see example below in Ingest in ELK).
Route 53 Resolver DNS queries
| Log | Route 53 Resolver Query Logging |
|---|---|
| Content | All DNS queries received by a VPC DNS resolver |
| Format | JSON |
| Delivery | 10-20 sec |
| Output | CloudWatch Logs group, S3 bucket or Kinesis Data Firehose delivery stream |
| Custom/Filter | No |
| Scope | VPC |
| Regional | Yes |
| ID | ARN: arn:aws:route53resolver:<region>:<acountID>:resolver-query-log-config/rqlc-<16digit> |
| Sharing | Yes with another AWS Accounts, OU/whole Organization through Resource Access Manager |
| Cost | Free. Indirect: CloudWatch/S3/Kinesis charges |
| Availability | August 2020 |
The Resolver Query Logging Configuration can only have one output and cannot be edited after creation.
Format
See below samples, JSON structure details format.
Sample
Real samples with a hosted private zone mhg.vpc and a public zone.
Private zone - Existing name:
{
"version": "1.000000",
"account_id": "1111222233334444",
"region": "eu-west-3",
"vpc_id": "vpc-5717186e",
"query_timestamp": "2020-11-01T09:25:02Z",
"query_name": "asset1.mhg.vpc.",
"query_type": "A",
"query_class": "IN",
"rcode": "NOERROR",
"answers": [
{
"Rdata": "10.5.0.34",
"Type": "A",
"Class": "IN"
}
],
"srcaddr": "172.31.24.160",
"srcport": "41366",
"transport": "UDP",
"srcids": {
"instance": "i-061fb32d2b51f86a3"
}
}
Private zone - No-existint name:
{
"version": "1.000000",
"account_id": "1111222233334444",
"region": "eu-west-3",
"vpc_id": "vpc-5717186e",
"query_timestamp": "2020-11-01T09:32:05Z",
"query_name": "test2.mhg.vpc.",
"query_type": "A",
"query_class": "IN",
"rcode": "NXDOMAIN",
"answers": [],
"srcaddr": "172.31.24.160",
"srcport": "37654",
"transport": "UDP",
"srcids": {
"instance": "i-061fb32d2b51f86a3"
}
}
Public zone - Existing name with multiple records:
{
"version": "1.000000",
"account_id": "1111222233334444",
"region": "eu-west-3",
"vpc_id": "vpc-5717186e",
"query_timestamp": "2020-11-01T10:06:16Z",
"query_name": "mhg.ovh.",
"query_type": "A",
"query_class": "IN",
"rcode": "NOERROR",
"answers": [
{
"Rdata": "13.32.145.91",
"Type": "A",
"Class": "IN"
},
{
"Rdata": "13.32.145.81",
"Type": "A",
"Class": "IN"
},
{
"Rdata": "13.32.145.126",
"Type": "A",
"Class": "IN"
},
{
"Rdata": "13.32.145.44",
"Type": "A",
"Class": "IN"
}
],
"srcaddr": "172.31.24.160",
"srcport": "58050",
"transport": "UDP",
"srcids": {
"instance": "i-061fb32d2b51f86a3"
}
}
Grok regex
Not applicable to JSON.
Route 53 Public DNS queries
| Log | Public DNS queries (public zone) |
|---|---|
| Content | DNS queries of the NS servers (resolvers, not final clients) |
| Format | Line with values separated by space |
| Delivery | 2-3 min |
| Output | CloudWatch Logs group |
| Custom/Filter | No |
| Scope | Route 53 public zone |
| Regional | No. Stored in us-east-1 only |
| ID | UUIDv4 (no ARN) |
| Sharing | Yes with another AWS Accounts, OU/whole Organization through Resource Access Manager |
| Cost | Free. Indirect: CloudWatch Logs charges |
| Availability | September 2017 |
CloudWatch Logs allows you to export streams to a S3 bucket, however you have to call yourself the CloudWatch API to initiate this asynchronous task. a scheduled Lambda function can be used to do the job.
Following samples come from S3 and includes the CloudWatch Logs timestamp because the Export Task API call task adds it in the S3 object.
Format
CloudWatch-timestamp Log-format-version Query-timestamp Hosted-zone-ID Query-name Query-type Response-code Layer-4-protocol Route-53-edge-location Resolver-IP-address EDNS-client-subnet
**Sample
2020-03-06T00:31:09.000Z 1.0 2020-03-06T00:31:09Z Z3M111CF8036L2 mhg.ovh AAAA NOERROR UDP VIE50 37.187.141.25 -
**Grok regex
%{TIMESTAMP_ISO8601:cwl_timestamp}\s%{NUMBER:log_version}\s%{TIMESTAMP_ISO8601:query_timestamp}\s%{WORD:hosted_zone_id}\s%{HOST:query_name}\s%{WORD:query_type}\s%{WORD:response_code}\s%{WORD:protocol}\s%{WORD:edge_location}\s%{IP:resolver_ip}\s(-|%{NOTSPACE:edns_client_subnet})
Flow Logs
| Log | Flow Logs |
|---|---|
| Content | IP Flow metadatas (ISO Layer 4) similar to NetFlow or IPFIX. |
| Format | Line with values separated by space |
| Delivery | 1 or 10 minutes (aggregation interval) |
| Output | CloudWatch Logs, S3 Bucket |
| Custom/Filter | Custom format available (fields and order) |
| Scope | VPC, Subnet or network interface (ENI). |
| Regional | Yes |
| ID | fl-<17charcaters> (No ARN) |
| Sharing | No |
| Cost | Free. Indirect: CloudWatch Logs/S3 charges |
| Availability | June 2015 (GA), August 2018 (S3 target) |
Format
Custom format with all fields
${version} ${account-id} ${vpc-id} ${subnet-id} ${instance-id} ${interface-id} ${type} ${srcaddr} ${dstaddr} ${pkt-srcaddr} ${pkt-dstaddr} ${srcport} ${dstport} ${protocol} ${tcp-flags} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}
Samples
Version number is 3 because of custom format.
3 013206742943 vpc-603f3c09 subnet-dc5cd691 i-00bc69353a1229ec3 eni-0cb464188f367fbc2 IPv4 52.95.155.48 172.31.40.102 52.95.155.48 172.31.40.102 443 37318 6 19 15577 21652528 1577191414 1577191424 ACCEPT OK
3 013206742943 vpc-603f3c09 subnet-dc5cd691 i-00bc69353a1229ec3 eni-0cb464188f367fbc2 IPv4 216.218.206.85 172.31.40.102 216.218.206.85 172.31.40.102 52862 5900 6 2 1 40 1577191398 1577191404 REJECT OK
Grok regex
^%{INT:fl_version:int}\s%{INT:account_id:int}\s%{HOSTNAME:vpc_id}\s%{HOSTNAME:subnet_id}\s%{HOSTNAME:instance_id}\s%{HOSTNAME:interface_id}\s%{WORD:flow_type}\s%{IP:ip_src}\s%{IP:ip_dst}\s%{IP:ip_src_pkt}\s%{IP:ip_dst_pkt}\s%{INT:port_src:int}\s%{INT:port_dst:int}\s%{INT:proto_num:int}\s%{INT:tcp_flag:int}\s%{INT:flow_packet:int}\s%{INT:flow_byte:int}\s%{INT:flow_start}\s%{INT:flow_end}\s%{WORD:flow_action}\s%{WORD:fl_status}$
CloudFront Access Logs
| Log | Cloudfront standard logs |
|---|---|
| Content | Web/RTMP access log of the CDNs similar to reverse proxy logs. |
| Format | Line with values separated by space |
| Delivery | Up to several times an hour. Can sometimes be delayed by up to 24 hours |
| Output | S3 Bucket |
| Custom/Filter | No |
| Scope | Distribution |
| Regional | No, stored in us-east-1 only |
| ID | None |
| Sharing | ? |
| Cost | Free. Indirect: S3 charges |
| Availability | May 2009 (GA), December 2019 (new fields) |
-
| Log | Real-time log configuration |
|---|---|
| Content | Web/RTMP access log of the CDN, similar to reverse proxy logs. |
| Format | Line |
| Delivery | Near real-time (1 sec) |
| Output | Kinesis data stream |
| Custom/Filter | Sampling rate (1-100%), fields (among 40) and Cache behavior |
| Scope | Attach configuration to Distribution(s) |
| Regional | No, stored in us-east-1 only |
| ID | ARN: arn:aws:cloudfront::<accountID>:realtime-log-config/<configID> |
| Sharing | To an external Kinesis data stream |
| Cost | Volume of log. Indirect: Kinesis |
| Availability | August 2020 |
Format
Standard log:
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields c-port time-to-first-byte x-edge-detailed-result-type sc-content-type sc-content-len sc-range-start sc-range-end
Sample
Standard log:
2019-12-26 09:12:26 FRA50-C1 3198 77.207.15.253 GET djm1t0idtm8px.cloudfront.net /about.html 200 https://mhg.ovh/ Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010.15;%2520rv:71.0)%2520Gecko/20100101%2520Firefox/71.0 - - Miss BCFMyYSZyL1u6ZixT2CLGNnFBcVg7P2Y-Rr93r68KwxN2d6OoFyVfw== mhg.ovh https 253 0.078 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss HTTP/2.0 - - 50024 0.078 Miss text/html - - -
Grok regex
Standard log:
^%{YEAR:date_year}\-%{MONTHNUM:date_month}\-%{MONTHDAY:date_day}\t%{TIME:time}\t%{HOSTNAME:edge_location}\t%{INT:res_byte:int}\t%{IP:ip_src}\t%{WORD:method}\t%{HOSTNAME:host}\t%{URIPATH:uri}\t%{INT:status}\t%{NOTSPACE:referer}\t%{NOTSPACE:user_agent}\t%{NOTSPACE:uri_query}\t%{NOTSPACE:cookie}\t%{WORD:edge_result}\t%{NOTSPACE:edge_req_id}\t%{HOSTNAME:header_host}\t%{WORD:protocol}\t%{INT:req_byte:int}\t%{BASE10NUM:edge_time}\t%{NOTSPACE:ip_xff}\t%{NOTSPACE:ssl_protocol}\t%{NOTSPACE:ssl_chiper}\t%{WORD:edge_result_response}\t%{NOTSPACE:protocol_version}\t%{NOTSPACE:fle_status}\t%{NOTSPACE:fle_field}\t%{INT:port_src}\t%{BASE10NUM:edge_time_first}\t%{NOTSPACE:edge_result_detail}\t%{NOTSPACE:res_content_type}\t%{NOTSPACE:res_content_len:int}\t%{NOTSPACE:res_range_start}\t%{NOTSPACE:res_range_end}$
GuardDuty findings
| Log | GuardDuty finding through EventBridge |
|---|---|
| Content | Security alerts |
| Format | JSON |
| Delivery | 5 min for new finding and 15min/1h/6h for next occurrences |
| Output | EventBridge Account default bus |
| Custom/Filter | Rule pattern |
| Scope | GuardDuty Detector |
| Regional | Yes |
| ID | ARN: arn:aws:events:<region>:<accountID>:rule/<ruleName> |
| Sharing | Possible with Rule target |
| Cost | Free. Indirect: EventBridge |
| Availability | November 2017 |
-
| Log | GuardDuty finding S3 Export |
|---|---|
| Content | Security alerts |
| Format | JSON |
| Delivery | 5 min for new finding and 15min/1h/6h for next occurrences |
| Output | S3 bucket |
| Custom/Filter | No |
| Scope | GuardDuty Detector |
| Regional | Yes |
| ID | 32 characters (no ARN) |
| Sharing | Possible to a external and unique S3 bucket for multiple Export configuration |
| Cost | Free. Indirect: S3 and KMS CMK charges |
| Availability | November 2019 |
Remarks
- If you use the Multi-Account/Organization feature in a GuardDuty administrator account, all findings related to member Accounts will be exported to the S3 bucket configured in the Administrator account.
- Charges: a KMS CMK key must be used to export to S3.
Format
n/a
Samples
Policy:IAMUser/RootCredentialUsage finding (An API was invoked using root credentials)
[
{
"schemaVersion": "2.0",
"accountId": "111111111111",
"region": "us-east-1",
"partition": "aws",
"id": "aab8f74669d66b7115893a8a87779XXX",
"arn": "arn:aws:guardduty:eu-west-3:111111111111:detector/caFC17dEAbDBCe5CA3EeEeEBaEXXXXXXX/finding/aab8f74669d66b7115893a8a8XXXXXXX",
"type": "Policy:IAMUser/RootCredentialUsage",
"resource": {
"resourceType": "AccessKey",
"accessKeyDetails": {
"accessKeyId": "ASIA2MSOSYTBPZMUXXXX",
"principalId": "111111111111",
"userType": "Root",
"userName": "Root"
}
},
"service": {
"serviceName": "guardduty",
"detectorId": "caFC17dEAbDBCe5CA3EeEeEBaEEcEXXX",
"action": {
"actionType": "AWS_API_CALL",
"awsApiCallAction": {
"api": "CreateServiceLinkedRole",
"serviceName": "iam.amazonaws.com",
"callerType": "Remote IP",
"remoteIpDetails": {
"ipAddressV4": "72.21.217.15",
"organization": {
"asn": "16509",
"asnOrg": "AMAZON-02",
"isp": "Amazon.com",
"org": "Amazon.com"
},
"country": {
"countryName": "United States"
},
"city": {
"cityName": "Ashburn"
},
"geoLocation": {
"lat": 39.0481,
"lon": -77.4728
}
},
"affectedResources": {}
}
},
"resourceRole": "TARGET",
"additionalInfo": {},
"evidence": null,
"eventFirstSeen": "2020-05-07T09:34:32Z",
"eventLastSeen": "2020-05-07T09:35:35Z",
"archived": false,
"count": 2
},
"severity": 2,
"createdAt": "2020-05-07T09:52:18.860Z",
"updatedAt": "2020-05-07T09:52:18.860Z",
"title": "API CreateServiceLinkedRole was invoked using root credentials.",
"description": "API CreateServiceLinkedRole was invoked using root credentials from IP address 72.21.217.15."
}
]
Grok regex
Not applicable to JSON.
Security Hub
| Log | Security Hub finding trough EventBridge |
|---|---|
| Content | Alerts received from other security tools |
| Format | JSON, AWS Security Finding Format (ASFF) |
| Delivery | Near real-time (1sec) ? |
| Output | EventBridge Account default bus |
| Custom/Filter | Rule pattern and 3 event types: All findings, Findings for custom actions, Insight results for custom actions |
| Scope | Security Hub |
| Regional | Yes |
| ID | EventBridge |
| Sharing | Another Account with EventBridge target |
| Cost | Free. Indirect: EventBridge |
| Availability | November 2018 |
Format
AWS published a standardized format, AWS Security Finding Format (ASFF), detailled on the documenttation.
Samples
GuardDuty finding:
{
"SchemaVersion": "2018-10-08",
"Id": "arn:aws:guardduty:eu-west-3:111122223333:detector/fcb76996ec7afe8c7fbef8e5c08eXXXX/finding/94b9ef45f2f0f4b2dc00006d77b8XXXX",
"ProductArn": "arn:aws:securityhub:eu-west-3::product/aws/guardduty",
"GeneratorId": "arn:aws:guardduty:eu-west-3:111122223333:detector/fcb76996ec7afe8c7fbef8e5c08eXXXX",
"AwsAccountId": "111122223333",
"Types": [
"Software and Configuration Checks/Policy:S3.BucketAnonymousAccessGranted"
],
"FirstObservedAt": "2020-08-11T17:09:59Z",
"LastObservedAt": "2020-08-11T17:09:59Z",
"CreatedAt": "2020-08-11T17:22:57.121Z",
"UpdatedAt": "2020-08-11T17:22:57.121Z",
"Severity": {
"Product": 8,
"Label": "HIGH",
"Normalized": 80
},
"Title": "Amazon S3 Public Anonymous Access was granted for S3 bucket bucket1.",
"Description": "The Amazon S3 bucket bucket1 was granted public anonymous access by Role1 calling PutBucketPolicy. If this behavior is not expected, it may indicate a configuration mistake or that your credentials are compromised.",
"SourceUrl": "https://eu-west-3.console.aws.amazon.com/guardduty/home?region=eu-west-3#/findings?macros=current&fId=94b9ef45f2f0f4b2dc00006d77b8XXXX",
"ProductFields": {
"aws/guardduty/service/archived": "false",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/organization/asnOrg": "XXXX",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/organization/org": "XXXX",
"aws/guardduty/service/additionalInfo": "",
"aws/guardduty/service/resourceRole": "TARGET",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/organization/isp": "XXXX",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/geoLocation/lat": "X.X",
"aws/guardduty/service/count": "1",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/ipAddressV4": "1.1.1.1",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/country/countryName": "XXXX",
"aws/guardduty/service/action/awsApiCallAction/callerType": "Remote IP",
"aws/guardduty/service/action/awsApiCallAction/serviceName": "s3.amazonaws.com",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/city/cityName": "XXXX",
"aws/guardduty/service/action/awsApiCallAction/api": "PutBucketPolicy",
"aws/guardduty/service/serviceName": "guardduty",
"aws/guardduty/service/evidence": "",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/geoLocation/lon": "X.X",
"aws/guardduty/service/detectorId": "fcb76996ec7afe8c7fbef8e5c08eXXXX",
"aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/organization/asn": "XXXX",
"aws/guardduty/service/eventFirstSeen": "2020-08-11T17:09:59Z",
"aws/guardduty/service/action/awsApiCallAction/affectedResources/AWS::S3::Bucket": "bucket1",
"aws/guardduty/service/eventLastSeen": "2020-08-11T17:09:59Z",
"aws/guardduty/service/action/actionType": "AWS_API_CALL",
"aws/securityhub/FindingId": "arn:aws:securityhub:eu-west-3::product/aws/guardduty/arn:aws:guardduty:eu-west-3:111122223333:detector/fcb76996ec7afe8c7fbef8e5c08eXXXX/finding/94b9ef45f2f0f4b2dc00006d77b8XXXX",
"aws/securityhub/ProductName": "GuardDuty",
"aws/securityhub/CompanyName": "Amazon"
},
"Resources": [
{
"Type": "AwsIamAccessKey",
"Id": "AWS::IAM::AccessKey:ASIA4HGNPFE47Q4MXXXX",
"Partition": "aws",
"Region": "eu-west-3",
"Details": {
"AwsIamAccessKey": {
"PrincipalId": "AROA4HGNPFE4XI5FLXXXX:user1",
"PrincipalType": "AssumedRole",
"PrincipalName": "Role1"
}
}
}
],
"WorkflowState": "NEW",
"Workflow": {
"Status": "NEW"
},
"RecordState": "ACTIVE"
}
Grok regex
Not applicable to JSON.
Usefull resources
S3 Access logs
| Log | Server access logging |
|---|---|
| Content | |
| Format | Line with values separated by space |
| Delivery | Few hours or less |
| Output | S3 bucket: the bucket source itself (not recommended) or any one the same Account and Region |
| Custom/Filter | No |
| Scope | S3 bucket |
| Regional | Yes |
| ID | |
| Sharing | No |
| Cost | Free. Indirect: S3 |
| Availability | Before 2009 (GA), March 2019 (6 new fields) |
-
| Log | Object-level logging (CloudTrail Data events) |
|---|---|
| Content | |
| Format | JSON |
| Delivery | |
| Output | S3 bucket |
| Custom/Filter | |
| Scope | |
| Regional | |
| ID | |
| Sharing | |
| Cost | |
| Availability |
Remarks
- AWS recommends that you use AWS CloudTrail data events instead of Amazon S3 access logs. CloudTrail data events are easier to set up and contain more information. Source
- CloudTrail object logging: CloudTrail supports logging Amazon S3 object level API operations such as GetObject, PutObject, and DeleteObject. (Nov 2016)
Format
Please refer to the AWS documentation
Samples
Object/Data event trough Cloudtrail: Athena Service reads an object on behalf of an IAM User that assumed a Role in the Account's bucket. The example show a bucket with Scott Piper's flaws.cloud CloudTrail logs.
{
"eventVersion": "1.07",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROA4HGNPFE4XI5FLXXXX:user1",
"arn": "arn:aws:sts::111122223333:assumed-role/OrganizationAccountAccessRole/user1",
"accountId": "111122223333",
"accessKeyId": "ASIAIAH3T5UEST3NXXXX",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROA4HGNPFE4XI5FLXXXX",
"arn": "arn:aws:iam::111122223333:role/Role1",
"accountId": "111122223333",
"userName": "Role1"
},
"attributes": {
"creationDate": "2020-10-31T16:41:18Z",
"mfaAuthenticated": "true"
}
},
"invokedBy": "athena.amazonaws.com"
},
"eventTime": "2020-10-31T17:05:36Z",
"eventSource": "s3.amazonaws.com",
"eventName": "GetObject",
"awsRegion": "us-east-1",
"sourceIPAddress": "athena.amazonaws.com",
"userAgent": "athena.amazonaws.com",
"requestParameters": {
"bucketName": "xxxx-flaws-cloud-cloudtrail",
"Host": "xxxx-flaws-cloud-cloudtrail.s3.amazonaws.com",
"key": "flaws_cloudtrail09.json.gz"
},
"responseElements": null,
"additionalEventData": {
"SignatureVersion": "SigV4",
"CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
"bytesTransferredIn": 0,
"AuthenticationMethod": "AuthHeader",
"x-amz-id-2": "YVT7gnHZDu7agzbOV+t8GECNPiJ31IErBUGuTEFkJyxvQ4HYylq5w0SBQOogF9u9Q+iee54wXXXX",
"bytesTransferredOut": 10977945
},
"requestID": "DE670E80ED3590E0",
"eventID": "b02731a7-41b4-4cd5-aabf-858d447bXXXX",
"readOnly": true,
"resources": [
{
"type": "AWS::S3::Object",
"ARN": "arn:aws:s3:::xxxx-flaws-cloud-cloudtrail/flaws_cloudtrail09.json.gz"
},
{
"accountId": "111122223333",
"type": "AWS::S3::Bucket",
"ARN": "arn:aws:s3:::xxxx-flaws-cloud-cloudtrail"
}
],
"eventType": "AwsApiCall",
"managementEvent": false,
"recipientAccountId": "111122223333",
"eventCategory": "Data"
}
Config (rule, snapshot)
TODO
CloudWatch Agent logs
TODO
WAF logs
TODO
Elastic Load Balancing Access Logs
TODO
https://aws.amazon.com/about-aws/whats-new/2014/03/06/elastic-load-balancing-announces-access-logs/
Amazon Redshift Logs
TODO
Amazon Relational Database Service (RDS) Logs
TODO
AWS Detailed Billing Reports
TODO
Ingest in ELK
Logstash template
The following logstash pipeline ingests:
- CloudTrail
- GuardDuty findings
- VPC Flow Logs (custom)
- CloudFront Logs
input {
s3 {
bucket => "bucket-cloudtrail"
region => "eu-west-3"
type => "cloudtrail"
}
s3 {
bucket => "bucket-guardduty"
region => "eu-west-3"
type => "guardduty"
}
s3 {
bucket => "bucket-flowlogs"
region => "eu-west-3"
type => "flowlogs"
}
s3 {
bucket => "bucket-cloudfront"
region => "eu-west-3"
type => "cloudfront"
sincedb_path => "/var/lib/logstash/plugins/inputs/s3/sincedb_org-mhg-cloudfront"
}
}
filter {
if [type] == "cloudtrail" {
mutate {
rename => { "type" => "es_type" }
}
json {
source => "message"
}
split {
field => "Records"
}
geoip {
source => "[Records][sourceIPAddress]"
target => "geoip"
}
}
if [type] == "guardduty" {
mutate {
rename => { "type" => "es_type" }
}
json {
source => "message"
}
}
if [type] == "flowlogs" {
grok {
match => { "message" => "^%{INT:fl_version:int}\s%{INT:account_id:int}\s%{HOSTNAME:vpc_id}\s%{HOSTNAME:subnet_id}\s%{HOSTNAME:instance_id}\s%{HOSTNAME:interface_id}\s%{WORD:flow_type}\s%{IP:ip_src}\s%{IP:ip_dst}\s%{IP:ip_src_pkt}\s%{IP:ip_dst_pkt}\s%{INT:port_src:int}\s%{INT:port_dst:int}\s%{INT:proto_num:int}\s%{INT:tcp_flag:int}\s%{INT:flow_packet:int}\s%{INT:flow_byte:int}\s%{INT:flow_start}\s%{INT:flow_end}\s%{WORD:flow_action}\s%{WORD:fl_status}$" }
}
if "_grokparsefailure" in [tags] {
drop { }
}
date {
match => [ "flow_start", "UNIX" ]
target => "@timestamp"
}
date {
match => [ "flow_end", "UNIX" ]
target => "flow_end"
}
mutate {
remove_field => [ "flow_start" ]
rename => { "type" => "es_type" }
}
}
if [type] == "cloudfront" {
grok {
match => { "message" => "^%{YEAR:date_year}\-%{MONTHNUM:date_month}\-%{MONTHDAY:date_day}\t%{TIME:time}\t%{HOSTNAME:edge_location}\t%{INT:res_byte:int}\t%{IP:ip_src}\t%{WORD:method}\t%{HOSTNAME:host}\t%{URIPATH:uri}\t%{INT:status}\t%{NOTSPACE:referer}\t%{NOTSPACE:user_agent}\t%{NOTSPACE:uri_query}\t%{NOTSPACE:cookie}\t%{WORD:edge_result}\t%{NOTSPACE:edge_req_id}\t%{HOSTNAME:header_host}\t%{WORD:protocol}\t%{INT:req_byte:int}\t%{BASE10NUM:edge_time}\t%{NOTSPACE:ip_xff}\t%{NOTSPACE:ssl_protocol}\t%{NOTSPACE:ssl_chiper}\t%{WORD:edge_result_response}\t%{NOTSPACE:protocol_version}\t%{NOTSPACE:fle_status}\t%{NOTSPACE:fle_field}\t%{INT:port_src}\t%{BASE10NUM:edge_time_first}\t%{NOTSPACE:edge_result_detail}\t%{NOTSPACE:res_content_type}\t%{NOTSPACE:res_content_len:int}\t%{NOTSPACE:res_range_start}\t%{NOTSPACE:res_range_end}$" }
add_field => { "timestamp" => "%{date_year} %{date_month} %{date_day} %{time}" }
}
if "_grokparsefailure" in [tags] {
drop { }
}
date {
match => [ "timestamp", "yyyy MM dd HH:mm:ss" ]
timezone => "UTC"
}
urldecode {
field => "user_agent"
}
urldecode {
field => "user_agent"
}
useragent {
source => "user_agent"
target => "user_agent_parser"
}
geoip {
source => "ip_src"
fields => [ "city_name", "country_code2", "country_name", "latitude", "longitude" ]
}
mutate {
remove_field => [ "date_year","date_month","date_day","time","cloudfront_fields","cloudfront_version","timestamp" ]
rename => { "type" => "es_type" }
}
}
mutate {
remove_field => [ "message" ]
}
}
output {
elasticsearch {
index => "lg-%{es_type}-%{+YYYY.MM.dd}"
user => "svc_***"
password => "***"
}
}