AWS let customer to generate and integrate some services logs. However, each log source has its own configuration making undertstanding quite difficult for new customers.

This post lists all main AWS services logging sources with a summary table, format, example and a Grok regex to parse log and ingest into a tool like Elastic Stack (ELK).

Summary table description

For each Service, the following summry will give you a clear overview of the log, its capabilities and limits.

Log Log source name (with doc link)
Content Nature of event
Format File format, value separator, possible custom format
Delivery Frequency of sending logs
Output Where we can stream log to another service/target
Filter Possibility to filter log content
Scope AWS context of regarding logs
Regional AWS Region linking of the log source
ID Amazon Resource Name (ARN) or ID format of the log resource (if exists)
Sharing Native sharing capabilities with other AWS Accounts
Cost Direct and indirect cost (without price)
Availability Launch date

Services logs

Native logging of most important AWS services.

CloudTrail logs

CloudTrail logs are splited into 3 parts, dependings of the input, output and sometimes event type.

Log CloudTrail Event History
Content Last 90 days of AWS API call details. It's enabled by default and cannot be disabled.
Included events types: Read and Write Management event (control plane). Formerly known as API Activity History.
Format JSON
Delivery 15 min
Output Must be queried (API or Console)
Custom/Filter apply only one filter at a time
Scope Account
ID None
Sharing No
Regional Yes. IAM Service events in us-east-1 only
Cost Free
Availability API Activity History: Mars 2015 (last 7 days on limited services and regions).
Event History: August 2017 (GA, all regions), June 2018 (All Management events and last 90 days)

-

Log CloudTrail trail
Content AWS API call details.
Included events types: Read/Write Management event (control plane), Data events (data plane) of S3 bucket and/or Lambda functions, Insights (unusual activity of write management events volume)
Format JSON
Delivery 15 min
Output at least to a S3 bucket and optionnaly to a CloudWatch Logs log group.
Custom/Filter Event type (Management, Data, Insight), Action type (Read, Write) and region (Current, All). KMS events can be excluded. A Organization trail can be created on the Master Account applied on all Accounts and regions .
Scope Trail per Account or per Organization (current and future Accounts)
Regional Yes. IAM Service events in us-east-1 only
ID ARN: arn:aws:cloudtrail:<region>:<accountID>:trail/<trailName>
Sharing No
Cost The first trail of Management events is free, the following are paying as well as other events types: Data and Insights. Indirect charges: CloudWatch/S3 charges
Availability November 2013 (GA)
November 2016 (S3 Data Events)
November 2019 (Insights)

https://aws.amazon.com/blogs/mt/analyzing-cloudtrail-in-cloudwatch

-

Log EventBridge default bus
Content AWS API call details (recorded by CloudTrail) can be catch in the EventBridge default bus of the Account. Only Write Management event (control plane) are available. Formerly known as 'CloudWatch Events' (same API).
Format JSON
Delivery near-real time (1 sec)
Output EventBridge Rule
Custom/Filter Service or/and action
Scope Account
ID ARN: arn:aws:events:<region>:<accountID>:rule/<ruleName>
Sharing Yes by using another Account bus as Rule target
Regional Yes. IAM Service events in us-east-1 only
Cost Free. Indirect: EventBridge rule target charges
Availability January 2016

Format

All Cloudtrail logs are generated in JSON. Content could be but complex to parse because JSON structure depends of each service and action details (What). However details about the Principal (Who) and Request Context (When and How) have the same structure.

Samples

I wrote a dedicated post. with some Cloudtrail samples.

AWS documentation lists some examples too.

Grok Regex

Not applicable to JSON. You should use json and split filter plugins in Logstash (see example below in Ingest in ELK).

Route 53 Resolver DNS queries

Log Route 53 Resolver Query Logging
Content All DNS queries received by a VPC DNS resolver
Format JSON
Delivery 10-20 sec
Output CloudWatch Logs group, S3 bucket or Kinesis Data Firehose delivery stream
Custom/Filter No
Scope VPC
Regional Yes
ID ARN: arn:aws:route53resolver:<region>:<acountID>:resolver-query-log-config/rqlc-<16digit>
Sharing Yes with another AWS Accounts, OU/whole Organization through Resource Access Manager
Cost Free. Indirect: CloudWatch/S3/Kinesis charges
Availability August 2020

The Resolver Query Logging Configuration can only have one output and cannot be edited after creation.

Format

See below samples, JSON structure details format.

Sample

Real samples with a hosted private zone mhg.vpc and a public zone.

Private zone - Existing name:

{
  "version": "1.000000",
  "account_id": "1111222233334444",
  "region": "eu-west-3",
  "vpc_id": "vpc-5717186e",
  "query_timestamp": "2020-11-01T09:25:02Z",
  "query_name": "asset1.mhg.vpc.",
  "query_type": "A",
  "query_class": "IN",
  "rcode": "NOERROR",
  "answers": [
    {
      "Rdata": "10.5.0.34",
      "Type": "A",
      "Class": "IN"
    }
  ],
  "srcaddr": "172.31.24.160",
  "srcport": "41366",
  "transport": "UDP",
  "srcids": {
    "instance": "i-061fb32d2b51f86a3"
  }
}

Private zone - No-existint name:

{
  "version": "1.000000",
  "account_id": "1111222233334444",
  "region": "eu-west-3",
  "vpc_id": "vpc-5717186e",
  "query_timestamp": "2020-11-01T09:32:05Z",
  "query_name": "test2.mhg.vpc.",
  "query_type": "A",
  "query_class": "IN",
  "rcode": "NXDOMAIN",
  "answers": [],
  "srcaddr": "172.31.24.160",
  "srcport": "37654",
  "transport": "UDP",
  "srcids": {
    "instance": "i-061fb32d2b51f86a3"
  }
}

Public zone - Existing name with multiple records:

{
    "version": "1.000000",
    "account_id": "1111222233334444",
    "region": "eu-west-3",
    "vpc_id": "vpc-5717186e",
    "query_timestamp": "2020-11-01T10:06:16Z",
    "query_name": "mhg.ovh.",
    "query_type": "A",
    "query_class": "IN",
    "rcode": "NOERROR",
    "answers": [
        {
            "Rdata": "13.32.145.91",
            "Type": "A",
            "Class": "IN"
        },
        {
            "Rdata": "13.32.145.81",
            "Type": "A",
            "Class": "IN"
        },
        {
            "Rdata": "13.32.145.126",
            "Type": "A",
            "Class": "IN"
        },
        {
            "Rdata": "13.32.145.44",
            "Type": "A",
            "Class": "IN"
        }
    ],
    "srcaddr": "172.31.24.160",
    "srcport": "58050",
    "transport": "UDP",
    "srcids": {
        "instance": "i-061fb32d2b51f86a3"
    }
}

Grok regex

Not applicable to JSON.

Route 53 Public DNS queries

Log Public DNS queries (public zone)
Content DNS queries of the NS servers (resolvers, not final clients)
Format Line with values separated by space
Delivery 2-3 min
Output CloudWatch Logs group
Custom/Filter No
Scope Route 53 public zone
Regional No. Stored in us-east-1 only
ID UUIDv4 (no ARN)
Sharing Yes with another AWS Accounts, OU/whole Organization through Resource Access Manager
Cost Free. Indirect: CloudWatch Logs charges
Availability September 2017

CloudWatch Logs allows you to export streams to a S3 bucket, however you have to call yourself the CloudWatch API to initiate this asynchronous task. a scheduled Lambda function can be used to do the job.

Following samples come from S3 and includes the CloudWatch Logs timestamp because the Export Task API call task adds it in the S3 object.

Format

CloudWatch-timestamp Log-format-version Query-timestamp Hosted-zone-ID Query-name Query-type Response-code Layer-4-protocol Route-53-edge-location Resolver-IP-address EDNS-client-subnet

**Sample

2020-03-06T00:31:09.000Z 1.0 2020-03-06T00:31:09Z Z3M111CF8036L2 mhg.ovh AAAA NOERROR UDP VIE50 37.187.141.25 -

**Grok regex

%{TIMESTAMP_ISO8601:cwl_timestamp}\s%{NUMBER:log_version}\s%{TIMESTAMP_ISO8601:query_timestamp}\s%{WORD:hosted_zone_id}\s%{HOST:query_name}\s%{WORD:query_type}\s%{WORD:response_code}\s%{WORD:protocol}\s%{WORD:edge_location}\s%{IP:resolver_ip}\s(-|%{NOTSPACE:edns_client_subnet})

Flow Logs

Log Flow Logs
Content IP Flow metadatas (ISO Layer 4) similar to NetFlow or IPFIX.
Format Line with values separated by space
Delivery 1 or 10 minutes (aggregation interval)
Output CloudWatch Logs, S3 Bucket
Custom/Filter Custom format available (fields and order)
Scope VPC, Subnet or network interface (ENI).
Regional Yes
ID fl-<17charcaters> (No ARN)
Sharing No
Cost Free. Indirect: CloudWatch Logs/S3 charges
Availability June 2015 (GA), August 2018 (S3 target)

Format

Custom format with all fields

${version} ${account-id} ${vpc-id} ${subnet-id} ${instance-id} ${interface-id} ${type} ${srcaddr} ${dstaddr} ${pkt-srcaddr} ${pkt-dstaddr} ${srcport} ${dstport} ${protocol} ${tcp-flags} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}

Samples

Version number is 3 because of custom format.

3 013206742943 vpc-603f3c09 subnet-dc5cd691 i-00bc69353a1229ec3 eni-0cb464188f367fbc2 IPv4 52.95.155.48 172.31.40.102 52.95.155.48 172.31.40.102 443 37318 6 19 15577 21652528 1577191414 1577191424 ACCEPT OK
3 013206742943 vpc-603f3c09 subnet-dc5cd691 i-00bc69353a1229ec3 eni-0cb464188f367fbc2 IPv4 216.218.206.85 172.31.40.102 216.218.206.85 172.31.40.102 52862 5900 6 2 1 40 1577191398 1577191404 REJECT OK

Grok regex

^%{INT:fl_version:int}\s%{INT:account_id:int}\s%{HOSTNAME:vpc_id}\s%{HOSTNAME:subnet_id}\s%{HOSTNAME:instance_id}\s%{HOSTNAME:interface_id}\s%{WORD:flow_type}\s%{IP:ip_src}\s%{IP:ip_dst}\s%{IP:ip_src_pkt}\s%{IP:ip_dst_pkt}\s%{INT:port_src:int}\s%{INT:port_dst:int}\s%{INT:proto_num:int}\s%{INT:tcp_flag:int}\s%{INT:flow_packet:int}\s%{INT:flow_byte:int}\s%{INT:flow_start}\s%{INT:flow_end}\s%{WORD:flow_action}\s%{WORD:fl_status}$

CloudFront Access Logs

Log Cloudfront standard logs
Content Web/RTMP access log of the CDNs similar to reverse proxy logs.
Format Line with values separated by space
Delivery Up to several times an hour. Can sometimes be delayed by up to 24 hours
Output S3 Bucket
Custom/Filter No
Scope Distribution
Regional No, stored in us-east-1 only
ID None
Sharing ?
Cost Free. Indirect: S3 charges
Availability May 2009 (GA), December 2019 (new fields)

-

Log Real-time log configuration
Content Web/RTMP access log of the CDN, similar to reverse proxy logs.
Format Line
Delivery Near real-time (1 sec)
Output Kinesis data stream
Custom/Filter Sampling rate (1-100%), fields (among 40) and Cache behavior
Scope Attach configuration to Distribution(s)
Regional No, stored in us-east-1 only
ID ARN: arn:aws:cloudfront::<accountID>:realtime-log-config/<configID>
Sharing To an external Kinesis data stream
Cost Volume of log. Indirect: Kinesis
Availability August 2020

Format

Standard log:

#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields c-port time-to-first-byte x-edge-detailed-result-type sc-content-type sc-content-len sc-range-start sc-range-end

Sample

Standard log:

2019-12-26 09:12:26    FRA50-C1    3198    77.207.15.253   GET djm1t0idtm8px.cloudfront.net    /about.html 200 https://mhg.ovh/    Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010.15;%2520rv:71.0)%2520Gecko/20100101%2520Firefox/71.0  -   -   Miss    BCFMyYSZyL1u6ZixT2CLGNnFBcVg7P2Y-Rr93r68KwxN2d6OoFyVfw==    mhg.ovh https   253 0.078   -   TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss    HTTP/2.0    -   -   50024   0.078   Miss    text/html   -   -   -

Grok regex

Standard log:

^%{YEAR:date_year}\-%{MONTHNUM:date_month}\-%{MONTHDAY:date_day}\t%{TIME:time}\t%{HOSTNAME:edge_location}\t%{INT:res_byte:int}\t%{IP:ip_src}\t%{WORD:method}\t%{HOSTNAME:host}\t%{URIPATH:uri}\t%{INT:status}\t%{NOTSPACE:referer}\t%{NOTSPACE:user_agent}\t%{NOTSPACE:uri_query}\t%{NOTSPACE:cookie}\t%{WORD:edge_result}\t%{NOTSPACE:edge_req_id}\t%{HOSTNAME:header_host}\t%{WORD:protocol}\t%{INT:req_byte:int}\t%{BASE10NUM:edge_time}\t%{NOTSPACE:ip_xff}\t%{NOTSPACE:ssl_protocol}\t%{NOTSPACE:ssl_chiper}\t%{WORD:edge_result_response}\t%{NOTSPACE:protocol_version}\t%{NOTSPACE:fle_status}\t%{NOTSPACE:fle_field}\t%{INT:port_src}\t%{BASE10NUM:edge_time_first}\t%{NOTSPACE:edge_result_detail}\t%{NOTSPACE:res_content_type}\t%{NOTSPACE:res_content_len:int}\t%{NOTSPACE:res_range_start}\t%{NOTSPACE:res_range_end}$

GuardDuty findings

Log GuardDuty finding through EventBridge
Content Security alerts
Format JSON
Delivery 5 min for new finding and 15min/1h/6h for next occurrences
Output EventBridge Account default bus
Custom/Filter Rule pattern
Scope GuardDuty Detector
Regional Yes
ID ARN: arn:aws:events:<region>:<accountID>:rule/<ruleName>
Sharing Possible with Rule target
Cost Free. Indirect: EventBridge
Availability November 2017

-

Log GuardDuty finding S3 Export
Content Security alerts
Format JSON
Delivery 5 min for new finding and 15min/1h/6h for next occurrences
Output S3 bucket
Custom/Filter No
Scope GuardDuty Detector
Regional Yes
ID 32 characters (no ARN)
Sharing Possible to a external and unique S3 bucket for multiple Export configuration
Cost Free. Indirect: S3 and KMS CMK charges
Availability November 2019

Remarks

  • If you use the Multi-Account/Organization feature in a GuardDuty administrator account, all findings related to member Accounts will be exported to the S3 bucket configured in the Administrator account.
  • Charges: a KMS CMK key must be used to export to S3.

Format

n/a

Samples

Policy:IAMUser/RootCredentialUsage finding (An API was invoked using root credentials)

[
  {
    "schemaVersion": "2.0",
    "accountId": "111111111111",
    "region": "us-east-1",
    "partition": "aws",
    "id": "aab8f74669d66b7115893a8a87779XXX",
    "arn": "arn:aws:guardduty:eu-west-3:111111111111:detector/caFC17dEAbDBCe5CA3EeEeEBaEXXXXXXX/finding/aab8f74669d66b7115893a8a8XXXXXXX",
    "type": "Policy:IAMUser/RootCredentialUsage",
    "resource": {
      "resourceType": "AccessKey",
      "accessKeyDetails": {
        "accessKeyId": "ASIA2MSOSYTBPZMUXXXX",
        "principalId": "111111111111",
        "userType": "Root",
        "userName": "Root"
      }
    },
    "service": {
      "serviceName": "guardduty",
      "detectorId": "caFC17dEAbDBCe5CA3EeEeEBaEEcEXXX",
      "action": {
        "actionType": "AWS_API_CALL",
        "awsApiCallAction": {
          "api": "CreateServiceLinkedRole",
          "serviceName": "iam.amazonaws.com",
          "callerType": "Remote IP",
          "remoteIpDetails": {
            "ipAddressV4": "72.21.217.15",
            "organization": {
              "asn": "16509",
              "asnOrg": "AMAZON-02",
              "isp": "Amazon.com",
              "org": "Amazon.com"
            },
            "country": {
              "countryName": "United States"
            },
            "city": {
              "cityName": "Ashburn"
            },
            "geoLocation": {
              "lat": 39.0481,
              "lon": -77.4728
            }
          },
          "affectedResources": {}
        }
      },
      "resourceRole": "TARGET",
      "additionalInfo": {},
      "evidence": null,
      "eventFirstSeen": "2020-05-07T09:34:32Z",
      "eventLastSeen": "2020-05-07T09:35:35Z",
      "archived": false,
      "count": 2
    },
    "severity": 2,
    "createdAt": "2020-05-07T09:52:18.860Z",
    "updatedAt": "2020-05-07T09:52:18.860Z",
    "title": "API CreateServiceLinkedRole was invoked using root credentials.",
    "description": "API CreateServiceLinkedRole was invoked using root credentials from IP address 72.21.217.15."
  }
]

Grok regex

Not applicable to JSON.

Security Hub

Log Security Hub finding trough EventBridge
Content Alerts received from other security tools
Format JSON, AWS Security Finding Format (ASFF)
Delivery Near real-time (1sec) ?
Output EventBridge Account default bus
Custom/Filter Rule pattern and 3 event types: All findings, Findings for custom actions, Insight results for custom actions
Scope Security Hub
Regional Yes
ID EventBridge
Sharing Another Account with EventBridge target
Cost Free. Indirect: EventBridge
Availability November 2018

Format

AWS published a standardized format, AWS Security Finding Format (ASFF), detailled on the documenttation.

Samples

GuardDuty finding:

{
  "SchemaVersion": "2018-10-08",
  "Id": "arn:aws:guardduty:eu-west-3:111122223333:detector/fcb76996ec7afe8c7fbef8e5c08eXXXX/finding/94b9ef45f2f0f4b2dc00006d77b8XXXX",
  "ProductArn": "arn:aws:securityhub:eu-west-3::product/aws/guardduty",
  "GeneratorId": "arn:aws:guardduty:eu-west-3:111122223333:detector/fcb76996ec7afe8c7fbef8e5c08eXXXX",
  "AwsAccountId": "111122223333",
  "Types": [
    "Software and Configuration Checks/Policy:S3.BucketAnonymousAccessGranted"
  ],
  "FirstObservedAt": "2020-08-11T17:09:59Z",
  "LastObservedAt": "2020-08-11T17:09:59Z",
  "CreatedAt": "2020-08-11T17:22:57.121Z",
  "UpdatedAt": "2020-08-11T17:22:57.121Z",
  "Severity": {
    "Product": 8,
    "Label": "HIGH",
    "Normalized": 80
  },
  "Title": "Amazon S3 Public Anonymous Access was granted for S3 bucket bucket1.",
  "Description": "The Amazon S3 bucket bucket1 was granted public anonymous access by Role1 calling PutBucketPolicy. If this behavior is not expected, it may indicate a configuration mistake or that your credentials are compromised.",
  "SourceUrl": "https://eu-west-3.console.aws.amazon.com/guardduty/home?region=eu-west-3#/findings?macros=current&fId=94b9ef45f2f0f4b2dc00006d77b8XXXX",
  "ProductFields": {
    "aws/guardduty/service/archived": "false",
    "aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/organization/asnOrg": "XXXX",
    "aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/organization/org": "XXXX",
    "aws/guardduty/service/additionalInfo": "",
    "aws/guardduty/service/resourceRole": "TARGET",
    "aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/organization/isp": "XXXX",
    "aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/geoLocation/lat": "X.X",
    "aws/guardduty/service/count": "1",
    "aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/ipAddressV4": "1.1.1.1",
    "aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/country/countryName": "XXXX",
    "aws/guardduty/service/action/awsApiCallAction/callerType": "Remote IP",
    "aws/guardduty/service/action/awsApiCallAction/serviceName": "s3.amazonaws.com",
    "aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/city/cityName": "XXXX",
    "aws/guardduty/service/action/awsApiCallAction/api": "PutBucketPolicy",
    "aws/guardduty/service/serviceName": "guardduty",
    "aws/guardduty/service/evidence": "",
    "aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/geoLocation/lon": "X.X",
    "aws/guardduty/service/detectorId": "fcb76996ec7afe8c7fbef8e5c08eXXXX",
    "aws/guardduty/service/action/awsApiCallAction/remoteIpDetails/organization/asn": "XXXX",
    "aws/guardduty/service/eventFirstSeen": "2020-08-11T17:09:59Z",
    "aws/guardduty/service/action/awsApiCallAction/affectedResources/AWS::S3::Bucket": "bucket1",
    "aws/guardduty/service/eventLastSeen": "2020-08-11T17:09:59Z",
    "aws/guardduty/service/action/actionType": "AWS_API_CALL",
    "aws/securityhub/FindingId": "arn:aws:securityhub:eu-west-3::product/aws/guardduty/arn:aws:guardduty:eu-west-3:111122223333:detector/fcb76996ec7afe8c7fbef8e5c08eXXXX/finding/94b9ef45f2f0f4b2dc00006d77b8XXXX",
    "aws/securityhub/ProductName": "GuardDuty",
    "aws/securityhub/CompanyName": "Amazon"
  },
  "Resources": [
    {
      "Type": "AwsIamAccessKey",
      "Id": "AWS::IAM::AccessKey:ASIA4HGNPFE47Q4MXXXX",
      "Partition": "aws",
      "Region": "eu-west-3",
      "Details": {
        "AwsIamAccessKey": {
          "PrincipalId": "AROA4HGNPFE4XI5FLXXXX:user1",
          "PrincipalType": "AssumedRole",
          "PrincipalName": "Role1"
        }
      }
    }
  ],
  "WorkflowState": "NEW",
  "Workflow": {
    "Status": "NEW"
  },
  "RecordState": "ACTIVE"
}

Grok regex

Not applicable to JSON.

Usefull resources

S3 Access logs

Log Server access logging
Content
Format Line with values separated by space
Delivery Few hours or less
Output S3 bucket: the bucket source itself (not recommended) or any one the same Account and Region
Custom/Filter No
Scope S3 bucket
Regional Yes
ID
Sharing No
Cost Free. Indirect: S3
Availability Before 2009 (GA), March 2019 (6 new fields)

-

Log Object-level logging (CloudTrail Data events)
Content
Format JSON
Delivery
Output S3 bucket
Custom/Filter
Scope
Regional
ID
Sharing
Cost
Availability

Remarks

  • AWS recommends that you use AWS CloudTrail data events instead of Amazon S3 access logs. CloudTrail data events are easier to set up and contain more information. Source
  • CloudTrail object logging: CloudTrail supports logging Amazon S3 object level API operations such as GetObject, PutObject, and DeleteObject. (Nov 2016)

Format

Please refer to the AWS documentation

Samples

Object/Data event trough Cloudtrail: Athena Service reads an object on behalf of an IAM User that assumed a Role in the Account's bucket. The example show a bucket with Scott Piper's flaws.cloud CloudTrail logs.

{
  "eventVersion": "1.07",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROA4HGNPFE4XI5FLXXXX:user1",
    "arn": "arn:aws:sts::111122223333:assumed-role/OrganizationAccountAccessRole/user1",
    "accountId": "111122223333",
    "accessKeyId": "ASIAIAH3T5UEST3NXXXX",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "principalId": "AROA4HGNPFE4XI5FLXXXX",
        "arn": "arn:aws:iam::111122223333:role/Role1",
        "accountId": "111122223333",
        "userName": "Role1"
      },
      "attributes": {
        "creationDate": "2020-10-31T16:41:18Z",
        "mfaAuthenticated": "true"
      }
    },
    "invokedBy": "athena.amazonaws.com"
  },
  "eventTime": "2020-10-31T17:05:36Z",
  "eventSource": "s3.amazonaws.com",
  "eventName": "GetObject",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "athena.amazonaws.com",
  "userAgent": "athena.amazonaws.com",
  "requestParameters": {
    "bucketName": "xxxx-flaws-cloud-cloudtrail",
    "Host": "xxxx-flaws-cloud-cloudtrail.s3.amazonaws.com",
    "key": "flaws_cloudtrail09.json.gz"
  },
  "responseElements": null,
  "additionalEventData": {
    "SignatureVersion": "SigV4",
    "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
    "bytesTransferredIn": 0,
    "AuthenticationMethod": "AuthHeader",
    "x-amz-id-2": "YVT7gnHZDu7agzbOV+t8GECNPiJ31IErBUGuTEFkJyxvQ4HYylq5w0SBQOogF9u9Q+iee54wXXXX",
    "bytesTransferredOut": 10977945
  },
  "requestID": "DE670E80ED3590E0",
  "eventID": "b02731a7-41b4-4cd5-aabf-858d447bXXXX",
  "readOnly": true,
  "resources": [
    {
      "type": "AWS::S3::Object",
      "ARN": "arn:aws:s3:::xxxx-flaws-cloud-cloudtrail/flaws_cloudtrail09.json.gz"
    },
    {
      "accountId": "111122223333",
      "type": "AWS::S3::Bucket",
      "ARN": "arn:aws:s3:::xxxx-flaws-cloud-cloudtrail"
    }
  ],
  "eventType": "AwsApiCall",
  "managementEvent": false,
  "recipientAccountId": "111122223333",
  "eventCategory": "Data"
}

Config (rule, snapshot)

TODO

CloudWatch Agent logs

TODO

WAF logs

TODO

Elastic Load Balancing Access Logs

TODO

https://aws.amazon.com/about-aws/whats-new/2014/03/06/elastic-load-balancing-announces-access-logs/

Amazon Redshift Logs

TODO

Amazon Relational Database Service (RDS) Logs

TODO

AWS Detailed Billing Reports

TODO


Ingest in ELK

Logstash template

The following logstash pipeline ingests:

  1. CloudTrail
  2. GuardDuty findings
  3. VPC Flow Logs (custom)
  4. CloudFront Logs
input {
    s3 {
        bucket => "bucket-cloudtrail"
        region => "eu-west-3"
        type => "cloudtrail"
    }
    s3 {
        bucket => "bucket-guardduty"
        region => "eu-west-3"
        type => "guardduty"
    }
    s3 {
        bucket => "bucket-flowlogs"
        region => "eu-west-3"
        type => "flowlogs"
    }
    s3 {
        bucket => "bucket-cloudfront"
        region => "eu-west-3"
        type => "cloudfront"
        sincedb_path => "/var/lib/logstash/plugins/inputs/s3/sincedb_org-mhg-cloudfront"
    }
}

filter {
    if [type] == "cloudtrail" {
        mutate {
            rename => { "type" => "es_type" }
        }
        json  {
            source => "message"
        }

        split {
            field => "Records"
        }

        geoip {
            source => "[Records][sourceIPAddress]"
            target => "geoip"
        }
    }
    if [type] == "guardduty" {
                mutate {
                        rename => { "type" => "es_type" }
                }
        json {
            source => "message"
        }

    }
    if [type] == "flowlogs" {
        grok {
            match => { "message" => "^%{INT:fl_version:int}\s%{INT:account_id:int}\s%{HOSTNAME:vpc_id}\s%{HOSTNAME:subnet_id}\s%{HOSTNAME:instance_id}\s%{HOSTNAME:interface_id}\s%{WORD:flow_type}\s%{IP:ip_src}\s%{IP:ip_dst}\s%{IP:ip_src_pkt}\s%{IP:ip_dst_pkt}\s%{INT:port_src:int}\s%{INT:port_dst:int}\s%{INT:proto_num:int}\s%{INT:tcp_flag:int}\s%{INT:flow_packet:int}\s%{INT:flow_byte:int}\s%{INT:flow_start}\s%{INT:flow_end}\s%{WORD:flow_action}\s%{WORD:fl_status}$" }
        }
        if "_grokparsefailure" in [tags] {
            drop { }
        }
        date {
               match => [ "flow_start", "UNIX" ]
                target => "@timestamp"
        }
        date {
                match => [ "flow_end", "UNIX" ]
                target => "flow_end"
        }
        mutate {
            remove_field => [ "flow_start" ]
            rename => { "type" => "es_type" }
        }
    }
        if [type] == "cloudfront" {
                grok {
             match => { "message" => "^%{YEAR:date_year}\-%{MONTHNUM:date_month}\-%{MONTHDAY:date_day}\t%{TIME:time}\t%{HOSTNAME:edge_location}\t%{INT:res_byte:int}\t%{IP:ip_src}\t%{WORD:method}\t%{HOSTNAME:host}\t%{URIPATH:uri}\t%{INT:status}\t%{NOTSPACE:referer}\t%{NOTSPACE:user_agent}\t%{NOTSPACE:uri_query}\t%{NOTSPACE:cookie}\t%{WORD:edge_result}\t%{NOTSPACE:edge_req_id}\t%{HOSTNAME:header_host}\t%{WORD:protocol}\t%{INT:req_byte:int}\t%{BASE10NUM:edge_time}\t%{NOTSPACE:ip_xff}\t%{NOTSPACE:ssl_protocol}\t%{NOTSPACE:ssl_chiper}\t%{WORD:edge_result_response}\t%{NOTSPACE:protocol_version}\t%{NOTSPACE:fle_status}\t%{NOTSPACE:fle_field}\t%{INT:port_src}\t%{BASE10NUM:edge_time_first}\t%{NOTSPACE:edge_result_detail}\t%{NOTSPACE:res_content_type}\t%{NOTSPACE:res_content_len:int}\t%{NOTSPACE:res_range_start}\t%{NOTSPACE:res_range_end}$" }

            add_field => { "timestamp" => "%{date_year} %{date_month} %{date_day} %{time}" }
                }
                if "_grokparsefailure" in [tags] {
                        drop { }
                }
                date {
            match => [ "timestamp", "yyyy MM dd HH:mm:ss" ]
            timezone => "UTC"
                }
        urldecode {
            field => "user_agent"
        }
        urldecode {
            field => "user_agent"
        }
        useragent {
            source => "user_agent"
            target => "user_agent_parser"
        }
        geoip {
            source => "ip_src"
            fields => [ "city_name", "country_code2", "country_name", "latitude", "longitude" ]
        }
                mutate {
                        remove_field => [ "date_year","date_month","date_day","time","cloudfront_fields","cloudfront_version","timestamp" ]
                        rename => { "type" => "es_type" }
                }
        }

    mutate {
        remove_field => [ "message" ]
    }
}

output {
    elasticsearch {
        index => "lg-%{es_type}-%{+YYYY.MM.dd}"
        user => "svc_***"
        password => "***"
    }
}

Like this post? Share it on: TwitterLinkedIn


Related Posts


Reading Time

~10 min read

Published

Category

AWS

Tags

Stay in Touch