‹ Return to How tos

Using access logs for a bucket in Amazon Lightsail to identify requests

Last updated: November 1, 2021

In this guide, we show you how to identify requests to a bucket using access logs. For more information, see Access logging for buckets in the Amazon Lightsail object storage service.

Contents

Querying access logs for requests using Amazon Athena

You can use Amazon Athena to query and identify requests to a bucket in access logs.

Lightsail stores access logs as objects in a Lightsail bucket. It is often easier to use a tool that can analyze the logs. Athena supports analysis of objects and can be used to query access logs.

Example

The following example shows how you can query bucket server access logs in Amazon Athena.

Note

To specify a bucket location in an Athena query, you need to format the target bucket name and target prefix where your logs are delivered as an S3 URI, as follows: s3://DOC-EXAMPLE-BUCKET1-logs/prefix/

  1. Open the Athena console at https://console.aws.amazon.com/athena/.

  2. In the Query Editor, run a command similar to the following.

    create database bucket_access_logs_db

    Note

    It's a best practice to create the database in the same AWS Region as your S3 bucket.

  3. In the Query Editor, run a command similar to the following to create a table schema in the database that you created in step 2. The STRING and BIGINT data type values are the access log properties. You can query these properties in Athena. For LOCATION, enter the bucket and prefix path as noted earlier.

    CREATE EXTERNAL TABLE `s3_access_logs_db.mybucket_logs`(
     `bucketowner` STRING, 
     `bucket_name` STRING, 
     `requestdatetime` STRING, 
     `remoteip` STRING, 
     `requester` STRING, 
     `requestid` STRING, 
     `operation` STRING, 
     `key` STRING, 
     `request_uri` STRING, 
     `httpstatus` STRING, 
     `errorcode` STRING, 
     `bytessent` BIGINT, 
     `objectsize` BIGINT, 
     `totaltime` STRING, 
     `turnaroundtime` STRING, 
     `referrer` STRING, 
     `useragent` STRING, 
     `versionid` STRING, 
     `hostid` STRING, 
     `sigv` STRING, 
     `ciphersuite` STRING, 
     `authtype` STRING, 
     `endpoint` STRING, 
     `tlsversion` STRING)
    ROW FORMAT SERDE 
     'org.apache.hadoop.hive.serde2.RegexSerDe' 
    WITH SERDEPROPERTIES ( 
     'input.regex'='([^ ]*) ([^ ]*) \\[(.*?)\\] ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) (\"[^\"]*\"|-) (-|[0-9]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) (\"[^\"]*\"|-) ([^ ]*)(?: ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*))?.*$') 
    STORED AS INPUTFORMAT 
     'org.apache.hadoop.mapred.TextInputFormat' 
    OUTPUTFORMAT 
     'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
    LOCATION
     's3://doc-example-bucket1-logs/prefix/'
    
  4. In the navigation pane, under Database, choose your database.

  5. Under Tables, choose Preview table next to your table name.

    In the Results pane, you should see data from the server access logs, such as bucketowner, bucket, requestdatetime, and so on. This means that you successfully created the Athena table. You can now query the bucket server access logs.

Example — Show who deleted an object and when (timestamp, IP address, and IAM user)

SELECT RequestDateTime, RemoteIP, Requester, Key 
FROM s3_access_logs_db.mybucket_logs 
WHERE key = 'images/picture.jpg' AND operation like '%DELETE%';

Example — Show all operations that were performed by an IAM user

SELECT * 
FROM s3_access_logs_db.mybucket_logs 
WHERE requester='arn:aws:iam::123456789123:user/user_name';

Example — Show all operations that were performed on an object in a specific time period

SELECT *
FROM s3_access_logs_db.mybucket_logs
WHERE Key='prefix/images/picture.jpg' 
 AND parse_datetime(RequestDateTime,'dd/MMM/yyyy:HH:mm:ss Z')
 BETWEEN parse_datetime('2017-02-18:07:00:00','yyyy-MM-dd:HH:mm:ss')
 AND parse_datetime('2017-02-18:08:00:00','yyyy-MM-dd:HH:mm:ss');

Example — Show how much data was transferred by a specific IP address in a specific time period

SELECT SUM(bytessent) AS uploadTotal,
 SUM(objectsize) AS downloadTotal,
 SUM(bytessent + objectsize) AS Total
FROM s3_access_logs_db.mybucket_logs
WHERE RemoteIP='1.2.3.4'
AND parse_datetime(RequestDateTime,'dd/MMM/yyyy:HH:mm:ss Z')
BETWEEN parse_datetime('2017-06-01','yyyy-MM-dd')
AND parse_datetime('2017-07-01','yyyy-MM-dd');

Identifying object access requests using Amazon S3 access logs

You can use queries on access logs to identify object access requests, for operations such as GET, PUT, and DELETE, and discover further information about those requests.

The following Amazon Athena query example shows how to get all PUT object requests for a bucket from the server access log.

Example — Show all requesters that are sending PUT object requests in a certain period

SELECT Bucket, Requester, RemoteIP, Key, HTTPStatus, ErrorCode, RequestDateTime
FROM s3_access_logs_db
WHERE Operation='REST.PUT.OBJECT' AND
parse_datetime(RequestDateTime,'dd/MMM/yyyy:HH:mm:ss Z') 
BETWEEN parse_datetime('2019-07-01:00:42:42','yyyy-MM-dd:HH:mm:ss')
AND 
parse_datetime('2019-07-02:00:42:42','yyyy-MM-dd:HH:mm:ss')

The following Amazon Athena query example shows how to get all GET object requests for Amazon S3 from the server access log.

Example — Show all requesters that are sending GET object requests in a certain period

SELECT Bucket, Requester, RemoteIP, Key, HTTPStatus, ErrorCode, RequestDateTime
FROM s3_access_logs_db
WHERE Operation='REST.GET.OBJECT' AND
parse_datetime(RequestDateTime,'dd/MMM/yyyy:HH:mm:ss Z') 
BETWEEN parse_datetime('2019-07-01:00:42:42','yyyy-MM-dd:HH:mm:ss')
AND 
parse_datetime('2019-07-02:00:42:42','yyyy-MM-dd:HH:mm:ss')

The following Amazon Athena query example shows how to get all anonymous requests to your S3 buckets from the server access log.

Example — Show all anonymous requesters that are making requests to a bucket in a certain period

SELECT Bucket, Requester, RemoteIP, Key, HTTPStatus, ErrorCode, RequestDateTime
FROM s3_access_logs_db.mybucket_logs
WHERE Requester IS NULL AND
parse_datetime(RequestDateTime,'dd/MMM/yyyy:HH:mm:ss Z') 
BETWEEN parse_datetime('2019-07-01:00:42:42','yyyy-MM-dd:HH:mm:ss')
AND 
parse_datetime('2019-07-02:00:42:42','yyyy-MM-dd:HH:mm:ss')

Note

  • You can modify the date range to suit your needs.

  • These query examples might also be useful for security monitoring. You can review the results for PutObject or GetObject calls from unexpected or unauthorized IP addresses/requesters and for identifying any anonymous requests to your buckets.

  • This query only retrieves information from the time at which logging was enabled.

Managing buckets and objects in Lightsail

These are the general steps to manage your Lightsail object storage bucket:

  1. Learn about objects and buckets in the Amazon Lightsail object storage service. For more information, see Object storage in Amazon Lightsail.

  2. Learn about the names that you can give your buckets in Amazon Lightsail. For more information, see Bucket naming rules in Amazon Lightsail.

  3. Get started with the Lightsail object storage service by creating a bucket. For more information, see Creating buckets in Amazon Lightsail.

  4. Learn about security best practices for buckets and the access permissions that you can configure for your bucket. You can make all objects in your bucket public or private, or you can choose to make individual objects public. You can also grant access to your bucket by creating access keys, attaching instances to your bucket, and granting access to other AWS accounts. For more information, see Security Best Practices for Amazon Lightsail object storage and Understanding bucket permissions in Amazon Lightsail.

    After learning about bucket access permissions, see the following guides to grant access to your bucket:

  5. Learn how to enable access logging for your bucket, and how to use access logs to audit the security of your bucket. For more information, see the following guides.

  6. Create an IAM policy that grants a user the ability to manage a bucket in Lightsail. For more information, see IAM policy to manage buckets in Amazon Lightsail.

  7. Learn about the way that objects in your bucket are labeled and identified. For more information, see Understanding object key names in Amazon Lightsail.

  8. Learn how to upload files and manage objects in your buckets. For more information, see the following guides.

  9. Enable object versioning to preserve, retrieve, and restore every version of every object stored in your bucket. For more information, see Enabling and suspending object versioning in a bucket in Amazon Lightsail.

  10. After enabling object versioning, you can restore previous versions of objects in your bucket. For more information, see Restoring previous versions of objects in a bucket in Amazon Lightsail.

  11. Monitor the utilization of your bucket. For more information, see Viewing metrics for your bucket in Amazon Lightsail.

  12. Configure an alarm for bucket metrics to be notified when the utilization of your bucket crosses a threshold. For more information, see Creating bucket metric alarms in Amazon Lightsail.

  13. Change the storage plan of your bucket if it's running low on storage and network transfer. For more information, see Changing the plan of your bucket in Amazon Lightsail.

  14. Learn how to connect your bucket to other resources. For more information, see the following tutorials.

  15. Delete your bucket if you're no longer using it. For more information, see Deleting buckets in Amazon Lightsail.