Skip to main content

How to collect from Amazon S3

  • Updated

Getting Started

Amazon S3 is a cloud storage service provided by Amazon Web Services. Companies can store data in S3 from multiple services. Onna connects directly with S3's API and can connect to a specific bucket to collect all files found as well as relevant metadata.

 

Onna's S3 integration is meant to work as a bridge to applications that Onna does not have a direct integration through an S3 bucket. 

 

For a short video overview of collecting from Amazon S3, please see below:

 

Integration Features

Type of Account Needed 

 

To create a bucket or access information on existing buckets on Amazon S3, you must have access to the Amazon Management Console. Learn more about creating a bucket by visiting Amazon's bucket documentation.

 

To integrate Amazon S3 in Onna, you will need:

 

  • Bucket name

  • Access Key ID 

  • Secret Access Key 

  • AWS region

 

For security purposes, we recommend working with your AWS admin and creating a role with access to specific buckets. For more information on how to do that, visit Amazon's IAM Role documentation. 

 

Note: The below permissions are required for each Amazon S3 bucket.

  • s3:ListAllMyBuckets

  • s3:ListBucket

  • s3:GetBucketLocation

  • s3:GetObject

Below are steps to grant the above permissions to the Amazon S3 bucket:

  1. Sign into the AWS Management Console and open the Amazon IAM console at https://console.aws.amazon.com/iam/.

  2. Add a new user with Programmatic Access.

3. Click on the ‘Create policy’ button that is seen below.

 

 

4. Select Service S3

 

5. Grant the below access for List and Read

  • ListAllMyBuckets

  • ListBucket

  • GetBucketLocation

  • GetObject

6. Under resources select ‘Specific’ and click ‘Add ARN’ for bucket. Here you can manually provide the list of ARNs for each bucket.

 

7. After the ARNs are provided for the required buckets click ‘Add ARN’ found under object. Here you can manually provide the list of ARNs for each bucket. For the field object provide an asterisk (*) within the field to grant access to all objects in the bucket.

8. Provide a name and description for the policy, then finally click create policy.

9. Below is an example of the policy as a JSON:

 

10. Navigate back to the user tab from the Amazon IAM console and click on the user name that should have access to the policy. Attach the policy to the appropriate user by clicking on ‘Add Permissions’. Once the policy has been successfully attached it will be listed under ‘Permissions policies’ for the user.

 

Files Collected

Onna's S3 integration can collect all objects from a specific bucket. You can use an existing bucket from within the organization's S3 or create a new one specifically for Onna. 

 

All files available through the S3 API and linked to the specific bucket are synced, including data stored in the bucket from other sources, historical information, and related metadata, including:

 

  • File Title

  • File creation 

  • File last modified

  • Extension

  • Size

  • MD5 hash

  • Creators

  • File URL in source

  • S3 Bucket name

Onna sync modes

We currently support two syncing methods - One-Time Sync and Auto-Sync. 

 

  • One-time sync: sync that collects files in a source during a specific time range or up until the date the source was added. 

  • Auto-sync: means that Onna will perform a full sync first and will keep the data source and Onna in mirrored sync. Any deletions from the data source will be deleted in Onna, as well.

Data Exports

You can export data and metadata in eDiscovery ready format. Exported files are available in the following formats:

 

  • dat 

  • CSV

  • Custom text file 

How to add an Amazon S3 collection

From My Sources click on Add new source and select Amazon S3

 


Next, you'll see the information that you need to add to connect an S3 source.

 

  • Source name:  the name that this source will be called in Onna. You can change this at anytime int he future.

  • Access ID: The AWS admin would provide this for the specific role

  • Access secret: The AWS admin would provide this for the particular role

  • Bucket name: You can add multiple buckets (one per line). The names must match the ones found in S3 

  • Region: This region must match the region the S3 bucket is present in

  • Synchronization mode: This will define how Onna interacts with the bucket. Choose either One-Time Sync or Auto-Sync

Once you've filled out this information, the Sync button will become available. Once you click Sync, Onna will begin to collect all of the files stored in the S3 bucket(s) specified. 

 

You will now see the Amazon S3 source appear in your My Sources section and will begin syncing.

 

Once the source starts the syncing process you'll be able to view them when you click on the source. You will start seeing results being populated.

 

From this screen, you can filter results by date range, categories, and extensions using the menu on the left. 

 

Accessing audit logs

Clicking on the information icon on the top right will take you to the source details where you can see how many files the source, its size, and other various information. In this same panel, you can scroll down and select the Audits button to see the source's collection audit logs.

 


You can learn more about accessing and navigating the source's collection audit logs in this article.

 

Ready to start collecting from Amazon S3? 

 

Contact Onna

 

 

Related articles