AWS S3 Bucket Monitor

Purpose

Use this engine to monitor the contents of an S3 bucket for new files. The bucket must be hosted on Amazon Web Services (AWS), as this engine relies extensively on its Simple Queue Service for handling the bucket events.

To make successful use of these engine, you will need IAM credentials for a user that has the following minimal policy attached:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:PutBucketNotification",
                "s3:ListBucket",
                "s3:GetBucketNotification",
                "sqs:DeleteMessage",
                "sqs:DeleteMessageBatch",
                "sqs:ReceiveMessage",
                "sqs:DeleteQueue",
                "sqs:GetQueueAttributes",
                "sqs:CreateQueue",
                "sqs:SetQueueAttributes"
            ],
            "Resource": [
                "arn:aws:sqs:*:*:rfi-file-monitor-s3-bucket-engine-*",
                "arn:aws:s3:::<i>your-bucket-name</i>/*",
                "arn:aws:s3:::<i>your-bucket-name</i>"
            ]
        }
    ]
}

When copy-pasting this policy into the AWS editor, do not forget to replace your-bucket-name, with the name of the bucket you plan on monitoring!

This engine marks all files as Saved when passing them to the Queue manager.

Options

• Bucket Name: the name of the bucket that will be monitored for changes.
• Process existing files in bucket: turn this option on to add existing objects to the queue manager before starting the bucket monitor.
• Access Key: the access key belonging to the IAM user that will be used by the engine.
• Secret Key: the secret key that is associated with the access key.

The following options are availabled when clicking the Advanced Settings button:

• Allowed filename patterns: enter a file extension e.g. *.txt, *.csv (always include the asterisk) to only process files of that type, any other file written to the directory will be ignored.
• Ignored filename patterns: enter a file extension e.g. *.txt, *.csv (always include the asterisk) to exclude files of these types, any other file written to the directory will be processed.

Exported File Format

S3Object

Author

Tom Schoonjans