Logstash Plugin for Amazon DynamoDB

The Logstash plugin for Amazon DynamoDB gives you a nearly real-time view of the data in your DynamoDB table. The Logstash plugin for DynamoDB uses DynamoDB Streams to parse and output data as it is added to a DynamoDB table. After you install and activate the Logstash plugin for DynamoDB, it scans the data in the specified table, and then it starts consuming your updates using Streams and then outputs them to Elasticsearch, or a Logstash output of your choice.

Logstash is a data pipeline service that processes data, parses data, and then outputs it to a selected location in a selected format. Elasticsearch is a distributed, full-text search server. For more information about Logstash and Elasticsearch, go to https://www.elastic.co/products/elasticsearch.

The following sections walk you through the process to:

1. Create a DynamoDB table and enable a new stream on the table.
2. Download, build, and install the Logstash plugin for DynamoDB.
3. Configure Logstash to output to Elasticsearch and the command line.
4. Run the Logstash plugin for DynamoDB.
5. Test Logstash by adding DynamoDB items to the table.

When this process is finished, you can search your data in the Elasticsearch cluster.

Prerequisites

The following items are required to use the Logstash plugin for Amazon DynamoDB:

1. Amazon Web Services (AWS) account with DynamoDB
2. A running Elasticsearch cluster—To download Elasticsearch, go to https://www.elastic.co/products/elasticsearch.
3. Logstash—To download Logstash, go to https://github.com/awslabs/logstash-input-dynamodb.
4. JRuby—To download JRuby, go to http://jruby.org/download.
5. Git—To download Git, go to http://git-scm.com/downloads
6. Apache Maven—To get Apache Maven, go to http://maven.apache.org/.

Before You Begin: Create a Source Table

In this step, you will create a DynamoDB table with DynamoDB Streams enabled. This will be the source table and writes to this table will be processed by the Logstash plugin for DynamoDB.

To create the source table

1. Open the DynamoDB console at https://console.aws.amazon.com/dynamodb/.
2. Choose Create Table.
3. On the Create Table page, enter the following settings:
   1. Table Name — SourceTable
   2. Primary Key Type — Hash
   3. Hash attribute data type — Number
   4. Hash Attribute Name — Id
   5. Choose Continue.
4. On the Add Indexes page, choose Continue. You will not need any indexes for this exercise.
5. On the Provisioned Throughput page, choose Continue.
6. On the Additional Options page, do the following:
   1. Select Enable Streams, and then set the View Type to New and Old Images.
   2. Clear Use Basic Alarms. You will not need alarms for this exercise.
   3. When you are ready, choose Continue.
7. On the Summary page, choose Create.

The source table will be created within a few minutes.

Setting Up the Logstash Plugin for Amazon DynamoDB

To use the Logstash plugin for DynamoDB, you need to build, install, run the plugin, and then you can test it.

IMPORTANT: in order to successfully build and install Logstash, you must have previously installed MAVEN to satisfy jar dependencies, and JRUBY to build and run the logstash gem.

To build the Logstash plugin for DynamoDB

At the command prompt, change to the directory where you want to install the Logstash plugin for DynamoDB and demo project.

In the directory where you want the Git project, clone the Git project:

git clone https://github.com/awslabs/logstash-input-dynamodb.git

Install the Bundler gem by typing the following:

jruby -S gem install bundler

NOTE: The jruby -S syntax ensures that our gem is installed with jruby and not ruby

The Bundler gem checks dependencies for Ruby gems and installs them for you.

To install the dependencies for the Logstash plugin for DynamoDB, type the following command:

jruby -S bundle install

To build the gem, type the following command:

jruby -S gem build logstash-input-dynamodb.gemspec

To install the gem, in the logstash-dynamodb-input folder type:

jruby -S gem install --local logstash-input-dynamodb-1.0.0-java.gem

To install the Logstash plugin for DynamoDB

Now that you have built the plugin gem, you can install it.

Change directories to your local Logstash directory.

In the Logstash directory, open the Gemfile file in a text editor and add the following line.

gem "logstash-input-dynamodb"

To install the plugin, in your logstash folder type the command:

bin/plugin install --no-verify logstash-input-dynamodb

To list all the installed plugins type the following command:

bin/plugin list

If the logstash-output-elasticsearch or logstash-output-stdout plugins are not listed you need to install them. For instructions on installing plugins, go to the Working with Plugins page in the Logstash documentation.

Running the Logstash Plugin for Amazon DynamoDB

NOTE: First, make sure you have Enabled Streams (see above) for your DynamoDB table(s) before running logstash. Logstash for DynamoDB requires that each table you are logging from have a streams enabled to work.

In the local Logstash directory create a logstash-dynamodb.conf file with the following contents:

input { 
    dynamodb{
      endpoint => "dynamodb.us-east-1.amazonaws.com" 
      streams_endpoint => "streams.dynamodb.us-east-1.amazonaws.com" 
      view_type => "new_and_old_images" 
      aws_access_key_id => "<access_key_id>" 
      aws_secret_access_key => "<secret_key>" 
      table_name => "SourceTable"
  }
} 
output { 
    elasticsearch {
      host => localhost 
    } 
    stdout { } 
}

Important

This is an example configuration. You must replace <access_key_id> and <secret_key> with your access key and secret key. If you have credentials saved in a credentials file, you can omit these configuration values.

To run logstash type:

bin/logstash -f logstash-dynamodb.conf

Logstash should successfully start and begin indexing the records from your DynamoDB table.

You can also change the other configuration options to match your particular use case.

You can also configure the plugin to index multiple tables by adding additional dynamodb { } sections to the input section.

The following table shows the configuration values.

Setting Description

Settings Id	Description
table_name	The name of the table to index. This table must exist.
endpoint	The DynamoDB endpoint to use. If you are running DynamoDB on your computer, use http://localhost:8000 as the endpoint.
streams_endpoint	The name of a checkpoint table. This does not need to exist prior to plugin activation.
view_type	The view type of the DynamoDB stream. ("new_and_old_images", "new_image", "old_image", "keys_only" Note: these must match the settings for your table's stream configured in the DynamoDB console.)
aws_access_key_id	Your AWS access key ID. This is optional if you have credentials saved in a credentials file. Note: If you are running DynamoDB on your computer, this ID must match the access key ID that you used to create the table. If it does not match, the Logstash plugin will fail because DynamoDB partitions data by access key ID and region.
aws_secret_access_key	Your AWS access key ID. Your AWS access key ID. This is optional if you have credentials saved in a credentials file.
perform_scan	A boolean flag to indicate whether or not Logstash should scan the entire table before streaming new records. Note: Set this option to false if your are restarting the Logstash plugin.
checkpointer	A string that uniquely identifies the KCL checkpointer name and CloudWatch metrics name. This is used when one worker leaves a shard so that another worker knows where to start again.
publish_metrics	Boolean option to publish metrics to CloudWatch using the checkpointer name.
perform_stream	Boolean option to not automatically stream new data into Logstash from DynamoDB streams.
read_ops	Number of read operations per second to perform when scanning the specified table.
number_of_scan_threads	Number of threads to use when scanning the specified table.
number_of_write_threads	Number of threads to write to the Logstash queue when scanning the table.
log_format	Log transfer format. "plain" - Returns the object as a DynamoDB object. "json_drop_binary" - Translates the item format to JSON and drops any binary attributes. "json_binary_as_text" - Translates the item format to JSON and represents any binary attributes as 64-bit encoded binary strings. For more information, see the JSON Data Format topic in the DynamoDB documentation.

Testing the Logstash Plugin for Amazon DynamoDB

The Logstash plugin for DynamoDB starts scanning the DynamoDB table and indexing the table data when you run it. As you insert new records into the DynamoDB table, the Logstash plugin consumes the new records from DynamoDB streams to continue indexing.

To test this, you can add items to the DynamoDB table in the AWS console, and view the output (stdout) in the command prompt window. The items are also inserted into Elasticsearch and indexed for searching.

To test the Logstash plugin for DynamoDB

Open the DynamoDB console at https://console.aws.amazon.com/dynamodb/.

In the list of tables, open (double-click) SourceTable.

Choose New Item, add the following data, and then choose PutItem:

Id—1 Message—First item

Repeat the previous step to add the following data items:

Id—2 and Message—Second item Id—3 and Message—Third item

Return to the command-prompt window and verify the Logstash output (it should have dumped the logstash output for each item you added to the console).

(Optional) Go back to the SourceTable in us-east-1 and do the following:

Update item 2. Set the Message to Hello world! Delete item 3.

Go to the command-prompt window and verify the data output.

You can now search the DynamoDB items in Elasticsearch.

For information about accessing and searching data in Elasticsearch, see the Elasticsearch documentation.