Skip to content

Latest commit

 

History

History

twitter-streams

README.adoc

Kafka Connect Twitter demo

Table of content

A demo setup demonstrates processing of twitter streams. Can be used in booth / conferences on analyze real-time social media data.

Example 1. Acknowledgements

The project structure inspired Confluent Platform demo [1] Actual demo logic is inspired by Robin’s blog [2] Playbook is available!

Appendix: Running demo on AWS

Prerequisites

  • Generate ssh key for github access

    ssh-keygen -t rsa -b 4096 -C "github@email"
cat .ssh/id_rsa.pub
optional install zsh and git
sudo yum install git
sudo yum install zsh  # (1)
sudo yum install jq   # (2)

  1. optional install zsh

  2. tool to work with JSON

    • clone repo

mkdir ~/temp/demo-scene
cd ~/temp/demo-scene
git init .
git remote add origin -f https://github.com/confluentinc/demo-scene/
git config core.sparsecheckout true
echo "twitter-streams/*" >> .git/info/sparse-checkout
git pull --depth=2 origin master
cd twitter-streams
ls -lh

  • intall JDK 8

sudo yum install java-1.8.0-openjdk-devel
sudo /usr/sbin/alternatives --config java
sudo /usr/sbin/alternatives --config javac
sudo yum remove java-1.7.0-openjdk # <1> optional, remove java 7
Note
 How find java in debian - JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")
Optional, install micro because it’s awesome
cd ~
mkdir -p .local/bin
cd .local/bin
curl https://getmic.ro | bash
Update twitter connector config
cd ~
cd twitter-streams/
micro connect_twitter.json

  • install docker and docker-compose

    sudo yum update
sudo yum install -y docker
sudo usermod -aG docker ec2-user
sudo service docker start
sudo su
curl -L "https://github.com/docker/compose/releases/download/1.21.2/docker-compose-$(uname -s)-$(uname -m)" > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
exit
docker-compose --version
1. https://github.com/confluentinc/cp-demo
2. https://www.confluent.io/blog/using-ksql-to-analyse-query-and-transform-data-in-kafka