Skip to content

Latest commit

 

History

History
158 lines (116 loc) · 4.13 KB

github-to-slack.md

File metadata and controls

158 lines (116 loc) · 4.13 KB
Raw

Github to Slack

This project creates an end-to-end event pipeline that detects changes in github stars & forks and publishes the result to Slack using connectors:

  • http-source: to read periodically from a github, parse the fields from the json output, and publishes the result to a topic.

  • http-sink: to listen to the same topic, detect changes, and publish the result to Slack.

Objective

Show an example on how to create a data streaming pipeline that reads from an HTTP API, transforms the output, and invokes a webhook to generate an alert.

Prerequsites

Step-by-Step

  1. Create http-source configuration file
  2. Create http-sink configuration file
  3. Download smartmodules
  4. Start Connectors
  5. Test Data Pipeline

Create http-source configuration file

Create an HTTP source connector configuration file called github.yaml :

apiVersion: 0.1.0
meta:
  version: 0.3.6
  name: github-stars-in
  type: http-source
  topic: stars-forks
  secrets:
    - name: GITHUB_TOKEN
http:
  endpoint: 'https://api.github.com/repos/infinyon/fluvio'
  method: GET
  headers: 
    - 'Authorization: token ${{ secrets.GITHUB_TOKEN }}'
  interval: 30s
transforms:
  - uses: infinyon/[email protected]
    with:
      spec:
        - operation: shift
          spec:
            "stargazers_count": "stars"
            "forks_count": "forks"

Github rate-limits API requests to 60 per hour, which you an extend to 5000 by creating an application token. Check out github documentation on how to create an Access Tokens.

Add the access token secret in InfinyOn Cloud :

fluvio cloud secret set GITHUB_TOKEN <access-token>

Create http-sink configuration file

Create an HTTP source connector configuration file called slack.yaml :

apiVersion: 0.1.0
meta:
  version: 0.2.8
  name: slack-stars-out
  type: http-sink
  topic: stars-forks
  secrets:
    - name: SLACK_TOKEN
http:
  endpoint: "https://hooks.slack.com/services/${{ secrets.SLACK_TOKEN }}"
  headers:
    - "Content-Type: application/json"
transforms:
  - uses: infinyon-labs/[email protected]
    lookback:
      last: 1
  - uses: infinyon/[email protected]
    with:
      spec:
        - operation: shift
          spec:
            "result": "text"

Check out Slack Webhooks on how to create a channel webhook token.

Add the access token secret in InfinyOn Cloud :

fluvio cloud secret set SLACK_TOKEN <webhook-token>

Download Smartmodules

Download the smartmodules used by the connectors to your cluster:

fluvio hub sm download infinyon/[email protected]
fluvio hub sm download infinyon-labs/[email protected]

Check fluvio smartmodule list to ensure they've been downloaded.

Start Connectors

Start source & sink connectors:

fluvio cloud connector create -config github.yaml
fluvio cloud connector create -config slack.yaml

Check fluvio cloud connector log to ensure they have been successfully provisioned.

Test Data Pipeline

Check the last values generated by the github connector:

$ fluvio consume -dT 1 stars-forks
{"stars":1770,"forks":138}

Produce a new value

$ fluvio produce stars-forks
> {"stars":1769,"forks":138}
OK

An alert with :star2: 1769 will show-up in your slack channel.

References