Data Engineering Zoomcamp - Week 2

Welcome to DE Zoomcamp with Mage!

Mage is an open-source, hybrid framework for transforming and integrating data. ✨

In this module, you'll learn how to use the Mage platform to author and share magical data pipelines. This will all be covered in the course, but if you'd like to learn a bit more about Mage, check out our docs here.

Get Started Assistance

Let's get started

This repo contains a Docker Compose template for getting started with a new Mage project. It requires Docker to be installed locally. If Docker is not installed, please follow the instructions here.

You can start by cloning the repo:

git clone https://github.com/mage-ai/mage-zoomcamp.git mage-zoomcamp

Navigate to the repo:

cd mage-data-engineering-zoomcamp

Rename dev.env to simply .env— this will ensure the file is not committed to Git by accident, since it will contain credentials in the future.

Now, let's build the container

docker compose build

Finally, start the Docker container:

docker compose up

Now, navigate to http://localhost:6789 in your browser! Voila! You're ready to get started with the course.

What just happened?

We just initialized a new mage repository. It will be present in your project under the name magic-zoomcamp. If you changed the varable PROJECT_NAME in the .env file, it will be named whatever you set it to.

This repository should have the following structure:

.
├── mage_data
│   └── magic-zoomcamp
├── magic-zoomcamp
│   ├── __pycache__
│   ├── charts
│   ├── custom
│   ├── data_exporters
│   ├── data_loaders
│   ├── dbt
│   ├── extensions
│   ├── interactions
│   ├── pipelines
│   ├── scratchpads
│   ├── transformers
│   ├── utils
│   ├── __init__.py
│   ├── io_config.yaml
│   ├── metadata.yaml
│   └── requirements.txt
├── Dockerfile
├── README.md
├── dev.env
├── docker-compose.yml
└── requirements.txt

Assistance

1. Mage Docs: a good place to understand Mage functionality or concepts.
2. Mage Slack: a good place to ask questions or get help from the Mage team.
3. DTC Zoomcamp: a good place to get help from the community on course-specific inquireies.
4. Mage GitHub: a good place to open issues or feature requests.