Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blog: Why Cloud Engineers need Pulumi ESC #13628

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Original file line number Diff line number Diff line change
@@ -0,0 +1,180 @@
---
title: "Why Every Cloud Engineer Needs Pulumi Esc for Secrets Management"
SaraDPH marked this conversation as resolved.
Show resolved Hide resolved
allow_long_title: true

# The date represents the post's publish date, and by default corresponds with
# the date and time this file was generated. Dates are used for display and
# ordering purposes only; they have no effect on whether or when a post is
# published. To influence the ordering of posts published on the same date, use
# the time portion of the date value; posts are sorted in descending order by
# date/time.
date: 2024-01-07T17:28:56Z
SaraDPH marked this conversation as resolved.
Show resolved Hide resolved

# The draft setting determines whether a post is published. Set it to true if
# you want to be able to merge the post without publishing it.
draft: false

# Use the meta_desc property to provide a brief summary (one or two sentences)
# of the content of the post, which is useful for targeting search results or
# social-media previews. This field is required or the build will fail the
# linter test. Max length is 160 characters.
meta_desc: Simplify and secure Kubernetes and multi-cloud secrets with Pulumi ESC. Learn how it integrates with ESO and revolutionizes your cloud operations.

# The meta_image appears in social-media previews and on the blog home page. A
# placeholder image representing the recommended format, dimensions and aspect
# ratio has been provided for you.
meta_image: meta.png

# At least one author is required. The values in this list correspond with the
# `id` properties of the team member files at /data/team/team. Create a file for
# yourself if you don't already have one.
authors:
- sara-huddleston

# At least one tag is required. Lowercase, hyphen-delimited is recommended.
tags:
- security
- esc
- external-secrets-operator
- kubernetes
- secrets-management


SaraDPH marked this conversation as resolved.
Show resolved Hide resolved
# The social copy used to promote this post on Twitter and Linkedin. These
# properties do not actually create the post and have no effect on the
# generated blog page. They are here strictly for reference.

# Here are some examples of posts we have made in the past for inspiration:
# https://www.linkedin.com/feed/update/urn:li:activity:7171191945841561601
# https://www.linkedin.com/feed/update/urn:li:activity:7169021002394296320
# https://www.linkedin.com/feed/update/urn:li:activity:7155606616455737345
# https://twitter.com/PulumiCorp/status/1763265391042654623
# https://twitter.com/PulumiCorp/status/1762900472489185492
# https://twitter.com/PulumiCorp/status/1755637618631405655

social:
twitter:
linkedin:

# See the blogging docs at https://github.com/pulumi/docs/blob/master/BLOGGING.md
# for details, and please remove these comments before submitting for review.
---

Managing secrets is one of the most critical responsibilities in cloud engineering. Secrets like API keys, database credentials, and encryption tokens are the backbone of secure and seamless cloud operations. However, the complexity of modern cloud-native and multi-cloud environments has made traditional secrets management solutions inadequate.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we talk about config too? All cloud engineers need config too, and we seem to forget that in every piece of content.


Enter **Pulumi ESC (Environments, Secrets, and Configuration)**—a breakthrough in taming secrets sprawl and streamlining configuration management across infrastructure. Let's explore why Pulumi ESC is a necessity for cloud engineers, its seamless integration with the External Secrets Operator (ESO), and the practical applications that make it the ultimate solution for modern secrets management.

<!--more-->

## In this article:

- The Challenge of Secrets Management in Modern Cloud Environments
- What is Pulumi ESC
- Key Features of Pulumi ESC
- Why Cloud Engineers Need Pulumi ESC
- The Future of Secrets Management with Pulumi ESC

## The Challenge of Secrets Management in Modern Cloud Environments

Secrets management has changed significantly over the past decade. Gone are the days when secrets could be manually maintained within static locations like configuration files or plain text databases. Nowadays, cloud environments demand a higher level of sophistication:

1. **Distributed Systems**: Even in multi-cloud and hybrid setups, secrets must be accessible across varied platforms without exposing vulnerabilities
2. **Dynamic Infrastructure**: Kubernetes, serverless architectures, and ephemeral environments must have secrets that adapt dynamically.
3. **Security Risks**: Hardcoded or poorly managed secrets can lead to catastrophic breaches, costing companies millions in data recovery and compliance penalties.
4. **Operational Burdens**: Manual secrets management is error-prone, inefficient, unscalable, and needs to align with DevOps and DevSecOps best practices.

Pulumi ESC addresses these issues head-on, redefining how cloud engineers manage secrets efficiently, securely, and at scale.

## What is Pulumi ESC?

Pulumi ESC is a secrets management and orchestration service from Pulumi designed to secure sensitive configurations across modern cloud environments. It supports seamless integration, enabling engineers to:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are not an orchestration service, but we are a "broker".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cleverguy25 "broker" doesn't sound as nice. Should we update the docs page too? At the bottom of https://www.pulumi.com/docs/esc/, in the "Why Pulumi ESC?" it says it is an orchestration service.

SaraDPH marked this conversation as resolved.
Show resolved Hide resolved

- Centrally manage secrets for applications and infrastructure.
- Synchronize secrets dynamically across Kubernetes and other environments with platforms like ESO.
- Implement automated secret rotation, ensuring compliance and minimizing vulnerabilities.
- Access secrets securely through Pulumi's SDKs, CLI, or API.
-
Whether integrated with Pulumi's Infrastructure as Code (IaC) platform or used as a standalone service, Pulumi ESC enables cloud engineers to streamline secrets management with centralized control.

## Key Features of Pulumi ESC

### 1. Seamless Integration with External Platforms

Pulumi ESC integrates with popular secrets providers, including AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, and HashiCorp Vault, making it adaptable for multi-cloud and hybrid cloud architectures.

### 2. Dynamic Secrets Synchronization with ESO

Partnering with the External Secrets Operator, Pulumi ESC synchronizes secrets securely into Kubernetes clusters. This eliminates hardcoding secrets into manifests or relying on unsecured manual processes.

### 3. Automated Rotation and Expiry

Pulumi ESC minimizes security risks by automating the rotation of secrets. This feature aligns secrets management with CI/CD processes for cloud engineers focused on DevOps, ensuring credentials remain valid only when needed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually do not do this, yet. It is on the roadmap

SaraDPH marked this conversation as resolved.
Show resolved Hide resolved

### 4. Secure by Design

Pulumi ESC follows a "secure by default" model by employing encryption, fine-grained access control, and detailed audit trails. Engineers can meet compliance regulations effortlessly while gaining full visibility into secret access patterns.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not call our current access control fine grained, it is something we are working on.

SaraDPH marked this conversation as resolved.
Show resolved Hide resolved

### 5. Language Flexibility

With SDKs available for Python, Go, JavaScript, and other major languages, cloud engineers can integrate Pulumi ESC directly into their CI/CD pipelines or custom applications.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only support python, go, and javascript/typescript at the moment.

SaraDPH marked this conversation as resolved.
Show resolved Hide resolved

## Why Cloud Engineers Need Pulumi ESC
Copy link
Contributor Author

@SaraDPH SaraDPH Dec 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to self: add a config-specific point


### Enhanced Security and Compliance

Cloud engineers often have to balance security and operations. Pulumi ESC centralizes secrets in a secure vault, ensuring encrypted storage, access control, and audit visibility. It also complies with industry standards like SOC 2, PCI-DSS, and GDPR.

### Streamlined Multi-Cloud Management

Managing secrets across AWS, Azure, and GCP can quickly become chaotic. Pulumi ESC's cross-platform integration simplifies this process, ensuring engineers can manage and sync secrets from a single interface.

### Simplified Kubernetes Secrets

Kubernetes' default secrets offer limited security and scalability. Pulumi ESC with ESO overcomes these limitations by securely synchronizing and managing secrets within Kubernetes clusters. This prevents unauthorized access and provides automated updates.

### Zero Downtime Through Automation

Manual secrets management often leads to errors such as expired credentials or outdated tokens. Pulumi ESC automates the entire lifecycle of secrets—creation, rotation, replication, and expiry—guaranteeing uninterrupted services.
Copy link
Contributor Author

@SaraDPH SaraDPH Dec 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to self: It doesn't seem like we can't claim this yet... rotation not available yet, check others.


### Developer-Friendly Workflows

Tools should make engineers' lives easier, not harder. Pulumi ESC's CLI, SDKs, and API provide intuitive ways to integrate into existing workflows. For cloud engineers leveraging Infrastructure as Code with Pulumi, managing secrets alongside the stack becomes effortless.

### Using Pulumi ESC with External Secrets Operator (ESO)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While we should mention ESO, we should also wait and mention the upcoming CSI provider.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When will the upcoming CSI provider be released?


For secrets management in Kubernetes environments, Pulumi ESC becomes even more powerful when paired with ESO. Here's how it works:

#### What is ESO?

External Secrets Operator is an open-source Kubernetes operator that syncs secrets from external providers (like Pulumi ESC) into Kubernetes as native secrets.

#### Benefits of Using Pulumi ESC with ESO:

1. Centralized Storage: Store secrets securely in Pulumi ESC and synchronize them with Kubernetes across multiple clusters.
2. Dynamic Updates: Whenever a secret is updated in ESC, ESO automatically replicates the changes into Kubernetes, eliminating manual intervention.
3. Granular Management: Define which secrets are synchronized and to which namespaces, ensuring tight access controls and minimizing risk.
4. Automated Secret Rotation: By leveraging ESC's rotation capabilities, ESO ensures Kubernetes receives refreshed credentials without requiring downtime.

### Example Use Case

Consider a cloud engineer deploying an application in Kubernetes that requires database credentials. Instead of hardcoding these credentials, the engineer can:

1. Store them securely in Pulumi ESC.
2. Use ESO to sync the secrets into the Kubernetes cluster.
3. Access these secrets securely from within the application.

This approach not only boosts security but also ensures compliance with enterprise-grade standards.

## The Future of Secrets Management with Pulumi ESC

Pulumi ESC is more than just a secrets management tool — it's the foundation of secure, scalable, and agile cloud operations. For cloud engineers grappling with multi-cloud complexity, Kubernetes deployments, and compliance concerns, it provides a streamlined, automated approach that enhances efficiency and security.

With rising cyber threats and stricter data regulations, adopting tools like Pulumi ESC is no longer optional. It's a competitive advantage for cloud engineers and organizations aiming to lead in today's cloud-native world.

## Next Steps to Secure Your Secrets

Secrets management doesn't have to be a headache! With **Pulumi ESC**, you can safeguard your infrastructure, ensure compliance, and eliminate manual errors—all while enhancing security and agility.

[Get started with Pulumi ESC today(https://www.pulumi.com/docs/esc/get-started/begin/) and experience the future of secrets management. Take control of your cloud environment—securely and effortlessly.

[Sign up for Pulumi ESC Free ➡️](https://app.pulumi.com/signup)
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading