Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add airflow config lint cli command for lint the configuration changes from Airflow 2.x to Airflow 3.0 #44908

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

sunank200
Copy link
Collaborator

@sunank200 sunank200 commented Dec 13, 2024

This PR introduces the airflow config lint command as part of the Airflow CLI config commands for migration tooling for Airflow 3.0. The purpose of this command is to help users transition from Airflow 2.x to 3.0 by identifying and providing actionable feedback on removed and renamed configuration parameters.

Implementation details:

  • Identify parameters that are no longer valid in Airflow 3.0 based on 44555
  • Highlights renamed parameters and provides guidance for transitioning to their new names and sections in the lint error.
  • Provider users options to specify sections, options, and ignore lists for granular control.

CLI Usage Examples:

  1. Lint all sections and options:

    airflow config lint
    
  2. Lint a specific section:

    airflow config lint --section core
    
  3. Lint a specific section and option:

    airflow config lint --section smtp --option smtp_user
    
  4. Ignore a specific section during linting:

    airflow config lint --ignore-section webserver
    
  5. Ignore a specific option during linting:

    airflow config lint --ignore-option smtp_user
    
  6. Enable verbose output for detailed feedback:

    airflow config lint --verbose
    

closes: #44555
related: #41641


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk
Copy link
Member

potiuk commented Dec 13, 2024

NAAAAJS

Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, looks really nice!
@sunank200 one comment but thats just a thought. Will it also be possible to add a short README or so in order for someone to deprecate any future options? (Although i think all are done)

return lint_message


CONFIGS_CHANGES = [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if we can get these dynamically instead?
We have newsfragments which can be the source. Just a thought to make this more manageable

Copy link
Member

@potiuk potiuk Dec 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think rather the opposite - if anything - those values here shoudl be the "source of truth" - and we could generate release notes for 3.0 from those - we already deleted a number of those deprecations from config I think, so this will be hte only place eventually where those configs will be stored.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, that makes sense. It was a thought I wanted to throw in. But I realised now that we have deprecated all that we should and so this serves well

Copy link
Member

@gopidesupavan gopidesupavan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome :)

@sunank200
Copy link
Collaborator Author

Awesome, looks really nice! @sunank200 one comment but thats just a thought. Will it also be possible to add a short README or so in order for someone to deprecate any future options? (Although i think all are done)

Readme for in how to deprecate options in future? @amoghrajesh can you elaborate on this, please

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Airflow 2 to 3 auto migration rules - airflow config
5 participants