Resolving Authorization Issues in Airflow with Azure Entra ID Integration

Resolving OAuth Authorization Errors in Airflow with Multiple Scripting Approaches

First Solution - Python Backend Script for OAuth Authorization

# Import required modules and configure OAuth settings
import os
from flask import Flask, redirect, url_for, session
from flask_oauthlib.client import OAuth
# Define environment variables
tenant_id = os.getenv("AAD_TENANT_ID")
client_id = os.getenv("AAD_CLIENT_ID")
client_secret = os.getenv("AAD_CLIENT_SECRET")
app = Flask(__name__)
app.secret_key = 'supersecretkey'
oauth = OAuth(app)
# Define OAuth configuration with Flask-OAuthlib
azure = oauth.remote_app('azure',
    consumer_key=client_id,
    consumer_secret=client_secret,
    request_token_params={'scope': 'openid email profile'},
    base_url=f"https://login.microsoftonline.com/{tenant_id}",
    access_token_url=f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token",
    authorize_url=f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/authorize"
)
@app.route('/login')
def login():
    return azure.authorize(callback=url_for('authorized', _external=True))
# OAuth authorization callback route
@app.route('/oauth-authorized/azure')
def authorized():
    response = azure.authorized_response()
    if response is None or response.get('access_token') is None:
        return 'Access Denied'
    # Handle successful authorization response
    session['azure_token'] = (response['access_token'], '')
    return redirect(url_for('home'))
@azure.tokengetter
def get_azure_oauth_token():
    return session.get('azure_token')
# Run the Flask app
if __name__ == '__main__':
    app.run()

Alternative Backend Approach - Airflow Configuration Using JWKS and OpenID for Secure Token Validation

Another backend solution with a focus on OpenID Connect and JSON Web Key Set configuration in Airflow

import os
from airflow.www.fab_security.manager import AUTH_OAUTH
# Required Airflow and custom modules for handling Azure OAuth
from airflow.auth.managers.fab.security_manager.override import FabAirflowSecurityManagerOverride
from airflow.utils.log.logging_mixin import LoggingMixin
class AzureAuthConfig:
    AAD_TENANT_ID = os.getenv('AAD_TENANT_ID')
    AAD_CLIENT_ID = os.getenv('AAD_CLIENT_ID')
    AAD_CLIENT_SECRET = os.getenv('AAD_CLIENT_SECRET')
AUTH_TYPE = AUTH_OAUTH
OAUTH_PROVIDERS = [{
    'name': 'azure',
    'remote_app': {
        'client_id': AzureAuthConfig.AAD_CLIENT_ID,
        'client_secret': AzureAuthConfig.AAD_CLIENT_SECRET,
        'authorize_url': f"https://login.microsoftonline.com/{AzureAuthConfig.AAD_TENANT_ID}/oauth2/v2.0/authorize",
        'access_token_url': f"https://login.microsoftonline.com/{AzureAuthConfig.AAD_TENANT_ID}/oauth2/v2.0/token",
        'jwks_uri': 'https://login.microsoftonline.com/common/discovery/v2.0/keys',
        'redirect_uri': 'https://airflow.xyz.com/oauth-authorized/azure'
    }},
# Ensure authentication maps to the correct role group in Azure
AUTH_ROLES_MAPPING = {
    "airflow_nonprod_admin": ["Admin"],
    "airflow_nonprod_op": ["Op"],
    "airflow_nonprod_viewer": ["Viewer"],
}

Frontend Script - JavaScript for OAuth Authorization Handling

A JavaScript approach for handling OAuth redirects and errors on the frontend

// JavaScript function to handle authorization redirect
const authorizeUser = () => {
  const oauthUrl = 'https://login.microsoftonline.com/your-tenant-id/oauth2/v2.0/authorize';
  const params = {
    client_id: 'your-client-id',
    redirect_uri: 'https://airflow.xyz.com/oauth-authorized/azure',
    response_type: 'token',
    scope: 'openid email profile'
  };
  const queryString = new URLSearchParams(params).toString();
  window.location.href = \`\${oauthUrl}?\${queryString}\`;
};
// Handle OAuth errors in the frontend
const handleOAuthError = (error) => {
  if (error === 'access_denied') {
    alert('Access Denied. Please contact your admin.');
  } else {
    alert('An unexpected error occurred.');
  }
};
// Bind function to login button
document.getElementById('login-btn').addEventListener('click', authorizeUser);

Essential Questions on Integrating Azure Entra ID with Airflow

1. What is the purpose of the AUTH_ROLES_MAPPING parameter in Airflow?
2. The AUTH_ROLES_MAPPING parameter connects Azure roles to Airflow roles, enabling automated role assignments based on group memberships in Azure. This simplifies access control by assigning appropriate permissions to users logging in via Azure Entra ID.
3. How does the jwks_uri work in the OAuth setup?
4. The jwks_uri defines the URI where Azure’s public keys can be retrieved for JWT token verification. This step is crucial for validating tokens’ authenticity, preventing unauthorized access.
5. Why is setting the redirect_uri in OAuth providers important?
6. The redirect_uri tells Azure where to send users after successful authentication. This is often set to the Airflow endpoint handling OAuth responses, allowing smooth integration between Azure and Airflow.
7. Can multiple roles be assigned to a single Azure Entra ID group?
8. Yes, multiple roles can be mapped to a single Azure group, allowing flexibility in assigning permissions. For instance, both "Admin" and "Viewer" roles can be associated with a group for overlapping permissions.
9. What is the best way to troubleshoot “Invalid JSON Web Key Set” errors?
10. Ensure the jwks_uri is correctly configured and accessible. Errors often occur if the endpoint is unreachable or if Azure Entra ID keys are incorrectly cached in Airflow.
11. How does the client_kwargs scope enhance security?
12. The client_kwargs scope limits the data Airflow can access from a user profile, enforcing restricted access to sensitive information, which is key for compliance in corporate settings.
13. Does enabling WTF_CSRF_ENABLED improve security?
14. Yes, WTF_CSRF_ENABLED provides Cross-Site Request Forgery protection for Airflow, preventing unauthorized requests. This flag is highly recommended in production environments for added security.
15. How can I handle a denied sign-in request?
16. Review user roles in Azure to confirm they are correctly assigned. Additionally, verify authorize_url and group mapping are correct, as these settings impact authentication success.
17. Can I use a different OAuth provider than Azure?
18. Yes, Airflow supports other OAuth providers like Google or Okta by adjusting the provider-specific parameters in OAUTH_PROVIDERS. Each provider may have unique URLs and configuration requirements.

Final Thoughts on Securing Airflow with Azure Entra ID

Integrating Azure Entra ID with Airflow can streamline authentication across organizations. By carefully configuring OAuth parameters like the jwks_uri and access token URLs, you’re establishing secure connections that minimize the risk of unauthorized access. This level of security is essential for any data-driven organization.

Role mappings in Azure allow for a scalable, role-based access strategy in Airflow. With these mappings, managing users and assigning permissions becomes more efficient, especially in larger teams. A clear understanding of these configurations can make your authorization setup more resilient to future security needs. 🔒