Configuration Management¶

Multi-source configuration with automatic validation and type safety.

Overview¶

The configuration module provides a flexible, type-safe approach to loading configuration from multiple sources with automatic validation using Pydantic.

Key Features¶

Multi-source loading: Combine defaults, files, environment variables, and secrets
Type-safe: Automatic validation with Pydantic models
Format support: YAML, JSON, TOML, HOCON, ENV files
Deep merging: Later sources override earlier ones
Environment overrides: Production values via environment variables
Optional sources: Development overrides without breaking production

Basic Usage¶

Define Configuration Model¶

from pydantic import BaseModel, Field

class DatabaseConfig(BaseModel):
    host: str = "localhost"
    port: int = 5432
    name: str
    password: str

class AppConfig(BaseModel):
    debug: bool = False
    database: DatabaseConfig
    api_key: str

Load Configuration¶

from dspu.config import Config

# Simple: from single file
config = Config.load_from_file(AppConfig, "config.yaml")

# Advanced: from multiple sources
from dspu.config import FileSource, EnvSource

config = Config.load(
    AppConfig,
    sources=[
        FileSource("config.yaml"),           # Base configuration
        EnvSource(prefix="APP_"),            # Environment overrides
    ],
)

Configuration Sources¶

FileSource¶

Load from configuration files:

from dspu.config import FileSource

# YAML
FileSource("config.yaml")

# JSON
FileSource("config.json")

# TOML
FileSource("config.toml")

# HOCON
FileSource("application.conf")

# Optional file (won't error if missing)
FileSource("config.local.yaml", required=False)

# Explicit format
FileSource("settings.txt", format="json")

EnvSource¶

Load from environment variables:

from dspu.config import EnvSource

# With prefix
EnvSource(prefix="APP_")  # Reads APP_DEBUG, APP_API_KEY, etc.

# Nested keys with separator
EnvSource(prefix="APP_", separator="__")
# APP_DATABASE__HOST → config.database.host
# APP_DATABASE__PORT → config.database.port

# No prefix (use all env vars)
EnvSource()

DictSource¶

Load from dictionary:

from dspu.config import DictSource

defaults = {
    "debug": False,
    "database": {
        "host": "localhost",
        "port": 5432,
    }
}

DictSource(defaults)

Source Priority¶

Later sources override earlier ones:

config = Config.load(
    AppConfig,
    sources=[
        DictSource(defaults),                # 1. Defaults (lowest priority)
        FileSource("config.yaml"),           # 2. Base config
        FileSource(f"config.{env}.yaml"),    # 3. Environment-specific
        FileSource("config.local.yaml", required=False),  # 4. Local overrides
        EnvSource(prefix="APP_"),            # 5. Environment variables (highest)
    ],
)

Example:

# config.yaml
debug: false
api_key: dev_key_123

# config.production.yaml
debug: false
api_key: prod_key_456

# Environment
export APP_DEBUG=true
export APP_API_KEY=runtime_key_789

Result: debug=true, api_key=runtime_key_789 (env vars win)

Deep Merging¶

Nested configurations are merged deeply:

# Source 1
{
    "database": {
        "host": "localhost",
        "port": 5432,
    }
}

# Source 2
{
    "database": {
        "port": 3306,  # Override port
        "name": "mydb",  # Add name
    }
}

# Result
{
    "database": {
        "host": "localhost",  # Kept from source 1
        "port": 3306,         # Overridden by source 2
        "name": "mydb",       # Added by source 2
    }
}

Validation¶

Field Constraints¶

Use Pydantic's validation features:

from pydantic import BaseModel, Field, EmailStr, HttpUrl

class AppConfig(BaseModel):
    # Range validation
    port: int = Field(ge=1, le=65535)
    workers: int = Field(ge=1, le=100)

    # String validation
    email: EmailStr
    api_url: HttpUrl
    api_key: str = Field(min_length=32)

    # Enum validation
    log_level: Literal["DEBUG", "INFO", "WARNING", "ERROR"]

Custom Validators¶

from pydantic import field_validator

class AppConfig(BaseModel):
    database_url: str

    @field_validator("database_url")
    @classmethod
    def validate_database_url(cls, v: str) -> str:
        if not v.startswith(("postgresql://", "mysql://")):
            raise ValueError("Invalid database URL scheme")
        return v

Conditional Validation¶

from pydantic import model_validator

class AppConfig(BaseModel):
    use_cache: bool
    cache_url: str | None = None

    @model_validator(mode='after')
    def check_cache_url(self):
        if self.use_cache and not self.cache_url:
            raise ValueError("cache_url required when use_cache is True")
        return self

Environment Variables¶

Naming Convention¶

Use prefixes and separators:

# Flat keys
export APP_DEBUG=true
export APP_API_KEY=secret

# Nested keys (use __)
export APP_DATABASE__HOST=localhost
export APP_DATABASE__PORT=5432
export APP_DATABASE__NAME=mydb

EnvSource(prefix="APP_", separator="__")

Type Coercion¶

Environment variables are automatically converted:

# Boolean
export APP_DEBUG=true     # → True
export APP_DEBUG=1        # → True
export APP_DEBUG=false    # → False
export APP_DEBUG=0        # → False

# Integer
export APP_PORT=8080      # → 8080

# List (JSON)
export APP_TAGS='["a","b","c"]'  # → ["a", "b", "c"]

# Dict (JSON)
export APP_META='{"key":"value"}'  # → {"key": "value"}

Common Patterns¶

Pattern 1: Local Development¶

import os

config = Config.load(
    AppConfig,
    sources=[
        FileSource("config.yaml"),                      # Base config
        FileSource("config.local.yaml", required=False), # Local overrides (gitignored)
        EnvSource(prefix="APP_"),                       # Environment overrides
    ],
)

Setup:

# .gitignore
config.local.yaml

# config.yaml (committed)
debug: false
database:
  host: localhost

# config.local.yaml (not committed)
debug: true
database:
  host: docker.local

Pattern 2: Docker Compose¶

config = Config.load(
    AppConfig,
    sources=[
        EnvSource(prefix="MYAPP_", separator="__"),
    ],
)

docker-compose.yml:

services:
  app:
    environment:
      MYAPP_DEBUG: "true"
      MYAPP_DATABASE__HOST: postgres
      MYAPP_DATABASE__PASSWORD: secret

Pattern 3: Kubernetes¶

config = Config.load(
    AppConfig,
    sources=[
        FileSource("/config/app.yaml"),  # From ConfigMap
        EnvSource(prefix="", separator="__"),  # From Secrets
    ],
)

Kubernetes:

# ConfigMap for non-sensitive config
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  app.yaml: |
    debug: false
    database:
      host: postgres-service

# Secret for sensitive data
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
data:
  DATABASE__PASSWORD: <base64-encoded>
  API_KEY: <base64-encoded>

Pattern 4: Environment-Specific¶

import os

env = os.getenv("ENV", "development")

config = Config.load(
    AppConfig,
    sources=[
        FileSource("config.yaml"),           # Base
        FileSource(f"config.{env}.yaml"),    # Environment-specific
        EnvSource(prefix="APP_"),            # Runtime overrides
    ],
)

Files:

config.yaml               # Base (committed)
config.development.yaml   # Dev overrides (committed)
config.staging.yaml       # Staging overrides (committed)
config.production.yaml    # Production overrides (committed)

Pattern 5: 12-Factor App¶

# Pure environment variable config
config = Config.load(
    AppConfig,
    sources=[
        EnvSource(),  # No prefix - use all env vars
    ],
)

Watching Configuration¶

Monitor configuration files for changes:

from dspu.config import WatchedConfig

# Create watched config
watched = WatchedConfig(
    AppConfig,
    sources=[FileSource("config.yaml")],
    poll_interval=5.0  # Check every 5 seconds
)

# Start watching
await watched.start()

# Get current config
config = watched.current

# Register callback for changes
def on_config_change(new_config: AppConfig):
    print("Configuration changed!")
    # Reload application...

watched.on_change(on_config_change)

# Stop watching
await watched.stop()

Best Practices¶

Security¶

✅ DO: - Use environment variables for secrets - Use secret managers in production (Vault, AWS Secrets Manager) - Keep sensitive config out of version control - Use .gitignore for local config files

❌ DON'T: - Don't commit secrets to git - Don't log sensitive configuration - Don't hardcode credentials - Don't use weak defaults

Organization¶

✅ DO: - Layer configuration (defaults → file → env → secrets) - Use meaningful names and prefixes - Document required vs optional settings - Provide sensible defaults - Validate early with Pydantic

❌ DON'T: - Don't use loose types (use validation) - Don't skip documentation - Don't mix concerns (separate DB, API, etc.) - Don't use magic values

Deployment¶

✅ DO: - Use environment-specific config files - Override with environment variables - Make local overrides optional - Test configuration loading

❌ DON'T: - Don't use same config for all environments - Don't require files in production (use env vars) - Don't skip error handling - Don't hardcode environment-specific values

Troubleshooting¶

ValidationError: Field required¶

Problem: Missing required field

Solution:

# Add default value
class AppConfig(BaseModel):
    api_key: str = "default_key"  # ✅

# Or make optional
class AppConfig(BaseModel):
    api_key: str | None = None  # ✅

ConfigurationError: File not found¶

Problem: Required config file missing

Solution:

# Make optional for local overrides
FileSource("config.local.yaml", required=False)  # ✅

Environment variables not working¶

Problem: Env vars not being picked up

Solution:

# Check prefix and separator
EnvSource(prefix="APP_", separator="__")

# Env var should be: APP_DATABASE__HOST
# Not: APP_DATABASE_HOST or DATABASE__HOST

Configuration Management¶

Overview¶

Key Features¶

Basic Usage¶

Define Configuration Model¶

Load Configuration¶

Configuration Sources¶

FileSource¶

EnvSource¶

DictSource¶

Source Priority¶

Deep Merging¶

Validation¶

Field Constraints¶

Custom Validators¶

Conditional Validation¶

Environment Variables¶

Naming Convention¶

Type Coercion¶

Common Patterns¶

Pattern 1: Local Development¶

Pattern 2: Docker Compose¶

Pattern 3: Kubernetes¶

Pattern 4: Environment-Specific¶

Pattern 5: 12-Factor App¶

Watching Configuration¶

Best Practices¶

Security¶

Organization¶

Deployment¶

Troubleshooting¶

ValidationError: Field required¶

ConfigurationError: File not found¶

Environment variables not working¶

Next Steps¶