# Programs API

> **Warning**: This API is experimental and may change or be removed in future versions without following normal deprecation procedures. Use at your own risk.
>
> When importing this module programmatically, you will see a `FutureWarning`. To suppress this warning:
> ```python
> import warnings
> warnings.filterwarnings('ignore', message='.*modflow_devtools.programs.*experimental.*')
> ```

The `modflow_devtools.programs` module provides programmatic access to MODFLOW and related programs in the MODFLOW ecosystem. It can be used with MODFLOW organization releases or custom program repositories.

This module builds on [Pooch](https://www.fatiando.org/pooch/latest/index.html) for file fetching and caching. While it leverages Pooch's capabilities, it provides an independent layer with:

- Registration, discovery and synchronization
- Installation and version management
- Platform-specific binary handling

Program registries can be synchronized from remote sources on demand. The user or developer can inspect and install programs published by the MODFLOW organization, from a personal fork, or from custom repositories.

<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->


- [Overview](#overview)
- [Usage](#usage)
  - [Syncing registries](#syncing-registries)
  - [Inspecting available programs](#inspecting-available-programs)
  - [Installing a program](#installing-a-program)
  - [Finding installed programs](#finding-installed-programs)
  - [Version management](#version-management)
  - [Using the default manager](#using-the-default-manager)
  - [Customizing program sources](#customizing-program-sources)
  - [Working with registries](#working-with-registries)
- [Program Addressing](#program-addressing)
- [Platform Support](#platform-support)
- [Cache Management](#cache-management)
- [Force Semantics](#force-semantics)
- [Automatic Synchronization](#automatic-synchronization)
- [Repository Integration](#repository-integration)
  - [Registry Generation](#registry-generation)
  - [Publishing Registries](#publishing-registries)
  - [Registry Format](#registry-format)
- [Relationship to pymake and get-modflow](#relationship-to-pymake-and-get-modflow)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

## Overview

The Programs API provides:

- **Program registration**: Index local or remote program repositories
- **Program discovery**: Browse available programs and versions
- **Program installation**: Install pre-built binaries for your platform
- **Version management**: Install multiple versions side-by-side and switch between them

Program metadata is provided by **registries** which are published by program repositories. On first use, `modflow-devtools` automatically attempts to sync these registries.

A program registry contains metadata for available programs including:

- **Program name and description**
- **Available versions**
- **Platform-specific distributions** (binaries and assets)

## Usage

### Syncing registries

Registries can be manually synchronized:

```python
from modflow_devtools.programs import ProgramSourceConfig

config = ProgramSourceConfig.load()

# Sync all configured sources
results = config.sync(verbose=True)

# Sync specific source
results = config.sync(source="modflow6", verbose=True)
```

Or via CLI (both forms are equivalent):

```bash
# Using the mf command
mf programs sync
mf programs sync --source modflow6
mf programs sync --force  # Force re-download of registry metadata

# Or using the module form
python -m modflow_devtools.programs sync
python -m modflow_devtools.programs sync --source modflow6
python -m modflow_devtools.programs sync --force  # Force re-download of registry metadata
```

**Note**: The `--force` flag on `sync` forces re-downloading of registry metadata even if already cached. This is separate from installation - see the "Force semantics" section below.

### Inspecting available programs

```python
from modflow_devtools.programs import ProgramSourceConfig

config = ProgramSourceConfig.load()

# Check sync status
status = config.status
for source_name, source_status in status.items():
    print(f"{source_name}: {source_status.cached_refs}")
```

Or by CLI (both forms are equivalent):

```bash
# Using the mf command
mf programs info  # Show sync status
mf programs list  # Show program summary
mf programs list --verbose  # Full list with details
mf programs list --source modflow6 --verbose  # Filter by source

# Or using the module form
python -m modflow_devtools.programs info  # Show sync status
python -m modflow_devtools.programs list  # Show program summary
python -m modflow_devtools.programs list --verbose  # Full list with details
python -m modflow_devtools.programs list --source modflow6 --verbose  # Filter by source
```

### Installing a program

```python
from modflow_devtools.programs import install_program

# Install latest available version
paths = install_program("mf6", verbose=True)

# Install specific version
paths = install_program("mf6", version="6.6.3", verbose=True)

# Install to custom directory
paths = install_program("mf6", version="6.6.3", bindir="/usr/local/bin")
```

Or via CLI (both forms are equivalent):

```bash
# Using the mf command
mf programs install mf6
mf programs install mf6@6.6.3
mf programs install mf6@6.6.3 --bindir /usr/local/bin

# Or using the module form
python -m modflow_devtools.programs install mf6
python -m modflow_devtools.programs install mf6@6.6.3
python -m modflow_devtools.programs install mf6@6.6.3 --bindir /usr/local/bin
```

### Finding installed programs

```python
from modflow_devtools.programs import _DEFAULT_MANAGER, list_installed

# Get path to installed executable
mf6_path = _DEFAULT_MANAGER.get_executable("mf6")

# Get specific version
mf6_path = _DEFAULT_MANAGER.get_executable("mf6", version="6.6.3")

# List all installed programs
installed = list_installed()
for program_name, installations in installed.items():
    for inst in installations:
        print(f"{program_name} {inst.version} in {inst.bindir}")
```

Or by CLI (both forms are equivalent):

```bash
# Using the mf command
mf programs history
mf programs history mf6 --verbose

# Or using the module form
python -m modflow_devtools.programs history
python -m modflow_devtools.programs history mf6 --verbose
```

### Version management

Multiple versions can be installed side-by-side. Switching between them is done by re-running `install` — the archive is already cached, so no re-download occurs:

```python
from modflow_devtools.programs import install_program

# Install multiple versions
install_program("mf6", version="6.6.3")
install_program("mf6", version="6.5.0")

# Switch back to 6.6.3 — re-copies from cache, no download
install_program("mf6", version="6.6.3")
```

Or by CLI:

```bash
mf programs install mf6@6.6.3
mf programs install mf6@6.5.0

# Switch back — fast, uses cached archive
mf programs install mf6@6.6.3
```

### Using the default manager

The module provides explicit access to the default manager used by `install_program()` etc.

```python
from modflow_devtools.programs import _DEFAULT_MANAGER

# Install programs
paths = _DEFAULT_MANAGER.install("mf6", version="6.6.3", verbose=True)

# Switch versions
_DEFAULT_MANAGER.select("mf6", version="6.5.0", verbose=True)

# Get executable path
mf6_path = _DEFAULT_MANAGER.get_executable("mf6")

# List installed programs
installed = _DEFAULT_MANAGER.list_installed()

# Uninstall specific version
_DEFAULT_MANAGER.uninstall("mf6", version="6.5.0")

# Uninstall all versions
_DEFAULT_MANAGER.uninstall("mf6", all_versions=True)
```

### Customizing program sources

Create a user config file to add custom sources or override defaults:

- **Windows**: `%APPDATA%/modflow-devtools/programs.toml`
- **macOS**: `~/Library/Application Support/modflow-devtools/programs.toml`
- **Linux**: `~/.config/modflow-devtools/programs.toml`

Example user config:

```toml
[sources.modflow6]
repo = "myusername/modflow6"  # Use a fork for testing
refs = ["develop"]
```

The user config is automatically merged with the bundled config, allowing you to test against forks or add private repositories.

### Working with registries

Access cached registry data directly:

```python
from modflow_devtools.programs import _DEFAULT_CACHE, ProgramSourceConfig

config = ProgramSourceConfig.load()

# Check sync status
status = config.status
for source_name, source_status in status.items():
    print(f"{source_name}: {source_status.cached_refs}")

# Load cached registry
registry = _DEFAULT_CACHE.load("modflow6", "6.6.3")
if registry:
    for program_name, metadata in registry.programs.items():
        print(f"{program_name} {metadata.version}")
        print(f"  Description: {metadata.description}")
        print(f"  Distributions: {[d.name for d in metadata.dists]}")
```

## Program Addressing

Programs are addressed using the format: `{program}@{version}`.

Examples:
- `mf6@6.6.3` - MODFLOW 6 version 6.6.3
- `zbud6@6.6.3` - MODFLOW 6 Zonebudget version 6.6.3
- `mp7@7.2.001` - MODPATH 7 version 7.2.001

## Platform Support

The Programs API automatically detects your platform and downloads the appropriate binaries:

- **linux**: Linux x86_64
- **mac**: macOS (Intel and Apple Silicon)
- **win64**: Windows 64-bit

Programs must provide pre-built binaries for supported platforms. Building from source is not supported—program repositories are responsible for releasing platform-specific binaries.

## Cache Management

Downloaded archives and installed binaries are cached locally:

- **Registries**: `~/.cache/modflow-devtools/programs/registries/{source}/{ref}/`
- **Archives**: `~/.cache/modflow-devtools/programs/archives/{program}/{version}/{platform}/`
- **Binaries**: `~/.cache/modflow-devtools/programs/binaries/{program}/{version}/{platform}/`
- **Metadata**: `~/.cache/modflow-devtools/programs/installations/{program}.json`

The cache enables:
- Fast re-installation without re-downloading
- Efficient version switching
- Offline access to previously installed programs

## Force Semantics

The `--force` flag has different meanings depending on the command:

**`sync --force`**: Forces re-downloading of registry metadata from GitHub
- Re-fetches `programs.toml` even if already cached
- Use when registry files have been updated on GitHub
- Does not affect installed programs or archives

**`install --force`**: Forces re-installation of program binaries
- Re-extracts from cached archive and re-copies to installation directory
- Does **not** re-sync registry metadata (use `sync --force` first if needed)
- Use when installation is corrupted or you want to reinstall to a different location
- Works offline if archive is already cached

**Common workflows**:
```bash
# Update to latest registry and install
mf programs sync --force
mf programs install mf6

# Repair broken installation (offline-friendly)
mf programs install mf6 --force

# Fresh install with latest metadata
mf programs sync --force
mf programs install mf6 --force
```

## Automatic Synchronization

Auto-sync is **opt-in** (experimental). To enable:

```bash
export MODFLOW_DEVTOOLS_AUTO_SYNC=1  # or "true" or "yes"
```

When enabled, `modflow-devtools` attempts to sync registries:
- On first access (best-effort, fails silently on network errors)
- Before installation
- Before listing available programs

Then manually sync when needed:

```bash
mf programs sync
# Or: python -m modflow_devtools.programs sync
```

## Repository Integration

Program repositories publish registry files (`programs.toml`) describing available programs and platform-specific distributions.

### Registry Generation

The `make_registry` tool generates registry files from local or remote assets.

**From local assets** (typical CI usage):

```bash
python -m modflow_devtools.programs.make_registry \
  --dists *.zip \
  --programs mf6 zbud6 libmf6 mf5to6 \
  --version 6.6.3 \
  --repo MODFLOW-ORG/modflow6 \
  --compute-hashes \
  --output programs.toml
```

**From existing GitHub release**:

```bash
python -m modflow_devtools.programs.make_registry \
  --repo MODFLOW-ORG/modflow6 \
  --version 6.6.3 \
  --programs mf6 zbud6 libmf6 mf5to6 \
  --compute-hashes \
  --output programs.toml
```

### Publishing Registries

Registry files are published as GitHub release assets alongside binary distributions.

For instance, to publish a registry in a GitHub Actions workflow:

```yaml
- name: Generate program registry
  run: |
    python -m modflow_devtools.programs.make_registry \
      --dists *.zip \
      --programs mf6 zbud6 libmf6 mf5to6 \
      --version ${{ github.ref_name }} \
      --repo ${{ github.repository }} \
      --compute-hashes \
      --output programs.toml

- name: Upload registry to release
  uses: softprops/action-gh-release@v1
  with:
    files: programs.toml
```

### Registry Format

The generated `programs.toml` file contains:

- Program metadata (description, license)
- Platform-specific distributions (linux, mac, win64)
- Asset filenames and SHA256 hashes
- Executable paths within archives

See the [developer documentation](dev/programs.md) for detailed registry format specifications.

## Relationship to pymake and get-modflow

The Programs API is designed to eventually supersede:
- **pymake's program database**: Registry responsibilities are delegated to program repositories
- **flopy's get-modflow**: Installation patterns adapted and enhanced for multi-version support

The Programs API provides:
- Decoupled releases (programs release independently of devtools)
- Multiple versions side-by-side
- Unified cache structure
- Comprehensive installation tracking
- Fast version switching