Web utilities
Some utility functions are provided for common web requests. Most use the GitHub API to query information or download artifacts and assets. See this project’s test cases (in particular test_download.py
) for detailed usage examples.
Note: to avoid GitHub API rate limits when using these functions, it is recommended to set the GITHUB_TOKEN
environment variable. If this variable is set, the token will be borne on requests sent to the API.
Queries
The following functions ask the GitHub API for information about a repository. The singular functions generally return a dictionary, while the plural functions return a list of dictionaries, with dictionary contents parsed directly from the API response’s JSON. The first parameter of each function is repo
, a string whose format must be owner/name
, as appearing in GitHub URLs.
For instance, to retrieve information about the latest executables release, then manually inspect available assets:
from modflow_devtools.download import get_release
release = get_release("MODFLOW-USGS/executables")
assets = release["assets"]
print([asset["name"] for asset in assets])
This yields ['code.json', 'linux.zip', 'mac.zip', 'win64.zip']
.
Equivalently, using the get_release_assets()
function to list the latest release assets directly:
from modflow_devtools.download import get_release_assets
assets = get_release_assets("MODFLOW-USGS/executables")
print([asset["name"] for asset in assets])
The simple
parameter, defaulting to False
, can be toggled to return a simple dictionary mapping asset names to download URLs:
from pprint import pprint
assets = get_release_assets("MODFLOW-USGS/executables", simple=True)
pprint(assets)
This prints:
{'code.json': 'https://github.com/MODFLOW-USGS/executables/releases/download/12.0/code.json',
'linux.zip': 'https://github.com/MODFLOW-USGS/executables/releases/download/12.0/linux.zip',
'mac.zip': 'https://github.com/MODFLOW-USGS/executables/releases/download/12.0/mac.zip',
'win64.zip': 'https://github.com/MODFLOW-USGS/executables/releases/download/12.0/win64.zip'}
Downloads
The download_artifact
function downloads and unzips the GitHub Actions artifact with the given ID to the given path, optionally deleting the zipfile afterwards. The repo
format is owner/name
, as in GitHub URLs. For instance:
from modflow_devtools.download import list_artifacts, download_artifact
repo = "MODFLOW-USGS/modflow6"
artifacts = list_artifacts(repo, max_pages=1, verbose=True)
artifact = next(iter(artifacts), None)
if artifact:
download_artifact(
repo=repo,
id=artifact["id"],
path=function_tmpdir,
delete_zip=False,
verbose=False,
)
The download_and_unzip
function is a more generic alternative for downloading and unzipping files from arbitrary URLs.
For instance, to download a MODFLOW 6.4.1 Linux distribution and delete the zipfile after extracting:
from modflow_devtools.download import download_and_unzip
url = f"https://github.com/MODFLOW-USGS/modflow6/releases/download/6.4.1/mf6.4.1_linux.zip"
download_and_unzip(url, "~/Downloads", delete_zip=True, verbose=True)
The function’s return value is the Path
the archive was extracted to.