Skip to content

Overhaul docs #170

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
leewujung wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Overhaul docs #170

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1e30290
purge old since no longer apply, add new content using jupyter book v2
leewujung Aug 10, 2026
f6b73a4
remove explicit linkify requirement
leewujung Aug 10, 2026
24cdcc8
test trigger rtd build
leewujung Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 12 additions & 27 deletions .readthedocs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,32 +1,17 @@
# .readthedocs.yaml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details

# Required
version: 2

# Set the OS, Python version and other tools you might need
build:
os: ubuntu-22.04
os: ubuntu-lts-latest
tools:
python: "3.10"
# You can also specify other tool versions:
# nodejs: "19"
# rust: "1.64"
# golang: "1.19"

# Build documentation in the "docs/" directory with Sphinx
sphinx:
configuration: docs/source/conf.py

# Optionally build your docs in additional formats such as PDF and ePub
# formats:
# - pdf
# - epub
python: "3.12"
nodejs: "22"
jobs:
build:
html:
- pip install --upgrade -r docs/requirements.txt
- cd docs && jupyter book build --html --strict
post_build:
- mkdir --parents $READTHEDOCS_OUTPUT/html/
- cp --recursive docs/_build/html/* $READTHEDOCS_OUTPUT/html/

# Optional but recommended, declare the Python requirements required
# to build your documentation
# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
python:
install:
- requirements: docs/requirements.txt
formats: []
245 changes: 51 additions & 194 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,207 +1,64 @@
<!-- <div>
<a href="https://echodataflow.readthedocs.io/en/latest/?badge=latest">
<img src="https://readthedocs.org/projects/echodataflow/badge/?version=latest"/>
</a>
<a href="https://codecov.io/gh/echostack-org/echodataflow" >
<img src="https://codecov.io/gh/echostack-org/echodataflow/graph/badge.svg?token=YTMVVHG585"/>
</a>
</div> -->
# Echodataflow

# Echodataflow: Streamlined Data Pipeline Orchestration
Echodataflow provides recipe-driven orchestration for fisheries acoustics workflows. It
combines [Prefect](https://www.prefect.io/), YAML deployment recipes, and processing tools
such as [Echopype](https://github.com/echostack-org/echopype) to run repeatable workflows on
edge, local, and cloud infrastructure.

Echodataflow streamlines echosounder data processing by combining [Prefect](https://www.prefect.io/)-based pipeline orchestration, YAML configuration, and [Echopype](https://github.com/echostack-org/echopype) into a modular tool for defining, configuring, and executing workflows.
Echodataflow `0.1.x` is deprecated. The repository currently contains the architecture being
prepared for the `0.2` release.

## Installation

**Note:** Echodataflow v.0.1.x have been deprecated. We will release v0.2.0 soon!
```shell
conda create -n echodataflow -c conda-forge python=3.12 uv
conda activate echodataflow
uv pip install "git+https://github.com/echostack-org/echodataflow.git"
```

Install Echopype separately when using the acoustic processing flows:

```shell
uv pip install echopype
```

## Installation
## Documentation

The documentation covers:

- installation and architecture;
- parameter and deployment recipes;
- Prefect server and worker setup on macOS and Linux;
- workflow deployment and troubleshooting;
- a reproducible simulated edge example; and
- development setup and adding operations, tasks, and flows.

Read the published documentation at
[echodataflow.readthedocs.io](https://echodataflow.readthedocs.io/) or build it locally:

```shell
uv pip install -e ".[docs]"
cd docs
jupyter book start
```

Mission-specific recipes are maintained in
[echodataflow-recipes](https://github.com/echostack-org/echodataflow-recipes).

1. Set up a computing environment using Conda:
```bash
conda create --name echodataflow -c conda-forge python=3.12
conda activate echodataflow
```

2. If you would like to run Echodataflow as an installed package,
install it from the repo like below:
```bash
pip install https://github.com/echostack-org/echodataflow.git # install from repo
```
This installs the `echodataflow-deploy` command, which can be run from any directory.
If you instead would like to install Echodataflow to develop it,
clone the repo and install it like below:
```bash
git clone https://github.com/echostack-org/echodataflow.git # clone the repo
pip install -e ".[test,lint,docs]" # install in editable mode with dev tools
```

3. Pip install the `segmentation_inference` package that contains a version of the hake segmentation model.
```bash
cd ..
git clone https://github.com/uw-echospace/segmentation_inference.git # clone the repo
cd segmentation_inference
pip install -e .
```


## Running the edge pipeline

Note: Starting the server and running work pool is unnecessary if local Mac Prefect background services are running.

1. Start the local Prefect server:
```shell
prefect server start
```

2. In a new terminal, create and run a work pool:
```shell
prefect worker start --pool "local"
```
If you run into the error below:
```shell
ValueError: `PREFECT_API_URL` must be set to start a Worker.
```
Run:
```shell
prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api
```

3. In a new terminal, download the recipes from the [echodataflow-recipes repository](https://github.com/echostack-org/echodataflow-recipes) by clonining it to your computer:
```
cd REPO_DIRECTORY # switch to where you want the recipes repo to sit
git clone https://github.com/echostack-org/echodataflow-recipes.git
```

4. Deploy and run the edge pipeline:
```shell
echodataflow-deploy run \
--default-work-pool-name local \
--param-config REPO_DIRECTORY/recipes/params/params_{MISSION_NAME}.yaml \
--deploy-spec REPO_DIRECTORY/recipes/deploy/deploy_{MISSION_NAME}.yaml
```
The deployment source is selected from the `source` section in the deploy recipe.
If not set, it is default to using the local codebase.


## Running the cloud pipeline

1. Start a cloud virtual machine using the Linux platform

2. Start up a system service that runs a Prefect worker

3. Establish connection with the cloud Prefect server

4. Download the recipes from the [echodataflow-recipes repository](https://github.com/echostack-org/echodataflow-recipes) by clonining it to your computer:
```
cd REPO_DIRECTORY # switch to where you want the recipes repo to sit
git clone https://github.com/echostack-org/echodataflow-recipes.git
```

5. Deploy and run the cloud pipeline:
```bash
echodataflow-deploy run \
--default-work-pool-name local \
--param-config REPO_DIRECTORY/recipes/params/params_{MISSION_NAME}.yaml \
--deploy-spec REPO_DIRECTORY/recipes/deploy/deploy_{MISSION_NAME}.yaml
```
The deployment source is selected from the `source` section in the deploy recipe.
If not set, it is default to using the local codebase.

6. Start up system services that hosts the 2 sets of visualization


## Running Local Prefect and auto mounting services on macOS (launchd)

To run a local Prefect server and worker as background services on macOS, you can
use launchd with the provided plist templates:

- `src/echodataflow/services/deploy_prefect_server.launchd.plist`
- `src/echodataflow/services/deploy_prefect_worker.launchd.plist`

These templates intentionally use direct one-line `ProgramArguments` commands, similar
to `.service` `ExecStart` usage, with no wrapper shell script required.

Included is a template and subsequent commands for auto mounting an SMB volume. These can be omitted if the volume is stable.

1. Copy and customize the templates for your user:
```shell
mkdir -p ~/.config/echodataflow ~/Library/LaunchAgents ~/.local/var/log/echodataflow
cp src/echodataflow/services/services.env.example_local ~/.config/echodataflow/services.env
cp src/echodataflow/services/deploy_prefect_server.launchd.plist ~/Library/LaunchAgents/org.echodataflow.prefect-server.plist
cp src/echodataflow/services/deploy_prefect_worker.launchd.plist ~/Library/LaunchAgents/org.echodataflow.prefect-worker.plist
cp src/echodataflow/services/auto_mount.launchd.plist ~/Library/LaunchAgents/org.echodataflow.auto-mount.plist
```

2. Edit `~/.config/echodataflow/services.env` as needed:
- Adjust `ECHODATAFLOW_ENV`
- Adjust `ECHODATAFLOW_HOME`
- Adjust `MAMBA_BIN`
- Adjust `PREFECT_POOL`
- Adjust `PREFECT_API_URL`
- Adjust SMB parameters as needed

3. Load and start services:
```shell
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.echodataflow.prefect-server.plist
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.echodataflow.prefect-worker.plist
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.echodataflow.auto-mount.plist
launchctl kickstart -k gui/$(id -u)/org.echodataflow.prefect-server
launchctl kickstart -k gui/$(id -u)/org.echodataflow.prefect-worker
launchctl kickstart -k gui/$(id -u)/org.echodataflow.auto-mount
```

4. Check status and logs:
```shell
# make sure "state = running" and "runs" not increasing
launchctl print gui/$(id -u)/org.echodataflow.prefect-server
launchctl print gui/$(id -u)/org.echodataflow.prefect-worker
launchctl print gui/$(id -u)/org.echodataflow.auto-mount
# -f to follow logs in real time
tail -n 100 ~/.local/var/log/echodataflow/prefect-server.err.log
tail -n 100 ~/.local/var/log/echodataflow/prefect-worker.err.log
tail -n 100 ~/.local/var/log/echodataflow/auto-mount.err.log
```

5. To stop and unload services:
```shell
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/org.echodataflow.prefect-server.plist
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/org.echodataflow.prefect-worker.plist
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/org.echodataflow.auto-mount.plist
```

6. SQLite health checks (local Prefect server):
```shell
sqlite3 ~/.prefect/prefect.db "PRAGMA quick_check;"
sqlite3 ~/.prefect/prefect.db "PRAGMA integrity_check;"
```

7. If server startup keeps failing with SQLite lock errors, reset local DB safely:
```shell
# stop services first
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/org.echodataflow.prefect-worker.plist
launchctl bootout gui/$(id -u) ~/Library/LaunchAgents/org.echodataflow.prefect-server.plist

# archive existing local Prefect DB files (do not delete first)
ts=$(date +%Y%m%d_%H%M%S)
mkdir -p ~/.prefect/db-backups/$ts
mv ~/.prefect/prefect.db* ~/.prefect/db-backups/$ts/ 2>/dev/null || true

# start server, then worker
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.echodataflow.prefect-server.plist
launchctl kickstart -k gui/$(id -u)/org.echodataflow.prefect-server
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/org.echodataflow.prefect-worker.plist
launchctl kickstart -k gui/$(id -u)/org.echodataflow.prefect-worker
```

Notes:
- `ThrottleInterval=30` in plist files helps avoid aggressive restart loops.
- `database is locked` usually means SQLite write contention, not corruption.
- For heavier multi-flow usage, move Prefect server DB to Postgres.
## Development

```shell
git clone https://github.com/echostack-org/echodataflow.git
cd echodataflow
conda create -n echodataflow-dev -c conda-forge python=3.12 uv
conda activate echodataflow-dev
uv pip install -e ".[test,lint,docs]"
pytest
```

See the development guide in the documentation for project structure and contribution
instructions.

## License

Echodataflow is licensed under the open source [Apache 2.0 license](https://opensource.org/license/Apache-2.0).
Echodataflow is distributed under the [Apache License 2.0](LICENSE).
20 changes: 0 additions & 20 deletions docs/Makefile
View file
Open in desktop

This file was deleted.

Loading