feat: ✨ add release process chapter

docs/release.qmd

@@ -1 +1,220 @@
 # Release process {#sec-release}
+
+A release is a snapshot of the data package at a specific point in time. It
+contains the data and metadata in their final state at that point in time and is
+tagged with a specific version number. Releases are how we track changes over
+time and make it easier to share or distribute data packages to researchers in a
+structured and predictable way.
+
+The release process is dependent on the type of data contained in the data
+package. For data packages that contain human (in particular health or
+sensitive) data, the release process is a bit more complicated, as will be
+explained later, than for data packages that contain data that doesn't fall
+under legal restrictions (e.g. GDPR). This is because the data must remain on
+secure servers and can't (usually) be uploaded to any public repository or
+archive. That means we can't use services like GitHub workflows nor upload any
+data for public access, so we can't use a continuous release process (or at
+least it is more difficult to).
+
+<!-- TODO: link to an example `release.yml` file -->
+
+For nonsensitive data, the release process is fairly simple. It can be done
+through a GitHub workflow (e.g. `release.yml`) that runs on a schedule or is
+triggered by specific events (e.g. a specific type of commit, described later in
+this chapter). The data and metadata can be attached as release artifacts on
+GitHub and/or be uploaded to public archives like [Zenodo](https://zenodo.org/).
+
+Regardless of the type of data, the general steps remain mostly the same with
+some notable differences. The difference is in *where* the release is done (the
+computing environment where the release process is executed, e.g. GitHub or on a
+secure server) and *what triggers* a release. We'll start with the triggers.
+
+## Triggers
+
+There are a few ways to trigger the release process: manually, on a schedule, or
+based on specific events whenever a change is merged into `main`. While we aim
+to practice [continuous
+delivery](https://en.wikipedia.org/wiki/Continuous_delivery) when we're able to,
+this isn't always the case for more complicated situations. For data that falls
+under legal regulations, you likely will need to use either a manual process or
+a schedule-based one by running a [cron job](https://en.wikipedia.org/wiki/Cron)
+on the server. For non-private data, you can use the merge/push-based process by
+using a GitHub workflow.
+
+Regardless of the underlying trigger, the actual release process is the same: a
+release is created based on specific text within the commit messages. Using
+commit messages to determine a release is called [semantic
+release](https://decisions.seedcase-project.org/why-semantic-release-with-cocogitto/),
+which uses [Semantic
+Versioning](https://decisions.seedcase-project.org/why-semver/index.html) and
+[Conventional
+Commits](https://decisions.seedcase-project.org/why-conventional-commits/) as
+its foundation.
+
+### Commits
+
+In order to determine whether changes should result in a release, commits must
+follow [Conventional
+Commits](https://decisions.seedcase-project.org/why-conventional-commits/) to
+structure the commit messages. The structure of a commit message looks like
+this:
+
+```text
+<type>(optional scope): <optional-emoji> <description>
+
+[optional body]
+
+[optional footer(s)]
+```
+
+The two main components of this structure for determining releases are the
+"type" and the "footer". The "type" is the first part of the commit message and
+is used to determine what type of change has been made. The "footer" is the last
+part of the commit message and is used to include a `BREAKING CHANGE` note if
+the change is a breaking change, i.e., a change that disrupt the compatibility
+of existing code with newer versions. A breaking change can also be indicated by
+appending a "!" to the commit "type". These two components determine which
+version to set for the release.
+
+Semantic versions are made up of three numbers: `MAJOR.MINOR.PATCH`, e.g.,
+`0.1.2`. The `MAJOR` version is incremented when there are breaking changes, the
+`MINOR` version is incremented when new features are added, and the `PATCH`
+version is incremented when fixes are made. For semantic releases, the commit
+"type" `feat` increases the `MINOR` version, while the commit "type" `fix`,
+`refactor`, or `perf` increases the `PATCH`. If there is a breaking change, with
+either `BREAKING CHANGE` in the footer or `<type>!` in the commit message, then
+the `MAJOR` version is increased.
+
+But how do you know which commit type to use? Unlike software development,
+developing data packages is quite different and it can be a bit more difficult
+to determine what a "feature", "fix", or "breaking change" is. To help determine
+the commit "type", we use aspects of [Data Package's semantic
+versioning](https://datapackage.org/recipes/data-package-version/) guide.
+
+Breaking changes with the `<type>!` or `BREAKING CHANGE` in the footer format
+*must only* happen after the first stable release of the data package. The first
+stable release is defined as when the data package has all expected or planned
+resources, metadata has been completed, and all "observational units" (e.g.
+participant or animal) have been measured. Essentially, when the planned study
+has been completed. Before that point, only `MINOR` and `PATCH` changes are
+allowed. This means that the version will remain at `0.MINOR.PATCH` until the
+stable release. Once a stable release has been made, a breaking change would
+occur if you:
+
+- Change the data package, resource, or column name or identifier.
+- Remove a resource or column from the data package.
+- Move a column into another resource.
+- Change a column type (e.g. from integer to string).
+- Change a column's constraints to be more restrictive (e.g. reduce the distance
+  between the minimum and maximum values).
+- Remove a participant's data (e.g. after they request their data be deleted).
+- Substantially change the meaning of the text in the metadata (e.g. a column's
+  description or a resource's title).
+
+A good guideline to use for `MINOR` (`feat`) commits would be if something *new*
+has been added or expanded on. Minor changes with the `feat` format would be if
+you:
+
+- Add a new resource.
+- Add data, either new rows or columns to an existing resource.
+- Change a column's constraints to be less restrictive (e.g. increase the
+  distance between the minimum and maximum values).
+- Add new text to the metadata, for example, when no metadata existed before,
+  but not correcting existing metadata (see `PATCH` below).
+
+A good guideline to use for `PATCH` (`fix`, `refactor`, or `perf`) commits would
+be if something has been *corrected* or *refined*. Before the stable release,
+many of the breaking change items above would be considered a patch change, as
+they generally don't add any new content. Patch changes with the `fix`,
+`refactor`, or `perf` commit type would be if you:
+
+- Correct errors in existing data, like a typo or data entry error. Depending on
+  the severity of the error, this could also be a breaking change.
+- Change the text of the metadata without changing the meaning, for example
+  fixing typos, grammatical errors, or clarifying the text without changing its
+  meaning.
+- Changes to how the data is processed so that it results in better compression
+  or other performance improvements.
+
+Any other commit types can be used, but they won't result in a releasable
+change. For example, before any metadata or resources have been created as the
+pipeline is being developed, you could use the `chore` commit type to indicate
+that you are working on the pipeline, but no data or metadata has been created
+yet.
+
+## Steps
+
+Now that we've covered the triggers, let's go over the actual steps involved in
+the release process, whether it is manual, schedule-based, or merge/push-based.
+[Cocogitto](https://decisions.seedcase-project.org/why-semantic-release-with-cocogitto/)
+manages all these steps via the `cog.toml` file.
+
+The release process runs the following steps to check and potentially update the
+release number:
+
+1. Check the commit history since the last release for any releasable changes.
+   If no releasable changes are found, then no release is created. Otherwise,
+   the process continues.
+
+2. Update the version based on the commit message and update the version in the
+   `pyproject.toml` file using
+   [`uv version`](https://docs.astral.sh/uv/reference/cli/#uv-version). If the
+   metadata format is `datapackage.json`, the version field uses the version in
+   `pyproject.toml` and will be updated automatically when the
+   `datapackage.json` file is (re)generated.
+
+3. Run the build process from start to end, excluding the step to pull from the
+   sources and saving to `raw/`. Raw data is only saved intentionally as a
+   separate commit/pull request. This is described in the build process in
+   @sec-build. The main artifacts of the build process are the
+   `<package-name>_<version>.tar` file for sensitive and nonsensitive data and
+   the `<package-name>_<version>.zip` for an artifact that can be publicly
+   uploaded when the data is sensitive.
+
+4. Generate the changelog based on the commit messages since the last release.
+   [git-cliff](https://decisions.seedcase-project.org/why-changelog-with-git-cliff/)
+   is used to generate the changelog.
+
+5. Commit the changes that were made in the `CHANGELOG.md` file and the metadata
+   file (e.g. `datapackage.json`), then create a tag for the new version on that
+   commit. No data in `raw/` is committed. Push the commit and the tag to
+   GitHub.
+
+6. Create a new GitHub release on GitHub from the new tag and changelog. Attach
+   the build artifacts to the release. For nonsensitive data and that is smaller
+   than 2 Gb (GitHub's limit), the `.tar` file is attached to the release. For
+   sensitive data or for data that is larger than 2 Gb, the `.zip` file is used
+   instead. Either way, the file is renamed to simply `<package-name>.zip` (or
+   `<package-name>.tar`), as the tag itself contains the version number.
+
+7. For nonsensitive data and that is smaller than 50 Gb (the limit for Zenodo),
+   upload the `<package-name>.tar` file to Zenodo. For sensitive data or data
+   that is larger than 50 Gb, upload the `<package-name>.zip` file instead.
+
+::: callout-important
+For data that is larger than GitHub's 2 Gb limit or Zenodo's 50 Gb limit, the
+best place to store the released `.tar` file is in your server in the
+`releases/` folder. You'll likely need to delete older releases if space becomes
+an issue on the server. Deleting older releases is fine, as the release process
+is designed to be reproducible, so you can always recreate the release by
+switching to the tagged commit and running the build process again.
+:::
+
+## Practical considerations
+
+As you develop a data package, there are a few things to keep in mind in order
+to make the release process easier.
+
+- We consider the first, non-stable (< 1.0.0) release to happen once there is
+  code that takes the first resource and its metadata from raw format into its
+  final resource state. The code must also be integrated into the `build.py`
+  file, so that the release can be reproducible.
+
+- Whenever you make a change, either directly to main or through a pull request,
+  you *always* need to make sure commits and pull requests are
+  [atomic](https://decisions.seedcase-project.org/why-atomic-commits-and-prs/).
+  This means that each commit or pull request contains only one *conceptual*
+  change. That's because the commit message (and consequently the changed files
+  in the commit) determine what type of release will be created. The commit
+  message will also be added to the changelog, so be aware of the message you
+  use.