diff --git a/articles/2021-05-24-commcare-events.md b/articles/2021-05-24-commcare-events.md index 5c7a2d5301c5..7f74705119ae 100644 --- a/articles/2021-05-24-commcare-events.md +++ b/articles/2021-05-24-commcare-events.md @@ -23,7 +23,7 @@ to Salesforce integration on your own. First, most people make use of CommCare's "Data Forwarding" feature to send form submissions and changes in cases (creation, update, closure, etc.) to OpenFn in real-time. You can read about that -[here](/adaptors/commcare#webhook-forward-cases-andor-forms-from-commcare-to-openfn-using-rest-service) +[here](/adaptors/commcare#webhook-or-data-forwarding-setup-commcare-to-openfn) but the key consideration at this planning stage is _when_ you'll be performing operations—`create(...)`, `update(...)`, `upsert(...)`, `query(...)`, `(bulk(...)`, etc.—in Salesforce and what data you'll have access to. @@ -369,8 +369,7 @@ this all up on your own are: 1. this site (docs.openfn.org), and 2. the [forum](https://community.openfn.org) (community.openfn.org) -Read through the -["What is an integration"](/documentation/tutorials/tutorial), +Read through the ["What is an integration"](/documentation/tutorials/tutorial), ["OpenFn Concepts"](/documentation/get-started/terminology), and ["Build"](/documentation/build/workflows) sections if you're a thorough, background-first kind of learner. If you crave snippets and sample job code, diff --git a/docs/build-for-developers/cli-collections.md b/docs/build-for-developers/cli-collections.md index fee0e3ae08c7..969197cc51c1 100644 --- a/docs/build-for-developers/cli-collections.md +++ b/docs/build-for-developers/cli-collections.md @@ -22,7 +22,7 @@ You can use the CLI to: - Experiment with query syntax to get the keys you need - Update mapping objects and lookup tables from local (or source-controlled) files -- Manually remove unneeded data +- Manually remove data :::tip @@ -59,10 +59,11 @@ We do this using Personal Access Tokens. See more details. One you have a PAT, you need to pass it in to the CLI. The easiest way to do -this is to set your `OPENFN_PAT` env var, which the CLI will use automatically. +this is to set an env var `OPENFN_API_KEY`, or use a `.env` file. The CLI will +automatically use this value for all requests. -If you're using multiple access tokens, you can pass `--token` to the CLI to -override the default. +You can also pass `--token` to the CLI to override the value loaded from your +environment. ```bash openfn collections get my-collection \* --token $MY_OPENFN_PAT @@ -78,16 +79,15 @@ all examples will work. ## Setting a server -By default, the CLI will point to our primary platform at -https://app.openfn.org. +By default, the CLI will point to our cloud app at https://app.openfn.org. If you're running from open source or using a different deployment, you'll also need to tell the CLI which Collections server to use. -You can do this by passing `--lightning` directly: +You can do this by passing `--endpoint` directly: ```bash -openfn collections get my-collection \* --lightning http://localhost:4000 +openfn collections get my-collection \* --endpoint http://localhost:4000 ``` Or by setting the `OPENFN_ENDPOINT` environment variable. @@ -103,6 +103,25 @@ openfn collections get my-collection \* --log debug ::: +## Project Name Uniqueness + +In Lightning releases prior to 2.17.0, collection names were globally unique. + +Since 2.17.0, collection names are scoped to a project, which means an OpenFn +instance can have multiple collections with the same name. + +Any requests to the collections API will attempt to resolve a collection name to +a single collection. But if there are conflicts, the server will return a 409 +error code. + +To resolve this, pass a project id + +```bash +openfn collections get --project_id 1d28c76c-e4ef-4e58-ac1e-464dc479946c +``` + +You can also set the project ID through an env var, or use the `-p` shortcut. + ## Fetching items You can fetch items from a Collection by passing a collection name and a key, or @@ -262,3 +281,25 @@ running the delete: ```bash openfn collections remove my-collection 2024* --dry-run ``` + +## Troubleshooting + +### Error 409: multiple collection names matched + +This means that you've requested a collection by name, but the server has +multiple collections with that name. + +You must scope your request to the correct project by including the project id +in your request. + +You can either pass this directly: + +``` +openfn collections get my-collection \* --project-id 1d28c76c-e4ef-4e58-ac1e-464dc479946c +``` + +Or set an env var (`.env` files are supported): + +``` +OPENFN_PROJECT_ID=1d28c76c-e4ef-4e58-ac1e-464dc479946c +``` diff --git a/docs/build-for-developers/cli-walkthrough.md b/docs/build-for-developers/cli-walkthrough.md index 1ddade3b940a..db0a512be33e 100644 --- a/docs/build-for-developers/cli-walkthrough.md +++ b/docs/build-for-developers/cli-walkthrough.md @@ -482,7 +482,7 @@ fn(state => { As you can see from our logs that helper function `dataValue` has a TypeError, to troubleshoot this you can go to the documentation for **dataValue -> -[docs.openfn.org/adaptors/packages/common-docs/#datavaluepath--operation](/adaptors/packages/common-docs/#datavaluepath--operation) +[docs.openfn.org/adaptors/packages/common-docs/#datavalue](/adaptors/packages/common-docs/#datavalue) ** According to the docs, dataValue takes a path as input, which is of the string @@ -557,9 +557,9 @@ each('posts[*]', state => { Notice how this code uses the `each` function, a helper function defined in -[language-common](/adaptors/packages/common-docs/#eachdatasource-operation--operation) -but accessed from this job that is using `language-http`. Most adaptors import -many functions from `language-common`. +[language-common](/adaptors/packages/common-docs/#each) but accessed from this +job that is using `language-http`. Most adaptors import many functions from +`language-common`. Run **openfn getPosts.js -a http -s tmp/state.json -o tmp/output.json** diff --git a/docs/deploy/options.md b/docs/deploy/options.md index bef0b3c53e2a..426fe8d172b0 100644 --- a/docs/deploy/options.md +++ b/docs/deploy/options.md @@ -46,11 +46,11 @@ local/government-managed servers, you might: preparations**: 1. Assess the **value of the solution** itself—is it solving the problems you thought it would? - 2. Asses your **data residency requirements**—do you need to run this + 2. Assess your **data residency requirements**—do you need to run this solution in country? 3. Assess your team’s technical **DevOps capacity**—how are other local deployments of DPGs going? - 4. Assess your countries computing, storage, and networking + 4. Assess your country's computing, storage, and networking infrastructure—what options\* are available for servers and network connectivity? 5. Determine if a **"zero-persistence"** cloud solution or a **locally diff --git a/docs/deploy/portability-v3.md b/docs/deploy/portability-v3.md new file mode 100644 index 000000000000..18a400bdcd58 --- /dev/null +++ b/docs/deploy/portability-v3.md @@ -0,0 +1,438 @@ +--- +title: Portability v3 (legacy) +--- + +The portability specification allows for the representations of entire workflow +projects "as code"; lets user moves between various deployment pathways (such as +cloud, local, hosted); and proposes a globally-applicable way of defining +workflow automation and system integration rules that might be applied across +workflow-engines/integration platforms across the sector. + +Nothing about the spec _must_ be specific to OpenFn or any one of our individual +products. We envision a future in which software built with Lightning, the +OpenFn Integration Toolkit, and entirely new and different integration/workflow +tools can adopt this specification. + +If you're interested in contributing to the specification, reach out to OpenFn +via the [community forum](https://community.openfn.org), write to us, or suggest +changes by submitting a pull request here. + +:::warning + +This is the legacy version of the OpenFn Portability spec. + +For the latest version, see [Portability](portability) + +::: + +## Projects "as code" + +Entire projects (groups of workflows with their associated triggers, edges, +credentials and jobs) can be represented as code. + +This improves the OpenFn developer experience by (a) allowing workflows to be +built and tested locally; (b) enabling project version control and an audit +trail of project changes; and (c) allowing users to port existing projects +between different instances (i.e., deployments) of Lightning. + +### Directory structure + +Many users keep OpenFn projects in git repositories, and this is a common +structure: + +``` +myProject/ +├── workflow-a/ +│ ├── job-1.js +│ ├── job-2.js +│ └── job-3.js +├── workflow-b/ +│ └── job-4.js +├── project.yaml +├── projectState.json +└── config.json +``` + +:::info Directory Structure + +There are commonly used 3 directory structure for OpenFn projects namely: +standard, production & test, and monorepo. + +::: + +### The project **_spec_** + +The project specification (or "spec") is often saved as a `project.yaml` file. +While most of the spec is written inline, many developers prefer to track their +job bodies in separate `.js` files and they then reference them with a relative +path. + +```yaml +name: openhie-project +description: Some sample +credentials: + jane-smith@test.com-HAPI-FHIR: + owner: jane-smith@test.com + name: HAPI FHIR +workflows: + OpenHIE-Workflow: + name: OpenHIE Workflow + jobs: + FHIR-standard-Data-with-change: + name: FHIR-standard-Data-with-change + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: + path: ./jobs/my-fancy-script.js + + Send-to-OpenHIM-to-route-to-SHR: + name: Send-to-OpenHIM-to-route-to-SHR + adaptor: '@openfn/language-http@latest' + enabled: true + credential: jane-smith@test.com-HAPI-FHIR + body: | + fn(state => { + console.log("hello github integration") + return state + }); + + Notify-CHW-upload-successful: + name: Notify-CHW-upload-successful + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: fn(state => state); + + Notify-CHW-upload-failed: + name: Notify-CHW-upload-failed + adaptor: '@openfn/language-http@latest' + enabled: true + credential: null + body: + path: ./jobs/notify-failure.js + + triggers: + webhook: + type: webhook + edges: + webhook->FHIR-standard-Data-with-change: + source_trigger: webhook + target_job: FHIR-standard-Data-with-change + condition: always + FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR: + source_job: FHIR-standard-Data-with-change + target_job: Send-to-OpenHIM-to-route-to-SHR + condition: on_job_success + Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful: + source_job: Send-to-OpenHIM-to-route-to-SHR + target_job: Notify-CHW-upload-successful + condition: on_job_success + Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed: + source_job: Send-to-OpenHIM-to-route-to-SHR + target_job: Notify-CHW-upload-failed + condition: on_job_failure +``` + +In this spec, you can see the different ways of defining a job's body: + +1. Inline body: Used in the `FHIR-standard-Data-with-change` and + `Send-to-OpenHIM-to-route-to-SHR` jobs. The body is directly written in the + YAML file. + +2. External file reference: Used in both `Notify-CHW-upload-successful` and + `Notify-CHW-upload-failed` jobs. The body is stored in separate files, + referenced by the path key. This allows for better organization of complex + job logic. + +When using file paths: + +- Paths are relative to the location of the `project.yaml` file. +- Ensure that the referenced files exist and contain valid job body code. +- This method is particularly useful for complex jobs or when you want to reuse + job bodies across different projects. + +### The project **_state_** + +The project state is a representation of a particular project as _on a specific +Lightning instance_. It is often saved as `projectState.json` and contains UUIDs +for resources on a particular Lightning deployment. + +```json +{ + "id": "8deff39d-8189-4bd7-9dc7-f9f08e7f2c60", + "name": "openhie-project", + "description": null, + "inserted_at": "2023-08-25T08:57:31", + "updated_at": "2023-08-25T08:57:31", + "scheduled_deletion": null, + "requires_mfa": false, + "project_credentials": { + "jane-smith@test.com-HAPI-FHIR": { + "id": "25f48989-d349-4eb8-99c3-923ebba5b116", + "name": "HAPI FHIR", + "owner": "jane-smith@test.com" + } + }, + "workflows": { + "OpenHIE-Workflow": { + "id": "27ae2937-0959-48b8-a597-b1646aae8c14", + "name": "OpenHIE Workflow", + "jobs": { + "Transform-data-to-FHIR-standard": { + "id": "e44f65bb-5038-4e17-8d93-b63cbe95254a", + "delete": true + }, + "Send-to-OpenHIM-to-route-to-SHR": { + "id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "name": "Send-to-OpenHIM-to-route-to-SHR", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "Notify-CHW-upload-successful": { + "id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b", + "name": "Notify-CHW-upload-successful", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "Notify-CHW-upload-failed": { + "id": "be85df30-0abd-4f8e-be17-501f67e18b8d", + "name": "Notify-CHW-upload-failed", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + }, + "FHIR-standard-Data": { + "id": "55016dda-42e3-4ee1-8a9c-24e3f23d42f1", + "delete": true + }, + "FHIR-standard-Data-with-change": { + "id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", + "name": "FHIR-standard-Data-with-change", + "adaptor": "@openfn/language-http@latest", + "body": "fn(state => state);\n", + "enabled": true + } + }, + "triggers": { + "webhook": { + "id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", + "type": "webhook" + } + }, + "edges": { + "webhook->Transform-data-to-FHIR-standard": { + "id": "b2c7407b-0ae9-4ca5-9d6b-ee624976fa54", + "delete": true + }, + "Transform-data-to-FHIR-standard->Send-to-OpenHIM-to-route-to-SHR": { + "id": "d22ed6f4-26a2-4c85-b261-cc110a6851e6", + "delete": true + }, + "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful": { + "id": "26c12f7f-7806-4008-87cd-6747998f95f4", + "condition": "on_job_success", + "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "source_trigger_id": null, + "target_job_id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b" + }, + "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed": { + "id": "0630ac96-4f67-4de7-8c3d-0bf3f89f80d9", + "condition": "on_job_failure", + "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", + "source_trigger_id": null, + "target_job_id": "be85df30-0abd-4f8e-be17-501f67e18b8d" + }, + "webhook->FHIR-standard-Data": { + "id": "5ce3a8ed-b9eb-464a-a2cd-ba55adc393c2", + "delete": true + }, + "FHIR-standard-Data->Send-to-OpenHIM-to-route-to-SHR": { + "id": "5f459cd9-2882-4a61-a2cc-ec45e58d4837", + "delete": true + }, + "webhook->FHIR-standard-Data-with-change": { + "id": "75e7f7d8-274b-410d-9600-730bbd535229", + "condition": "always", + "source_job_id": null, + "source_trigger_id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", + "target_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe" + }, + "FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR": { + "id": "1e5ba385-2c49-4241-8cd2-042c99a810ec", + "condition": "on_job_success", + "source_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", + "source_trigger_id": null, + "target_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32" + } + } + } + } +} +``` + +## Using the CLI interact with projects + +The project spec and project state can be used for a variety of reasons, e.g. +one could generate the state and spec as backups of the project or one could +generate these files and use them for auditing and record keeping, etc. The +OpenFn [CLI](https://github.com/OpenFn/kit/tree/main/packages/cli) comes with +commands that can be used to pull project configurations down from a running +Lightning server, and to deploy or push updates to existing projects on a +Lightning server. To learn more about automated version control via pull and +deploy, head over to our [Version Control](../manage-projects/link-to-gh.md) +docs. + +:::info Don't have the CLI yet? + +Install it by running `npm install -g @openfn/cli` + +::: + +Before using the CLI, configure it either by passing in environment variables: + +``` +OPENFN_ENDPOINT=https://app.openfn.org +OPENFN_API_KEY=yourSecretApiToken +``` + +Or through a `config.json` file: + +```json +{ + // Required, can be overridden or set with `OPENFN_API_KEY` env var + "apiKey": "***", + + // Optional: can be set using the -p, defaults to project.yaml + "specPath": "project.yaml", + + // Optional: can be set using -s, defaults to .state.json + "statePath": ".state.json", + + // Optional: defaults to OpenFn.org's API, can be overridden or set with + // `OPENFN_ENDPOINT` env var + "endpoint": "https://app.openfn.org" +} +``` + +More details on the CLI can be found +[here](https://github.com/OpenFn/kit/tree/main/packages/cli#basic-usage). + +### `openfn pull` to generate spec & state + +To generate the spec and state files for an existing project, use: + +```sh +openfn pull {YOUR-PROJECT-UUID} -c ./config.json +``` + +This command will save (or overwrite) a project spec and state file based on the +path you've set in your configuration. + +### `openfn deploy` to create new projects + +To deploy a new project to a Lightning instance from a project spec (without a +project state) file use: + +```sh +openfn deploy -c config.json +``` + +### `openfn deploy` to update existing projects + +With a valid project state defined in your `config.json`, the same +`openfn deploy` command will beam up your changes as described by a difference +between your project spec and what's found on the server. + +```sh +openfn deploy -c config.json +Checking https://demo.openfn.org/api/provision/4adf2644-ed4e-4f97-a24c-ab35b3cb1efa for existing project. +Project found. +[CLI] ♦ Changes: + { + workflows: [ + { + jobs: [ + { +- body: "fn(state => {\n console.log(\"ok\")\n return state\n});" ++ body: "fn(state => {\n console.log(\"some changes here!\")\n return state\n});\n" + } + ... + ... + ... + ] + } + ] + } + +? Deploy? yes +[CLI] ♦ Deployed. +``` + +## Getting Help with the cli + +The cli package comes with an inbuilt `help`. Adding `--help` to a command such +as `openfn deploy --help` will result in a help message describing the command +and the options available when using this command. See an example below + +```sh +openfn deploy --help +openfn deploy + +Deploy a project's config to a remote Lightning instance + +Options: + --version Show version number [boolean] + --help Show help [boolean] + -c, --config, --config-path The location of your config file [default: "./.config.json"] + --no-confirm Skip confirmation prompts (e.g. 'Are you sure?') [boolean] + --describe Downloads the project yaml from the specified instance [boolean] + -l, --log Set the log level [string] + --log-json Output all logs as JSON objects [boolean] + -p, --project-path The location of your project.yaml file [string] + -s, --state-path Path to the state file +``` + +## Troubleshooting + +This section covers solutions to some errors you might come across when using +OpenFn pull or deploy in your projects. + +### Extraneous Workflow ID + +#### Description + +This error occurs when you run `openfn deploy` and there is a mismatch between +between IDs of workflows in your projectSpec and your OpenFn instance. When this +occurs, the error will be written out in an error object as shown below: + +``` +[CLI] ✘ Failed to deploy project openfn-data-buffers-prototype: +{ + "errors": { + "workflows": { + "1-ingest-messages": { + "base": [ + "extraneous parameters: workflow_id" + ] + }, + "2-calculate-indicators": { + "base": [ + "extraneous parameters: workflow_id" + ] + } + } + } +``` + +#### Solution + +Run `openfn pull` to update your local instance and keep IDs in sync, +incorporate your changes and run `openfn deploy` again. + +## Other Versions + +- [Portability Spec v2](portability-versions#v2) +- [Portability Spec v1](portability-versions#v1) diff --git a/docs/deploy/portability-versions.md b/docs/deploy/portability-versions.md index 8c7d5cd4dd24..4a3bf7867526 100644 --- a/docs/deploy/portability-versions.md +++ b/docs/deploy/portability-versions.md @@ -2,12 +2,20 @@ title: Versions of the Portability Proposal --- -OpenFn is currently designing a portable project configuration schema that can -be used to import or export projects between OpenFn/platform and OpenFn/engine. +Our commitment to portability hasn't changed within OpenFn's lifetime - but our +approach and implementation of this commitment have taken many forms. -## [Current](/documentation/deploy/portability#the-project-spec) +This document provides a reference to older versions of the specification. -[See the current specification here.](/documentation/deploy/portability#the-project-spec) +## v3 + +The v3 standard was created for the v2 platform and is linked to the Lightning +project. + +V3 uses the legacy CLI deploy commands and protocols. It is still fully supported +by the app and CLI, but is being phased out as of May 2026. + +[See the v3 specification here](/documentation/deploy/portability-v3) ## v2 diff --git a/docs/deploy/portability.md b/docs/deploy/portability.md index cc6d1474526c..275e0a6f12eb 100644 --- a/docs/deploy/portability.md +++ b/docs/deploy/portability.md @@ -2,440 +2,183 @@ title: Portability --- -## Intent - -The portability specification allows for the representations of entire workflow -projects "as code", lets user move between various deployment pathways (cloud, -local, DIY, etc.) and proposes a globally-applicable way of **_specifying -workflow automation_** and **_systems integration_** that might be applied -across workflow-engines/integration platforms across the sector. Nothing about -the spec _must_ be specific to OpenFn or any one of our individual products. We -envision a future in which software built with Lightning, the OpenFn Integration -Toolkit, and entirely new and different integration/workflow tools can adopt -this specification. +The Portability Specification is an idea right at the heart of OpenFn Projects. +It is both a technical standard and an ongoing commitment. It ensures that code +written in an OpenFn application can be: -If you're interested in contributing to the specification, reach out to OpenFn -via the [community forum](https://community.openfn.org), write to us, or suggest -changes by submitting a pull request here. - -## Projects "as code" +- Deployed to another OpenFn instance (critical for production services running + in-country) +- Executed on a local machine (great news for developers building workflows or + adaptors) +- Ejected from OpenFn entirely and executed through a generic JavaScript runtime -Entire projects (groups of workflows with their associated triggers, edges, -credentials and jobs) can be represented as code. This improves the OpenFn -developer experience by (a) allowing workflows to be built and tested locally; -(b) enabling project version control and an audit trail of project changes; and -(c) allowing users to port existing projects between different instances (i.e., -deployments) of Lightning. +This manifesto drives the core functionality of OpenFn Sync, CLI Deploy, Sandbox +merging, and Project export/import from the app. -### Directory structure - -Many users keep OpenFn projects in git repositories, and this is a common -structure: - -``` -myProject/ -├── workflow-a/ -│ ├── job-1.js -│ ├── job-2.js -│ └── job-3.js -├── workflow-b/ -│ └── job-4.js -├── project.yaml -├── projectState.json -└── config.json -``` +:::info Legacy Portability Specifications -:::info Directory Structure +Our commitment to portability hasn't changed within OpenFn's lifetime - but our +approach and implementation of this commitment have taken many forms. -There are 3 commonly used directory structures for OpenFn projects, namely: -standard, production & test, and monorepo. To learn more, please see the OpenFn -[GitHub configuration documentation](/documentation/link-to-GitHub#structuring-your-github-repository). +This document describes the latest Portability Specification, published in +May 2026. For older specifications, see +[Portability Versions](portability-versions) ::: -### The project **_spec_** - -The project specification (or "spec") is often saved as a `project.yaml` file. -While most of the spec is written inline, many developers prefer to track their -job bodies in separate `.js` files and they then reference them with a relative -path. - -```yaml -name: openhie-project -description: Some sample -credentials: - jane-smith@test.com-HAPI-FHIR: - owner: jane-smith@test.com - name: HAPI FHIR -workflows: - OpenHIE-Workflow: - name: OpenHIE Workflow - jobs: - FHIR-standard-Data-with-change: - name: FHIR-standard-Data-with-change - adaptor: '@openfn/language-http@latest' - enabled: true - credential: null - body: - path: ./jobs/my-fancy-script.js - - Send-to-OpenHIM-to-route-to-SHR: - name: Send-to-OpenHIM-to-route-to-SHR - adaptor: '@openfn/language-http@latest' - enabled: true - credential: jane-smith@test.com-HAPI-FHIR - body: | - fn(state => { - console.log("hello github integration") - return state - }); - - Notify-CHW-upload-successful: - name: Notify-CHW-upload-successful - adaptor: '@openfn/language-http@latest' - enabled: true - credential: null - body: fn(state => state); - - Notify-CHW-upload-failed: - name: Notify-CHW-upload-failed - adaptor: '@openfn/language-http@latest' - enabled: true - credential: null - body: - path: ./jobs/notify-failure.js - - triggers: - webhook: - type: webhook - edges: - webhook->FHIR-standard-Data-with-change: - source_trigger: webhook - target_job: FHIR-standard-Data-with-change - condition: always - FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR: - source_job: FHIR-standard-Data-with-change - target_job: Send-to-OpenHIM-to-route-to-SHR - condition: on_job_success - Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful: - source_job: Send-to-OpenHIM-to-route-to-SHR - target_job: Notify-CHW-upload-successful - condition: on_job_success - Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed: - source_job: Send-to-OpenHIM-to-route-to-SHR - target_job: Notify-CHW-upload-failed - condition: on_job_failure -``` - -In this spec, you can see the different ways of defining a job's body: - -1. Inline body: Used in the `FHIR-standard-Data-with-change` and - `Send-to-OpenHIM-to-route-to-SHR` jobs. The body is directly written in the - YAML file. - -2. External file reference: Used in both `Notify-CHW-upload-successful` and - `Notify-CHW-upload-failed` jobs. The body is stored in separate files, - referenced by the path key. This allows for better organization of complex - job logic. - -When using file paths: - -- Paths are relative to the location of the `project.yaml` file. -- Ensure that the referenced files exist and contain valid job body code. -- This method is particularly useful for complex jobs or when you want to reuse - job bodies across different projects. - -### The project **_state_** - -The project state is a representation of a particular project as _on a specific -Lightning instance_. It is often saved as `projectState.json` and contains UUIDs -for resources on a particular Lightning deployment. - -```json -{ - "id": "8deff39d-8189-4bd7-9dc7-f9f08e7f2c60", - "name": "openhie-project", - "description": null, - "inserted_at": "2023-08-25T08:57:31", - "updated_at": "2023-08-25T08:57:31", - "scheduled_deletion": null, - "requires_mfa": false, - "project_credentials": { - "jane-smith@test.com-HAPI-FHIR": { - "id": "25f48989-d349-4eb8-99c3-923ebba5b116", - "name": "HAPI FHIR", - "owner": "jane-smith@test.com" - } - }, - "workflows": { - "OpenHIE-Workflow": { - "id": "27ae2937-0959-48b8-a597-b1646aae8c14", - "name": "OpenHIE Workflow", - "jobs": { - "Transform-data-to-FHIR-standard": { - "id": "e44f65bb-5038-4e17-8d93-b63cbe95254a", - "delete": true - }, - "Send-to-OpenHIM-to-route-to-SHR": { - "id": "977b87ff-f347-42b5-832f-6ae2ca726f32", - "name": "Send-to-OpenHIM-to-route-to-SHR", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - }, - "Notify-CHW-upload-successful": { - "id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b", - "name": "Notify-CHW-upload-successful", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - }, - "Notify-CHW-upload-failed": { - "id": "be85df30-0abd-4f8e-be17-501f67e18b8d", - "name": "Notify-CHW-upload-failed", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - }, - "FHIR-standard-Data": { - "id": "55016dda-42e3-4ee1-8a9c-24e3f23d42f1", - "delete": true - }, - "FHIR-standard-Data-with-change": { - "id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", - "name": "FHIR-standard-Data-with-change", - "adaptor": "@openfn/language-http@latest", - "body": "fn(state => state);\n", - "enabled": true - } - }, - "triggers": { - "webhook": { - "id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", - "type": "webhook" - } - }, - "edges": { - "webhook->Transform-data-to-FHIR-standard": { - "id": "b2c7407b-0ae9-4ca5-9d6b-ee624976fa54", - "delete": true - }, - "Transform-data-to-FHIR-standard->Send-to-OpenHIM-to-route-to-SHR": { - "id": "d22ed6f4-26a2-4c85-b261-cc110a6851e6", - "delete": true - }, - "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-successful": { - "id": "26c12f7f-7806-4008-87cd-6747998f95f4", - "condition": "on_job_success", - "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", - "source_trigger_id": null, - "target_job_id": "86b743a3-fd00-4629-b9fb-d5f38fb56d0b" - }, - "Send-to-OpenHIM-to-route-to-SHR->Notify-CHW-upload-failed": { - "id": "0630ac96-4f67-4de7-8c3d-0bf3f89f80d9", - "condition": "on_job_failure", - "source_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32", - "source_trigger_id": null, - "target_job_id": "be85df30-0abd-4f8e-be17-501f67e18b8d" - }, - "webhook->FHIR-standard-Data": { - "id": "5ce3a8ed-b9eb-464a-a2cd-ba55adc393c2", - "delete": true - }, - "FHIR-standard-Data->Send-to-OpenHIM-to-route-to-SHR": { - "id": "5f459cd9-2882-4a61-a2cc-ec45e58d4837", - "delete": true - }, - "webhook->FHIR-standard-Data-with-change": { - "id": "75e7f7d8-274b-410d-9600-730bbd535229", - "condition": "always", - "source_job_id": null, - "source_trigger_id": "530cde0b-0de4-4f68-8834-0a4356a2fe53", - "target_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe" - }, - "FHIR-standard-Data-with-change->Send-to-OpenHIM-to-route-to-SHR": { - "id": "1e5ba385-2c49-4241-8cd2-042c99a810ec", - "condition": "on_job_success", - "source_job_id": "28dd0846-a6ae-40c0-8ab4-3e0a6b487afe", - "source_trigger_id": null, - "target_job_id": "977b87ff-f347-42b5-832f-6ae2ca726f32" - } - } - } - } -} -``` - -## Using the CLI interact with projects - -:::warning - -This document describes the legacy OpenFn Sync format used by older versions of -the CLI and GitHub Sync. +Nothing about the spec _must_ be specific to OpenFn or any one of our individual +products. We envision a future in which software built with Lightning, the +OpenFn Integration Toolkit, and entirely new and different integration/workflow +tools can adopt this specification. -See the [CLI Sync](/documentation/sync) docs for the best experience syncing and -deploying projects. - -::: - -The project spec and project state can be used for a variety of reasons, e.g. -one could generate the state and spec as backups of the project or one could -generate these files and use them for auditing and record keeping, etc. The -OpenFn [CLI](https://github.com/OpenFn/kit/tree/main/packages/cli) comes with -commands that can be used to pull project configurations down from a running -Lightning server, and to deploy or push updates to existing projects on a -Lightning server. To learn more about automated version control via pull and -deploy, head over to our [Version Control](../manage-projects/link-to-gh.md) -docs. - -:::info Don't have the CLI yet? - -Install it by running `npm install -g @openfn/cli` - -::: - -Before using the CLI, configure it either by passing in environment variables: +If you're interested in contributing to the specification, reach out to OpenFn +via the [community forum](https://community.openfn.org), write to us, or suggest +changes by submitting a pull request here. -``` -OPENFN_ENDPOINT=https://app.openfn.org -OPENFN_API_KEY=yourSecretApiToken -``` +## Projects As Code -Or through a `config.json` file: +A core tenet of OpenFn Projects is that they can be represented as code, on a +file system or a git branch. -```json -{ - // Required, can be overridden or set with `OPENFN_API_KEY` env var - "apiKey": "***", +This improves the OpenFn developer experience by: - // Optional: can be set using the -p, defaults to project.yaml - "specPath": "project.yaml", +1. Allowing workflows to be built and tested locally +2. Enabling project version control and an audit trail of project changes +3. Allowing users to port existing projects between different instances (i.e., + deployments) of Lightning. - // Optional: can be set using -s, defaults to .state.json - "statePath": ".state.json", +## Project Spec - // Optional: defaults to OpenFn.org's API, can be overridden or set with - // `OPENFN_ENDPOINT` env var - "endpoint": "https://app.openfn.org" -} -``` +The unit of portability - the thing that encodes a Project and allows it to be +shared, synced, deployed and edited - is called a Project Spec. It is an +abstract definition of a project, a blueprint which can be deployed to many +places. -More details on the CLI can be found -[here](https://github.com/OpenFn/kit/tree/main/packages/cli#basic-usage). +This structure defines a set of workflows, and for each workflow, its core +configuration, and the sequence of steps which it executes. We usually represent +this structure as YAML, because it's convenient for humans and machines, but it +can be represented in any text format. -### `openfn pull` to generate spec & state +With a copy of a project spec, users can: -To generate the spec and state files for an existing project, use: +- Import a project into an OpenFn app instance +- Execute workflows locally with the CLI +- Deploy a project to an OpenFn app instance +- Merge sandbox projects locally -```sh -openfn pull {YOUR-PROJECT-UUID} -c ./config.json -``` +Keys are regularly added to this structure as new features are introduced. We +expect and ensure that these keys are supported in all applications of the spec. -This command will save (or overwrite) a project spec and state file based on the -path you've set in your configuration. +Project specs can be exported from the app via the Settings page. -### `openfn deploy` to create new projects +Workflows can also be interchanged independently, using the same spec. So you +can import a Workflow to an existing project, or execute it locally without +cloning a whole project. -To deploy a new project to a Lightning instance from a project spec (without a -project state) file use: +## Spec Example -```sh -openfn deploy -c config.json -``` +Here is an example Project spec in YAML format: -### `openfn deploy` to update existing projects - -With a valid project state defined in your `config.json`, the same -`openfn deploy` command will beam up your changes as described by a difference -between your project spec and what's found on the server. - -```sh -openfn deploy -c config.json -Checking https://demo.openfn.org/api/provision/4adf2644-ed4e-4f97-a24c-ab35b3cb1efa for existing project. -Project found. -[CLI] ♦ Changes: - { - workflows: [ - { - jobs: [ - { -- body: "fn(state => {\n console.log(\"ok\")\n return state\n});" -+ body: "fn(state => {\n console.log(\"some changes here!\")\n return state\n});\n" - } - ... - ... - ... - ] - } - ] - } - -? Deploy? yes -[CLI] ♦ Deployed. +```yaml +id: portability-example +name: Portability Example +collections: + - cache +credentials: + - name: local login + owner: editor@openfn.org +workflows: + - name: Event-based workflow + steps: + - id: transform-data + name: Transform data + expression: fn(s => s) + adaptor: '@openfn/language-common@latest' + - id: webhook + type: webhook + webhook_reply: before_start + enabled: true + next: + transform-data: + disabled: false + condition: always + id: event-based-workflow + start: webhook + - name: Scheduled workflow + steps: + - id: common + name: Common + expression: fn(s => s) + adaptor: '@openfn/language-common@3.3.1' + - id: cron + type: cron + enabled: true + cron_expression: 00 00 * * 1-5 + cron_cursor_job_id: get-data + next: + get-data: + disabled: false + condition: always + - id: get-data + name: Get data + expression: fn(s => s) + adaptor: '@openfn/language-http@latest' + configuration: editor@openfn.org|local login + next: + throw-error: + disabled: false + condition: on_job_failure + common: + disabled: false + condition: '!state.error' + label: sometimes + never: + disabled: true + condition: on_job_success + - id: never + name: never + expression: fn(s => s) + adaptor: '@openfn/language-http@7.2.10' + - id: throw-error + name: throw error + expression: fn(s => s) + adaptor: '@openfn/language-common@3.3.1' + id: scheduled-workflow + start: cron ``` -## Getting Help with the cli +The latest schema for a project spec file is defined in TypeScript +[here](https://github.com/OpenFn/kit/blob/main/packages/lexicon/portability.d.ts) -The cli package comes with an inbuilt `help`. Adding `--help` to a command such -as `openfn deploy --help` will result in a help message describing the command -and the options available when using this command. See an example below +## Syncing Projects -```sh -openfn deploy --help -openfn deploy +For more details about how a project can be deployed, executed, pulled and +edited, see our extensive documentation on [CLI Sync](/documentation/sync). -Deploy a project's config to a remote Lightning instance +## Linked Resources -Options: - --version Show version number [boolean] - --help Show help [boolean] - -c, --config, --config-path The location of your config file [default: "./.config.json"] - --no-confirm Skip confirmation prompts (e.g. 'Are you sure?') [boolean] - --describe Downloads the project yaml from the specified instance [boolean] - -l, --log Set the log level [string] - --log-json Output all logs as JSON objects [boolean] - -p, --project-path The location of your project.yaml file [string] - -s, --state-path Path to the state file -``` - -## Troubleshooting - -This section covers solutions to some errors you might come across when using -OpenFn pull or deploy in your projects. - -### Extraneous Workflow ID +While we designed Projects with portability in mind - some features are +intrinsically NOT portable. -#### Description +For example, credentials contain highly secure tokens, which by design and +nature should be very hard to extract from the OpenFn platform. So credentials +aren't really portable. When exporting a project, sensitive credentials should +not be included in that exported, plain-text document. -This error occurs when you run `openfn deploy` and there is a mismatch between -IDs of workflows in your projectSpec and your OpenFn instance. When this -occurs, the error will be written out in an error object as shown below: - -``` -[CLI] ✘ Failed to deploy project openfn-data-buffers-prototype: -{ - "errors": { - "workflows": { - "1-ingest-messages": { - "base": [ - "extraneous parameters: workflow_id" - ] - }, - "2-calculate-indicators": { - "base": [ - "extraneous parameters: workflow_id" - ] - } - } - } -``` +Similarly, collections are a feature which is tied closely to a specific +deployment of an OpenFn platform. The data of collections is not covered by the +portability spec (although with the right permissions data can be synced between +collections). -#### Solution +These kinds of non-portable resources are not part of a project, but they are +LINKED to a project. -Run `openfn pull` to update your local instance and keep IDs in sync, -incorporate your changes and run `openfn deploy` again. +Usually resources are linked by name. Credentials and collections just declare a +dependency on a thing with a given name, which must be resolved at runtime. The +CLI has tools to do that, and when deploying to a target instance the instance +may need to be pre-configured to have matching resources. -## Other Versions +## Legacy Portability Specifications -- [Portability Spec v2](portability-versions#v2) -- [Portability Spec v1](portability-versions#v1) +For older versions of our approach, see +[Portability Versions](portability-versions) diff --git a/docs/deploy/requirements.md b/docs/deploy/requirements.md index 3c24072d9afc..dee382b455f9 100644 --- a/docs/deploy/requirements.md +++ b/docs/deploy/requirements.md @@ -63,7 +63,7 @@ independently of the Javascript worker node pool & pods. the following specs will help prevent unwanted downtime. 1. **GKE requests:** cpu@ "500m", memory@ "1024Mi" 2. **GKE limits:** memory@ "2560Mi" -2. For a simple non-Kubernetes/HA deployments, the minimum recommended machines +2. For simple non-Kubernetes/HA deployments, the minimum recommended machines are: - **Application machine:** 2 vCPU (roughly a single core of a 2.6 GHz Intel Xeon E5) with 3.75 GB memory and 15 gb of storage for the application @@ -75,8 +75,9 @@ independently of the Javascript worker node pool & pods. days (if any) of message data you’d like to store on the app itself and cannot be determined without estimates for message/run throughput. If scaling physical storage is not difficult for your particular deployment, - start at 40gb. 3. A Postgres (at least v14.2) instance (as we run this on a - _separate server_) from the application for greater stability. + start at 40gb. + 1. A Postgres (at least v14.2) instance, run on a _separate server_ from + the application for greater stability. 3. If both the application and database are hosted on the same machine (which is not recommended) that machine should have roughly the sum of the requirements above. diff --git a/docs/jobs/job-writing-guide.md b/docs/jobs/job-writing-guide.md index e45a186770c1..83d08435bcb6 100644 --- a/docs/jobs/job-writing-guide.md +++ b/docs/jobs/job-writing-guide.md @@ -13,7 +13,9 @@ of key patterns in the OpenFn ecosystem which it is important to learn. :::tip -If you're writing jobs on the platform app (Lightning), you can use the [AI Assistant](/documentation/build/ai-assistant) to help you. You'll find it in the Inspector. +If you're writing jobs on the platform app (Lightning), you can use the +[AI Assistant](/documentation/build/ai-assistant) to help you. You'll find it in +the Inspector. ::: @@ -101,7 +103,7 @@ minute. :::caution As of July 2024, callbacks are going to be phased out of the adaptor APIs. See -[Promise-like Operations](#promise-like-operations) for tips on how to use +[Promise-like Operations](#operations-and-promises) for tips on how to use callbacks with adaptors APIs that don't explicitly support them. ::: @@ -1159,27 +1161,30 @@ fn(state => { ## Referencing credential secrets in your job code -If you want to reference any credential secrets in your job code, you can still map keys from your `state.configuration`. See example below that will dynamically map the username and password from your `configuration` (or "credential" if using the app) into your http request body. +If you want to reference any credential secrets in your job code, you can still +map keys from your `state.configuration`. See example below that will +dynamically map the username and password from your `configuration` (or +"credential" if using the app) into your http request body. ```js post('/api/v1/auth/login', { - body: { + body: { username: $.configuration.username, //map the UN from credential - password: $.configuration.password //map the PW from credential - }, - headers: {'content-type': 'application/json'}, - }) + password: $.configuration.password, //map the PW from credential + }, + headers: { 'content-type': 'application/json' }, +}); ``` :::info OpenFn scrubs Configuration & Functions from final state OpenFn will automatically scrub the `configuration` key and any functions from -your final state, as well as from logs if running workflows on the app. This is to help ensure that your credential secrets are kept secure and won't be leaked into History. +your final state, as well as from logs if running workflows on the app. This is +to help ensure that your credential secrets are kept secure and won't be leaked +into History. ::: - -