Init Swamp by wpride · Pull Request #3 · canvas/sdk

wpride · 2024-08-06T01:03:05Z

No description provided.

vercel · 2024-08-06T01:03:09Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sdk	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 14, 2024 0:27am

drx · 2024-08-06T02:43:04Z

+RUN npm run build
+
+# Expose the port the app runs on
+EXPOSE 80


This just has to match ENV PORT=... above; can add comment

drx · 2024-08-06T02:44:44Z

+  "main": "src/index.ts",
+  "scripts": {
+    "build": "tsc",
+    "HACK-comment": "TZ=UTC is necessary because SQLite will not respect system timezone",


It seems "//" is "idiomatic" https://stackoverflow.com/questions/14221579/how-do-i-add-comments-to-package-json-for-npm-install

drx · 2024-08-06T02:46:00Z

+  const sqlEngine = new InMemoryDuckDb(store);
+  const swamp = new Swamp(store, metaStore, sqlEngine);
+  swamp.initialize();
+  swamp.addLoader(runPostgres, "postgres").withCadence(60 * 60);


(Adding naive questions from my first reading to reveal non-obvious names) What's cadence?

drx · 2024-08-06T02:47:04Z

@@ -0,0 +1,74 @@
+import axios from 'axios';


What is jsonplaceholder?

Just the name of this demo API; can kill it https://jsonplaceholder.typicode.com/

drx · 2024-08-06T02:48:24Z

+    return response.data;
+  } catch (error) {
+    throw error;
+  }


What's the purpose of catching and rethrowing the error? Lots of theses try catch blocks

Many such catches

drx · 2024-08-06T02:50:17Z

+  }
+}
+
+// eslint-disable-next-line @typescript-eslint/no-unused-vars


This lint seems unnecessary. Doesn't it have a setting where you can exclude things that start with _?

drx · 2024-08-06T02:51:50Z

+  if (cursor.next) {
+    return {
+      'page[cursor]': cursor.next,
+      'filter': `greater-than(datetime,${cursor.since})`,


Is there an id cursor available? This could fail if there are two records with the same datetime

drx · 2024-08-06T02:52:37Z

+      cursor: nextCursor,
+    };
+  } catch (e) {
+    console.log((e as any).response?.data);


cmd+f remove all the console.logs

drx · 2024-08-06T02:53:06Z

+
+async function getRecords(apiKey: string, _cursor: Cursor): Promise<{ cursor: Cursor, inserts: Inserts, hasMore: boolean }> {
+  const { data, included, cursor } = await getKlaviyoEvents(apiKey, _cursor);
+  const events = data.map((event: any) => {


Have you tried unknown instead of any?

drx · 2024-08-06T02:53:58Z

+    const flow = event.attributes.event_properties.$flow;
+    const messageId = event.attributes.event_properties.$message;
+
+    const campaignId = flow ? null : messageId;


What's flow?

drx · 2024-08-06T02:54:56Z

+  };
+}
+
+const INITIAL_CURSOR = { since: '2024-07-30T21:48:59Z' };


drx · 2024-08-06T02:55:08Z

+export async function runKlaviyo(_secrets: LoaderSecrets, cursor: Cursor | null): Promise<LoaderResponse<Cursor>> {
+  const secrets = Secrets.safeParse(_secrets);
+  if (secrets.error) {
+	  console.error(`could not parse secrets: ${secrets.error}`);


Strange formatting here

drx · 2024-08-06T02:55:41Z

+	  console.error(`could not parse secrets: ${secrets.error}`);
+	  throw new Error('failed to parse secrets');
+  }
+  const { inserts, hasMore, cursor: newCursor } = await getRecords(secrets.data.apiKey, cursor || INITIAL_CURSOR);


Suggested change

const { inserts, hasMore, cursor: newCursor } = await getRecords(secrets.data.apiKey, cursor || INITIAL_CURSOR);

const { inserts, hasMore, cursor: newCursor } = await getRecords(secrets.data.apiKey, cursor ?? INITIAL_CURSOR);

Could probably cmd+f the codebase for this

drx · 2024-08-06T22:28:44Z

+
+export type Inserts = Record<string, RecordBatch>;
+
+export const DuckDBRawType = z.enum(['VARCHAR', 'DOUBLE', 'DATE', 'TIMESTAMP', 'JSON', 'BOOLEAN', 'BIGINT', 'BLOB']);


why do we have duckdb things in core

drx · 2024-08-06T22:29:16Z

+export class LoaderExecutor<SecretsType, CursorType> extends BaseTransformer {
+  loader: Loader<SecretsType, CursorType>;
+
+  rateLimitMs?: number;


Suggested change

rateLimitMs?: number;

rateLimitMillis?: number;

drx · 2024-08-06T22:29:49Z

+    while (true) {
+      await this.maybeRun(false);
+      await sleep(1000 * 60 * 5);
+      continue;


Suggested change

continue;

drx · 2024-08-06T22:30:25Z

+
+export function sleep(ms: number) {
+  return new Promise((resolve) => {
+    setTimeout(resolve, ms);


node has a built-in for this

drx · 2024-08-06T22:33:07Z

+  initialize(messageBroker: MessageBroker): Promise<void>;
+}
+
+type DuckDbCredentials = S3Credentials & { type: 's3' };


drx · 2024-08-06T22:33:56Z

+        values.push(value);
+      }
+    }
+    columns.push('_swamp_updated_at');


_updated_at?

drx · 2024-08-06T22:34:29Z

+    if (this.uploadQueue.length === 0) {
+      return null;
+    }
+    return this.uploadQueue.shift()!;


what's up with this ! guy?

drx · 2024-08-06T22:34:58Z

+    }
+  }
+
+  withS3Credentials(credentials: S3Credentials): InMemoryDuckDb {


s/s3/storage/g?

drx · 2024-08-06T22:36:49Z

+  return filePath;
+}
+
+export function createS3SecretStatement(credentials: S3Credentials): string {


these s3 things seem like they could be in their own file

drx · 2024-08-06T22:38:21Z

+
+type IntermediateType = 'string' | 'number' | 'date' | 'datetime' | 'boolean' | 'bigint' | 'blob' | 'array' | 'object';
+
+function duckDBTypeForIntermediate(intermediate: IntermediateType): DuckDBRawType {


quack quack

Init swamp

e342fcd

rename and cleanup

51cfec6

drx reviewed Aug 6, 2024

View reviewed changes

Fix S3 syncing bug

ba5548b

vercel Bot deployed to Preview August 6, 2024 22:23 View deployment

drx reviewed Aug 6, 2024

View reviewed changes

Will Pride added 2 commits September 2, 2024 12:53

Add cursor and proxy

4cc2f06

Before writer refactor

c6058ca

Will Pride added 9 commits September 3, 2024 15:50

Refactor out writer and query code

2dde77c

Increment

b749414

Add writer

8e045d1

Refactor to create one file and restore from s3

8b44180

Clean up logs

dd3c3b5

This is going pretty good

a6e6bd4

Working

f867260

Remove log

3675be4

Add subscriptions

3faa1f1

vercel Bot deployed to Preview September 14, 2024 00:27 View deployment

	const { inserts, hasMore, cursor: newCursor } = await getRecords(secrets.data.apiKey, cursor \|\| INITIAL_CURSOR);
	const { inserts, hasMore, cursor: newCursor } = await getRecords(secrets.data.apiKey, cursor ?? INITIAL_CURSOR);


		export type Inserts = Record<string, RecordBatch>;

		export const DuckDBRawType = z.enum(['VARCHAR', 'DOUBLE', 'DATE', 'TIMESTAMP', 'JSON', 'BOOLEAN', 'BIGINT', 'BLOB']);


		type IntermediateType = 'string' \| 'number' \| 'date' \| 'datetime' \| 'boolean' \| 'bigint' \| 'blob' \| 'array' \| 'object';

		function duckDBTypeForIntermediate(intermediate: IntermediateType): DuckDBRawType {

Conversation

wpride commented Aug 6, 2024

Uh oh!

vercel Bot commented Aug 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

drx Aug 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vercel Bot commented Aug 6, 2024 •

edited

Loading

drx Aug 6, 2024 •

edited

Loading