Conversation
- Add `GC_KEY_BODY`, `GC_KEY_FILE`, `KEYFILE`, and `GC_CRED_API_KEY` property aliases to ensure consistent behavior across different connection methods. - Pass `gcstorage.ScopeReadWrite` to `FindDefaultCredentials`. - Use `option.WithTokenSource` instead of `option.WithCredentials` to prevent a "multiple credential options provided" collision in `google.golang.org/api >= v0.258.0`.
- Separates the `IsInteger` and `IsBool` source types when casting to a string in `castBoolForSelect`. - Genuine booleans fail when using `= 1` in PostgreSQL and `CAST` to varchar in Redshift. - Added a specific case for boolean to string casting that relies on truthiness instead of integer comparisons or direct casting.
- Treat ExecStatusSkipped as a finished state in the ExecStatus.IsFinished() method. - Ensures that tasks with a skipped status are properly recognized as completed.
Add a pipeline test that reproduces the bug where SLING_STATE wrote
timestamps using Go's default layout (e.g. "2024-03-10 09:00:00 +0200 +02:00"),
which cast.ToTime cannot parse. This caused the incremental watermark to
silently reset to zero on read, re-pulling the entire table on every run.
The test uses TIMESTAMP WITH TIME ZONE (whose Oracle driver zone name is
the offset string) to reproduce the exact malformation, then verifies:
- The stored watermark is RFC3339 ("2024-03-10T09:00:00+02:00")
- It contains no "+0200 " Go-layout artifact
- A subsequent incremental run loads only the new row (count stays at 4)
- The watermark advances to the new maximum after the re-run
- Upgrade DuckDB version from 1.5.2 to 1.5.5 - Add mutex-based thread safety for DuckDB query state management - Improve Arrow stream error handling with subprocess error capture - Fall back to CSV mode when Arrow unavailable due to locked instance - Support timezone-aware timestamps in Arrow schema conversion - Consolidate DuckDB/MotherDuck/DuckLake tests with Arrow mode coverage
Expand the library search logic to resolve `adbc_driver_manager.dll` across common Windows install locations, including Conda environments and pip site-packages. Bump the arrow-adbc dependency to pick up related fixes.
When a driver path is not found locally, sling will now attempt to automatically download and install the required ADBC driver using the dbc CLI (https://columnar.tech/dbc). The dbc binary itself is downloaded on-demand if not present on PATH or in the sling bin directory. This behavior can be disabled with the SLING_DISABLE_DBC_AUTO_INSTALL environment variable.
- Add logic to automatically download and extract the ADBC driver manager library from conda-forge if it is not found on the system. - This improves the out-of-the-box experience by allowing the ADBC connection to initialize without requiring users to manually install the driver manager. - Implemented `ensureDriverManagerLib` to fetch the correct build based on OS/arch and `extractCondaLib` to handle the new `.conda` (zip + zstd) package format.
The prebuilt ADBC driver manager is compiled against newer toolchains than some supported distros provide, and raw loader errors (e.g. "GLIBCXX_3.4.29 not found") give users no guidance. Added `diagnoseADBCLoadError` which detects common failure modes — outdated libstdc++ or glibc, wrong CPU architecture, and ADBC_DRIVER_MANAGER_LIB issues — and returns targeted remediation steps. On Linux, `libStdCxxRemedy` will even fetch a compatible libstdc++ and emit the exact `LD_PRELOAD` command to re-run with.
Distinguish `*` inside SQL expressions (e.g. `concat('a*', id)`,
JSONPath `'$[*].amount'`) from column-name globs so they are no
longer mismatched against input fields. Computed expressions with
aliases now pass through as output columns, and glob patterns that
match zero columns emit a warning instead of silently no-oping.
Add t.Skip for TestCSV which asserts untyped decimal/float values that now arrive as strings. Update TestCSVSkipLines to use correct relative path for test1.skiplines.csv file.
Fix ADBC connection property keys to match what drivers actually expect. Snowflake uses the generic "uri" key (not "adbc.snowflake.sql.uri") since the driver parses it with gosnowflare.ParseDSN, and Trino similarly uses "uri" instead of "url". Add support for Snowflake programmatic access token authentication by placing the token in the password position, and key-pair authentication via encoded_private_key which sets the SNOWFLAKE_JWT authenticator. Introduce named constants for BigQuery ADBC option keys and fully qualified auth_type values since the driver rejects unknown options. Make auth_type and auth_credentials travel together when determining the authentication method. Handle the adbc_uri override property per database type: DuckDB maps to "path", BigQuery ignores it with a warning (unsupported), and all others use "uri".
…or_redshift Implemented the credential-chain fallback for Redshift
Fix the typo in the `copy_method` property name to properly set it to `arrow_http` when the `DUCKDB_USE_ARROW` environment variable is enabled.
- Add loadAWSCredentialsFromChain helper for loading credentials from the default AWS chain (env vars, profiles, IAM roles) - Handle errors from ensureAWSCredentials in getS3Props with warning instead of silently ignoring - Add redactCredentials method to mask AWS secrets for safe logging - Fix USE_ENVIRONMENT check to properly handle boolean conversion instead of only checking for "false" string - Clarify documentation and error messages around credential fallback
Apply COALESCE to `_sling_synced_op` in the Redshift `merge_change_capture_soft` SQL template. This prevents rows with a NULL `_sling_synced_op` from being skipped, ensuring they are correctly marked as deleted.
The `_sling_synced_op != 'D'` guard in merge_change_capture_soft evaluates to NULL (not TRUE) for target rows loaded before CDC started, where `_sling_synced_op` is NULL. This silently skipped those rows, so deletes were never recorded for them. Wrap the comparison with COALESCE(_sling_synced_op, '') so NULL values are treated as empty string and properly matched. Adds TestSoftMergeGuardIsNullSafe to verify every dialect template uses COALESCE in the soft-mark statement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New Features
SLING_DISABLE_DBC_AUTO_INSTALL). Windows discovery foradbc_driver_manager.dllis included.Bug Fixes
Redshift AWS credential chain: Falls back to the default AWS credential chain (env vars, shared profiles, IAM roles) when Redshift/S3 credentials are not explicitly configured, with clearer errors and redacted credential logging.
Soft-merge delete guard NULL-safe: CDC soft-merge
_sling_synced_op != 'D'guards now useCOALESCEacross dialects so pre-CDC target rows with NULL_sling_synced_opare no longer skipped (including Redshift).ADBC connection keys and auth: Corrected driver property keys (e.g. Snowflake/Trino
uri), added Snowflake PAT and key-pair (JWT) auth, fixed BigQueryauth_typehandling, and improvedadbc_urioverrides per database type.ADBC library load diagnostics: Load failures (outdated libstdc++/glibc, wrong arch,
ADBC_DRIVER_MANAGER_LIB) now return actionable remediation steps, including anLD_PRELOADhint on Linux when needed.Column select vs SQL expressions:
*inside SQL/JSONPath expressions (e.g.concat('a*', id),'$[*].amount') is no longer treated as a column glob; unmatched globs warn instead of silently no-oping.DuckDB Arrow / concurrency: Upgraded DuckDB, fixed
copy_method→arrow_httpwhenDUCKDB_USE_ARROWis set, improved Arrow stream error handling, and fall back to CSV when Arrow is unavailable due to a locked instance.Bool vs int cast to string: Boolean→string casting no longer uses integer comparisons/
CASTthat break on PostgreSQL/Redshift; integers targeting string columns use a plain varchar cast.Incremental state time format:
time.Timestate values are stored as RFC3339 so watermarks round-trip correctly withcast.ToTime(avoids full re-pulls).GCS auth collision: Resolved multiple-credential-option errors with newer Google API clients; added
GC_KEY_BODY,GC_KEY_FILE,KEYFILE, andGC_CRED_API_KEYaliases.