Skip to main content

Postgres

Airbyte's certified Postgres connector offers the following features:

  • Replicate data from tables, views and materilized views. Other data objects won't be replicated to the destination like indexes, permissions.
  • Multiple methods of keeping your data fresh, including Change Data Capture (CDC) and replication using the xmin system column.
  • All available sync modes, providing flexibility in how data is delivered to your destination.
  • Reliable replication at any table size with checkpointing and chunking of database reads.

The contents below include a 'Quick Start' guide, advanced setup steps, and reference information (data type mapping, and changelogs). See here to troubleshooting issues with the Postgres connector.

Please note the required minimum platform version is v0.58.0 for this connector.

Quick Start

Here is an outline of the minimum required steps to configure a Postgres connector:

  1. Create a dedicated read-only Postgres user with permissions for replicating data
  2. Create a new Postgres source in the Airbyte UI using xmin system column
  3. (Airbyte Cloud Only) Allow inbound traffic from Airbyte IPs

Once this is complete, you will be able to select Postgres as a source for replicating data.

Step 1: Create a dedicated read-only Postgres user

These steps create a dedicated read-only user for replicating data. Alternatively, you can use an existing Postgres user in your database.

The following commands will create a new user:

CREATE USER <user_name> PASSWORD 'your_password_here';

Now, provide this user with read-only access to relevant schemas and tables. Re-run this command for each schema you expect to replicate data from:

GRANT USAGE ON SCHEMA <schema_name> TO <user_name>;
GRANT SELECT ON ALL TABLES IN SCHEMA <schema_name> TO <user_name>;
ALTER DEFAULT PRIVILEGES IN SCHEMA <schema_name> GRANT SELECT ON TABLES TO <user_name>;

Step 2: Create a new Postgres source in Airbyte UI

From your Airbyte Cloud or Airbyte Open Source account, select Sources from the left navigation bar, search for Postgres, then create a new Postgres source.

To fill out the required information:

  1. Enter the hostname, port number, and name for your Postgres database.
  1. You may optionally opt to list each of the schemas you want to sync. These are case-sensitive, and multiple schemas may be entered. By default, public is the only selected schema.
  1. Enter the username and password you created in Step 1.
  2. Select an SSL mode. You will most frequently choose require or verify-ca. Both of these always require encryption. verify-ca also requires certificates from your Postgres database.
  3. Select Standard (xmin) from available replication methods. This uses the xmin system column to reliably replicate data from your database.
    1. If your database is particularly large (> 500 GB), you will benefit from configuring your Postgres source using logical replication (CDC).

Step 3: (Airbyte Cloud Only) Allow inbound traffic from Airbyte IPs.

If you are on Airbyte Cloud, you will always need to modify your database configuration to allow inbound traffic from Airbyte IPs. You can find a list of all IPs that need to be allowlisted in our Airbyte Security docs.

Now, click Set up source in the Airbyte UI. Airbyte will now test connecting to your database. Once this succeeds, you've configured an Airbyte Postgres source!

Advanced Configuration

Setup using CDC

Airbyte uses logical replication of the Postgres write-ahead log (WAL) to incrementally capture deletes using a replication plugin:

  • See here to learn more on how Airbyte implements CDC.
  • See here to learn more about Postgres CDC requirements and limitations.

We recommend configuring your Postgres source with CDC when:

  • You need a record of deletions.
  • You have a very large database (500 GB or more).
  • Your table has a primary key but doesn't have a reasonable cursor field for incremental syncing (updated_at).

These are the additional steps required (after following the quick start) to configure your Postgres source using CDC:

  1. Provide additional REPLICATION permissions to read-only user
  2. Enable logical replication on your Postgres database
  3. Create a replication slot on your Postgres database
  4. Create publication and replication identities for each Postgres table
  5. Enable CDC replication in the Airbyte UI

Step 1: Prepopulate your Postgres source configuration

We recommend following the steps in the quick start section to confirm that Airbyte can connect to your Postgres database prior to configuring CDC settings.

For CDC, you must connect to primary/master databases. Pointing the connector configuration to replica database hosts for CDC will lead to failures.

Step 2: Provide additional permissions to read-only user

To configure CDC for the Postgres source connector, grant REPLICATION permissions to the user created in step 1 of the quick start:

ALTER USER <user_name> REPLICATION;

Step 3: Enable logical replication on your Postgres database

To enable logical replication on bare metal, VMs (EC2/GCE/etc), or Docker, configure the following parameters in the postgresql.conf file for your Postgres database:

ParameterDescriptionSet value to
wal_levelType of coding used within the Postgres write-ahead loglogical
max_wal_sendersThe maximum number of processes used for handling WAL changesmin: 1
max_replication_slotsThe maximum number of replication slots that are allowed to stream WAL changes1 (if Airbyte is the only service reading subscribing to WAL changes. More than 1 if other services are also reading from the WAL)

To enable logical replication on AWS Postgres RDS or Aurora:

  • Go to the Configuration tab for your DB cluster.
  • Find your cluster parameter group. Either edit the parameters for this group or create a copy of this parameter group to edit. If you create a copy, change your cluster's parameter group before restarting.
  • Within the parameter group page, search for rds.logical_replication. Select this row and click Edit parameters. Set this value to 1.
  • Wait for a maintenance window to automatically restart the instance or restart it manually.

To enable logical replication on Azure Database for Postgres, change the replication mode of your Postgres DB on Azure to logical using the replication menu of your PostgreSQL instance in the Azure Portal. Alternatively, use the Azure CLI to run the following command:

az postgres server configuration set --resource-group group --server-name server --name azure.replication_support --value logical
az postgres server restart --resource-group group --name server

Step 4: Create a replication slot on your Postgres database

Airbyte requires a replication slot configured only for its use. Only one source should be configured that uses this replication slot.

For this step, Airbyte requires use of the pgoutput plugin. To create a replication slot called airbyte_slot using pgoutput, run as the user with the newly granted REPLICATION role:

SELECT pg_create_logical_replication_slot('airbyte_slot', 'pgoutput');

The output of this command will include the name of the replication slot to fill into the Airbyte source setup page.

Step 5: Create publication and replication identities for each Postgres table

For each table you want to replicate with CDC, follow the steps below:

  1. Add the replication identity (the method of distinguishing between rows) for each table you want to replicate:
ALTER TABLE tbl1 REPLICA IDENTITY DEFAULT;

In rare cases, if your tables use data types that support TOAST or have very large field values, consider instead using replica identity type full: ALTER TABLE tbl1 REPLICA IDENTITY FULL;. Ensure that TOAST-able tables use non-TOAST-able primary keys (integers, varchars, etc), and there will only be a modest increase in resource utilization, in addition to increased WAL storage size.

  1. Create the Postgres publication. You should include all tables you want to replicate as part of the publication:
CREATE PUBLICATION airbyte_publication FOR TABLE <tbl1, tbl2, tbl3>;`

The publication name is customizable. Refer to the Postgres docs if you need to add or remove tables from your publication in the future.

note

The Airbyte UI currently allows selecting any tables for CDC. If a table is selected that is not part of the publication, it will not be replicated even though it is selected. If a table is part of the publication but does not have a replication identity, that replication identity will be created automatically on the first run if the Airbyte user has the necessary permissions.

Step 6: Enable CDC replication in Airbyte UI

In your Postgres source, change the replication mode to Logical Replication (CDC), and enter the replication slot and publication you just created.

Postgres Replication Methods

The Postgres source currently offers 3 methods of replicating updates to your destination: CDC, xmin and standard (with a user defined cursor). Both CDC and xmin are the most reliable methods of updating your data.

CDC

Airbyte uses logical replication of the Postgres write-ahead log (WAL) to incrementally capture deletes using a replication plugin. To learn more how Airbyte implements CDC, refer to Change Data Capture (CDC). We recommend configuring your Postgres source with CDC when:

  • You need a record of deletions.
  • You have a very large database (500 GB or more).
  • Your table has a primary key but doesn't have a reasonable cursor field for incremental syncing (updated_at).

If your goal is to maintain a snapshot of your table in the destination but the limitations prevent you from using CDC, consider using the xmin replication method.

Xmin

Xmin replication is the new cursor-less replication method for Postgres. Cursorless syncs enable syncing new or updated rows without explicitly choosing a cursor field. The xmin system column which (available in all Postgres databases) is used to track inserts and updates to your source data.

This is a good solution if:

  • There is not a well-defined cursor candidate to use for Standard incremental mode.
  • You want to replace a previously configured full-refresh sync.
  • Your database doesn't incur heavy writes that would lead to transaction ID wraparound.
  • You are not replicating non-materialized views. Non-materialized views are not supported by xmin replication.

Connecting with SSL or SSH Tunneling

SSL Modes

Airbyte Cloud uses SSL by default. You are not permitted to disable SSL while using Airbyte Cloud. You will most frequently choose require or verify-ca. Both of these always require encryption. verify-ca also requires certificates from your Postgres database.

Here is a breakdown of available SSL connection modes:

  • disable to disable encrypted communication between Airbyte and the source
  • allow to enable encrypted communication only when required by the source
  • prefer to allow unencrypted communication only when the source doesn't support encryption
  • require to always require encryption. Note: The connection will fail if the source doesn't support encryption.
  • verify-ca to always require encryption and verify that the source has a valid SSL certificate
  • verify-full to always require encryption and verify the identity of the source

SSH Tunneling

If you are using SSH tunneling, as Airbyte Cloud requires encrypted communication, select SSH Key Authentication or Password Authentication if you selected disable, allow, or prefer as the SSL Mode; otherwise, the connection will fail.

For SSH Tunnel Method, select:

  • No Tunnel for a direct connection to the database
  • SSH Key Authentication to use an RSA Private as your secret for establishing the SSH tunnel
  • Password Authentication to use a password as your secret for establishing the SSH tunnel

Connect via SSH Tunnel

You can connect to a Postgres instance via an SSH tunnel.

When using an SSH tunnel, you are configuring Airbyte to connect to an intermediate server (also called a bastion or a jump server) that has direct access to the database. Airbyte connects to the bastion and then asks the bastion to connect directly to the server.

To connect to a Postgres instance via an SSH tunnel:

  1. While setting up the Postgres source connector, from the SSH tunnel dropdown, select:
    • SSH Key Authentication to use a private as your secret for establishing the SSH tunnel
    • Password Authentication to use a password as your secret for establishing the SSH Tunnel
  2. For SSH Tunnel Jump Server Host, enter the hostname or IP address for the intermediate (bastion) server that Airbyte will connect to.
  3. For SSH Connection Port, enter the port on the bastion server. The default port for SSH connections is 22.
  4. For SSH Login Username, enter the username to use when connecting to the bastion server. Note: This is the operating system username and not the Postgres username.
  5. For authentication:
    • If you selected SSH Key Authentication, set the SSH Private Key to the private Key that you are using to create the SSH connection.
    • If you selected Password Authentication, enter the password for the operating system user to connect to the bastion server. Note: This is the operating system password and not the Postgres password.

Generating a private key for SSH Tunneling

The connector supports any SSH compatible key format such as RSA or Ed25519. To generate an RSA key, for example, run:

ssh-keygen -t rsa -m PEM -f myuser_rsa

The command produces the private key in PEM format and the public key remains in the standard format used by the authorized_keys file on your bastion server. Add the public key to your bastion host to the user you want to use with Airbyte. The private key is provided via copy-and-paste to the Airbyte connector configuration screen to allow it to log into the bastion server.

Limitations & Troubleshooting

To see connector limitations, or troubleshoot your Postgres connector, see more in our Postgres troubleshooting guide.

Data type mapping

According to Postgres documentation, Postgres data types are mapped to the following data types when synchronizing data. You can check the test values examples here. If you can't find the data type you are looking for or have any problems feel free to add a new test!

Postgres TypeResulting TypeNotes
bigintnumber
bigserial, serial8number
bitstringFixed-length bit string (e.g. "0100").
bit varying, varbitstringVariable-length bit string (e.g. "0100").
boolean, boolboolean
boxstring
byteastringVariable length binary string with hex output format prefixed with "\x" (e.g. "\x6b707a").
character, charstring
character varying, varcharstring
cidrstring
circlestring
datestringParsed as ISO8601 date time at midnight. CDC mode doesn't support era indicators. Issue: #14590
double precision, float, float8numberInfinity, -Infinity, and NaN are not supported and converted to null. Issue: #8902.
hstorestring
inetstring
integer, int, int4number
intervalstring
jsonstring
jsonbstring
linestring
lsegstring
macaddrstring
macaddr8string
moneynumber
numeric, decimalnumberInfinity, -Infinity, and NaN are not supported and converted to null. Issue: #8902.
pathstring
pg_lsnstring
pointstring
polygonstring
real, float4number
smallint, int2number
smallserial, serial2number
serial, serial4number
textstring
timestringParsed as a time string without a time-zone in the ISO-8601 calendar system.
timetzstringParsed as a time string with time-zone in the ISO-8601 calendar system.
timestampstringParsed as a date-time string without a time-zone in the ISO-8601 calendar system.
timestamptzstringParsed as a date-time string with time-zone in the ISO-8601 calendar system.
tsquerystring
tsvectorstring
uuidstring
xmlstring
enumstring
tsrangestring
arrayarrayE.g. "["10001","10002","10003","10004"]".
composite typestring

Changelog

Expand to review
VersionDatePull RequestSubject
3.6.222024-10-0246900Fixed a bug where source docs won't render on Airbyte 1.1
3.6.212024-10-0246322Support CDC against a read-replica (continuation)
3.6.202024-10-0146299Make postgres source compile and use the latest CDK
3.6.192024-09-1745639Adopt latest CDK to use the latest apache sshd mina to handle tcpkeepalive requests.
3.6.182024-08-2844878Enable tcpKeepAlive for jdbc connection.
3.6.172024-08-2744841Adopt latest CDK.
3.6.162024-08-1544119Fix incorrect final state on initial read in CDC mode.
3.6.152024-08-1243945Add missing replication slot config error.
3.6.142024-08-0843418Adopt latest CDK.
3.6.132024-07-3042869Adopt latest CDK.
3.6.122024-07-3042550Correctly report stream states.
3.6.112024-07-2942852Bump CDK version to latest to use new bug fixes on error translation.
3.6.102024-07-2342417Handle null error message in ConnectorExceptionHandler.
3.6.92024-07-2342421Remove final transient error emitter iterators.
3.6.82024-07-2241622Bump CDK version to latest.
3.6.72024-07-2242411Hide the "initial load timeout in hours" field by default in UI
3.6.62024-07-2241622Fix bug in CDC syncing
3.6.52024-07-2242024Fix a bug on resuming from a failed attempt.
3.6.42024-07-1742087Translate more errors for Postgres source.
3.6.32024-07-1942122Improve wass error message + logging.
3.6.22024-07-1842108Disable incremental sync for view streams in xmin replication mode
3.6.12024-07-0540716Fix typo in connector specification
3.6.02024-07-1740208Start using the new error Postgres source error handler that comes with a new error translation layer.
3.5.22024-07-1742068Add analytics for WASS case occurrence.
3.5.12024-07-1742055Add debezium heartbeat timeout back to shutdown debezium.
3.5.02024-07-1741651Implement WASS algo - large initial snapshots shouldn't block CDC.
3.4.262024-07-1541654Allow null value for array typed columns in CDC.
3.4.252024-07-1241651Throw transient error if tables of interest and undergoing full vacuum.
3.4.242024-07-0541067Fix Postgres sending duplicated streams
3.4.232024-07-0140757Rollback 3.4.22.
3.4.212024-07-0140516Remove dbz hearbeat.
3.4.202024-06-2340559Remove strict check for stream states of unknown types
3.4.192024-06-2340223Revert the changes introduced in version 3.4.15.
3.4.182024-06-1439349Full refresh stream sending internal count metadata.
3.4.172024-06-1339460Bump postgres JDBC driver version
3.4.162024-05-2939474Adopt latest CDK.
3.4.152024-05-2938773Connect with adaptiveFetch=true.
3.4.142024-06-0839353Upgrade Debezium to 2.6.2
3.4.132024-06-0438875read() throws config exception upon detecting transaction ID wraparound.
3.4.122024-06-0438836check() throws config error upon detecting transaction ID wraparound.
3.4.112024-06-0438848Improve UI message and doc on xmin
3.4.102024-05-2938584Set is_resumable flag in discover.
3.4.92024-05-2938775Publish CDK
3.4.92024-05-2838716Publish CDK
3.4.82024-05-2838716Stream status for postgres
3.4.72024-05-2038365Rollback a previously version (3.4.6)
3.4.52024-05-1638303Streams not in the CDC publication still have a cursor and PK.
3.4.42024-05-1538208disable counts in full refresh stream in state message.
3.4.32024-05-1338104Handle transient error messages.
3.4.22024-05-1038171Bug fix on final state setup.
3.4.12024-05-1038130Bug fix on old PG where ctid column not found when stream is a view.
3.4.02024-04-2937112resumeable full refresh.
3.3.332024-05-0738030Mark PG hot standby error as transient.
3.3.322024-04-3037758Correct previous release to disable debezium retries
3.3.312024-04-3037754Add CDC logs
3.3.302024-04-3037726Remove debezium retries
3.3.292024-04-2337509remove excessive logs
3.3.282024-04-2337509Better error messages on switching between sync modes.
3.3.272024-04-2237445Remove legacy bad values handling code.
3.3.262024-04-1036982Populate airyte_meta.changes for xmin path
3.3.252024-04-1036981Track latest CDK
3.3.242024-04-1036865Track latest CDK
3.3.232024-04-0236759Track latest CDK
3.3.222024-04-0136739Fix useLocalCdk flag.
3.3.212024-03-2536584Adopt Kotlin CDK.
3.3.202024-03-2536432Failure to serialize values from Postgres DB shouldn't fail sync.
3.3.192024-03-1236333Use newest CDK - deprecate dbz iterator
3.3.182024-03-1235599Use newest CDK
3.3.172024-03-1235939Use lsn_commit value instead of lsn_proc for CDC checkpointing logic.
3.3.162024-03-1135904Adopt Java CDK 0.23.1- debezium retries.
3.3.152024-02-2934724Add record count in state message.
3.3.142024-03-0635842Add logging to understand cases with a large number of records with the same LSN.
3.3.132024-02-2735675Fix invalid cdc error message.
3.3.122024-02-2235569Fix logging bug.
3.3.112024-02-2035304Add config to throw an error on invalid CDC position and enable it by default.
3.3.102024-02-1335036Emit analytics message for invalid CDC cursor.
3.3.92024-02-1335224Adopt CDK 0.20.4
3.3.82024-02-0834751Adopt CDK 0.19.0
3.3.72024-02-0834781Add a setting in the setup page to advance the LSN.
3.3.62024-02-0734892Adopt CDK v0.16.6
3.3.52024-02-0734948Adopt CDK v0.16.5
3.3.42024-01-3134723Adopt CDK v0.16.3
3.3.32024-01-2634573Adopt CDK v0.16.0
3.3.22024-01-2434465Check xmin only if user selects xmin sync mode.
3.3.12024-01-1034119Adopt java CDK version 0.11.5.
3.3.02023-12-1933437Remove LEGACY state flag
3.2.272023-12-1833605Advance Postgres LSN for PG 14 & below.
3.2.262023-12-1133027Support for better debugging tools.
3.2.252023-11-2932961Bump debezium wait time default to 20 min.
3.2.242023-11-2832686Better logging to understand dbz closing reason attribution.
3.2.232023-11-2832891Fix CDK dependency in build.
3.2.222023-11-2232656Adopt java CDK version 0.5.0.
3.2.212023-11-0731856handle date/timestamp infinity values properly
3.2.202023-11-0632193Adopt java CDK version 0.4.1.
3.2.192023-11-0332050Adopt java CDK version 0.4.0.
3.2.182023-11-0129038Fix typo (s/Airbtye/Airbyte/)
3.2.172023-11-0132068Bump Debezium 2.2.0Final -> 2.4.0Final
3.2.162023-10-3131976Speed up tests involving Debezium
3.2.152023-10-3031960Adopt java CDK version 0.2.0.
3.2.142023-10-2431792Fix error message link on issue with standby
3.2.142023-10-2431792fail sync when debezeum fails to shutdown cleanly
3.2.132023-10-1631029Enforces encrypted traffic settings when env var DEPLOYMENT_MODE = CLOUD.
3.1.132023-10-1331309Addressed decimals being incorrectly deserialized into scientific notation.
3.1.122023-10-1231328Improvements to initial load of tables in older versions of postgres.
3.1.112023-10-1131322Correct pevious release
3.1.102023-09-2930806Cap log line length to 32KB to prevent loss of records.
3.1.92023-09-2530534Fix JSONB[] column type handling bug.
3.1.82023-09-2030125Improve initial load performance for older versions of PostgreSQL.
3.1.72023-09-0529672Handle VACUUM happening during initial sync
3.1.62023-08-2429821Set replication_method display_type to radio, update titles and descriptions, and make CDC the default choice
3.1.52023-08-2229534Support "options" JDBC URL parameter
3.1.42023-08-2128687Under the hood: Add dependency on Java CDK v0.0.2.
3.1.32023-08-0328708Enable checkpointing snapshots in CDC connections
3.1.22023-08-0128954Fix an issue that prevented use of tables with names containing uppercase letters
3.1.12023-07-3128892Fix an issue that prevented use of cursor columns with names containing uppercase letters
3.1.02023-07-2528339Checkpointing initial load for incremental syncs: enabled for xmin and cursor based only.
3.0.22023-07-1828336Add full-refresh mode back to Xmin syncs.
3.0.12023-07-1428345Increment patch to trigger a rebuild
3.0.02023-07-1227442Set _ab_cdc_lsn as the source defined cursor for CDC mode to prepare for Destination v2 normalization
2.1.12023-07-0626723Add new xmin replication method.
2.1.02023-06-2627737License Update: Elv2
2.0.342023-06-2027212Fix silent exception swallowing in StreamingJdbcDatabase
2.0.332023-06-0126873Add prepareThreshold=0 to JDBC url to mitigate PGBouncer prepared statement [X] already exists.
2.0.322023-05-3126810Remove incremental sync estimate from Postgres to increase performance.
2.0.312023-05-2526633Collect and log information related to full vacuum operation in db
2.0.302023-05-2526473CDC : Limit queue size
2.0.292023-05-1825898Translate Numeric values without decimal, e.g: NUMERIC(38,0), as BigInt instead of Double
2.0.282023-04-2725401CDC : Upgrade Debezium to version 2.2.0
2.0.272023-04-2624971Emit stream status updates
2.0.262023-04-2625560Revert some logging changes
2.0.252023-04-2425459Better logging formatting
2.0.242023-04-1925345Logging : Log database indexes per stream
2.0.232023-04-1924582CDC : Enable frequent state emission during incremental syncs + refactor for performance improvement
2.0.222023-04-1725220Logging changes : Log additional metadata & clean up noisy logs
2.0.212023-04-1225131Make Client Certificate and Client Key always show
2.0.202023-04-1124859Removed SSL toggle and rely on SSL mode dropdown to enable/disable SSL
2.0.192023-04-1124656CDC minor refactor
2.0.182023-04-0624820Fix data loss bug during an initial failed non-CDC incremental sync
2.0.172023-04-0524622Allow streams not in CDC publication to be synced in Full-refresh mode
2.0.162023-04-0524895Fix spec for cloud
2.0.152023-04-0424833Fix Debezium retry policy configuration
2.0.142023-04-0324609Disallow the "disable" SSL Modes
2.0.132023-03-2824166Fix InterruptedException bug during Debezium shutdown
2.0.122023-03-2724529Add CDC checkpoint state messages
2.0.112023-03-2324446Set default SSL Mode to require in strict-encrypt
2.0.102023-03-2324417Add field groups and titles to improve display of connector setup form
2.0.92023-03-2220760Removed redundant date-time datatypes formatting
2.0.82023-03-2224255Add field groups and titles to improve display of connector setup form
2.0.72023-03-2124207Fix incorrect schema change warning in CDC mode
2.0.62023-03-2124271Fix NPE in CDC mode
2.0.52023-03-2121533Add integration with datadog
2.0.42023-03-2124147Fix error with CDC checkpointing
2.0.32023-03-1424000Removed check method call on read.
2.0.22023-03-1323112Add state checkpointing for CDC sync.
2.0.02023-03-0623112Upgrade Debezium version to 2.1.2
1.0.512023-03-0223642Revert : Support JSONB datatype for Standard sync mode
1.0.502023-02-2721695Support JSONB datatype for Standard sync mode
1.0.492023-02-2423383Fixed bug with non readable double-quoted values within a database name or column name
1.0.482023-02-2322623Increase max fetch size of JDBC streaming mode
1.0.472023-02-2222221Fix previous versions which doesn't verify privileges correctly, preventing CDC syncs to run.
1.0.462023-02-2123105Include log levels and location information (class, method and line number) with source connector logs published to Airbyte Platform.
1.0.452023-02-0922221Ensures that user has required privileges for CDC syncs.
2023-02-1523028
1.0.442023-02-0622221Exclude new set of system tables when using pg_stat_statements extension.
1.0.432023-02-0621634Improve Standard sync performance by caching objects.
1.0.422023-01-2321523Check for null in cursor values before replacing.
1.0.412023-01-2520939Adjust batch selection memory limits databases.
1.0.402023-01-2421825Put back the original change that will cause an incremental sync to error if table contains a NULL value in cursor column.
1.0.392023-01-2021683Speed up esmtimates for trace messages in non-CDC mode.
1.0.382023-01-1720436Consolidate date/time values mapping for JDBC sources
1.0.372023-01-1720783Emit estimate trace messages for non-CDC mode.
1.0.362023-01-1121003Handle null values for array data types in CDC mode gracefully.
1.0.352023-01-0420469Introduce feature to make LSN commit behaviour configurable.
1.0.342022-12-1320378Improve descriptions
1.0.332022-12-1218959CDC : Don't timeout if snapshot is not complete.
1.0.322022-12-1220192Only throw a warning if cursor column contains null values.
1.0.312022-12-0219889Check before each sync and stop if an incremental sync cursor column contains a null value.
2022-12-0219985Reenable incorrectly-disabled wal2json CDC plugin
1.0.302022-11-2919024Skip tables from schema where user do not have Usage permission during discovery.
1.0.292022-11-2919623Mark PSQLException related to using replica that is configured as a hot standby server as config error.
1.0.282022-11-2819514Adjust batch selection memory limits databases.
1.0.272022-11-2816990Handle arrays data types
1.0.262022-11-1819551Fixes bug with ssl modes
1.0.252022-11-1619004Use Debezium heartbeats to improve CDC replication of large databases.
1.0.242022-11-0719291Default timeout is reduced from 1 min to 10sec
1.0.232022-11-0719025Stop enforce SSL if ssl mode is disabled
1.0.222022-10-3118538Encode database name
1.0.212022-10-2518256Disable allow and prefer ssl modes in CDC mode
1.0.202022-10-2518383Better SSH error handling + messages
1.0.192022-10-2118263Fixes bug introduced in 15833 and adds better error messaging for SSH tunnel in Destinations
1.0.182022-10-1918087Better error messaging for configuration errors (SSH configs, choosing an invalid cursor)
1.0.172022-10-1718041Fixes bug introduced 2022-09-12 with SshTunnel, handles iterator exception properly
1.0.162022-10-1315535Update incremental query to avoid data missing when new data is inserted at the same time as a sync starts under non-CDC incremental mode
1.0.152022-10-1117782Handle 24:00:00 value for Time column
1.0.142022-10-0317515Fix an issue preventing connection using client certificate
1.0.132022-10-0117459Upgrade debezium version to 1.9.6 from 1.9.2
1.0.122022-09-2717299Improve error handling for strict-encrypt postgres source
1.0.112022-09-2617131Allow nullable columns to be used as cursor
1.0.102022-09-1415668Wrap logs in AirbyteLogMessage
1.0.92022-09-1316657Improve CDC record queueing performance
1.0.82022-09-0816202Adds error messaging factory to UI
1.0.72022-08-3016114Prevent traffic going on an unsecured channel in strict-encryption version of source postgres
1.0.62022-08-3016138Remove unnecessary logging
1.0.52022-08-2515993Add support for connection over SSL in CDC mode
1.0.42022-08-2315877Fix temporal data type bug which was causing failure in CDC mode
1.0.32022-08-1814356DB Sources: only show a table can sync incrementally if at least one column can be used as a cursor field
1.0.22022-08-1115538Allow additional properties in db stream state
1.0.12022-08-1015496Fix state emission in incremental sync
2022-08-1015481Fix data handling from WAL logs in CDC mode
1.0.02022-08-0515380Change connector label to generally_available (requires upgrading your Airbyte platform to v0.40.0-alpha)
0.4.442022-08-0515342Adjust titles and descriptions in spec.json
0.4.432022-08-0315226Make connectionTimeoutMs configurable through JDBC url parameters
0.4.422022-08-0315273Fix a bug in 0.4.36 and correctly parse the CDC initial record waiting time
0.4.412022-08-0315077Sync data from beginning if the LSN is no longer valid in CDC
2022-08-0314903Emit state messages more frequently (⛔ this version has a bug; use 1.0.1 instead
0.4.402022-08-0315187Add support for BCE dates/timestamps
2022-08-0314534Align regular and CDC integration tests and data mappers
0.4.392022-08-0214801Fix multiple log bindings
0.4.382022-07-2614362Integral columns are now discovered as int64 fields.
0.4.372022-07-2214714Clarified error message when invalid cursor column selected
0.4.362022-07-2114451Make initial CDC waiting time configurable (⛔ this version has a bug and will not work; use 0.4.42 instead)
0.4.352022-07-1414574Removed additionalProperties:false from JDBC source connectors
0.4.342022-07-1713840Added the ability to connect using different SSL modes and SSL certificates.
0.4.332022-07-1414586Validate source JDBC url parameters
0.4.322022-07-0714694Force to produce LEGACY state if the use stream capable feature flag is set to false
0.4.312022-07-0714447Under CDC mode, retrieve only those tables included in the publications
0.4.302022-06-3014251Use more simple and comprehensive query to get selectable tables
0.4.292022-06-2914265Upgrade postgresql JDBC version to 42.3.5
0.4.282022-06-2314077Use the new state management
0.4.262022-06-1713864Updated stacktrace format for any trace message errors
0.4.252022-06-1513823Publish adaptive postgres source that enforces ssl on cloud + Debezium version upgrade to 1.9.2 from 1.4.2
0.4.242022-06-1413549Fixed truncated precision if the value of microseconds or seconds is 0
0.4.232022-06-1313655Fixed handling datetime cursors when upgrading from older versions of the connector
0.4.222022-06-0913655Fixed bug with unsupported date-time datatypes during incremental sync
0.4.212022-06-0613435Adjust JDBC fetch size based on max memory and max row size
0.4.202022-06-0213367Added convertion hstore to json format
0.4.192022-05-2513166Added timezone awareness and handle BC dates
0.4.182022-05-2513083Add support for tsquey type
0.4.172022-05-1913016CDC modify schema to allow null values
0.4.162022-05-1412840Added custom JDBC parameters field
0.4.152022-05-1312834Fix the bug that the connector returns empty catalog for Azure Postgres database
0.4.142022-05-0812689Add table retrieval according to role-based SELECT privilege
0.4.132022-05-0510230Explicitly set null value for field in json
0.4.122022-04-2912480Query tables with adaptive fetch size to optimize JDBC memory consumption
0.4.112022-04-1111729Bump mina-sshd from 2.7.0 to 2.8.0
0.4.102022-04-0811798Fixed roles for fetching materialized view processing
0.4.82022-02-2110242Fixed cursor for old connectors that use non-microsecond format. Now connectors work with both formats
0.4.72022-02-1810242Updated timestamp transformation with microseconds
0.4.62022-02-1410256(unpublished) Add -XX:+ExitOnOutOfMemoryError JVM option
0.4.52022-02-0810173Improved discovering tables in case if user does not have permissions to any table
0.4.42022-01-269807Update connector fields title/description
0.4.32022-01-249554Allow handling of java sql date in CDC
0.4.22022-01-139360Added schema selection
0.4.12022-01-059116Added materialized views processing
0.4.02021-12-138726Support all Postgres types
0.3.172021-12-018371Fixed incorrect handling "\n" in ssh key
0.3.162021-11-287995Fixed money type with amount > 1000
0.3.152021-11-268066Fixed the case, when Views are not listed during schema discovery
0.3.142021-11-178010Added checking of privileges before table internal discovery
0.3.132021-10-267339Support or improve support for Interval, Money, Date, various geometric data types, inventory_items, and others
0.3.122021-09-306585Improved SSH Tunnel key generation steps
0.3.112021-09-025742Add SSH Tunnel support
0.3.92021-08-175304Fix CDC OOM issue
0.3.82021-08-134699Added json config validator
0.3.42021-06-093973Add AIRBYTE_ENTRYPOINT for Kubernetes support
0.3.32021-06-083960Add method field in specification parameters
0.3.22021-05-263179Remove isCDC logging
0.3.12021-04-212878Set defined cursor for CDC
0.3.02021-04-212990Support namespaces
0.2.72021-04-162923SSL spec as optional
0.2.62021-04-162757Support SSL connection
0.2.52021-04-122859CDC bugfix
0.2.42021-04-092548Support CDC
0.2.32021-03-282600Add NCHAR and NVCHAR support to DB and cursor type casting
0.2.22021-03-262460Destination supports destination sync mode
0.2.12021-03-182488Sources support primary keys
0.2.02021-03-092238Protocol allows future/unknown properties
0.1.132021-02-021887Migrate AbstractJdbcSource to use iterators
0.1.122021-01-251746Fix NPE in State Decorator
0.1.112021-01-251765Add field titles to specification
0.1.102021-01-191724Fix JdbcSource handling of tables with same names in different schemas
0.1.92021-01-141655Fix JdbcSource OOM
0.1.82021-01-131588Handle invalid numeric values in JDBC source
0.1.72021-01-081307Migrate Postgres and MySql to use new JdbcSource
0.1.62020-12-091172Support incremental sync
0.1.52020-11-301038Change JDBC sources to discover more than standard schemas
0.1.42020-11-301046Add connectors using an index YAML file