Release Spark Connector 2.11.0

Support Spark 3.3 and fixed some bugs:

1. Added support for Spark 3.3.
2. Upgraded the version of the PostgreSQL JDBC Driver that tests depend on to avoid the security vulnerability CVE-2022-31197.
3. Updated the connector to use the Snowflake JDBC driver 3.13.22 and the Snowflake Ingest SDK 0.10.8.

NOTE:

1. Starting from version 2.11.0, the Snowflake Connector for Spark supports Spark 3.1, 3.2 and 3.3.
   Version 2.11.0 of the Snowflake Connector for Spark does not support Spark 3.0. Note that previous versions of the connector continue to support Spark 3.0.
2. For Snowflake GCP accounts, the Snowflake JDBC driver versions 3.13.16 through 3.13.21 do not work with the Spark connector.

Release Spark Connector 2.10.1

Fixed some critical issues:

1. Removed unnecessary dependencies on libraries to avoid the security vulnerabilities CVE-2020-8908 and CVE-2018-10237.
2. Added support for using the JDBC data type TIMESTAMP_WITH_TIMEZONE when reading data from Snowflake.
3. Changed the logic for checking for the existence of a table before saving a DataFrame to Snowflake:
   • The connector now reuses the existing connection (rather than creating a new connection) to avoid potential problems with token expiration.
   • If the table name is not fully qualified (i.e. does not include the schema name), the connector now checks for the table under the schema specified by sfSchema, rather than the schema that is currently in use in the session.
     Note: If you need to save a DataFrame to a table in a schema other than sfSchema, specify the schema as part of the fully qualified name of the table, rather than executing USE SCHEMA to change the current schema.
4. Improved performance by avoiding unnecessary parse_json() calls in the COPY INTO TABLE command when writing a DataFrame with ArrayType, MapType or StructType columns to Snowflake.
5. Added the getLastSelectQueryId and getLastCopyLoadQueryId methods to the Utils class. These methods return the query ID of the last query that read data from Snowflake and the last COPY INTO TABLE statement that was executed (respectively).

Release Spark Connector 2.10.0

• Support Spark 3.2
• Fix an issue that string "null" is regarded as NULL when pushding

Note:

1. From this version, Spark connector will support spark 3.0, 3.1 and 3.2, Spark 2.4 is not supported any more

Release Spark Connector 2.9.3

Fixed some critical issues:

1. Modified the connector to avoid executing a CREATE TABLE command when writing an empty DataFrame to a Snowflake table if the target table exists and the following options are set for the connector: “usestagingtable” is set to “off” and “truncate_table” is set to “on”.
2. Added support for writing a DataFrame to a Snowflake table when the Snowflake account has FORCE_REGIONAL_S3_ENDPOINTS_FOR_PRESIGNED_URLS enabled.
3. Fixed a potential memory leak that could occur when a DataFrame reads from Snowflake tables but the results are not consumed completely.
4. Updated the connector to use the latest version of the Snowflake JDBC Driver (3.13.14).

Release Spark Connector 2.9.2

Fixed some critical issues:

1. For the DataFrame read and write queries, the connector now logs the query ID and the URL to display the query details.
2. When a Spark application fails or is cancelled, the connector now cancels all running queries executed by that Spark application.
3. Added the configuration options timestamp_ntz_output_format, timestamp_ltz_output_format, and timestamp_tz_output_format for specifying the output format for timestamp values. By default, these are set as "YYYY-MM-DD HH24:MI:SS.FF3", "TZHTZM YYYY-MM-DD HH24:MI:SS.FF3", and "TZHTZM YYYY-MM-DD HH24:MI:SS.FF3". If they are set to “sf_current”, which means that the connector uses the formats specified for the session.
4. Updated the connector to use the latest version of the Snowflake JDBC Driver (3.13.10).

Release Spark Connector 2.9.1

Fixed some critical issues:

1. Improved the way in which the connector retrieves metadata for a schema or query. In the past, the connector executed a query to get the metadata. With this release, the connector uses a JDBC PreparedStatement to retrieve just the ResultSet metadata.
2. Fixed an issue with pushdowns when you call limit() after sort() in a DataFrame transformation.
3. Updated the connector to use the latest version of the Snowflake JDBC Driver (3.13.6).
4. Updated the connector to use the latest version of the Snowflake Ingest SDK (0.10.3).

Release Spark Connector 2.9.0

Support spark 3.1 and some bug fixes.

1. Added support for Spark 3.1.
2. Removed the requirement that you must have the privilege to set and get system properties.
3. Updated the connector to use the latest version of the Snowflake JDBC Driver (3.13.3).
4. Updated the connector to use the latest version of the Snowflake Ingest SDK (0.10.2).

Note:

1. With this new version of the connector, Snowflake continues to support the latest three Spark versions: Spark 2.4, 3.0 and 3.1.
2. Version 2.9.0 of the Snowflake Connector for Spark does not support Spark 2.3. Note that previous versions of the connector continue to support Spark 2.3.
3. Spark Connector 2.9.0 is NOT compatible with version 3.13.2 (and earlier versions) of the JDBC Driver.

Release Spark Connector 2.8.6

This release is used to confirm the new spark connector release process to work. It's functionality is equal to Spark Connector 2.8.5.

Release Spark Connector 2.8.5

Fixed some critical issues:

1. Added pushdown support for the EqualNullSafe operator.
2. In cases where Date column data is converted to Number column data (e.g. an integer or long), avoid performing a pushdown, which would result in conversion errors.
3. In cases where DataFrame.dropDuplicates() is called with only some of the columns in the DataFrame (e.g. DataFrame.dropDuplicates("C1") on a DataFrame with the columns "C1" and "C2"), avoid performing a pushdown, which would result in a syntax error.
4. Added support to write a DataFrame that has no partitions to the Snowflake database. If the target table doesn’t exist, an empty table is created. If the target table exists and the write mode is "overwrite", the target table is overwritten with an empty table.
5. Disabled the pushdown of some window functions in order to avoid inconsistent results.
6. Fixed an issue with writing a DataFrame to a Snowflake table when the DataFrame has a column with a complex type (e.g. MapType) and the column name is Snowflake SQL keyword.
7. Updated the connector to use the latest version of the Snowflake JDBC Driver (3.13.2).

Release Spark Connector 2.8.4

Added some pushdown supports and fixed some critical issues.

1. Fixed an issue that occurred when writing a DataFrame with Map type column if the name of the column contains spaces.
2. Fixed an issue that occurred when writing to a table with overwrite mode if the schema name isn't specified as part of the table name, the table is not in the schema specified by the “sfSchema” option, and the PUBLIC schema contains a table with the same name.
3. Added pushdown support for the COALESCE() function when reading from the Snowflake database.
4. Added pushdown support for the windows functions of RANK() and DENSE_RANK() when reading from the Snowflake database.
5. Updated the connector to use the latest version of the Snowflake JDBC Driver (3.12.17).
6. Updated the connector to use the latest version of the Snowflake Ingest SDK (0.10.1).