v.2.3.1

• Full pushdowns are automatically enabled.

v2.3.0

• Updated main branch to support Spark 2.2, and LT-support branch to support Spark 2.1.
• Utils.runQuery() now supports a java.util.Map options map.

v2.2.8

Added a new optional parameter, parallelism, to control the size of the thread pool used in uploads and downloads.

Example:

df.write
.format(SNOWFLAKE_SOURCE_NAME)
.option("parallelism", "12")
.mode(SaveMode.Overwrite)
.save()

v2.2.7

• We now enforce a tighter version dependency on snowflake-jdbc, as specified in the project definition.
• The connector no longer uses disallowed prepared statements for DDL via JDBC.

v2.2.6

• Fixed a regression (NoSuchMethodException) when using the internal temp-stage with newer versions of JDBC. The new minimum version of JDBC, to be used with versions 2.2.6+ of the connector, is 3.2.4.

v2.2.4

• Changed SnowflakeRelation and MergedParameters printout.

v2.2.3

• Fixed an issue where DDLs with identifiers using one or more of the reserved SQL keywords (ORDER, SELECT, etc.) would fail.

• Fixed an issue when returning empty-string results with non-nullable columns.

v2.2.2

• Fixed an issue with identifier quoting for quoted column names for filter pushdowns.

v2.2.1

• Fixed an issue when loading empty result sets from Snowflake.

v2.2.0 - External S3 No Longer Needed!

Use of an external S3 bucket location for data staging is deprecated.

In version 2.2.0, users will no longer need to supply an external S3 bucket for staging of data for movement to/from Spark/Snowflake.

This means that the tempdir, awsAccessKey, and awsSecretAccessKey parameters no longer need to be provided in the connector options.

This brings the minimum parameter set to the following (although you may still need to specify one or more additional options based on your specific use case):

var sfOptions = Map(
    "sfURL" -> "<account_name>.snowflakecomputing.com",
    "sfUser" -> "<user_name>",
    "sfPassword" -> "<password>"
)

val df: DataFrame = sqlContext.read
    .format("net.snowflake.spark.snowflake")
    .options(sfOptions)
    .option("dbtable", "<table>")
    .load()

If you still desire to stage data in an S3 location of your choice, the deprecated parameters are still functional, but may be removed in a later version.