v.2.4.7

Bug fixed:

• Errors in loading from query #79

v.2.4.6

fixed bugs:

• always drop tables after renamed staging tables
• column name mismatch in pushdown function

v.2.4.5

Bug Fixed:

• Can not insert Timestamp value into Date column.

v.2.4.4

Support variant data types(MapType, StructType, and ArrayType)

• Write to Snowflake
  • Spark-Snowflake Connector can correctly write the variant data type to Snowflake table now. (In previous versions, all variant data types would be converted to StringType before written to Snowflake table.)
  • The staging data file format will be JSON instead of CSV, if the Spark Dataframe contains variant data types.
  • Example:
    Spark Dataframe Schema

StructType(
    Array(
      StructField("ARR", ArrayType(IntegerType), false),
      StructField("OB",StructType(
        Array(
          StructField("str", StringType, false)
        )
      ), false),
      StructField("MAP", MapType(StringType,StringType), false),
      StructField("NUM", IntegerType, false)
    )
  )

Result Snowflake Table Schema

(ARR VARIANT, OB VARIANT, MAP VARIANT, NUM INTEGER)

• Read from Snowflake
  • If a schema containing variant data types provided during reading from Snowflake, the connector will convert all variant data to the corresponding types.
  • Otherwise, variant data will be converted to StringType by default.
  • Example:
    Snowflake Table Schema

(NUM INTEGER, STR STRING, ARR VARIANT)

Read From Snowflake

  val schema = new StructType(
    Array(
      StructField("NUM", IntegerType, false),
      StructField("STR", StringType, false),
      StructField("ARR", ArrayType(IntegerType), false)
  )

  val df = sqlContext.read
      .format(SNOWFLAKE_SOURCE_NAME)
      .options(sfOption)
      .schema(schema)
      .load()

v.2.4.3

• Support Column Mapping function on external stage
• Bug fixed:
  • No default value of truncate_table variable in v2.4.2

v.2.4.2

• Add new parameter truncate_table
  • Value: on/off
  • Default: off
  • Detail: If it is on, the schema of original table will be kept in overwrite mode when writing data.
• Add new parameter continue_on_error
  • Value: on/off
  • Default: off
  • Detail: if it is on, the on_error parameter will be continue in COPY command.

v.2.4.1

• Fixed an issue when including line feed in string values
• Fixed an issue when using Azure external stages in IT test

v.2.4.0

• Support Azure internal and external stage

v.2.3.3

• Introduced purge parameter to enable/disable automatic purging external storage when writing data to Snowflake.

v.2.3.2

• Support for Spark 2.3.
• truncate_columns can be passed along with other parameters for use in the COPY Dataframe write to Snowflake.
  If set to true, TRUCATECOLUMNS will be true in the load statement.
• Support for column-mapping. Columns may be written out-of-order, or to an arbitrary set of equal quantity, type-compatible columns from a Dataframe to a Snowflake table. Example:

    df.write.format(SNOWFLAKE_SOURCE_NAME).options(connectorOptionsNoTable)
      .option("dbtable", dbtable)
      .option("columnmap", Map("one" -> "sf_col2", "two" -> "sf_col1").toString())
      .mode(SaveMode.Append).save()

will write column "one" of the Spark Dataframe to a "sf_col2" and a column "two" of the Dataframe to a "sf_col1" in the target Snowflake table.