v.2.4.4

Support variant data types(MapType, StructType, and ArrayType)

• Write to Snowflake
  • Spark-Snowflake Connector can correctly write the variant data type to Snowflake table now. (In previous versions, all variant data types would be converted to StringType before written to Snowflake table.)
  • The staging data file format will be JSON instead of CSV, if the Spark Dataframe contains variant data types.
  • Example:
    Spark Dataframe Schema

StructType(
    Array(
      StructField("ARR", ArrayType(IntegerType), false),
      StructField("OB",StructType(
        Array(
          StructField("str", StringType, false)
        )
      ), false),
      StructField("MAP", MapType(StringType,StringType), false),
      StructField("NUM", IntegerType, false)
    )
  )

Result Snowflake Table Schema

(ARR VARIANT, OB VARIANT, MAP VARIANT, NUM INTEGER)

• Read from Snowflake
  • If a schema containing variant data types provided during reading from Snowflake, the connector will convert all variant data to the corresponding types.
  • Otherwise, variant data will be converted to StringType by default.
  • Example:
    Snowflake Table Schema

(NUM INTEGER, STR STRING, ARR VARIANT)

Read From Snowflake

  val schema = new StructType(
    Array(
      StructField("NUM", IntegerType, false),
      StructField("STR", StringType, false),
      StructField("ARR", ArrayType(IntegerType), false)
  )

  val df = sqlContext.read
      .format(SNOWFLAKE_SOURCE_NAME)
      .options(sfOption)
      .schema(schema)
      .load()