Does not validate by stripping \u0000 from json

[jdcaperon]

The client does not validate VARIANT values by stripping the null char, this means that it's possible to submit values that are not able to parsed by Snowflake. See support case 00872717

snowflake-ingest-java/src/main/java/net/snowflake/ingest/streaming/internal/DataValidationUtil.java

Lines 89 to 124 in a9ed16d

	/**
	* Validates and parses input as JSON. All types in the object tree must be valid variant types,
	* see {@link DataValidationUtil#isAllowedSemiStructuredType}.
	*
	* @param input Object to validate
	* @return JSON tree representing the input
	*/
	private static JsonNode validateAndParseSemiStructuredAsJsonTree(
	String columnName, Object input, String snowflakeType, final long insertRowIndex) {
	if (input instanceof String) {
	String stringInput = (String) input;
	verifyValidUtf8(stringInput, columnName, snowflakeType, insertRowIndex);
	try {
	return objectMapper.readTree(stringInput);
	} catch (JsonProcessingException e) {
	throw valueFormatNotAllowedException(
	columnName, snowflakeType, "Not a valid JSON", insertRowIndex);
	}
	} else if (isAllowedSemiStructuredType(input)) {
	return objectMapper.valueToTree(input);
	}
	throw typeNotAllowedException(
	columnName,
	input.getClass(),
	snowflakeType,
	new String[] {
	"String",
	"Primitive data types and their arrays",
	"java.time.*",
	"List<T>",
	"Map<String, T>",
	"T[]"
	},
	insertRowIndex);
	}

This is pretty bad, because it means that tables can get into a state where they are unqueryable when selecting the damaged VARIANT column.

[jdcaperon]

Potentially we can have a similar system to https://community.snowflake.com/s/article/How-to-load-Data-into-snowflake-removing-multiple-invalid-UTF-8-Characters-without-doing-a-seperate-transformation-step

[sfc-gh-xhuang]

@jdcaperon we have this filed on our backlog to resolve.
have you stripped the values from your input as a workaround in the meantime?

[jdcaperon]

@sfc-gh-xhuang yes we are doing our own prevention for this issue in the meantime