Merge develop into main

[asparke2]

• Generated the aggregated failure summary.

• Chris shared the new compare py file for development.

• Moved the metadata check after weight columns added.

• Fill null in columns ends with applicable.

• Remove compare_3.py from index

• patched with more detail

• Since we need call the add columns explictly, we may should have a hint in template.

Pull request overview

• Fixes #ISSUENUMBERHERE (IF RELEVANT)

Pull Request Author

This pull request makes changes to (select all the apply):

• Documentation
• Infrastructure (includes apptainer image, buildstock batch, dependencies, continuous integration tests)
• Sampling
• Workflow Measures
• Upgrade Measures
• Reporting Measures
• Postprocessing

Author pull request checklist:

• Tagged the pull request with the appropriate label (documentation, infrastructure, sampling, workflow measure, upgrade measure, reporting measure, postprocessing) to help categorize changes in the release notes.
• Added tests for new measures
• Updated measure .xml(s)
• Register values added to comstock_column_definitions.csv
• Both options_lookup.tsv files updated
• 10k+ test run
• Change documentation written
• Measure documentation written
• ComStock documentation updated
• Changes reflected in example .yml files
• Changes reflected in README.md files
• Added 'See ComStock License' language to first two lines of each code file
• Implements corresponding measure tests and indexing path in test/measure_tests.txt or/and test/resource_measure_tests.txt
• All new and existing tests pass the CI

Review Checklist

This will not be exhaustively relevant to every PR.

• Perform a code review on GitHub
• All related changes have been implemented: data and method additions, changes, tests
• If fixing a defect, verify by running develop branch and reproducing defect, then running PR and reproducing fix
• Reviewed change documentation
• Ensured code files contain License reference
• Results differences are reasonable
• Make sure the newly added measures has been added with tests and indexed properly
• CI status: all tests pass

ComStock Licensing Language - Add to Beginning of Each Code File

# ComStock™, Copyright (c) 2023 Alliance for Sustainable Energy, LLC. All rights reserved.
# See top level LICENSE.txt file for license terms.

[wenyikuang]

Just raise a headsup:

ComStock/postprocessing/comstockpostproc/comstock.py

Lines 1971 to 1995 in 625309d

	for up_id in up_ids:
	# Write CSV version
	file_name = f'ComStock wide upgrade{up_id}.csv'
	file_path = os.path.abspath(os.path.join(self.output_dir, file_name))
	logger.info(f'Exporting to: {file_path}')
	try:
	national_aggregation.filter(pl.col(self.UPGRADE_ID) == up_id).sink_csv(file_path)
	except pl.exceptions.InvalidOperationError:
	logger.warn('Warning - sink_csv not supported for metadata write in current polars version')
	logger.warn('Falling back to .collect.write_csv')
	national_aggregation.filter(pl.col(self.UPGRADE_ID) == up_id).collect().write_csv(file_path)
	# Write Parquet version
	file_name = f'ComStock wide upgrade{up_id}.parquet'
	file_path = os.path.abspath(os.path.join(self.output_dir, file_name))
	logger.info(f'Exporting to: {file_path}')
	try:
	national_aggregation.filter(pl.col(self.UPGRADE_ID) == up_id).sink_parquet(file_path)
	except pl.exceptions.InvalidOperationError:
	logger.warn('Warning - sink_parquet not supported for metadata write in current polars version')
	logger.warn('Falling back to .collect.write_parquet')
	national_aggregation.filter(pl.col(self.UPGRADE_ID) == up_id).collect().write_parquet(file_path)
	# Export dictionaries corresponding to the exported columns
	self.export_data_and_enumeration_dictionary()

We are saving fkt instead of the original self.data in comstock;

I think that's need some clarification about the workflow of the data there....