PUDL v2023.12.01 Data Release

[zaneselvans]

Whew, it's been almost an entire year, but we've just put out a new PUDL data release!

• Just want to download all the data? See this Zenodo archive and find the "Download all" button or click here (~10GB).
• For a more detailed description of what went into this release, see the full release notes.
• For documentation about what's in the database, see the PUDL Data Dictionary for this release.

Who should use PUDL v2023.12.01?

• This is an interim release that provides continuity for existing PUDL users who depend on the current database structure, and gives them access to the final 2022 data.
• In the next few weeks we are going to be merging in a reorganization of the database structure, which will hopefully make it easier for users to find the appropriate data for their needs. Now that we are distributing hundreds of tables, our inconsistent legacy naming patterns were a big source of friction and confusion for newer users.
• If you depend on the current database table names directly (e.g. you are using the pudl.sqlite database generated by our nightly builds) and you do not want to migrate to the new database structure immediately, you should switch to using this data release for the time being.
• When PR Feature branch: Rename core + output assets to match new naming protocols #2818 is merged into dev in the next couple of weeks and the database structure changes, it will probably break whatever you've built on top of the old database structure.
• Note that after that merge happens, any new data which is integrated into PUDL will only appear in the new database. All ongoing development will be using the new database structure.
• The output layer implemented by the PudlTabl object remains available, but is now deprecated, and will be removed some time in 2024. The class now just queries the database rather than constructing dataframes itself. If you rely only on the software output layer, it will continue working after the database rename happens, but no new database tables will be made available via the PudlTabl output class. If you need to continue using this software interface for the time being, you should install catalystcoop.pudl v2023.12.1 which corresponds to the version of the software which was used to create this release, or checkout the v2023.12.01 tag from the PUDL repository and install it locally.

Who should not use PUDL v2023.12.01?

• If you aren't already dependent on our existing database structure, you should not set up any new dependencies on this release.
• Instead, wait for Feature branch: Rename core + output assets to match new naming protocols #2818 to be merged in, and get set up with the subsequent data release which will use clearer naming conventions.

What's New?

Data only and lots of it!

• This is our first data-only release. You no longer need to install any particular software to access all of the data that we produce.
• We're also distributing much more data now than in previous releases. This includes both new data sources, all the final agency data from 2022, and derived tables that were previously only available through the software interface.
• This has increased the size of our main database, pudl.sqlite which is now about 12GB (previously it was about 1GB). As a result, we've started compressing the SQLite files for distribution using gzip, which shrinks them by about 75%.
• For simplicity and to reduce overall download sizes, we've switched to only distributing the monolithic EPA CEMS Parquet output. Internally it is still split into year-state row groups, so you can query it efficiently.

For an exhaustive list of the tables available in the database in this release, see the PUDL Data Dictionary for v2023.12.01

New data:

Some of the newly available data sources & tables in the pudl.sqlite database include:

• Many new tables from the FERC Form 1 (thanks to our work with RMI)
• Monthly year-to date data for EIA923
• A crosswalk table that links the EPA CAMD emissions units to the EIA generation units in EPA CEMS.
• EIA-861 data from 2001-2022 (Note: this data is not yet as deeply cleaned and integrated as the EIA-860 & EIA-923)
• FERC-714 hourly demand data for 2006-2020 (we have yet to extract this data from the new XBRL format for 2021 and later).
• Emissions control and cooling equipment tables from the EIA-860

Derived tables that are directly available in the database which you may not have seen before if you weren't using the software layer:

• Annual and monthly generator-level statistics including fuel consumption, heat rates, net generation, and capacity factor.
• County-level historical utility and balancing authority service territories from 2001-2022, compiled from EIA-861.
• A crosswalk that connects the FERC-714 respondents to their EIA equivalent entities, whether they are utilities or balancing authorities.
• Historical estimated hourly electricity demand by state.
• A machine-learning enabled crosswalk that connects FERC Form 1 plant financial data to the corresponding collections of EIA generator ownership slice, so that FERC Form 1 non-fuel operating and capital costs can be correlated with EIA generator attributes, fuel costs, and emissions control equipment information.

Other databases:

We are also distributing SQLite versions of several FERC data sources previously only available as Visual FoxPro DBF files or XBRL, including:

• FERC-1 (1994-2020 from DBF, 2021-2022 from XBRL)
• FERC-2 (1996-2020 from DBF, 2021-2022 from XBRL)
• FERC-6 (2000-2020 from DBF, 2021-2022 from XBRL)
• FERC-60 (2006-2020 from DBF, 2021-2022 from XBRL)
• FERC-714 (2021-2022, from XBRL)

These relatively unprocessed data sources are nonetheless far more accessible as SQLite databases than in their originally published formats.

We are treating PUDL as an application not a library

• We no longer consider PUDL a library which should be included as a dependency by other Python packages.
• We are using lockfiles to control our software environment, which includes more than 500 direct and indirect dependencies.
• If you currently depend on the PUDL software package, we encourage you to move to depending only on the data releases as soon as possible.
• The software remains free and open source, and licensed under the MIT license, but it's really a purpose-built data processing pipeline that isn't readily adaptable to other uses, and we think that users will be better served if we focus on providing frequent, comprehensive, versioned data releases that can be easily integrated into a variety of other applications and analyses without requiring users to manage the software that produced the data as well.
• If you need to continue installing the catalystcoop.pudl software package, for now we are doing software releases alongside any data release, to archive the software that was used to produce the data release.

How do I access this data release?

There are several distribution points for our data releases. Which one is appropriate for you will depend on your use case:

• For long-term reproducible, archival access, you can rely on this Zenodo archive which is also appropriate for citation in academic or other publications using DOI: 10.5281/zenodo.10275052
• If you want to explore the PUDL data interactively through a Jupyter Notebook, it is available on Kaggle. The v2023.12.01 data release corresponds to v9 of the Kaggle dataset.
• PUDL is part of the AWS Open Data Registry. If you are comfortable using the AWS CLI or the S3 API you can access this data release in this free and publicly accessible bucket: s3://pudl.catalyst.coop/v2023.12.01
• It's also available in this public requester-pays bucket in GCS: gs://pudl.catalyst.coop/v2023.12.01

Where's the code?

The software used to produce this data release can be obtained in several ways:

• By cloning the PUDL GitHub repository and checking out the v2023.12.01 release tag.
• By installing PUDL from the Python Package Index: pip install catalystcoop.pudl==2023.12.1
• By downloading the release from GitHub
• By downloading an archive of the release from Zenodo. That archive is also appropriate for citation in any academic publication using the DOI: 10.5281/zenodo.10266492