Climate Trace GHG Emissions refresh by HarishC727 · Pull Request #1867 · datacommonsorg/data

HarishC727 · 2026-02-05T13:17:23Z

Check list - https://docs.google.com/spreadsheets/d/1vaZzhpRbcuOTCHMRp9hC9rDYan9ZrMvFmpLF-JGje8g/edit?gid=1086229089#gid=1086229089

gemini-code-assist · 2026-02-05T13:17:51Z

Summary of Changes

Hello @HarishC727, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request establishes a robust and automated pipeline for integrating ClimateTrace's Greenhouse Gas emissions data. By leveraging a new Python script, the system efficiently fetches, processes, and standardizes vast amounts of country and gas-specific emissions data, making it readily available for analysis and ingestion into platforms like Data Commons. This enhancement significantly improves the freshness and accessibility of critical environmental data.

Highlights

New ClimateTrace Data Processor: Introduced a Python-based system to download, process, and segregate Greenhouse Gas (GHG) emissions data from the ClimateTrace API.
Parallel Data Acquisition: Implemented efficient parallel downloading of country-specific and gas-specific data zip files.
Data Segregation by Gas Type: The system now segregates emissions data into separate CSV files for CO2, CH4, N2O, CO2e_20yr, and CO2e_100yr.
Data Commons Integration: Added comprehensive mapping files (climate_pvmap_sectors.csv, climate_pvmap_subsectors.csv, statvar_remap.csv) and configuration for seamless ingestion into Data Commons.
Automated Refresh Schedule: A manifest.json file is included to define the import specifications and set up a cron schedule for regular data refreshes.

Changelog

statvar_imports/climate_trace/README.md
- Added a detailed guide for the ClimateTrace GHG Emissions Data Processor, covering its features, data sources, setup, usage, and project structure.
statvar_imports/climate_trace/check_country.csv
- Introduced a list of ISO country codes to ensure specific countries are always included in the data acquisition process.
statvar_imports/climate_trace/climate_pvmap_sectors.csv
- Created a mapping file to standardize sector-level ClimateTrace data for Data Commons, including various greenhouse gases and emission categories.
statvar_imports/climate_trace/climate_pvmap_subsectors.csv
- Provided a more granular mapping file for subsector-level ClimateTrace data, enhancing the detail of emission source categorization for Data Commons.
statvar_imports/climate_trace/common_metadata.csv
- Added a configuration file to define essential metadata parameters for the data processing pipeline.
statvar_imports/climate_trace/common_output.tmcf
- Included a template MCF file to specify the output structure for the processed ClimateTrace data, aligning it with Data Commons requirements.
statvar_imports/climate_trace/download_and_segregate_by_gas.py
- Implemented the core Python script for dynamic country list generation, parallel downloading of zipped ClimateTrace data, and segregation of emissions by gas type.
statvar_imports/climate_trace/manifest.json
- Defined the import specifications for ClimateTrace data, including the execution scripts, input/output paths, and a cron schedule for automated updates.
statvar_imports/climate_trace/statvar_remap.csv
- Added a remapping file to convert internal statistical variable names to Data Commons identifiers, particularly for subsector data.
statvar_imports/climate_trace/test_data/sample_input.csv
- Added a sample CSV file to serve as input for testing the data processing logic.
statvar_imports/climate_trace/test_data/sample_output.csv
- Included a sample CSV file demonstrating the expected output format after data processing.
statvar_imports/climate_trace/test_data/sample_output.tmcf
- Provided a sample template MCF file for validating the structure of the processed test output.

Activity

This pull request represents the initial implementation of the ClimateTrace GHG Emissions data refresh pipeline. All changes are new file additions, setting up the infrastructure for this data integration.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request adds a new data processing pipeline for Climate Trace GHG emissions data. The changes include a Python script for downloading and segregating data, along with necessary configuration files, documentation, and test data. My review has identified a critical issue in manifest.json due to a typo that would break the import. I've also found a data correctness issue in one of the pvmap files and several opportunities to improve code maintainability in the Python script by using the logging module. Additionally, there are minor issues in the documentation and a configuration file that I've provided suggestions for.

statvar_imports/climate_trace/manifest.json

statvar_imports/climate_trace/climate_pvmap_subsectors.csv

statvar_imports/climate_trace/README.md

statvar_imports/climate_trace/download_and_segregate_by_gas.py

statvar_imports/climate_trace/statvar_remap.csv

pulkeet-wq · 2026-02-06T09:15:03Z

/gcbrun

statvar_imports/climate_trace/climate_pvmap_subsectors.csv

statvar_imports/climate_trace/climate_pvmap_sectors.csv

saanikaaa · 2026-02-16T11:27:28Z

Pls add PR checklist in description

HarishC727 · 2026-02-16T11:47:43Z

Pls add PR checklist in description

Done

saanikaaa · 2026-02-16T12:44:13Z

LGTM

ajaits · 2026-03-02T12:55:56Z

statvar_imports/climate_trace/common_metadata.csv

@@ -0,0 +1,6 @@
+parameter,value
+header_rows,1
+output_columns,"observationAbout,observationDate,variableMeasured,value"


what about unit, observationPeriod?

ajaits · 2026-03-02T12:58:27Z

statvar_imports/climate_trace/download_and_segregate_by_gas.py

+            final_df.to_csv(output_filename, index=False)
+            logging.info(f"  -> Successfully created {output_filename} with {len(final_df)} rows.\n")
+        except Exception as e:
+            logging.error(f"  -> An error occurred during the final processing for {gas}: {e}\n")


please raise the exception again after log statement so that import script can stop on error.

ajaits · 2026-03-02T13:01:29Z

statvar_imports/climate_trace/manifest.json

+      "provenance_description": "Climate Trace data provides comprehensive, granular, and up-to-date greenhouse gas emissions data using satellite imagery and other public data sources.",
+      "scripts": [
+        "download_and_segregate_by_gas.py",
+        "../../tools/statvar_importer/stat_var_processor.py --input_data=input_files/all_countries_ch4.csv --pv_map=climate_pvmap_sectors.csv --config_file=common_metadata.csv --output_path=output/sectors_CH4_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf",


How large are the input files?

if they are small enough (<100Mb), can we merge the pvmap and process all CSvs together?

../../tools/statvar_importer/stat_var_processor.py --input_data=input_files/*.csv --pv_map=climate_pvmap.csv --config_file=common_metadata.csv --output_path=output/climate_trace_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf

ajaits · 2026-03-02T13:14:40Z

statvar_imports/climate_trace/climate_pvmap_sectors.csv

+2028-01-01 00:00:00,observationDate,2028,,,,,,,,,,,,
+2029-01-01 00:00:00,observationDate,2029,,,,,,,,,,,,
+2030-01-01 00:00:00,observationDate,2030,,,,,,,,,,,,
+emissions_quantity,value,{Number},populationType,Emissions,measuredProperty,amount,measurementQualifier,Annual,#Aggregate,sum,observationPeriod,P1Y,unit,MetricTon


Can you please add a note in the README on what data is being aggregated?

ajaits · 2026-03-02T13:15:09Z

statvar_imports/climate_trace/common_metadata.csv

@@ -0,0 +1,6 @@
+parameter,value
+header_rows,1
+output_columns,"observationAbout,observationDate,variableMeasured,value"


please add measurementMethod as well if data is being aggregated.

Added - unit, measurementmethod and observationperiod

Climate Trace GHG Emissions refresh

416c26c

gemini-code-assist bot reviewed Feb 5, 2026

View reviewed changes

HarishC727 and others added 2 commits February 6, 2026 08:48

resolved gemini comments

15b2c25

Merge branch 'master' into climate_trace_ref

6b45269

HarishC727 added 2 commits February 13, 2026 08:46

Merge branch 'master' into climate_trace_ref

a17f6ee

Merge branch 'master' into climate_trace_ref

3593055

saanikaaa reviewed Feb 16, 2026

View reviewed changes

statvar_imports/climate_trace/climate_pvmap_subsectors.csv Show resolved Hide resolved

saanikaaa reviewed Feb 16, 2026

View reviewed changes

statvar_imports/climate_trace/climate_pvmap_sectors.csv Show resolved Hide resolved

resolved internal comments

be09dc6

ajaits reviewed Mar 2, 2026

View reviewed changes

Merge branch 'master' into climate_trace_ref

b6a6327

Conversation

HarishC727 commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot commented Feb 5, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pulkeet-wq commented Feb 6, 2026

Uh oh!

Uh oh!

Uh oh!

saanikaaa commented Feb 16, 2026

Uh oh!

HarishC727 commented Feb 16, 2026

Uh oh!

saanikaaa commented Feb 16, 2026

Uh oh!

ajaits Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HarishC727 commented Feb 5, 2026 •

edited

Loading

ajaits Mar 2, 2026 •

edited

Loading