problems with exports while running adjoint

[kaysuselj]

Your name

kay suselj

Your affiliation

JPL

Please provide a clear and concise description of your question or discussion topic.

I am trying to run an older GCHP version from Colin Lee and have problems exporting outputs when running adjoint model. I modified Colin's setup by including different version of CMS-Flux emissions restart file and start time -- I don't have an access to the restarts and emissions provided by Colin.

My understanding is that after running the adjoint variables are to be written in the Adjoint and SFEmission files, but all the fields there are zero. Warnings in HEMCO.log indicate that there is a problem with the CMS-Flux emission files and adjoints of emission:
--> LOCATION: DiagnCont_PrepareOutput (hco_diagn_mod.F90)
HEMCO WARNING: Diagnostics counter is zero - return empty array: SFEmisCO2_Total

HEMCO.log did not indicate any problems with the forward run simulation.

Any help would be appreciated!

Thanks,

Kay
ExtData.rc.txt
HEMCO_Config.rc.txt
HEMCO_Diagn.rc.txt
HEMCO.log.txt
HISTORY.rc.txt

[lizziel]

Hi Kay,

To start you should enable all HEMCO debug prints by updating HEMCO_Config.rc verbose and warnings. What version of GEOS-Chem are you using? The latest version uses a logical for HEMCO verbose, but your config file indicates you are using a much older version. If that is the case you can set verbose and warnings to 3.

The warnings in HEMCO you see indicate that the emissions containers in HEMCO were not updated. Every time they are updated with new data the diagnostics counter is incremented. If it is always zero then the output will be zero.

So the next question is why aren't they being updated. Is data being read for them? Depending on your GCHP version, you can set a logging entry for ExtData to debug to enable prints about files being read. Instructions for this are in current ReadTheDocs, on the debugging page.

As far as I can tell your ExtData.rc and HEMCO_Diagn.rc are correct, at least for the entries that have actual input files specified (not /dev/null). The wildcard for me is the backwards flag in HISTORY.rc. I have not run the adjoint so I am not sure what that is doing. The GCHP log is currently the only place where MAPL history output is sent. Do you have information there about the collections being written?

[kaysuselj]

Hi Lizzie,

Thanks. My versions of the code are:
GEOS-Chem 13.0.0 (dev)
HEMCO 3.0.0 (dev)

I did change the verbose versions in HEMCO_Config.rc and logging.yml files. I am attaching the log files -- I do not see anything weird in there, would you mind taking a look (the attached allPEs.log includes only a fraction of the file)? The HISTORY collection seem to be initialized properly.

There is an error message in the PETXX.ESMF_LogFile. Does that indicate the problem, and if so what does it mean.

Thanks,

Kay
allPEs.log.txt
gchp.log.txt
HEMCO.log.txt
PET00.ESMF_LogFile.txt

[lizziel]

Hi Kay,

It looks like your run crashed with an MPI issue. Did it ever run to completion?

13.0.0 is very old at this point and not something we support. I suggest trying to build the latest version with adjoint enabled and tackling the inevitable compile errors in the adjoint-only code. Comparing it side-by-side with the forward model version of the code along with the compiler errors should make it doable to update. With that version you will have MAPL 2.26.2 and the most up-to-date debug tools. With the older version I don't think GMAO will be able to help much since it has a very, very old version of MAPL.