NEMSIO vs. NETCDF

[MarcusGDaniels]

Hi,

I'm trying to set up initial conditions using GFSv.16 and I find that one of the scripts ush/global_chgres_driver.sh wants to load nemsio files instead of netcdf as is provided here https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20221221/00/atmos/ (for example, gfs.t00z.sfcanl.nc).

I looked at both v16.2.2 and v16.3.4 and they both have this way of loading.

Because I don't have access to HPSS, the instructions on the wiki won't work for me.

Thanks for any suggestions!

Marcus

[GeorgeGayno-NOAA]

The global_chgres_driver.sh script and its program were deprecated and removed from the official UFS_UTILS fork.

[GeorgeGayno-NOAA]

Do you have access to any NOAA RDHPCS machines?

[MarcusGDaniels]

Hi George, No, I am standing up a build of GFS on another supercomputer. Marcus

…

________________________________ From: GeorgeGayno-NOAA ***@***.***> Sent: Tuesday, January 3, 2023 1:47 PM To: NOAA-EMC/global-workflow ***@***.***> Cc: Marcus Daniels ***@***.***>; Author ***@***.***> Subject: Re: [NOAA-EMC/global-workflow] NEMSIO vs. NETCDF (Discussion #1200) Do you have access to any NOAA RDHPCS machines? — Reply to this email directly, view it on GitHub<#1200 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AACVVP3FROYFBF5NTUKYIZTWQSF5HANCNFSM6AAAAAATGDKKWU>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[GeorgeGayno-NOAA]

I don't know of a way to generate those files on your own.

[MarcusGDaniels]

Any suggestions on who to ask? It seems like a bug these were dropped.

[GeorgeGayno-NOAA]

I don't think it is a bug. It is a lack of disk space. There is just no room for the 80 ENKF atmospheric files.

[MarcusGDaniels]

It isn't possible to independently reproduce the ensemble predictions. That can't be a good thing from a scientific perspective? Also, why share half of them? It seems to be a waste of that disk space too. At one point according to the screenshot above, they were being shared. It seems as though they were pulled back.

[MarcusGDaniels]

Pull the NetCDF data you want to use to your machine. Then, clone and compile the latest version of UFS_UTILS 'develop'. I can help you run the 'gdas_init' utility without HPSS access.

Hi, I have both the atmospheric and surface netcdf ensemble sets now. What is the next step?

[aerorahul]

@MarcusGDaniels
If you have need for a single case of initial conditions, we can provide.
However, please note that this data is not publicly available due to its size.
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/ is the holding tank for a few days of current data.
The ensemble data would be under enkfgdas, but as you note is not complete for your needs.
There must have been a Public Notification Statement issued when it was decided to cut down the data volume. A PNS typically has a window to gather public comment.

[MarcusGDaniels]

A single case of initial conditions would certainly be of interest. If funding is needed for more storage, I can investigate getting it.

[MarcusGDaniels]

It seems like there must be code somewhere for generating the 80 variants?

[MarcusGDaniels]

We'd like to do production uncertainty characterization at C768. C48 would enable us to rationalize the workflow, which would be valuable.

What I mean about the 80 is if there exists code to expand a single gdas set into a set of 80? Is it correct to say that each of the 80 members is "raw" data? Or is each member derived by an algorithm?

[MarcusGDaniels]

Will the rocoto workflow of eobs/ediag/eomg/eupd/econ/esfc/efcs/echgres/epos do the job? (Assuming I could get that to work.)
Or will there still be dependencies missing?

[MarcusGDaniels]

How was this process bootstrapped? Would it converge toward the set of mem*/atmfXXX files you have at C768 starting with C48 or just copies of a sfc and atm files like at the URL below? How much disk space is it, in fact? Do you offer a way to buy this data as a subscription?

https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gdas.20230314/00/atmos/

[MarcusGDaniels]

Hi again,

Could I try the 80 of C48?

Is it the case that chgres will let me mix those with the full resolution surface data?
If so, might it possible to make those C48 runs part of the workflow so they would be published each day?

Thanks,

Marcus

[MarcusGDaniels]

In case the issue is S3 bandwidth costs, I can provide a login to do a direct rsync.