Welcome, Guest
Username: Password: Remember me
Forum on HARMONIE Climate Branch

TOPIC: Spectral nudging or hindcast mode

Spectral nudging or hindcast mode 3 years 5 months ago #2161

  • Oskar Landgren
  • Oskar Landgren's Avatar
  • Fresh Boarder
  • Using HCLIM for Arctic climate projections
  • Posts: 6
Hi Bert and everyone!
Hope you are all well! I have a few questions about spectral nudging and hindcast mode.
I have set up HCLIM38h1 on ECMWF to run with ALARO over a pan-Arctic domain at 24 km. (The polar stereographic projection works after some updates by David Lindstedt to hor_interpol_rotlatlon.F90 and prep_grib_grid.F90 (both in src/surfex/SURFEX)).
The original plan was to try to participate in a voluntary exercise from Polar CORDEX, where a comparison is made with measurement data from a ship campaign in the summer of 2014 (ACSE, Arctic Clouds in Summer Experiment).
Because the simulations will be directly compared to point measurements a spectral nudging approach would be preferable, but as far as I know that is not possible in HCLIM with ALARO or ALADIN, but only with AROME. Is that correct?
At least LESPCPL is not defined for ALARO in the nam/harmonie_namelists.pm, but for AROME and MUSC:
L683: %arome / NEMELBC0A / LESPCPL
L3415: %musc / NEMELBC0A / LESPCPL
Is there another variable that could enable spectral nudging in HCLIM with ALARO (or ALADIN for that matter)?

Then I read about the hindcast mode that you have implemented, but I unfortunately could not manage not make it work.
What is the proper way to run it?
If I start an experiment directly with SIMULATION_TYPE=hindcast, I do not get any run showing in ECFLOW, and just an error in the ECF.log saying
ping server(ecgb11:5241) succeeded in 00:00:00.013736 ~13 milliseconds
ecflow_client --port 5241 --host ecgb11 --begin H38h1
Error: request( --delete force yes /H38h1 :fa0o ) failed! Server replied with: 'PathsCmd:Delete: Could not find node at path '/H38h1''

ecflow_client: Caught exception whilst parsing arguments:

LoadDefsCmd::LoadDefsCmd. Failed to parse file harmonie.def
Failed to parse complete ( ( 0 eq '0' ) )
at TASK:/H38h1/MakeCycleInput/Hour/Cycle/Observations/Prepradar because Parsing failed
length = 18446744073709551615
stopped at: ": eq '0' ) )"

args: ecflow_client --port 5241 --host ecgb11 --load=harmonie.def

Error: request( --begin=H38h1 :fa0o ) failed! Server replied with: 'BeginCmd::doHandleRequest: Begin failed as suite 'H38h1' is not loaded.'

fa0o ecgb11 5241

In sms/config_exp.h it says that the hindcast mode "initialise surface with file from previous cycle".
Should I first start an experiment with e.g. SIMULATION_TYPE=climate, let it initialize and run for a day, then change to SIMULATION_TYPE=hindcast, and then use "Harmonie prod" or something?

Best regards,
Last Edit: 3 years 5 months ago by Oskar Landgren.

Spectral nudging or hindcast mode 3 years 5 months ago #2162

  • Bert van Ulft
  • Bert van Ulft's Avatar
  • Expert Boarder
  • Posts: 109
  • Thank you received: 22
Hi Oskar,

mmm, I have no experience with the spectral nudging option. You could try to simply add the relevant namelist variables for ALARO as well and see what happens. There is a commit by Samuel Viana for harmonie-43 which enables spectral nudging for AROME: hirlam.org/trac/changeset/16655. Perhaps that commit can push you in the right direction.
Regarding the hindcast option, your error seems to be missing, could you post it please. One thing to check is that SURFEX_LSELECT is switched off in sms/config_exp.h, because that option is currently is not compatible with the hindcast option. But this is checked in CheckOptions.pl.



Spectral nudging or hindcast mode 3 years 5 months ago #2163

I have no idea about the possibility to use spectral nudging for ALARO, but I agree with Bert that you could start by just copying the whole LESPCPL section from AROME to ALARO and see what happens:
'NEFRSPCPL' => '1,',
'NEK0' => 'ZNEK0,',
'NEK1' => 'ZNEK1,',
'NEN1' => 'ZNEN1,',
'NEN2' => 'ZNEN2,',
'SPNUDDIV' => '0.01,',
'SPNUDQ' => '0.0,',
'SPNUDT' => '0.01,',
'SPNUDVOR' => '0.01,',

Don't pay attention to my commit for harmonie-43, this was done because the spectral nudging code was rearranged from cy38 to cy43 so I had to rearrange the namelists accordingly.

About one year ago Patrick discussed with Xiaohua Yang about the use of spectral nudging, I'm going to just paste here the e-mails (read from bottom to top) as I think they might be useful for you. Mmm now I see he mentions a script where apparently some spectral nudging constants are pretty linked to AROME...you might also contact Xiaohua for help.


Hi Patrick and Magnus

Sorry for having overlooked this thread. In the NWP code we introduced upper boundary relaxation scheme LUNBC devised by Mariano Hortal with inspiration from lateral boundary relaxation, and disabled in that connection upper boundary spectral nudging. If for climate modelling purpose you choose to use the latter instead, please make sure to disable LUNBC. I don't know how the two shall work together but the first assumption is to choose either and not both.

If you have selected the spectral nudging LESPCPL=.TRUE., than the hirlam script shall have worked out the coefficient automatically, unless you use a different vertical levelling. In that case the explanation text in the HARMONIE script can be followed.

mvh Xiaohua

Xiaohua Yang, Ph D
Danish Meteorological Institute
Lyngbyvej 100, 2100 Copenhagen, Denmark


From: Samuelsson Patrick [Patrick.Samuelsson [at] smhi [dot] se]
Sent: 26 October 2017 18:17
To: Xiaohua Yang; magnus.lindskog [at] smhi [dot] se
Cc: svianaj [at] aemet [dot] es
Subject: SV: mixing of ecmwf at upper levels

Hi Xiaohua,

Thanks for your reply but it is not totally clear for me how the setting works...
So we use upper boundary spectral nudging by default in our operational AROME?

Should one play around with these numbers (100000 and 50000) depending on domain and expected setup?

gs=$( echo "$GSIZE" | cut -f 1 -d . )
ZNEK0=$( expr \( "$NLON" \* "$gs" \) / 100000 || echo "" )
ZNEK1=$( expr \( "$NLON" \* "$gs" \) / 50000 || echo "" )

The reason I ask is because we plan to set up cy43h in climate mode (without any data assimilation) and would like to give AROME a bit less freedom to invent its own "large" scales but force the simulation to follow more closely the large scales of the forcing model (e.g. ERA-Interim).



Ursprungligt meddelande
Från: Xiaohua Yang [mailto:Xiaohua [at] dmi [dot] dk]
Skickat: den 26 oktober 2017 17:04
Till: Lindskog Magnus; Samuelsson Patrick
Ämne: RE: mixing of ecmwf at upper levels

Hej Magnus,

If you use upper boundary spectral nudging you can modify the script scr/forecast_model_settings.sh to adapt your vertical levels The following lines in that scripts describes principle

# Domain dependent settings for transition zones in AROME spectral nudging # horizontally it is approximately 50 and 100 km in wavelength # vertically a height correspnding to around 50 and 100 hpa

mvh Xiaohua

Xiaohua Yang, Ph D
Danish Meteorological Institute
Lyngbyvej 100, 2100 Copenhagen, Denmark
Last Edit: 3 years 5 months ago by Samuel Viana Jiménez. Reason: Protect email addresses

Spectral nudging or hindcast mode 3 years 4 months ago #2164

  • Oskar Landgren
  • Oskar Landgren's Avatar
  • Fresh Boarder
  • Using HCLIM for Arctic climate projections
  • Posts: 6
Thank you Samuel and Bert for your replies!

Bert, I have updated the original post with the error log in ECF.log. It seems to be about not being able to find radar data, but assimilation is turned off so it is a bit strange. The only possibly suspicious settings I can find in my sms/config_exp.h are that I'm using the default OBSMONITOR=plotlog:obstat and OBSEXTR=bufr. This did not matter when running in climate mode however, and changing them to "no" does not help.
I'm afraid I don't quite understand the syntax of the msms/harmonie.tdf file
(lines 183-203 posted here)
Warning: Spoiler! [ Click to expand ]

but if I comment out that Prepradar call I get a new error in ECF.log:
sh: HH2LL.pl: command not found

This is likely from line 110 in scr/submission.db:
105: if ( $ENV{SIMULATION_TYPE} eq 'climate' ) {
106:   if ( $dtg > 0 ) {
107:     $LL=`export DTG=$dtg ; Update_LL 2>&1`;
108:   }
109: } else {
110:  $LL = qx( DTG=$dtg; HH2LL.pl $HH $ensmbr );
111: }
which leads me to think that I'm not in the right mode. Should there be a "eq 'hindcast'" check here as well?
Or maybe I have something wrong in my config_exp.h
Warning: Spoiler! [ Click to expand ]


Samuel, I copied the lines you sent, set LESPCPL to true (and LUNBC to false in config_exp.h), but the simulation crashes during Forecast after 9 timesteps, with very high U and V wind values, >200 m/s. The traceback is cryptic but seems to refer to ACTQSAT (or possibly src/arp/phys_dmn/acnebcond.F90:411, which is ZAW(JLON)=ZALFI*SQRT(ZDC(JLON)/PQW(JLON,JLEV)) )
but it's probably just something that has propagated due to the too high winds, so the error may be hidden earlier.
I'll try to dig a bit further, but if you have any tips I'm all ears.
Should I scale some of the parameters by the lateral resolution (AROME 2.5 km vs my ALARO at 12 km)?

(Note that the hindcast and the spectral nudging of course are two separate experiments. I'm not trying to implement both in the same experiment.)
Last Edit: 3 years 4 months ago by Oskar Landgren.

Spectral nudging or hindcast mode 3 years 4 months ago #2165

  • Bert van Ulft
  • Bert van Ulft's Avatar
  • Expert Boarder
  • Posts: 109
  • Thank you received: 22
Hi Oskar,

the problem with Prepradar seems to be caused by comparing a number with a string:
complete ( ( $ENV{RADAR_OBS} eq '0' ) )
Changing this to the following seems to fix this
complete ( ( $ENV{RADAR_OBS} == 0 ) )
But the same lines exist in the trunk, so I'm not sure why we see this problem all of sudden.
The HH2LL problem I introduced while manually copying changes to get it working on the ECMWF again, but only tested the climate setup. The HH2LL.pl script isn't in our scr directory. Simply adding it doesn't work, because it needs the new HH_LIST, LL_LIST variables, which we don't have yet in HCLIM. Probably the easiest solution is to revert a part of submission.db, change:
109 	} else { 
110 	  $LL = qx( DTG=$dtg; HH2LL.pl $HH $ensmbr ); 
111 	} 

to what it was before:
} else {        
  if ( $ENV{LLMAIN} ) {         
    # Run shorter forecasts at 06/18 if LLMAIN is set
    $llshort = sprintf("%02d", &Env('FCINT',$ensmbr));  
    if ( $HH % 12 ) { $LL = $llshort; } else { $LL = $ENV{LLMAIN}; }    
  } else {

I hope this gets you a bit further again. Would be worthwhile to commit these fixes once you have everything working.

best wishes,


Spectral nudging or hindcast mode 3 years 4 months ago #2197

  • Oskar Landgren
  • Oskar Landgren's Avatar
  • Fresh Boarder
  • Using HCLIM for Arctic climate projections
  • Posts: 6
Hi Bert!
I discovered that because I had set OBSMONITOR=no, then sqlite was not built and Makeup crashed in
   USE sqlite,   ONLY      : SQLITE_DATABASE
Fatal Error: Can't open module file 'sqlite.mod' for reading at (1): No such file or directory
compilation terminated.
Perhaps I have an error earlier on, because I don't see why the obsmon module should be used at all in this setup.
Anyhow, setting OBSMONITOR=plotlog:obstat as before, then sqlite is built as it should but the simulation crashes at MakeCycleInput/Hour/Cycle/Observations/Oulan with message:
/scratch/ms/no/fa0o/hm_home/hctest/lib/scr/Oulan: line 161:
/scratch/ms/no/fa0o/hm_home/hctest/bin/ShuffleBufr: No such file or directory
/scratch/ms/no/fa0o/hm_home/hctest/lib/scr/Oulan failed

Changing back to OBSMONITOR=no and running with BUILD=no I get a crash in
/scratch/ms/no/fa0o/hm_home/hctest/lib/scr/Climate: line 15:
cd: /scratch/ms/no/fa0o/hm_home/hctest/20130904_00: No such file or directory
This is because the Postprocessing step starts before Forecast, and I believe SaniDisk removes the directory.

There are a lot of other differences to the structure of the hindcast run. I was expecting it to be more like the climate run, but I guess you have made it from some NWP routines.
For example, in Climate mode the tasks under Postprocessing/PPtasks were as follows:
Extract4ver, convertFA, Archive_c2a, Disk_cleaning
In hindcast mode I get:
Does that mean that we cannot use convertFA (as in climate mode) to get results in NetCDF format from the hindcast mode?

After some digging I found that the premature start of the Postprocessing/Hour/Cycle task was a bug that Ulf had corrected in trunk in hirlam.org/trac/changeset/15509
(On a side note, the variable names in msms/harmonie.tdf are not the same as the ones I can get in ecFlow, so it's a bit challenging to debug. E.g. harmonie.tdf mentions $ENV{StartHourPP}, but ecFlow only lists StartHour, which is the start hour of the simulation. Is there a way of checking the values of these variables without modifying the script to print them?)

Noting that there are very many differences between our harmonie.tdf and the one in trunk I just copied the trigger for the Postprocessing/Hour/Cycle and now the Postprocessing wait a bit longer but not enough for Date to complete. Anyhow I believe I can solve this if I look a bit more at the triggers in harmonie.tdf.
Rerunning Climate manually worked, but afterwards Oulan crashed again as above, complaining about not being able to find SHUFFLEBUFR. I assume this should not be used if I have turned off obs verification. Do you have any ideas?

Did you use another branch when you had hindcast working? Or are there such big differences when running on ECMWF compared to what you did at the KNMI HPC that it could cause this?

All the best,

Spectral nudging or hindcast mode 3 years 4 months ago #2199

  • Bert van Ulft
  • Bert van Ulft's Avatar
  • Expert Boarder
  • Posts: 109
  • Thank you received: 22
Hi Oskar,

the hindcast is indeed something in between an NWP and climate simulation and uses the harmonie.tdf playfile, instead of climsim.tdf. I set this up in the 38h1.2 NWP version when we started experimenting with HARMONIE as a climate model at KNMI, because at that time HCLIM was based on 38h1.1 and did not run properly with AROME physics. Afterwards I copied these modifications to HCLIM, but apparently didn't test it very well.
Sqlite is not build if OBSMONITOR=no, but still used somewhere. As a workaround you could remove the line:

complete [ $ENV{OBSMONITOR} eq 'no' ]

from msms/build.inc, just below task Make_sqlite3. That way sqlite is always build and you can leave OBSMONITOR set to no.
I just started a test run on the ECMWF and run into the sqlite/obsmon problems as well. Oulan kept crashing, even after setting OBSMONITOR to no. So either we need to build these executables (see ./util/makeup/defaults.mk), or exclude those tasks if running in hindcast mode.
I did not see the problem with Climate you describe. Also Postprocessing seemed to wait as it should.
However, Listen failed, because variable SMSNAME was not set. I modified msms/smsmeter so it is the same as scr/meter from trunk and now that seems to work fine. convertFA wasn't in harmonie.tdf, but can be added quite easily.
Have a look at the experiment on ecgate in ~nkl/hm_home/HCLIM38h1_test, which uses RACMO boundaries and run with cycles of LL=36 and FCINT=24. Modified in there:
  • don't run Oulan for hindcasts
  • Prepradar fix
  • fix for racmo as host model
  • added support for convertFA
  • Always build sqlite3
  • working in ecflow
This works for the first cycle (cold start), but for the second cycle it crashes, because the ICMSHFULL file cannot be found around line 285, so you need to modify the Forecast a bit so it also works if SURFEX_LSELECT is not used.
You can view variables that have been set at the start of the simulation in $SCRATCH/hm_home/$EXP/_ecFlow_submit_env on ecgate. When using mSMS there is an equivalent with mSMS in the name. If that isn't enough, often adding 'set -x' at the beginning of a script is the easiest solution.
Bit of a messy message, but I hope it gets you a bit further.

best wishes,

The following user(s) said Thank You: Oskar Landgren

Spectral nudging or hindcast mode 3 years 4 months ago #2200

  • Oskar Landgren
  • Oskar Landgren's Avatar
  • Fresh Boarder
  • Using HCLIM for Arctic climate projections
  • Posts: 6
Thank you so much!
Time to create page: 0.162 seconds