Welcome, Guest
Username: Password: Remember me
Forum on HARMONIE Surface development

TOPIC: Surface Data Assimilation

Surface Data Assimilation 5 years 2 months ago #1702


A joint HIRLAM/ALADIN/LACE/SURFEX Surface Working Week, focusing much on SURFEX Data Assimilation in SODA, was just held in Zagreb Croatia. See Workshop wiki here:

One of the outcomes of the Workshop was that we agreed to communicate our SODA development via a common branch in the SURFEX repository:
along with development descriptions on this wiki:

We would prefer to use the SURFEX open source environment for communication but while some issues still remain to be solved there we offer the HIRLAM environment meanwhile.

Thus, for the moment, if you wish to announce or keep yourself updated with SURFEX Data Assimilation development following the Zagreb working week please consider to sign up on this forum subject.



Surface Data Assimilation 5 years 1 month ago #1748

Hi all,

Forward an email below regarding a new mailing list devoted to land surface data assimilation.



From: This email address is being protected from spambots. You need JavaScript enabled to view it. [ This email address is being protected from spambots. You need JavaScript enabled to view it. ] För Pullen, Samantha
Sent: den 29 november 2016 17:37
To: This email address is being protected from spambots. You need JavaScript enabled to view it.
Subject: Invitation to join an international land surface DA mailing list

Dear WG3 Members,

At the Met Office, the team working on land surface data assimilation has recently set up an international land surface data assimilation mailing list. This can be used for purposes such as discussion of research and developments at our respective institutes, circulating relevant publications and advertising relevant workshops and conference sessions, and raising issues and initiatives of interest to the land DA community.

Those of you who attended the Granada meeting will already have heard about the list, and some of you have already subscribed. If not,
I’d like to invite you to join the mailing list, and/or forward the invitation to other colleagues with an interest in land surface DA. The list is managed at the Met Office (current list owners Breo Gomez and Samantha Pullen), but is intended for the wider international community.

To subscribe to the list (use plain text):
Send an email to: This email address is being protected from spambots. You need JavaScript enabled to view it.
In the body write : subscribe land-surface-da

To write to the list:
Mail to: This email address is being protected from spambots. You need JavaScript enabled to view it.

To contact list owners:
This email address is being protected from spambots. You need JavaScript enabled to view it.
This email address is being protected from spambots. You need JavaScript enabled to view it.

Please forward this information to any colleagues who might be interested,

Many thanks,

Dr Samantha Pullen NWP SAF Manager Satellite Applications
Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
Tel: +44 (0)1392 886876 Fax: +44 (0)1392 885681
E-mail: This email address is being protected from spambots. You need JavaScript enabled to view it. www.metoffice.gov.uk

Surface Data Assimilation 5 years 2 weeks ago #1772


During the autumn 2016 a few of us have discussed how to proceed with surface assimilation development considering that our NWP system now is more and more used in EPS mode and that we should incorporate more advanced surface processes (prognostic variables). In the long term we have expressed that we should head for a fully coupled 3D-EnKF-atmosphere-surface assimilation system but we need some steps in between. Which are these steps? Should we first develop a fully operational EKF surface assimilation system including more advanced surface processes? Or, should we go directly to an EnKF system in combination with the more advanced surface processes? Or, should we have the EKF system only as a development step but never make it fully operational?

Thus, there are some strategic questions on how to proceed. It may seem early to bring this discussion up now considering that we are still struggling with Force-restore and OI but I would like to share opinions and thought at least since the working plan for some of us involves aspects of these questions quite soon.

In our current NWP system with Force-restore and 1 patch we perform assimilation of 5 surface prognostic variables; 2 soil temperatures (TG1, TG2), 2 soil moistures (WG1, WG2) and snow water equivalent (SWE). In a future system, with more advanced surface processes, we may have 2 patches, diffusion soil (default 14 temperature layers) and explicit snow scheme (default 12 layers). Some of these layers represent relatively slow processes (e.g. deep soil layers) and may not be included in the assimilation, but still quite a few variables will be considered for assimilation. As an example let's assume

( ( 5 top-mid soil layers ) * 2 variables + ( 5 snow layers * 2 variables ) ) * 2 patches = 40 variables

Running EKF surface assimilation in an EPS NWP system with let's say 10 members would mean that all assimilated surface variables would be assimilated separately for each ensemble member. That would mean for example 40 variables times 10 members = 400 EKF-perturbed offline SURFEX runs! Hmhm, is that affordable? Maybe yes considering that we could run the perturbed runs in parallel....?

But, if we go for an EnKF surface assimilation system, how many perturbed members would then be reasonable? Less than 40 per EPS member? But more efficiently, and more physically correct, would be to run a fully coupled 3D-EnKF-atmosphere-surface assimilation system, right?

With our EPS colleagues we are also currently discussing how to proceed with surface perturbations in EPS to increase the spread, i.e. perturbing for example soil temperatures and soil moistures. Essentially such EPS surface perturbations should not be different from soil perturbations in an EnKF surface assimilation system, right? So, running EnKF in combination with EPS would solve two perturbation needs in one go, right?

Ok, many questions! In my mind right now, after been talking to a few of you, a reasonable way forward could be to combine EKF surface assimilation with more advanced surface processes as a development step but that such a system necessarily does not become operational. In such a development step we learn how more advanced surface processes work in combination with quite a well established assimilation method. Such a system could then be used as a reference when we go for EnKF methods...

Please share your ideas and thoughts!


Surface Data Assimilation 5 years 6 days ago #1774


Here are some thoughts on this from my perspective.


Coupled DA for Atmosphere-Land-Ocean using an ensemble method. Several institutes are heading in this direction. See proceedings from last years "International workshop on coupled data assimilation": www.meteo.fr/cic/meetings/2016/CDAW2016/presentations.html


Take steps towards fully coupled DA using physically consistent ensemble members for atmosphere, surface and ocean but using separate assimilations for a start.

- Replace deterministic CANARI with ensemble OI to assimilate t2m and rh2m (snow information assimilated together with other SURFEX variables).

- It would be really nice to use EnKF for surface DA and use the EPS members for forcing. Keep the surface and atmosphere members tied together to assure physical consistency between them. Avoid ensemble transform like methods where only the first and second moments need to be represented by the members after data assimilation. Maybe 10 members are too few to capture the evolution of some 40 surface state variables but then again maybe not - do we know how correlated they are?

- When it comes to EKF I agree with Patrick's suggestion to continue and experiment with EFK and deterministic forcing to learn about more advanced surface schemes (e.g. correlation between different control variables) since the EKF set-up is already up and running.

I'll contribute to this work during the spring by running SURFEX off-line and force with perturbed MetCoOp data from our MARS archive. The idea is to spin up a SURFEX ensemble for a winter season (3-6 months). The perturbations in the forcing will be generated from NMC like statistics and have horizontal correlations. From this experiment I hope to learn about horizontal and vertical correlations between SURFEX control variables (exact set-up not yet decided). This could help answer the question about the degrees of freedom for the surface model (could 10 members suffice) and also if it would be beneficial to do a 3D EnKF for the surface, taking horizontal correlations into consideration, instead of using independent 1D filters for each grid point.


Surface Data Assimilation 5 years 37 minutes ago #1783

  • Bart van den Hurk
  • Bart van den Hurk's Avatar
Indeed, the way forward is to go for EnKF with a physically realistic surface scheme. What I doubt, though, is whether a soil scheme with 40 prognostic variables requires all variables to be initialized independently. These variables themselves are highly correlated and there are no 40 independent degrees of freedom in the system. So whatever system is to be explored, some research should be devoted to the required dimensionality of the DA in order to optimize the use of computing resources.

Surface Data Assimilation 4 years 11 months ago #1784


I am not sure if there is overall agreement within the consortium on a 3D coupled EnKF as the way forward for the different components of the Earth system. Others may favour an EDA approach, following the ECMWF. Until such agreement is there we should focus on DA for the land surface.

I agree that EKF-developments focussing on advanced surface processes is useful to study e.g. covariance modelling and filter behaviour and thus should continue.

If you run an ensemble of EKFs, like EDA, with 10 members you run 10 perturbed versions of SURFEX in parallel, each coupled to an EKF. The filters have as you say a control vector of 40 variables, which is expensive to run. However you may be able to limit the control vector to those variables directly affected by and closest to the observations, resulting in perhaps 10 variables.

I would not use the EKF in an EPS. That is what the EnKF seems to be invented for. Then you have an ensemble of 10 members and one filter that solves the Kalman equations. Cheaper to run but the EPS is no longer embarrassingly parallel and no seamless shift from deterministic DA.

If we look at the land surface as part of Harmon-EPS an EnKF-type DA-system is a natural option. Two aspects come to mind, coupled DA (not here) and creating perturbations. The latter should focus, as discussed here, on model uncertainty.

Perturbation of the Land Surface in a calibrated LSM of a coupled system needs to focus on the sources of uncertainty in the simulation of the surface fluxes.

Using a calibrated model anticipates on the work currently being done by me to reduce biases in ISBA. It is not a mandatory prerequisite though.

We need to distinguish between cause and consequence (of a cause elsewhere). Uncertainty in the land surface model (SURFEX) is present in the atmospheric forcing, the model structure, in model variables, model parameters and observations.

The uncertainty in the atmospheric forcing is external, imported through the atmospheric part of the ensemble members. This uncertainty then propagates into the model state variables and other outputs of SURFEX (consequence). Within SURFEX sources of uncertainty exist that cause a modulation of the imposed uncertainty.

Such sources of uncertainty lie in the model parameters of the description of the soil mechanics and the diagnostics/prognostics to compute the surface fluxes.These model parameters are assumed to be constant on the time-scales of interest. The uncertainty arises in their real value. Specified values often originate from laboratory experiments, which are not universally applicable. Real values are uncertain both spatially and in time, e.g. because of lack of knowledge (cause) of e.g. the local composition of the soil, which affects the specification of hydraulic parameters, the soil depth, which determines the total available water content or the minimum stomatal resistance to scale LAI in determining the canopy resistance, etc.

Variation in model parameters changes the behaviour of the model in a non-linear way and an LSM has many parameters. A subset of those parameters may contain critical values in their value range that could lead to considerable variability in model output(s) of interest. Knowledge of and control over those individual sensitive parameters may dampen or amplify extreme model behaviour in the ensemble members. On top of this, interactions between parameters holding a certain value even if the individual parameters are not sensitive can lead to large variability in model behaviour as well.

Uncertainty in observations translates into instrument errors and errors of representativeness. The latter includes errors related to the fact that different sensors producing the same observations do not always measure the same physical quantity. Moreover, observations are often sparse, both spatially and temporally, and parts of the model are unobserved and represent a constant source of uncertainty (deep soil).

Parameter sensitivity can be a way to identifying important sources of uncertainty in the model. The goal is to gain, through sensitivity analysis, the knowledge on parameters that explain a large part of the variability in model outputs of interest (surface fluxes), both individually and through interaction. This knowledge will enable us to reduce systematic errors in the model behaviour based on historic information (observations). Improvement of the accuracy of the predictions of future states of the land surface may be achieved by combining this knowledge on parameter behaviour with new information (e.g. observations) and apply it to not only correct the state of the model but also its behaviour by estimating the evolution of the sensitive parameters.

Data assimilation can add value to the forecast skill of the model when biases in key model variables are sufficiently small. Hence the relevance of using a calibrated model. What sufficiently means depends on the application and the magnitude of random errors. It is difficult to generalize.

A useful in-between step towards Harmon-EPS DA in my view is doing off-line ensemble DA experiments with a calibrated version of SURFEX to research the merits of a joint state and parameter estimation approach in the assimilation of soil moisture observations from e.g. ASCAT,AMSR2,SMAP and using atmospheric forcing from a 10-member Harmon-EPS run.

In such experiments a calibrated ISBA-DIF scheme could be used in the model structure because it is likely to lead to more realistic behaviour on longer time scales than Force-Restore. This could provide the potential for improving the accuracy in the description of the surface fluxes in periods of drought, extreme precipitation and prolonged cloudiness.

Knowledge gained on sensitive parameters can be used to experiment with specific sensitive parameters using sampling techniques catered to their probability distributions. In case sensitivity through interaction is relevant, a constraint between parameters should be implemented in the sampling to enforce desired behaviour. An EnKF type assimilation scheme is envisaged initially, or possibly a hybrid approach in combination with a particle filter for the parameter estimation.


Surface Data Assimilation 4 years 11 months ago #1785

  • Sander Tijm
  • Sander Tijm's Avatar
  • Administrator
  • Posts: 25
  • Thank you received: 2
Dear all,

I have some more practical points that might be discussed during the hangout. I will not be able to join myself, but based on the research that I presented last year in Lisbon and the results achieved afterwards I have some thoughts that I would like to share with you.

In Lisbon last year I presented the findings of Nadia Bloemendaal, who found that an erroneous yearly cycle of LAI caused an overestimation of evaporation in Spring and that there was a general overestimation of the evaporation in Summer. This leads to a reduction in soil moisture which leaves parts of the model without enough moisture for the plants to evaporate. This behaviour already starts in March, when the weather is warm enough.

One observation that gives clues on where the problem may come from comes from the shape of the areas with a very low evaporation. When these areas are not too large they have a shape that is quite similar to the shape of areas with high clay percentage.

What very probably happens is the following. There is an overestimation of evaporation that the DA tries to reduce by removing soil water. But a reduction in soil water has a different impact on the amount of soil water that is available for evaporation through vegetation, as the wilting point lies higher for soils with a high clay fraction. The same reduction of soil water therefore has a different impact for different soil types and when there is a sharp boundary between different soil types then the DA impact on one soil type may reduce the soil water below wilting point on the adjacent soil type.

For the DA the most important lesson therefore is that it is not the soil moisture itself that should be changed but the available water for evaporation (which is (the soil moisture minus wilting point) times the root zone depth). If you change this parameter by a certain fraction then the DA will not change the soil moisture below the wilting point (or a certain threshold above the wilting point) in the different areas.

After Lisbon Nadia has done some experiments where she tried to reduce this effect. The most simple one was with a reduction of the LAI with a factor of 2 or 4. The reduction of LAI with a factor of 2 resulted in a much reduced DA impact (measured through the standard deviation of the available water for evaporation parameter) and therefore may be a very simple way of improving the evaporation and reducing the areas with soil moisture below wilting point. A more elaborate change would be to include a vegetation model or LAI model in HARMONIE that may be changed through assimilation of the satellite observations of the vegetation index and which follows the yearly cycle of LAI much better.

So maybe these more practical points can be discussed also. But if that is not the case we can always discuss this during the ASM in Helsinki, as I am planning to go there.

Kind regards,

Time to create page: 0.103 seconds