Welcome, Guest
Username: Password: Remember me
HELPDESK

Here we can describe more what should be posted here

TOPIC: MARS_stage_bd runs out of time

MARS_stage_bd runs out of time 9 months 6 days ago #2508

Dear colleagues,

In running two CY43h2.1.1 validation experiments I am stuck since the MARS server reboot at ECMWF last Wednesday. It is the task MARS_stage_bd. The logfile keeps saying 'no error reported' but it is already running for more than 5 hours again. Soon it will abort with an out of time message. I have tried simply requeueing but also removed $WDIR and then a prod. No cure yet

Any clue would be appreciated very much. One of the experiments is

nl/nlf/hm_home/cy43knmi_control_feb21

It basically does 3h forecasts to keep 3dvar going.

Thanks!
Jan

MARS_stage_bd runs out of time 9 months 6 days ago #2509

Miraculously, the experiment is running again and the logfile MARS_stage_bd says 'Task complete before submission' (!??) Thank you very much.

MARS_stage_bd runs out of time 9 months 6 days ago #2510

  • Eoin Whelan
  • Eoin Whelan's Avatar
  • OFFLINE
  • Gold Boarder
  • Posts: 222
  • Thank you received: 45
I have observed that MARS is painfully slow. Patience required!

Not sure how much we can do.

Eoin
The following user(s) said Thank You: Jan Barkmeijer

MARS_stage_bd runs out of time 9 months 6 days ago #2511

What I did not understand is that the logfile now says: 'Task completed before submission' . Also for newer DTG's. Before that it spend order 5h on the cca.

MARS_stage_bd runs out of time 9 months 6 days ago #2512

  • Eoin Whelan
  • Eoin Whelan's Avatar
  • OFFLINE
  • Gold Boarder
  • Posts: 222
  • Thank you received: 45
Clever new development from Ulf. The MARS stage task now leaves evidence of past success and will no re-stage data.

The task, by default, stages 2 days of data.

MARS_stage_bd runs out of time 9 months 6 days ago #2513

Thank you Eoin and Ulf, that explains I think.
Jan

MARS_stage_bd runs out of time 7 months 1 week ago #2554

Hi!

Task MARS_stage_bd is sometimes very slow. By archiving LBC data we hope to speed up experimentation.

But what is the best approach when the task abort due to a timing issue after 9h or so. Does a simple requeue suffice or should I also empty $WRK or ...

Thanks,
Jan

MARS_stage_bd runs out of time 7 months 6 days ago #2555

  • Ulf Andrae
  • Ulf Andrae's Avatar
  • OFFLINE
  • Administrator
  • Posts: 308
  • Thank you received: 35
Jan,

For the MARS_stage_bd task there is no need to clean anything before you rerun it.

Ulf

MARS_stage_bd runs out of time 7 months 6 days ago #2556

Thanks Ulf,

I simply keep requeuing then.

Jan
Time to create page: 0.085 seconds