Comparing different weather forecasts

Jon

Member
Has anyone took the time to compare different forecast providers for caving areas to see which is the most reliable?

I contacted a few of the providers that I use recently to find out about the differences in data and modelling and got the following replies. I'd be interested in anyone's thoughts.

Met office:
"Thank you for your email in connection with our forecasts. We are world-leading when it comes to the accuracy of our forecasts and the trust in them is high. I've included some links below which should help you make an informed decision about why you can rely on our forecasting. I hope this helps:

About our forecasts: https://www.metoffice.gov.uk/guide/weather/about-forecasts

How we take observations: https://www.metoffice.gov.uk/guide/weather/observations-guide

Forecast accuracy: https://www.metoffice.gov.uk/about-us/who/accuracy/forecasts

If you have any further queries or need additional information, please contact our Weather Desk on +44 (0)1392 88 5680 where one of our advisors will be happy to help.  The number is open 24 hours a day, every day of the year and it will help us if you can quote the reference number given above."


Yr.no:
"Thank you for your interest in our Yr forecasts. 
Our main priorities are Norway, the Arctic region and the other Nordic countries. For these areas we use high resolution models (2.5 km) for short term forecasts and EC ensemble forecasts for medium range. We also do extensive post-processing/statistical corrections of the model output for these areas. This is being funded by the Norwegian government. Unfortunately we do not have the resources to do this for the rest of the world.

For the rest of the world:

We use the EC-HRES model from ECMWF. It is described here:
https://www.ecmwf.int/en/forecasts/datasets/set-i
This is the leading global weather forecasting model. However, the resolution of this model is currently 10 km (weather is calculated at points approx 10km apart). For areas with complex topography the terrain is therefore not very well resolved. We also verify forecasts for locations outside Norway.

Local weather service with a forecasters opinion is what best.?

Meteo Group:
"Many thanks for your request.
In WeatherPro there are several types of data: observational and forecast data, as well as radar and satellite images.
As a MeteoGroup product we primarily use our own multi-model approach (MOS) for the forecasts in WeatherPro. Our approach is mostly based on data from the leading global weather models, e.g. ECMWF (European Centre for Medium-Range Weather Forecasts) and other local models like the American GFS (Global Forecast System) and the British UKMO (UK Met Office). The forecast data is updated at several, undetermined times each day.
The observational data offered in WeatherPro comes from our own private weather measurement network (1300 in Germany and Switzerland) and from the national weather services (whose data is supplied by the official World Meteorological Organisation WMO weather stations worldwide).
We would like to be able to offer updated data at more regular intervals, but the WMO weather stations only provide once hourly data (only weather stations at airports refresh their data every 20 minutes). Radar images from the national weather services are updated every 15 minutes, and satellite images are received from the geostationary satellites at intervals of between 15 minutes and 3 hours.

The BBC and MeteoGroup are in collaboration for some months."


XCWeather:
"The forecast I display is the Global Forecast System (GFS) which is produced by the US National Weather Service. I take the utmost care to ensure that the GFS data is displayed exactly as provided.


Performance statistics and verification data for the GFS are available here: http://www.emc.ncep.noaa.gov/GFS/perf.php

I'd always strongly recommend checking several forecasts before setting out, in particular, the MetOffice who use a different weather model. All models have their strengths and weaknesses and the MetOffice may spot problems that the GFS misses (and visa versa).

You can find more information about the limitations of numerical weather models and how best to interpret the data on the XCWeather FAQ page and also by following this link. http://weather.mailasail.com/Franks-Weather/Numerical-Weather-Prediction "









Sent from my Lenovo YT3-X50F using Tapatalk

 

mikem

Well-known member
Basically, if several agree then they are more likely to be correct, but they are still just models & sometimes each will be nearest...

Mike
 

Jon

Member
I realise that, I'm interested to know if anyone has kept some records comparing forecast to actual.
mikem said:
Basically, if several agree then they are more likely to be correct, but they are still just models & sometimes each will be nearest...

Mike

Sent from my Lenovo YT3-X50F using Tapatalk

 

mikem

Well-known member
I haven't kept records, but yr.no generally predicts more rain than metcheck & the reality normally falls somewhere between the two.
 

Pitlamp

Well-known member
Bear in mind one of the main variables affecting forecast accuracy is time of year.

When a depression comes to us off the Atlantic at this time of year the results are fairly easy to predict. The detail is mainly in the timing.

In 6 months time, when localised convectional rainfall occurs, all bets are off!
 

Graigwen

Active member
Since Meteo group became involved in the BBC forecasts they seem to seriously underestimate intensity and duration of rain in NW Kent.  Very odd.

.
 

mikem

Well-known member
Another problem is that many of the weather stations use tipping collectors for rainfall & they over-record when windy, so can't even rely on what's happened.

Mike
 

Jon

Member
And this from mylocalweather.org.uk

"

Jonathan, many thanks for taking the time and interest in contacting mlw.
 
I cannot comment on other parties weather forecasts only on mlw?s.
 
All our weather stations are configured for each specific location, coordinates etc., in their software they have built in algorithms and calculate a weather forecast based on what is happening according to the instrumentation at that precise location (temperature, humidity, solar and wind ? THSW) and what is likely to happen, the clue is in the word ?likely? because as we know weather is not an exact science unless it?s staring you in the face. What I mean by that is that there may be a an overall prevailing weather situation but each locality can have its own microclimate dependent on its geography, so because we have a weather station in every major Yorkshire Dale we are able to know what is happening at a single moment in time in different areas, our data is transmitted every 6 to 10 seconds and logged every minute and a forecast calculated??.. but not necessarily correct!
 
What mlw can do for you as a caver is to offer you ?real time? data of rainfall in each location so you can make a judgement for yourself on the situation BUT please note the weather stations sometime go down/off line/stop transmitting, this can be for a variety of reasons, loss of power, weather station and/or computer, loss of internet, rain tippler blocked and a number of other reasons, so be aware they only offer a guide and sometimes may not be working at all.
 
I hope I have given an insight as to the workings of mlw and that the system is of use to you."


Sent from my SM-G930T using Tapatalk

 

Oceanrower

Active member
I tend to check 4 or 5 online weather sites before taking part in an activity

And then pick the one I like the most...
 

Flotsam

Active member
Although the standard of forecasts is far better than it was when I was caving, 30-40 years ago, they aren't infallible and a risk assessment should be made by the caver. I sail these days but even though wind strength and direction should be the easiest parameters to predict the forecast is frequently very inaccurate. Wind strengths are regularly under estimated. The BBC wind forecasts are terrible IMO. Rain is probably(?) harder to forecast so caution is called for. I use a site called windy.com which has all the current models available for comparison and it's surprising how different they can be. I tend to use the synoptic charts a lot, the ones with the isobars on, they give me the best insight into what the wind might be doing.

I remember kitting up at Langcliffe Pot 40 odd years ago only to find as the rope was lowered down the entrance shaft the heavens opened and a veritable deluge followed, a narrow escape! I did get flooded most of the way nto Birks Fell for about eight hours (Thirza Holden in another part of the cave).
 

Bratchley

New member
The question of which one is the most accurate is an extremely complex one and there will not really be an answer to it (short enough for anyone to care about!). It depends on a number of factors, not least which model the provider uses, and how much the model output is 'intervened' by a forecaster.

Elevated area forecasting (most caving areas count here...) is always more difficult because boundary layer Meteorology is increasingly complicated when surface topography interacts with the passing airmass (i.e. air forced upwards by a sloping hill causing instability in conditionally stable air). Often, because of the scales involved, fully automated lower-resolution weather prediction models may 'miss' surface undulations.

As an example, resolutions of the commonly used models within the UK Met Office are as follows:
- UKV (United Kingdom only) 1.5km
- Euro4 (UK and Europe) 4km
- Global (...guess) 10km

This means that smaller scale weather is more likely to be resolved with the UKV, whereas the E4 and Global might miss things (localised showers, for instance).

Other forecast models used by other companies (European Commision ECMWF, American GFS etc), will have similar characteristics.

A graph of forecast model accuracy when tracking a tropical depression (and eventually hurricane) can be seen here, this is the best example I have of a significant comparison between them based on actual events:
https%3A%2F%2Fblueprint-api-production.s3.amazonaws.com%2Fuploads%2Fcard%2Fimage%2F589842%2F764af78a-af2a-4791-8854-3777fb93fd06.jpg


Up to 24hrs out, UKMO was best. After that, ECMWF took over, while the UKMO started to deviate further. The GFS lagged behind both in general (not ideal, because the hurricane hit the USA).

The ECMWF is a medium range forecast, more so than the UK Global (and other) models. The UKMO, GFS etc are used 'operationally', as in, the model is 'run' on supercomputers multiple times a day in order to give updated forecasts as often as possible. The ECMWF model is not spooled up as often. This is important because of the assimilation of data.

Every numerical weather prediction model will 'assimilate' data, this can and does include:
- Remote sensing data (satellite readings)
- Surface observations (wind, temperature, dew point etc)

This takes considerable time, as the amount of data is enormous and is used to form the 'initial' conditions before the numerical model uses this to predict change with time. This means that more time spent assimilating data can mean a more accurate forecast, especially in the medium and longer ranges. The UK Global runs every 6 hours, the E4 is the same (it doesn't do its own assimilation, rather it uses that of the Global), and the UKV runs every 3 hours. In comparison, the ECMWF runs every 12 hours, which gives it much more time to collect observational data.

This means in the short term, due to the reduced number of 'updates', it is less accurate, but it is often more accurate long term (more than 24 hours ahead).

In the event of a smaller geographical area model run (i.e. the UK's UKV 1.5km resolution model) it will also use models with a larger geographical coverage to form it's boundary conditions. This is also important, as large errors in the larger model fields that affect conditions at the smaller model field boundary can affect the conditions forecast within the smaller models.

Therefore, automated models will be more or less accurate depending on where you are looking for. In the UK, the Met Office will likely be more accurate up to 24 hours ahead, with the ECMWF being better after this - however it is worth bearing in mind that the ECMWF Highest Resolution is 9km, so may have associated weaknesses with small scale weather. In Europe, the Met Office model used there will be a lower resolution (4km) than in the UK, still beating the ECMWF, but again after 24 hours the EC's longer data assimilation may give it the edge. Globally, outside of Europe, this will be exacerbated further with the GM having 10km resolution. Therefore for Norway, the Norwegian Model will likely be better than the UK Met Office's Euro 4km as it is the area they prioritise for their highest resolution forecasts, with additional human input. For the USA, locally, their own GFS will likely be better.

On a broadscale level, when looking at larger scale systems such as tropical depressions, lower resolution matters less. The ECMWF and UKMO are generally better than the GFS at this, even for the Atlantic and US areas. Some of this in the short term is down to the mathematics of the models, but EC's edge at Medium range is also again to do with data assimilation.

Private companies (not state owned ones like Met Office or ECMWF) often take raw model data from multiple state organisations and compare them and see which seems to be behaving itself the most, however this is not always an easy task and doesn't always give the best results as their resources are often limited and their knowledge of the individual model weaknesses and strengths may be lacking. MeteoGroup are one of these, and now run the BBC weather forecast (except for warnings - still Met Office). Government Met organisations also consider data from each other, but tend to err towards their own unless deviation for the better is obvious.

Essentially, this question has no answer, because it depends where you are caving. You haven't specified, but if the UK is where you are interested in, the Met Office are likely to come out best on average because they prioritise the UK in their model development.

Of course, it is weather, so judgment can't be passed on only a few events - but a comparison between multiple events over time with all models hasn't been done to my knowledge, because a fair analysis would be difficult as for the UK, the Met Office models are better tailored. I can safely say that the UKMO would win out long-term for any UK weather event analysis.

Josh
 

Jon

Member
Thanks Josh, that is comprehensive. I'll need to read it a few times to understand!

Sent from my Lenovo YT3-X50F using Tapatalk

 

MarkS

Moderator
On the morning of a trip I usually find that by far the best indicator is a rain radar, such as the Met Office.

Sometimes it's hard to tell if a forecast of x mm of rain is due to some scattered showers or the edge of a big area of rain, but the radar certainly helps with that sort of situation, and you're not left wondering how the predicted weather animations have been derived.
 

JasonC

Well-known member
A fascinating subject, thanks for taking time to provide the info.
I often compare BBC (Meteo) and Metcheck to decide whether to cycle to work or catch the bus :)

They often disagree in detail, especially in showery conditions.

But what I struggle with is how to interpret a prediction of, say, "20% chance of 1mm rain" - does this mean 1mm of rain will fall in 20% of the area, or there is a 1 in 5 chance of 1mm rain over the entire area, or maybe there will be something in the range 0-1mm, with less more likely than more ?
 

TheBitterEnd

Well-known member
I always thought that comparing forecast sites would in itself make quite a good website.

Most forecasts come with a probability as a percentage. 10% chance of rain means they are fairly certain that rain is unlikely; 90%, it's going to rain, 50% -  they have no idea  ;)

<edit>
To answer Jason's point (which coincidentally he typed while I was typing this): As I understand it, the percentage is a percentage of the likely hood of the forecast being correct. So they run the forecast model many times with slightly different input parameters and see what the outcome is. If 9 out of 10 runs predicts rain then it?s going to rain. If 5 out of 10 runs predict rain then slight changes in conditions (temp, pressure, etc) could make all the difference.
 

Flotsam

Active member
There is a comparison site called weatherweb.net. He's a meteorological boffin and will do custom forecasts for a fee. Some of his site is free, some subscription.
 

Bratchley

New member
Any website that compared all forecast models would have a huge array of conditions at any given moment, only if they all agreed would it actually be of any use.

Put it this way, by human nature we'd always tend to swing either pessimistically or optimistically, which means when we get presented with 0-100 every time, we'd always either go for 0 or 100.

Therefore you'd either get caught out more often or never get anything done.  If you always went in the middle, you may as well have just guessed.

Too much data and too much choice is not always a good thing.
 
Top