Model validation

Model/data comparisons and model validation are made using satellite SST, sea surface height anomaly (altimeter) data and any available Argo temperature and salinity profiles for both the SAROM historical re-analaysis and for the real-time data assimilating now-casts and forecasts. In addition, hind-cast comparisons are to be made with mooring data (currents, temperatures etc) as well as CTD casts obtained through the Southern Australian Integrated Marine Observing System (SAIMOS) that is led by SARDI.

Comparisons with Satellite Data

The historical reanalysis was run over a 5 year period from 2010-2014 on a three day assimilation cycle. The system is run as a long time series of short forecasts where each forecast is compared to all available satellite observations prior to assimilation into the next analysis. This enables forecast statistics to be generated in order to assess overall system performance, to check how well the model can be constrained to the observations and to assess the stability and reliability of the system. The reanalysis is used to estimate whether the system has skill in predicting the details of events such as coastal upwelling and marine heatwaves. We use “innovation” statistics to estimate forecast error. An innovation is the difference between the model and observation at the observation time and location. The most common innovation based metric is the root mean square deviation (RMSD) and its mean deviation, or bias. RMSD tends to be dominated by a relatively small number of innovation elements with large magnitudes and may not accurately represent the performance of the system. These elements may correspond to either less observed or more chaotic parts of the model, or be caused by observations with large errors – as the metric does not account for observation error. To reduce the impact of such elements we use the mean absolute deviation (MAD).

Figure 1. Time series of forecast innovation errors for sea level anomaly (SLA) and sea surface temperature (SST) for the 2010-2014 reanalysis using the SAROM model.
Figure 1. Time series of forecast innovation errors for sea level anomaly (SLA) and sea surface temperature (SST) for the 2010-2014 reanalysis using the SAROM model.

Figure 1 shows a time series of forecast innovation errors from the reanalysis. This shows that the system is stable over the 5 year period and that there is no seasonal variation or long-term trend in the errors. At times there are spikes involving individual forecasts, however the system returns to its average skill usually by the next forecast. These forecasts with larger than usual errors are typically caused by errors in atmospheric forcing, such as misplacement in the timing and locations of synotpic weather patterns.

Table 1 quantifies the overall performance of the reanalysis and shows the total number of observations that were available during the period. It can be seen that analysis MAD is significantly lower than forecast MAD, yet forecast MAD is reasonably good for the region. This means that the analysis is able to fit the data to the model fields in a constructive manner, without overfitting the data, which would lead to larger forecast MAD. Also the results indicate the system is relatively unbiased for SLA, SST and SAL (in-situ salinity), however, is slightly biased warm for sub-surface temperature.

Table 1

Variable (units)

Number of Forecasts

Forecast MAD

Forecast Bias

Analysis MAD

Analysis Bias

# Obs

SLA (cm)

658

5.11

0.0138

3.08

0.00411

190649

SST (K)

658

0.386

-0.0517

0.222

-0.0155

7908794

TEM (K)

658

0.532

-0.209

0.323

-0.168

14657

SAL (psu)

658

0.112

0.0587

0.073

0.0465

14719

Table 1: Mean forecast innovation errors for the 2010-2014 reanalysis with respect to altimetry (SLA – sea level anomaly), remotely sensed sea surface temperature (SST), and in-situ temperature (TEM) and salinity (SAL) from Argo floats.

Table 2 shows forecast errors of the real-time system for the period 1st January 2017 to 14th March 2017. Whilst there is overlapping periods of time over which the same observations are used due to the way the real-time system runs1, the results sample errors from the different ACCESS-R forecasts applied to each forecast and from the different initial and open boundary conditions (in the realtime system, this is OceanMAPS version 3). The errors are similar to the reanalysis, except for SLA, which is significantly larger. These cannot be fairly compared as the period of time and number of observations are different.

Table 2

Variable (units)

Number of Forecasts

Forecast MAD

Forecast Bias

Analysis MAD

Analysis Bias

# Obs

SLA (cm)

1152

7.27

0.006

2.93

0.00107

374789

SST (K)

1152

0.399

-0.133

0.219

-0.0134

27070770

TEM (K)

1152

0.484

-0.0147

0.291

-0.0117

18020

19392

1152

0.134

0.0987

0.063

0.0373

14719

Table 2: Mean forecast innovation errors for the period 1st January 2017 to 14th March 2017 for the real-time system with respect to altimetry (SLA – sea level anomaly), remotely sensed sea surface temperature (SST), and in-situ temperature (TEM) and salinity (SAL) from Argo floats.

Comparisons with IMOS and other data

Three moorings and associated CTD transects have been maintained by SARDI since 2010 and the locations and cruise paths are shown in the Figure below. There are instances of data gaps due to mooring failure, bad weather and lack of resources. A comparison between these and other IMOS data with the SAROM re-analysis model output is made below.

Figure 2. The SAIMOS mooring facility mooring sites (green circles) for SAM5CB (Coffin Bay region), SAM8SG (Mouth Spencer Gulf) and NRSKAI (National reference station off Kangaoo Island.and CTD stations (blue dots). The location of the AEMP mooring in SW Spencer Gulf is indicated by the green rectangle
Figure 2. The SAIMOS mooring facility mooring sites (green circles) for SAM5CB (Coffin Bay region), SAM8SG (Mouth Spencer Gulf) and NRSKAI (National reference station off Kangaoo Island.and CTD stations (blue dots). The location of the AEMP mooring in SW Spencer Gulf is indicated by the green rectangle

Weather-Band Comparisons

Model results and Observational data from three IMOS moorings were low-pass filtered with a 3-day cut-off to investigate the model performance for winter and summer periods. Periods were selected for which we had near continuous observational data. The vertically averaged currents of the observations were used to determine the principal axis (of major current direction) and the model results were taken at this angle.

In general the weather band comparisons show (Figures 3) very good agreement for strong events at the two off-shore moorings (NRSKAI and SAM5CB) and the weaker currents in the more sheltered SAM8SG mooring are reproduced, but show less agreement in detail. This may be due to the weaker Gulf circulation being dominated by thermohaline effects that are most difficult to model.

NRSKAI is the national reference station near Kangaroo Island.

Figure 3a: Vertically integrated principal axis currents at the NRSKAI site for summer (top) and winter (bottom) periods. Currents derived from the SAROM reanalysis and bottom mounted ADCP.
Figure 3a: Vertically integrated principal axis currents at the NRSKAI site for summer (top) and winter (bottom) periods. Currents derived from the SAROM reanalysis and bottom mounted ADCP.

SAM5CB is an SA-IMOS mooring located near Coffin Bay.

Figure 3b: Vertically integrated principal axis currents at the SAM5CB site for summer (top) and winter (bottom) periods. Currents derived from the SAROM reanalysis and bottom mounted ADCP.
Figure 3b: Vertically integrated principal axis currents at the SAM5CB site for summer (top) and winter (bottom) periods. Currents derived from the SAROM reanalysis and bottom mounted ADCP.

SAM8SG is an SA-IMOS mooring at the mouth of Spencer Gulf.

Figure 3c: Vertically integrated principal axis currents at the SAM8SG site for summer (top) and winter (bottom) periods. Currents derived from the SAROM reanalysis and bottom mounted ADCP.
Figure 3c: Vertically integrated principal axis currents at the SAM8SG site for summer (top) and winter (bottom) periods. Currents derived from the SAROM reanalysis and bottom mounted ADCP.

Inter-annual Comparisons

Model results and observational data from the three IMOS moorings were used to look at longer period inter-annual variations. The principal axis were determined from the observations and the vertically integrated velocities were low pass filtered with a 30-day cut-off.

Figure 4: vertically averaged 30-day low pass filtered currents along principal axis for the 5+ year period of the SAROM reanalysis.
Figure 4: vertically averaged 30-day low pass filtered currents along principal axis for the 5+ year period of the SAROM reanalysis.

From Figure 4, the SAROM model is able to reproduce the periodicity of the annual cycle in currents, but it does appear to have a tendency to over-estimate the stronger events over this time scale.  However, this may be related to problems with ADCPs resolving the surface layer during strong wind and wave events.

Vertical Shear

The vertical shear was investigated by separating the 3D NRSKAI principal axis currents into 3 depth bins: Surface (~0-30m), Mid-depth (~30-60m), and Bottom (60-100m).  A 3-day low pass filter was applied to the model results and observational data from the bottom mounted ADCP.

Figure 5: Vertical Shear at the NRSKAI mooring - principal axis currents for surface, mid-depth, and bottom along principal axis.
Figure 5: Vertical Shear at the NRSKAI mooring - principal axis currents for surface, mid-depth, and bottom along principal axis.

The vertical shear is well represented in the SAROM model results at the NRSKAI (Figure 5). The ADCP observations are sparser in the surface layer during strong wind and wave events due to the interference with bubbles formed by breaking waves.

Temperature and Salinity Transects

The reference transect shown in Figure 2 is commonly repeated during SAIMOS surveys and is a useful comparison with model results. Two features which we wish to be able to reproduce are the Summer Upwelling and the saline outflow from Spencer Gulf that occurs from late Autumn to early Spring.

Summer Upwelling

Figure 6a: Temperature (C) during a Summer Upwelling Event, CTD Observations (top) and Model Transect (bottom). The data from the northern most station is represented on the left.
Figure 6a: Temperature (C) during a Summer Upwelling Event, CTD Observations (top) and Model Transect (bottom). The data from the northern most station is represented on the left.
Figure 6b: Salinity (PSU) during Summer Upwelling Event, CTD Observations (top) and Model Transect (bottom). The data from the northern most station is represented on the left.
Figure 6b: Salinity (PSU) during Summer Upwelling Event, CTD Observations (top) and Model Transect (bottom). The data from the northern most station is represented on the left.

A typical summer upwelling event is illustrated in Figures 6. The model reproduces an upwelling but the bottom temperatures are not as low as the data. The temperatures and salinities inside the Gulf (to the left of the plots) are lower in the model but this may be related to the bias found in the surface forcing at the top of the Gulf which is now being addressed.

Winter Gulf Outflow

Figure 7a: Temperature (C) during Gulf Outflow Event, CTD Observations (top) and Model Transect (bottom). The data from the northern most station is represented on the left.
Figure 7a: Temperature (C) during Gulf Outflow Event, CTD Observations (top) and Model Transect (bottom). The data from the northern most station is represented on the left.
Figure7b: Salinity (PSU) during Gulf Outflow Event, CTD Observations (top) and Model Transect (bottom). The data from the northern most station is represented on the left.
Figure7b: Salinity (PSU) during Gulf Outflow Event, CTD Observations (top) and Model Transect (bottom). The data from the northern most station is represented on the left.

The Spencer Gulf outflow usually begins in mid to late April and occurs when the extremely saline waters at the head of the Gulf begin to cool and flow out near the bottom.  The outflow is very vigorous and appears to be unstable, shedding eddies as a result of baroclinic instability. Capturing the general properties of the outflow is an important test of any model of Spencer Gulf.  In Figures 7 a strong outflow event from late autumn (May 2014) shows good agreement between the model and the observations.

Comparison to IMOS HF-Radar

The SAROM real-time system can routinely be compared to IMOS HF-Radar at http://oceancurrent.imos.org.au/sst.php. Figure 8 shows an example for the 6th March 2017 from both of the real-time systems. The SAROM surface currents are in very good qualitative agreement with the radar derived surface currents west of Kangaroo Island and south of the Bonney Coast. The total surface current is a challenge to predict as it depends on the interplay of all mechanisms that generate it - winds, tides, waves and features of the ocean circulation such as mesoscale eddies, which arise from gradients in water-density. The predictability in this case appears to comes from having decent winds from ACCESS-R, tides and background mesocale features. Data from other times have been compared and SAROM shows very good agreement with the radar data more than 80% of the time. The IMOS ocean current is derived from altimetry, which captures the mesoscale geostrophic currents and is reliable in the deep ocean when good data is available, however, cannot be expected to predict a complex surface flow, driven by many factors, such as that shown here.

Figure 8: IMOS HF-RADAR and SST (left) and SAROM real-time SST and surface currents (right) for the 6th March 2017. Note different colour scales for SST.
Figure 8: IMOS HF-RADAR and SST (left) and SAROM real-time SST and surface currents (right) for the 6th March 2017. Note different colour scales for SST.


1 SAROM makes 4 forecasts per day - each one goes back 12 days behind real-time and does a series of 4 analyses using a 3 day cycle prior to making the real-time nowcast/forecast.

Disclaimer

This is a research and experimental product and no liability is assumed for the use of the eSA-Marine system or its products including now-casts and forecasts. In particular, this includes all conditions outlined in the general PIRSA disclaimer that can be accessed here or at the bottom of this page. Moreover, other sources of ocean now-cast/forecasts and or data may not align or be consistent with output from the eSA-Marine System which may be in error and/or out of date. Users should obtain their own independent advice. No liability is assumed for the eSA-Marine products presented here.

Page Last Reviewed: 11 Jan 2017
Top of page