Model validation
Model/data comparisons and model validation are made using satellite SST, sea surface height anomaly (altimeter) data and any available Argo temperature and salinity profiles for both the SAROM historical re-analaysis and for the real-time data assimilating now-casts and forecasts. In addition, hind-cast comparisons are to be made with mooring data (currents, temperatures etc) as well as CTD casts obtained through the Southern Australian Integrated Marine Observing System (SAIMOS) that is led by SARDI.
Comparisons with Satellite Data
The historical reanalysis was run over a 5 year period from 2010-2014 on a three day assimilation cycle. The system is run as a long time series of short forecasts where each forecast is compared to all available satellite observations prior to assimilation into the next analysis. This enables forecast statistics to be generated in order to assess overall system performance, to check how well the model can be constrained to the observations and to assess the stability and reliability of the system. The reanalysis is used to estimate whether the system has skill in predicting the details of events such as coastal upwelling and marine heatwaves. We use “innovation” statistics to estimate forecast error. An innovation is the difference between the model and observation at the observation time and location. The most common innovation based metric is the root mean square deviation (RMSD) and its mean deviation, or bias. RMSD tends to be dominated by a relatively small number of innovation elements with large magnitudes and may not accurately represent the performance of the system. These elements may correspond to either less observed or more chaotic parts of the model, or be caused by observations with large errors – as the metric does not account for observation error. To reduce the impact of such elements we use the mean absolute deviation (MAD).
Figure 1 shows a time series of forecast innovation errors from the reanalysis. This shows that the system is stable over the 5 year period and that there is no seasonal variation or long-term trend in the errors. At times there are spikes involving individual forecasts, however the system returns to its average skill usually by the next forecast. These forecasts with larger than usual errors are typically caused by errors in atmospheric forcing, such as misplacement in the timing and locations of synotpic weather patterns.
Table 1 quantifies the overall performance of the reanalysis and shows the total number of observations that were available during the period. It can be seen that analysis MAD is significantly lower than forecast MAD, yet forecast MAD is reasonably good for the region. This means that the analysis is able to fit the data to the model fields in a constructive manner, without overfitting the data, which would lead to larger forecast MAD. Also the results indicate the system is relatively unbiased for SLA, SST and SAL (in-situ salinity), however, is slightly biased warm for sub-surface temperature.
Variable (units) | Number of Forecasts | Forecast MAD | Forecast Bias | Analysis MAD | Analysis Bias | # Obs |
SLA (cm) | 658 | 5.11 | 0.0138 | 3.08 | 0.00411 | 190649 |
SST (K) | 658 | 0.386 | -0.0517 | 0.222 | -0.0155 | 7908794 |
TEM (K) | 658 | 0.532 | -0.209 | 0.323 | -0.168 | 14657 |
SAL (psu) | 658 | 0.112 | 0.0587 | 0.073 | 0.0465 | 14719 |
Table 1: Mean forecast innovation errors for the 2010-2014 reanalysis with respect to altimetry (SLA – sea level anomaly), remotely sensed sea surface temperature (SST), and in-situ temperature (TEM) and salinity (SAL) from Argo floats.
Table 2 shows forecast errors of the real-time system for the period 1st January 2017 to 14th March 2017. Whilst there is overlapping periods of time over which the same observations are used due to the way the real-time system runs1, the results sample errors from the different ACCESS-R forecasts applied to each forecast and from the different initial and open boundary conditions (in the realtime system, this is OceanMAPS version 3). The errors are similar to the reanalysis, except for SLA, which is significantly larger. These cannot be fairly compared as the period of time and number of observations are different.
Variable (units) | Number of Forecasts | Forecast MAD | Forecast Bias | Analysis MAD | Analysis Bias | # Obs |
SLA (cm) | 1152 | 7.27 | 0.006 | 2.93 | 0.00107 | 374789 |
SST (K) | 1152 | 0.399 | -0.133 | 0.219 | -0.0134 | 27070770 |
TEM (K) | 1152 | 0.484 | -0.0147 | 0.291 | -0.0117 | 18020 |
19392 | 1152 | 0.134 | 0.0987 | 0.063 | 0.0373 | 14719 |
Table 2: Mean forecast innovation errors for the period 1st January 2017 to 14th March 2017 for the real-time system with respect to altimetry (SLA – sea level anomaly), remotely sensed sea surface temperature (SST), and in-situ temperature (TEM) and salinity (SAL) from Argo floats.
Comparisons with IMOS and other data
Three moorings and associated CTD transects have been maintained by SARDI since 2010 and the locations and cruise paths are shown in the Figure below. There are instances of data gaps due to mooring failure, bad weather and lack of resources. A comparison between these and other IMOS data with the SAROM re-analysis model output is made below.
Weather-Band Comparisons
Model results and Observational data from three IMOS moorings were low-pass filtered with a 3-day cut-off to investigate the model performance for winter and summer periods. Periods were selected for which we had near continuous observational data. The vertically averaged currents of the observations were used to determine the principal axis (of major current direction) and the model results were taken at this angle.
In general the weather band comparisons show (Figures 3) very good agreement for strong events at the two off-shore moorings (NRSKAI and SAM5CB) and the weaker currents in the more sheltered SAM8SG mooring are reproduced, but show less agreement in detail. This may be due to the weaker Gulf circulation being dominated by thermohaline effects that are most difficult to model.
NRSKAI is the national reference station near Kangaroo Island.
SAM5CB is an SA-IMOS mooring located near Coffin Bay.
SAM8SG is an SA-IMOS mooring at the mouth of Spencer Gulf.
Inter-annual Comparisons
Model results and observational data from the three IMOS moorings were used to look at longer period inter-annual variations. The principal axis were determined from the observations and the vertically integrated velocities were low pass filtered with a 30-day cut-off.
From Figure 4, the SAROM model is able to reproduce the periodicity of the annual cycle in currents, but it does appear to have a tendency to over-estimate the stronger events over this time scale. However, this may be related to problems with ADCPs resolving the surface layer during strong wind and wave events.
Vertical Shear
The vertical shear was investigated by separating the 3D NRSKAI principal axis currents into 3 depth bins: Surface (~0-30m), Mid-depth (~30-60m), and Bottom (60-100m). A 3-day low pass filter was applied to the model results and observational data from the bottom mounted ADCP.
The vertical shear is well represented in the SAROM model results at the NRSKAI (Figure 5). The ADCP observations are sparser in the surface layer during strong wind and wave events due to the interference with bubbles formed by breaking waves.
Temperature and Salinity Transects
The reference transect shown in Figure 2 is commonly repeated during SAIMOS surveys and is a useful comparison with model results. Two features which we wish to be able to reproduce are the Summer Upwelling and the saline outflow from Spencer Gulf that occurs from late Autumn to early Spring.
Summer Upwelling
A typical summer upwelling event is illustrated in Figures 6. The model reproduces an upwelling but the bottom temperatures are not as low as the data. The temperatures and salinities inside the Gulf (to the left of the plots) are lower in the model but this may be related to the bias found in the surface forcing at the top of the Gulf which is now being addressed.
Winter Gulf Outflow
The Spencer Gulf outflow usually begins in mid to late April and occurs when the extremely saline waters at the head of the Gulf begin to cool and flow out near the bottom. The outflow is very vigorous and appears to be unstable, shedding eddies as a result of baroclinic instability. Capturing the general properties of the outflow is an important test of any model of Spencer Gulf. In Figures 7 a strong outflow event from late autumn (May 2014) shows good agreement between the model and the observations.
Comparison to IMOS HF-Radar
The SAROM real-time system can routinely be compared to IMOS HF-Radar at http://oceancurrent.imos.org.au/sst.php. Figure 8 shows an example for the 6th March 2017 from both of the real-time systems. The SAROM surface currents are in very good qualitative agreement with the radar derived surface currents west of Kangaroo Island and south of the Bonney Coast. The total surface current is a challenge to predict as it depends on the interplay of all mechanisms that generate it - winds, tides, waves and features of the ocean circulation such as mesoscale eddies, which arise from gradients in water-density. The predictability in this case appears to comes from having decent winds from ACCESS-R, tides and background mesocale features. Data from other times have been compared and SAROM shows very good agreement with the radar data more than 80% of the time. The IMOS ocean current is derived from altimetry, which captures the mesoscale geostrophic currents and is reliable in the deep ocean when good data is available, however, cannot be expected to predict a complex surface flow, driven by many factors, such as that shown here.
1 SAROM makes 4 forecasts per day - each one goes back 12 days behind real-time and does a series of 4 analyses using a 3 day cycle prior to making the real-time nowcast/forecast.
Disclaimer
This is a research and experimental product and no liability is assumed for the use of the eSA-Marine system or its products, including now-casts and forecasts. This includes all conditions outlined in the general PIRSA disclaimer which can also be accessed at the bottom of this page.
Moreover, other sources of ocean now-cast/forecasts and/or data may not align or be consistent with output from the eSA-Marine System, which may be in error and/or out of date.
Users should obtain their own independent advice. No liability is assumed for the eSA-Marine products presented here.