Goal Attainment Scaling: Environmental Impact Evaluation in the Upstream Petroleum Industry

October 1997 (updated December 2001)
Michael Malavazos (Petroleum and Geothermal Group, Department of Primary Industries and Resources South Australia)

Abstract

The Department of Primary Industries and Resources South Australia (PIRSA) has developed an alternative method of monitoring and evaluating the environmental impact of petroleum operations, as part of the development and implementation of an objective-and-management-system-based regulatory regime. Goal attainment scaling (GAS) was developed in the 1960s and 1970s as a tool for monitoring and evaluation in human services. Apparently it has not been used in environmental impact monitoring before.

This paper outlines the experience gained to date on the application of GAS as an environmental evaluation tool for the restoration activities carried out on abandoned well sites in the Cooper Basin situated in the arid environment in the northeastern corner of South Australia. The application of GAS has also been extensively used for the assessment of the environmental restoration of seismic lines in the Cooper Basin.

Introduction

PIRSA commenced a review the South Australian Petroleum Act 1940 in 1996.  As a result of this review the South Australian Petroleum Act 2000 replaced the old Act in September 2000. One of the main thrusts of the new Petroleum Act has been to introduce an objective-and management-system-based regime to regulate the upstream petroleum industry in relation to environmental, public safety and resource development issues.

The upstream petroleum industry includes all exploration and production activities. Such activities include seismic surveys, drilling and operating wells, and the construction and operation of production facilities and pipelines.

The fundamental requirements of the new regulatory regime to be effective are:

• a credible means for establishing objectives relevant and appropriate to the activities being regulated;
• a practical means for evaluating the level of attainment of those objectives; and
• an independent means for assessing the performance of company management systems, from process and outcome perspectives.

In early 1994 the Petroleum Division of PIRSA (then the Department of Mines and Energy Resources of South Australia) began assessing the suitability of Goal Attainment Scaling (GAS) as an environmental evaluation tool to satisfy the above requirements for the new regulatory regime.

In late 1994 PIRSA and Santos Ltd commenced assessing the appropriateness of GAS as an evaluation tool for environmental objectives of restoration activities on abandoned well sites and seismic lines (Sharp, 1994). This concept has subsequently also been introduced to the mining industry (Sharp, 1996; Malavazos, 1995, 1996) and to environmental interest groups (Sharp, 1997).

This paper presents the results of the trials carried out prior to October 1997 on the use of GAS as an environmental evaluation tool on the restoration activities of abandoned petroleum well sites. The well sites included in this evaluation are located in the arid dunefield and floodplain landsystems in the South Australian sector of the Cooper Basin situated in the northeastern corner of the state. The paper focuses on the evolution of the use of GAS for such a purpose from early 1995 to March 1997. Acquired field data is used to illustrate and discuss various findings. GAS has also been successfully applied to the assessment of seismic line rehabilitation in the Cooper Basin. This success has recently been demonstrated in the assessment of Environmental Outcomes of the Western Prospects seismic survey in the Coongie Lakes Control Zone in the South Australian sector of the Cooper Basin (Cockshell, 1998).

Goal Attainment Scaling Theory

An important feature of GAS is that for any goal (be it environmental or otherwise) stakeholders are involved in evaluating and seeking consensus on the most important aspects of the goals to be achieved within a particular timeframe, and the range of expected outcomes of the activities undertaken within such a timeframe. The technique forces participants to document the expected outcomes ranging from much less than expected (-2), less than expected (-1), expected (0), more than expected (+1) and much more than expected (+2) outcomes for each objective.

The various outcomes can take the form of descriptive statements, statements supported by photographs or quantitative data.

Field Trials and Findings

From a study on the environmental impacts of seismic lines and well sites in the Cooper Basin (Fatchen & Woodburn, 1997) it was concluded that the environmental impacts to the landsystems are mainly visual and not ecological. The study suggested that a return of the physical configuration of an affected region would be an acceptable indicator of the level of return of the regional ecology. Therefore, setting the environmental objectives for the restoration activities of abandoned well sites and seismic lines, the focus was on objectives which could be assessed visually. These include:

• minimising visual impact through effective earthworking of restored well sites into the surrounding contours and ensuring soil colour and texture is returned to a compatible level with that of the surrounds; and
  
• the extent of revegetation and re-establishment of native plant species, as assessed visually.

Objective 1: Minimise Visual Impact
The assessment of this objective was initially perceived to consist of two major components or goals (Malavazos & Dobrzinski, 1995); firstly, an assessment of the effectiveness of the earthworks in restoring the original contours of the site; and secondly, an assessment of the extent to which the site is stabilised against erosion. The second component was considered to be a visual impact on the grounds that effects of erosion on well sites in dunefield and floodplain landsystems are considered to be more significant aesthetically rather than ecologically (Low and Moss, 1996).

The range of GAS outcome criteria used to initially assess this objective on the basis of these two goals are given in Table 1. The results gathered since early 1995 (Malavazos & Wright, 1995 to 1997), are presented in the form of frequency charts in Figures 1 and 2. These results show a skewed distribution towards the +1 and +2 scores.

Figure 1: Results of GAS assessment for Objective: Minimise Visual Impact on basis of earthwork effectiveness

Figure 2: Results of GAS assessment for Objective: Minimise Visual Impact on basis of stabilising against erosion

 

 

 

 

 

 

 

 

For the goal of earthworking well sites effectively, it was apparent from the criteria used in Table 1 that the well sites situated on flat plains, whether in dunefield or floodplain landsystems, were effectively restored. Also, the original expectation of erosion along the edges of restored well sites was not an issue for well sites situated on flat plains. However, it was found that well sites situated on top of sand dunes were more difficult to earthwork effectively and often remained unsightly. Erosion problems were also evident on well sites situated on top of sand dunes, and even for the majority of such cases the erosion down the edges of the lease was relatively minor and unobtrusive in relation to the surrounding landscape.

The GAS outcome criteria for this objective were subsequently reviewed in the light of these findings. The revised scales separated the assessment of restored well sites into two categories. Firstly, those situated on flat plains such as interdunal plains, dunal swales and floodplains and secondly those situated on top of dunes (Table 2). Since erosion was only a major issue for well sites situated on top of dunes, erosion is now no longer assessed as a separate item but has been incorporated with earthworking assessment criteria for assessing only those well sitessituated on top of dunes.

Objective 2: Revegetate Well Sites
The revegetation objective is very much dependent on the type of vegetation, soil types and moisture content in the soil (which in turn is dependent on the timing and amount of rainfall in the region after the completion of the restoration activities). From observations made in the field for this region (Malavazos & Dobrzinski, 1995) it was decided that at least 5 years from the completion of the restoration works would be needed to ensure sufficient rainfall has occurred to make assessment for revegetation possible. Prior to 5 years, it was considered that the assessment would need to be carried out on an "enabling" objective, that is, a reliable short-term indicator for the achievement of longer-term vegetation outcomes. Therefore two sets of GAS outcome criteria were initially set up to assess the revegetation objective, one for well sites restored for less than 5 years and one for greater than 5 years (Table 1).

Assessments using these two sets of criteria since early 1995 (Malavazos & Wright, 1995 to 1997) gave the results shown in Figures 3 and 4. The skew towards the positive observed on Figure 3 for well sites less than 5 years indicated that earthworking practices for facilitating longer term revegetation was not an issue of non-compliance. Also the original expectation of minor gully erosion along the edges of the lease was not as significant as first thought. As stated previously, gully erosion was only evident on those well sites situated on top of sand dunes.

Figure 3: Results of GAS assessment for Objective: Minimise impact on vegetation for well sites restored less than 5 years

Figure 4: Results of GAS assessment for Objective: Minimise impact on vegetation for well sites restored for more than 5 years

The skew towards the positive observed on Figure 4 for sites restored for more than 5 years resulted from the effects of assessing well sites of varying timeframes ranging up to 15 years since the completion of restoration work. Significant annual and biannual vegetation was apparent on many of the well sites where restoration was completed for less than 5 years. The implications are that further GAS outcome criteria for the assessment of the revegetation objective are needed to be developed for more incremental timeframes. The problems faced in attempting to do this are discussed below.

As a result of the above findings, the GAS outcome criteria for the revegetation objective were reviewed for both timeframes (Table 2). Particularly, for the criteria used to assess well sites restored for less than 5 years where all reference to erosion was removed and the annual and biannual vegetation observations were incorporated. MESA has now developed a guideline for the assessment of restored well sites in the Cooper Basin using the GAS outcome criteria given in Table 2. The guideline utilises colour photographs to illustrate examples of the outcomes for each GAS score.

Guide for using GAS to Assess Restored Well Sites

As a result of these trials, a field guide for the environmental assessment of abandoned well sites in the Cooper Basin was developed by PIRSA (Malavazos, 1998; PIRSA, 1998). This guide uses the criteria outlined in Table 2 and incorporates the use of photographs of examples to illustrate each criteria. Another purpose of the guide is to also allow other stakeholders, such as any member of the community, to be able to assess the restoration of a well site. This provides for openness and transparency in the regulatory process which are two of the key principles of the new Petroleum Act.

Limitations of GAS

Limitations for the strict theoretical application of GAS as an environmental evaluation tool as observed and expressed by various groups include:

• The variability in climate, soil conditions and vegetation types within each landsystem will make it difficult to develop specific GAS outcome criteria for assessing environmental restoration at incremental timeframes. This variability was highlighted as a restriction to the strict application of GAS theory for assessing environmental rehabilitation because it makes it impossible to collect data which represent random samples from a single population (Stokes, 1997).

• The difficulty faced in trying to develop incremental timeframe GAS outcome criteria due to practicality of use in the field. For instance ensuring enough samples for each incremental timeframe are taken to allow statistically valid analysis of the outcomes (Santos, 1996).

• The GAS concept was considered as inappropriate for cases where the defined -2 outcome may be totally catastrophic and unacceptable in environmental terms (Conservation Groups, 1997). The conservation groups also expressed concern on the use of GAS for assessing long-term objectives using short-term outcomes, that is the use of enabling objectives such as ripping practices etc for facilitating long-term revegetation. The main concern related to addressing liability issues of short-term operators.

Modification of GAS

To help address some of these concerns and to ensure that GAS is a useful management tool it was suggested that the GAS outcome criteria need to be developed and used to monitor environmental performance on a progressive basis (Santos, 1996; Stokes, 1997). That is, the 0 outcome to be defined as the expected outcome at time 0, the +1 outcome to be defined as the expected outcome for some later time and the +2 outcome to be defined as the ultimate expectation. In this case, the -1 and -2 outcomes were proposed to be used for initiating urgent action.

Over time the distribution will be expected to skew towards the +2 until all sites ultimately meet the +2 outcome (Stokes, 1997). Santos (1997) also expressed their intent to use GAS as a management tool for monitoring progress for each objective towards the ultimate desirable expectation of +2. Santos admitted that the time dependency of some objectives such as revegetation will require them to use at least two sets of scales such as those proposed in Table 2.

Future Work

To ensure that GAS is being used appropriately and that the results reported from the field are valid and an honest reflection of the environmental performance, an independent third party audit of the assessment is needed. PIRSA is intending to appoint independent third party auditors for this purpose. Such third parties may include suitably qualified representatives of environmental conservation groups or independent ecologists, these independent audits will also help to ascertain any biases in the use of the GAS criteria by either PIRSA and/or Santos personnel.

Further research and investigation into the environmental outcome criteria used in GAS is currently underway to follow on from the initial findings of Fatchen and Woodburn (1997). Such research is aimed at identifying environmental outcome indicators which may either validate the ones currently being used or the ones which need to be incorporated into the GAS criteria to make them more appropriate.

Finally, more extensive consultation will be required with other stakeholders to validate the current objectives and assessment criteria.

Conclusions

In three years of trial and application progress has been observed in environmental management and evaluation from both a Santos and PIRSA perspective, including:

• The specification of the GAS outcome criteria and acknowledgment that these are used to assess performance against objectives has been a major contributor in facilitating change in the culture of field staff and contractors and increasing their environmental awareness (Santos, 1997). Particularly, it is apparent that the implementation of the GAS tool has instilled a bias towards the +1 and +2 articulated expected outcomes.
  
• The dynamic nature of GAS has encouraged a culture of continuous improvement in the management of environmental impacts.
  
• The graphical representation of the GAS results has provided an effective management reporting tool on the state of company environmental performance.

One of the requirements of the proposed regulatory regime is for industry to develop management systems designed to achieve environmental objectives. An environmental management system includes both internal and external audits to check and verify the procedural compliance (which focuses on the process of management, that is, the means upon which a company achieves its objectives) and the achievement of objectives (that is, the actual outcomes which the company achieves).

In relation to auditing and reporting the achievement of environmental objectives (i.e. actual environmental outcomes achieved by the environmental management system (EMS)), GAS is being integrated into the EMS to report on the level of achievement to both regulators and company management (Figure 5).

Figure 5: Integration of GAS within the company Environmental Management System

Click here for the References for this Paper