student i define to be the average scaled mark obtained by student i across all ... 2 This is often referred to as the final combined mark (for WACE c...

0 downloads 0 Views 120KB Size

100 Royal Street East Perth, Western Australia 6004 Telephone (08) 9318 8000 Facsimile (08) 9225 7050 http://www.tisc.edu.au/

AVERAGE MARKS SCALING

§1. Introduction In Western Australia a wide range of courses1 is available for senior secondary students. For each course there is an external examination taken at the end of Year 12. The examination results are then combined with school assessments. This process involves standardisation and moderation. The way in which this is done is described on the Curriculum Council website at www.curriculum.wa.edu.au. Thus, for each student a composite unscaled mark2 is generated for each course taken. However, the cohort of students taking a particular course P may be academically more able than the cohort of students taking course Q . This means that a composite unscaled mark of, say, 60 in course P represents a higher level of attainment than the same mark in course Q . Since marks in different courses are added to form a Tertiary Entrance Aggregate (TEA), equity considerations require that the marks in courses P and Q should be scaled. This means that the marks in course P should be scaled up relative to the marks in course Q (or the marks in course Q scaled down.) Of course, the situation is more complex than just establishing the appropriate relativity of the marks in courses P and Q . In fact there are many courses that students can take and so it is the relativities of all courses which scaling needs to address. The key concept is to generate a measure of academic ability, based on students’ achievements, for the cohort of students taking a particular course. The scaling method used in Western Australia3 is based on the premise that the best measure of an individual student’s academic performance is that student’s average scaled mark across all courses taken. Suppose that average is denoted by student i define

t i . That is to say, for a particular

t i to be the average scaled mark obtained by student i across all courses taken by

student i .

Now consider the cohort of students taking a particular course j . For each of these students there will be an average scaled mark

t i . The average of the t i across all the students taking course j will be denoted by

1

The terminology ‘course’ is used in the new structure introduced by the Curriculum Council. Previously the term ‘subjects’ was used to describe what have now become courses. 2 This is often referred to as the final combined mark (for WACE courses, this is after adjustment between stages and the stage 3 increment has been applied as appropriate). 3 Average Marks Scaling has been used in New South Wales for many years. The main architect of that system was Professor E. Senata, Department of Mathematical Statistics, University of Sydney. The New South Wales system was adapted for use in Western Australia by Dr M.T. Partis, Director, Secondary Education Authority, in 1997 and introduced in 1998. Prior to that the Australian Scaling Test was used as the anchor variable in the scaling process. Page 1 of 7 September 2010 Curtin University · Edith Cowan University · Murdoch University · The University of Western Australia

T j . That is to say, for a particular course j define T j to be the average of the average scaled marks t i obtained by all students taking course j . The Average Marks Scaling (AMS) process uses T j as a proxy measure4 of the academic ability of the cohort of students taking course j .

At the end of the process the value of

TP for course P and the value of TQ for course Q can be calculated.

The mean5 mark for the cohort of students taking course P is then scaled to

TP , whilst the mean mark for

the cohort of students taking course Q is scaled to TQ . This gives the appropriate relativity between the two courses. Adjustments are also made to the standard deviations of the course distributions, but these turn out to be minor compared to the adjustments to the means. It is worth stressing the main feature of Average Marks Scaling. The description above seems to suggest that the scaled marks need to be known in order for scaling to be carried out. However, at the heart of the process is the equating of the average scaled mark in course j and the anchor variable T j .

The

mathematics involved is set out below, but the key equation

Average scaled mark in course j of the cohort of students taking course j = The anchor variable T j for the course j

…… (1)

provides the central focus of the analysis. To clarify the concepts involved it is worth considering a numerical example. Suppose that student i has a scaled mark of 63 in course j . In what follows this scaled mark will be denoted by y ij .

y ij = 63

That is to say,

Suppose that student i takes four other courses obtaining scaled marks in those of 47, 71, 58 and 66. Then the average of this student’s five scaled marks will be 61.

t i = 61

That is to say,

It is important to note that, in this instance, the values of y ij and

t i are not the same. This is because they

are measuring different things. y ij is measuring the student’s performance in course j , whilst

t i is

measuring the student’s performance across five courses.

Now consider the cohort of all students taking course j . corresponding values of y ij and

For each of these students there will be

t i . Hence, the average values of y ij and t i for the whole cohort can be

4

Tj

5

Throughout this document the terms ‘mean’ and ‘average’ will be regarded as synonymous.

is sometimes described as the anchor variable for the scaling process. Page 2 of 7 September 2010 Curtin University · Edith Cowan University · Murdoch University · The University of Western Australia

determined.

The scaling process is designed to make these averages identical.

In simple terms the

unscaled marks in course j are moved up or down to achieve this outcome.

The adjustment of the composite unscaled marks to scaled marks uses a linear conversion, the details of which are explained below. It is important to note the following points:

§2.

•

the AMS scaling process preserves the ranking of students and the shape of the distribution in each course;

•

the mean scaled score across all courses and all students (the global mean) is predetermined, and currently set at 606.

•

the standard deviation parameter for the AMS process is currently set at 137.

Re-standardisation using z-scores

Although examination marks and school assessments are standardised at an earlier stage of the process, it is mathematically convenient to standardise again in such a way that the composite unscaled mark distribution for each course has a mean of 0 and a standard deviation of 1. Let w ij be the composite unscaled mark for student i in course j . For each course j let

µ j and σ j be the

mean and standard deviation of the w ij . Now put

zij =

w ij − µij

σj

.

The z ij values generated in this way are often referred to as z -scores. By definition it follows that the

z -scores for each course will have a mean of 0 and a standard deviation of 1. The AMS process uses the z ij values defined above to generate scaled marks y ij . The conversion to scaled marks can be regarded as a three-part process. First, add 60 to restore the overall mark distribution to the predetermined global mean. Second, add a term d j for each course j which determines whether the marks in that particular course are scaled up or down relative to the global mean. This means that some d j will be positive and some negative. Third, add a term 13 c j z ij , which alters the standard deviation for each course from 1 to 13 c j . The value of 13 is predetermined to produce an appropriate global standard deviation. The c j will be calculated for each course j , but in practice the values turn out to be always close to 1. ________________________________

6 7

The scaling mean was 58 prior to 2009 The scaling standard deviation was 14 prior to 2010

The combination of the three steps outlined above gives rise to the following equation: The scaled mark y ij for student i in course j is given by Page 3 of 7 September 2010 Curtin University · Edith Cowan University · Murdoch University · The University of Western Australia

y ij = 60 + d j + 13c j z ij .

…… (2)

The parameters d j and c j need to be evaluated for each course j . It should be emphasised that whilst equation (2) gives an algebraic definition of the scaled marks y ij , the arithmetical values of the y ij can only be calculated after the parameters d j and c j have been determined. The way in which the parameters d j and c j are evaluated is set out below.

§3.

Calculating averages

In this section the technical details of working out the average of the average scaled marks, that is to say T j for each course j , are developed. For this purpose it is useful to introduce a function

α ij which depends on

whether a particular student is taking a course or not. Define

⎧ α ij = ⎨ ⎩

1

if student i takes course j ;

0

if student i does not take course j .

n be the total number of students taking the examinations8. Let m be the total number of courses available in the examinations. Let

The number of students n j taking course j is then given by n

n j = ∑α ij i=1

The number of courses

mi taken by student i is given by m

mi = ∑α ij j=1

For a particular student i the average scaled mark, denoted by given by

ti =

t i , over all courses taken by that student is

1 m ∑α y mi k =1 ik ik

___________________________

8

In practice a subset of the total number of students, known as the scaling population, is used. The intention is to exclude, for example, students taking only one course.

For all the students taking a particular course j the average of the average scaled marks, denoted by T j , is given by n

Tj

1 = ∑α ij t i n j i=1 Page 4 of 7 September 2010

Curtin University · Edith Cowan University · Murdoch University · The University of Western Australia

=

= The significance of

§4.

1 nj

1 nj

n

∑α i=1

⎧1 m ⎫ ⎨ α y ∑ ik ik ⎬ ij ⎭ ⎩ m i k =1

⎧1 ⎩ mi

n

∑ α ij ⎨ i =1

m

∑ α (60 + d k =1

ik

k

⎫ + 13c k z ik )⎬ ⎭

t i and T j was explained in §1. T j is the anchor variable for the scaling process.

Matrix representation

m equations, corresponding to the values of j

In §3 the formula for T j was derived. This generates running from 1 through to

m . The next stage in the process is to recast these m equations in a matrix

format. From the previous section it follows that

T j = ∑ (60b jk + d k b jk + 13c k a jk ) m

k =1

where

αα 1 b jk = ∑ ij ik n j i=1 mi

1 n α ij α ik zik a jk = ∑ n j i=1 mi

n

and

This leads to the matrix equation

c 1 d T ~ = 60 B ~ + B ~ + 13 A ~

….. (3)

where

T ~ is the m × 1 column vector [T j ]; A and B are the m × m matrices [a jk ] and [b jk ], respectively;

1~ is the m × 1 column vector with each entry equal to 1; and c~ and d are the m × 1 column vectors

~

[c ] and [d ], respectively. j

j

Analysis of the b jk which are the elements of matrix B then gives

B 1~ = 1~ Hence, equation (3) simplifies to

T = 60 1 + B d~ + 13 A c~ ~ ~

§5.

….. (4)

Average scaled mark in course j Page 5 of 7 September 2010 Curtin University · Edith Cowan University · Murdoch University · The University of Western Australia

From equation (2) the scaled mark y ij for student i in course j is given by

y ij = 60 + d j + 13c j z ij . Hence, for a particular course j , the average of the y ij is given by n

1 1 α ij y ij = ∑ n j i=1 nj

n

∑α i =1

ij

(60 + d j + 13c j z ij )

⎛ 1 = 60 + d j + 13c j ⎜ ⎜n ⎝ j

n

∑α i =1

ij

⎞ z ij ⎟ ⎟ ⎠

= 60 + d j since the z -scores for course j have a mean of 0. From equation (1) this gives

T j = 60 + d j Hence,

T ~ = 60 1~ + d~

.…. (5)

From equations (4) and (5) it follows that

c d 1 60 1~ + d ~ = 60 ~ + B ~ + 13 A ~ Simplifying this matrix equation gives

(I − B) d~ = 13 A c~ where I is the

§6.

.…. (6)

m × m identity matrix.

Calculation of the key parameters

In matrix equation (6) the only unknowns are the column vectors c~ and d . The c j entries which make up ~ the column vector c~ can be thought of as the standard deviations for each course j . This can be evaluated by considering the z -scores for all students taking course j . The standard deviation c j for the z -scores in course j is given by a complicated (but standard) formula, namely 2 ⎧ ⎧ ⎞⎫ ⎞⎫ ⎛ m ⎛ m 2 ⎪ ⎪ ⎜ ∑ α ik zik ⎟ ⎪ ⎜ ∑ α ik zik ⎟ ⎪ ⎪ 1⎪n 1⎪n 2 k =1 ⎟ ⎟ ⎪⎬ ⎜ ⎬ − 2 ⎨∑ α ij ⎜ k =1m c j = ⎨∑ α ij m ⎟ n j ⎪ i=1 ⎜ ⎟ n j ⎪ i=1 ⎜ α ik ⎟ ⎪ α ik ⎟ ⎪ ∑ ∑ ⎜ ⎜ ⎪⎩ ⎪⎩ ⎠ ⎪⎭ ⎠ ⎪⎭ ⎝ k =1 ⎝ k =1

Page 6 of 7 September 2010 Curtin University · Edith Cowan University · Murdoch University · The University of Western Australia

The arithmetical values for the c j can now be substituted into equation (6). Solving equation (6) gives the values of d j . Equation (2) now allows the scaled marks y ij to be calculated for all students in all courses.

§7.

Example showing how a student’s scaled mark is calculated

Consider a student whose unscaled combined mark in course j is 65.73. For the cohort of students taking course j suppose that the mean and standard deviation of the unscaled combined marks are 59.51 and 12.20, respectively. Now derive the z -score corresponding to the student’s unscaled mark of 65.73. This is

z ij =

wij − µ j

σj

=

65.73 − 59.51 = 0.51 12.20

For course j suppose that the scaling parameters turn out to be c j = 1.02 and d j = 5.22. Then equation (2) gives the scaled mark y ij as

y ij = 60 + d j + 13c j z ij = 60 + 5.22 + 13(1.02)(0.51) = 71.98 ************************************

Page 7 of 7 September 2010 Curtin University · Edith Cowan University · Murdoch University · The University of Western Australia

Our partners will collect data and use cookies for ad personalization and measurement. Learn how we and our ad partner Google, collect and use data. Agree & close