Group Project: Designing and Piloting a Language Test

Abstract

A group project designing a language test for adult students preparing for the Cambridge Advanced exam - covering test purpose and usefulness, the design of speaking, reading, listening and writing items, the pilot test, and guidelines for users. Written with Un Lai, Shuai Xie and Yi Liu.

The Rationale for the Pilot Test

The Test Takers

The content of the test is based on adult students (aged 20-35 years old) at TAFE Queensland who will be attending a pathway course, which includes cookery, automobile services, and carpentry, after passing at least four tests in a 12-week Cambridge Advanced (CAE) course. The class demographic is broad as the students are from Thailand, Japan, Brazil, Colombia, Chile, Spain, Italy, Croatia, the Philippines, and Indonesia. The learners were in a 12-week First Certificate in English (FCE) course and had to pass at least four exams prior to entering the CAE class. Therefore, the learners are expected to have prior knowledge of the type of testing materials in this course. The content of the test will be designed to assess the students' language proficiency in areas that are relevant to their intended pathway course, such as reading, listening, writing, and speaking skills. The content of the test will also be aligned with the Common European Framework of Reference for Languages (CEFR) levels, which is the recognized standard for language proficiency assessment in Cambridge testing criteria. In determining the test content, a checklist has been implemented, and it includes important considerations such as validity, reliability, authenticity, practicality, and impact (see appendix 1). The test content has been designed to meet these criteria, ensuring that the test accurately measures the students' language proficiency, is fair and unbiased, and provides a useful tool for evaluating their progress.

1.1.2 Textbook and Test Book

The 12-week CAE course at TAFE is 5 days a week for 4 hours a day. The teachers use the Advanced Expert coursebook to teach the regular grammar, listening, speaking, reading, and writing components, which are aligned with the CAE exam (Bell & Gower, 2014).

Figure 1. Test Components of the CAE examination. On Thursdays, students focus on writing skills and practice writing the components tested in the official exam (see Figure 1). On Fridays, students practice speaking in the form of presentations, or speaking test practice as illustrated in Figure 1. The weekly tests are administered on Mondays, which are from the Cambridge Advanced in English Test Book (Assessment, 2016). This formative achievement test is used to measure and evaluate learners’ English language skills and knowledge gained from their weekly classes.

Test Content

1.2.1 Test Purpose

Instead of focusing on the content of each unit, TAFE teachers align the test numbers from the test book with the unit numbers from the coursebook when preparing students for the CAE test. This is because the actual CAE test is designed to be random, and the tests in the test book cover the original components of the exam. As a result, this approach allows for a better alignment with the test format and provides students with a more realistic practice experience. In preparation for the Certificate in Advanced English (CAE) exam, TAFE instructors administer mock tests under authentic exam conditions in weeks 1, 6, and 12 of the course. While the CAE exam itself is randomized, students must pass a minimum of 4 out of the 12 tests in the course to progress to the pathway course of their choice at TAFE. A careful analysis of past test scores revealed that unit 5 had consistently yielded the lowest scores. To address this issue, we took a meticulous approach to redesign the test, focusing on assessing the students' language proficiency in areas that were relevant to the unit they were studying. Our primary objective was to ensure that the test items accurately reflected the level of difficulty, degree of clarity, and appropriateness of the material covered in the unit. Our aim was to provide students with a more effective learning experience that would better prepare them for the pathway course. The differences between the two tests will be explained in detail within the various skill sections. The administration of the test was thoughtfully designed to ensure its reliability and validity. As such, the grammar component of the test was excluded since the reading and listening sections already incorporate multiple-choice, short-answer, and open-ended questions to assess learners’ grammatical skills. This deliberate decision was made to provide an equitable and impartial evaluation of the students' language proficiency based solely on what they have learned in the unit, rather than utilizing generalised tests that are commonly employed at TAFE Queensland. The test's reliability will be ensured through the use of standardized administration conditions and established procedures for scoring and reporting the results, as specified in the Cambridge English Qualifications Handbook for Teachers (Cambridge Assessment English, 2015).

1.2.2 Overview of the Test

The aim of the formative assessment is to assess the English language proficiency and academic knowledge of students who have finished Unit 5. The test is divided into four sections, namely listening, reading, speaking, and writing, and is modelled after the components of CAE. The successful completion of four out of twelve tests is a requirement for students seeking admission to a TAFE course, thereby demonstrating their English proficiency. The primary objective of the pilot test is to assist teachers in monitoring their students' English language development after completion of Unit 5, and to offer valid washback that encourages the enhancement of both students' four language skills and teachers' teaching methodologies. To assess students' language proficiency, a performance-based model has been implemented, as per Fulcher's (2010) recommendations. This model is suitable for evaluating learners' intended communicative effect and their achievement in communicative abilities. To measure the test's usefulness, the CEFR framework was chosen as the basis for assessment and results measurement, as it aligns with the language functions and communicative situations learners can handle and produce confidently. The framework provides students with a clear understanding of their strengths and areas for improvement, facilitating their language development effectively.

1.3 Test Usefulness

The purpose of the language proficiency test is to evaluate the learners' aptitude in English. In light of this, we have chosen to modify the original CAE exam structure to better align with the learners' language skills. To comprehensively evaluate learners' language proficiency, we have included all four areas of listening, speaking, reading, and writing. The original test is geared towards advanced learners who have completed the entire course, and therefore may not be suitable for the current level of the learners. To address this, we have adjusted the level of difficulty of the test questions to match the current proficiency level of the learners. To ensure the validity and relevance of the revised content, learners will initially take the original test, and the modifications will be based on their actual test results. The revised tests are aligned with the themes of the selected unit, which is centred on "controversy," and include engaging content to stimulate learners' interest. In order to achieve the learning objectives of this unit, we have placed particular emphasis on the speaking and writing tests. The speaking test items have been modified to address controversial topics, while also reflecting the themes of the selected unit on commonly discussed issues, such as the environment and climate. Similarly, the writing tests have been revised to feature travel-related contexts, which students are likely to encounter in the classroom. The revised listening and reading tests aim to highlight controversy by updating the questions and texts with themes that reflect current societal concerns around ecological and social development, particularly in Part 6 of the reading test. This section presents a reflective article on the impact of human activity on climate change, using examples of environmental pollution and posing thought-provoking questions. Completing these tests will not only challenge candidates' abilities to comprehend, infer, and analyse information but also encourage them to think critically about the real-world problems raised in the materials and appreciate the diverse cultures represented. In addition, some uplifting stories included in the tests may inspire candidates and evoke a sense of wonder about the world. As previously discussed, learners in the classroom originate from diverse backgrounds and possess varying levels of learning abilities, as well as diverse career aspirations. As a result, these factors necessitate the incorporation of an adaptive testing framework that is mindful of the learners' individual needs. To this end, the speaking tests are tailored to account for learners' proficiency levels, where the examiners use varying assessment criteria to promote a constructive learning experience that helps to alleviate anxiety and encourage optimal performance. Similarly, the writing test is designed to match the learners' current level of knowledge and ability, thus fostering confidence in their academic writing skills. This approach ensures that the assessment is both fair and effective in facilitating learners' progress towards their academic goals. The entire test covers the assessment of the four communicative competencies, and each part of the test examines at least two communicative competencies simultaneously. Our tests mimic the skill mobilization of real-world language use as closely as possible so that learners to gain comprehensive English language development in one test. The time for implementing the test will also be planned by the teacher according to the teaching plan. The test will be divided into four parts, the time for each part is: Speaking (16 minutes), Listening (40 minutes), Reading (1 hour 30 minutes), and Writing (1 hour). Due to the long-time of each section, it is difficult to complete all the tests in one class, The tests will be completed separately in the class time according to the class type, which can ensure that each test can have sufficient time. To ensure test usefulness, the checklist of Bachman and Palmer (1996, pp. 150-155) is used for evaluation (See Appendix 1).

1.4 The Pilot Test

When designing a language proficiency test, it is crucial to consider the five principles of test development, including validity, reliability, practicality, authenticity, washback, and transparency. Validity refers to the test's ability to accurately measure the construct it claims to measure. A valid test should measure the targeted language skill without introducing other confounding factors as we mentioned above. Reliability, on the other hand, refers to the test's consistency in producing similar results when administered multiple times or by different raters. A reliable test should produce consistent results, regardless of the rater or the occasion on which it is administered, which is why we hope that the teachers at TAFE will take into account our pilot test and keep using it in the future to verify these results. The test's practicality is another key consideration, as it should be easy to administer, score, and interpret by those who need to use the test results, which is why we have consulted with the FCE and CAE teachers at TAFE before we distributed our test to the students. They gave us valid feedback to readjust some of our question formulations and corrected some of the question structures. The test's authenticity is also important, as it should simulate real-life tasks and contexts that are meaningful and relevant to the learner. This is the main reason why we have developed our pilot test to accommodate the learners and give them a test which resembled their prior learning. In addition to this, the test's positive washback can motivate learners to learn the material being tested, leading to improved teaching and learning. Finally, the test's transparency is essential, as it should be clear and easily understood by learners and other stakeholders, reducing confusion and increasing the usefulness of the test results. By taking into account these factors, language proficiency tests can be designed to accurately and reliably assess a learner's language skills in a practical, authentic, motivating, and transparent way. As such, the language proficiency assessment for this unit comprises four parts: listening, reading, speaking, and writing, and has been designed in accordance with these six principles of test development. The selection of Unit 5 "Controversy" (See appendix 2) of the coursebook aims to test the students' competence in the four skills and their understanding of the unit's content. By administering this test, students can assess their level of competency in the four skills and the knowledge covered in the unit, identify their areas of strength and weakness, and prepare for future tests and assignments. Adhering to these principles ensures that the test will accurately and reliably measure the students' language proficiency while being practical, authentic, motivating, and transparent in its design.

2. Item design

Overview Speaking Unit 5 of the Advanced Expert Coursebook is about Controversy. The topics mentioned in the unit are predominantly about the environment, however, test 5 focuses on isolated places, painting, and what sort of help people need in life. The topics in the test measure illocutionary competence, whereas a speaking construct should refer to the ability, knowledge, and processes involved in the production of spoken language (Fulcher, 2003). These skills include pronunciation, grammar, vocabulary, fluency, and discourse management. A speaking construct is typically assessed through tasks or activities that require the learners to use these abilities in order to communicate effectively in a given situation. Fulcher (2003) emphasizes that it is important to define the construct clearly to develop valid and reliable assessments of speaking ability. This is why we have adapted the test according to the unit taught in the prior week. The redesigned test involves controversial topics about the environment, climate change, and problems people discuss in general. The students have also watched two documentaries related to controversial topics (i.e., The World’s Water Crisis, and The Game Changers), these documentaries discuss the importance of water to humans and the planet, and how the planet is running out of drinkable water, which is linked to the meat industry. As homework, they watched the second documentary which is about athletes who have opted for a vegan diet and how their bodies can perform without meat. This documentary is controversial and is linked to the meat industry as the cause of deforestation, the reduction of oxygen, and the increase of carbon dioxide in the atmosphere.

2.1.2 The Original Test Items

The topics in the original test measure illocutionary competence as the topics are random and do not resemble unit 5 (see Appendix 3), whereas a speaking construct should refer to the ability, knowledge, and processes involved in the production of spoken language used in class (Fulcher, 2003). These constructs include pronunciation, grammar, vocabulary, fluency, and discourse management. A speaking construct is typically assessed through tasks or activities that require the students to use these abilities in order to communicate effectively in a given situation. Fulcher (2003) emphasizes that it is important to define the construct clearly in order to develop valid and reliable assessments of speaking ability. This is why we have adapted the test according to the unit taught in the prior week.

2.1.3 The Redesigned Speaking Test

The redesigned test involves controversial topics about the environment, climate change, and problems people discuss in general where learners can implement language learned in class (see appendix 2). The selected areas include the ecological impact of pets, the decline of the bee population, the impact of tourism, and the impact of water. This will stimulate and reinforce the learners to focus on the speaking constructs as learned and practised in class for positive washback, because every individual has their own style and manner of speaking, based on their personal experiences in life. The tasks were carefully selected to avoid negative washback of simply memorizing information, but to include the ecological impact of pets, the decline of the bee population, the impact of tourism, the meat industry, and the impact of water (see unit 5). This will stimulate and reinforce the learners to focus on the speaking constructs as learned and discussed in class (Fulcher, 2003). Based on the learners’ needs, we have decided to omit Part 1 of the speaking test as it is irrelevant, time-consuming, and to reduce anxiety by focusing on the testing components. Part 1 involves an introduction of the students where they explain where they are from, during normal exam conditions, this is necessary if they do not know their partner and interlocutor to build rapport in the official exam. In this scenario, the learners have been together for 12 weeks in the FCE class and 5 weeks in CAE. They know each other and their teacher, reducing this part of the test, allows the teacher to complete all tests in one day (i.e., 4 hours), rather than spread out over two days. Additionally, it allows the learners to have more time to speak in Parts 2 and 3 of the tests. As it has been noted that students often struggle to answer the questions within one minute by the FCE and CAE teachers at TAFE. In sections 3 and 4, we will explain and show the item use and design in detail.

Reading Original Reading Test

Reading skills are considered “part of general language-understanding processes” (Alderson, 2020). According to Adetunji (2017), communicative competence can be manifested and tested through extensive reading, allowing learners to process information, which is an indispensable tool within human communication. Module 5 of the Advanced Expert Coursebook includes cross-text multiple matching sections where learners are taught to (1) sort out the general messages of some texts after reading the title and the introduction of them, (2) look for key information which will help them answer the questions, and (3) organize texts coherently. Based on the learners’ prior knowledge of the testing materials, their communicative competence, including sociolinguistic, grammatical, strategic, and discourse competencies (Savignon, 2017) are predominantly tested in the original reading test, Test 5 (see appendix). As mentioned in the rationale, learners are to study at TAFE where they are likely to analyze different reading materials. The reading articles of this reading test are authentic texts imitating real-life magazines, journals, etc. The authenticity of the tasks allows learners to cope with their future tests or assignments. Moreover, some of the tasks are also replicative activities that learners have done in class. The following table shows a brief description of the original test: Duration Part No. of Items Task Type Constructs Multiple-choice cloze Tests candidates’ lexical competence (i.e., semantic precision) Open cloze Tests candidates’ awareness of grammar and vocabulary. Word formation Tests candidates’ lexical competence (i.e., affixation) Key Word transformation Tests candidates’ medium of responses regarding grammar, vocabulary, and collocation. Tests candidates’ comprehension skills, ability to spot key information, and understanding of text structure. Tests candidates’ understanding of specific opinions/information. Tests candidates’ ability to grasp global meaning of text, understanding of text structure, cohesion, and development. Tests candidates’ understanding of specific opinions/information.

Parts 1 to 4 of the original test consist of open cloze, word formation, and keyword transformation task types which mainly test students’ linguistic competence and medium of responses. However, when measuring reading comprehension, requesting candidates to write short phrases is less valid a procedure than other question types (i.e., multiple-choice questions) (Henning, 1987). Thus, the first four parts have been eliminated in the redesigned test to maintain validity and to better align with the module content and test objectives. As for parts 5 to 8, questions were revised and improved based on piloting results such as item discriminability and difficulty checks, that had been tabulated in an Excel document (Henning, 1987). It is important to note that parts 5, 6, 7, and 8 in the original reading test have been adjusted to parts 1, 2, 3, and 4 in the revised reading test. As in the above, the modified version of the reading test. The following are the general reviews of the revised reading test: •Part 1 consists of an article and six multiple-choice questions which are designed to test candidates’ comprehension skills, including their understanding of opinions and messages expressed within the passage. Candidates are expected to read closely to deduce contextual meanings. During the stage of item analysis, it is noticeable that most of the candidates have chosen the incorrect option “D” in question 5. Having reviewed option D, it is indeed misleading as it includes matching words in the passage which do not provide correct information. As specified by Henning (1987), candidates are likely to adept matching materials without comprehending the passage. Therefore, option D was modified. Little modifications have been made to other items. •Part 2 is a cross-text multiple matching test which is similar to the reading activity in the textbook. It is important to note that part 6 of the reading test is about a controversial topic regarding how we can reduce climate change. Not only does it test candidates’ comprehension skills, but it also elevates their understanding and perception of the topic. Part 6, particularly question #39, was modified due to relatively low test scores. Little modifications have been made to other items. •Part 3 of the original test requires candidates to select six out of seven extracts and fit them into the gaps in the magazine article. Gapped text tests candidates’ prior knowledge of text structure and their competence to make the text coherent and well-developed. As reflected in the piloting results, most candidates have chosen the extra extract as one of their six selections. Therefore, before the revised test is implemented, the extra extract which does not provide useful information is eliminated to alleviate candidates’ confusion (Popham, 1990). In addition, as four parts of the original test have been eliminated, the test duration has been shortened from 90 minutes to 60 minutes. Scoring is subjected to the revised key answer sheet provided.

2.3 Listening

2.3.1 Original Listening Test

Brown and Abeywickrama (2010) mention that the process of listening performance itself is an invisible and inaudible process from the auditory signal to the inner meaning of the ear and brain. It is a learning performance that cannot be directly observed. In addition, more than 45% of human communication is spent listening. This shows the importance of listening skills in overall language ability. Therefore, in order to test students' listening ability at this stage, teachers must observe students' listening ability through listening tests. To test the listening ability of learners in this stage. Unit 5 of the textbook focuses on the decline of the bee population. Using an audio clip, students are asked to answer short questions based on the listening material. Next, fill in the blanks according to the recording. These two basic exercises mainly examine students' language competence and strategic competence. One is to test students' ability to summarise listening materials; the other is to test students' ability to process vocabulary spelling and key information in listening materials. For the listening practice in the textbook, this listening test mainly sets questions on language competence and strategic competence. To pass the listening test, students should first master the vocabulary spelling of the learning stage. Second, students should be able to understand long conversations or speeches. And be able to understand the key message of the speaker. In the first part of the test, students were given three conversations and asked to choose the answers to the questions based on the conversations. The conversation included a conversation between a student teacher and her tutor about teaching, a discussion between a couple of friends about a concert, and their thoughts on a new company policy. In the original test, students need to quickly understand and discriminate the meaning of the question stem and options. Due to the short examination time and weak text comprehension ability of students, the correct rate of equalisation is about 50%, and some questions get high scores because the question stem is simple and easy to understand. The second part of the test provided a speech to the scientist addressing the robot he had created. Fill in the blanks according to the recording. First, test students' vocabulary and spelling abilities. Second, it examines the ability of students to capture key information. Thirdly, the question is rewritten according to the audio, and the student's ability to replace synonyms in English is tested. In the original test, students performed well on some of the simpler words but made more spelling mistakes on the more complex words. At the same time, some key information captured for context understanding, students' performance ability is poor. Unable to achieve quick semantic conversion and positioning. The third part of the test provides a recording of an interview discussing the history of the London Underground. This part examines students' ability to paraphrase sentences and paraphrase conversations. Listen to the recording to determine the correct choice. This part of the question is easy to understand, so the score is relatively good. Individuals with bewildering topics stem, the overall level of students is poor. The fourth part of the test provides five short voice excerpts of people talking about museums they have visited. This part requires students to perform out-of-order matching, which contains multiple interfering items that are not given. This part tests students' ability to capture key information and answer strategies. Because of the way the question is set, the visual effect is confused, and the question item is all paraphrased. The students' lack of problem-solving strategies and language ability resulted in very low scores in this part of the test. It can be seen that students' spelling ability is relatively good, but the complex vocabulary is still unfamiliar. Some words are simply misspelled but heard correctly, which suggests that students should also improve their vocabulary proficiency. In addition, students' ability to summarise the listening text and capture key information is insufficient. The revised test will reduce the difficulty of distracting items and interpretations in the questions.

2.3.2 Revised Listening Test

The revised test focuses on questions with low test scores. Part of the original choice was kept. Considering that students are prone to making mistakes in the question, analyse the cause of the mistake to modify. Test the first part of the main content of modified options for middle school students who cannot understand the meaning of options. And a relatively simple rewrite. The second part of the test examines students' vocabulary and spelling. The questions with more errors in the original test were kept and new dictation vocabulary was added. The third and fourth sections of the test measure students' ability to paraphrase recognition. Assess by providing stimulating sentences or dialogues, asking students to choose the correct answer from several options. During this period students should integrate the correct answers from the context and match them with the choices. The fourth part of topic design is dazzling, and strategic difficulty. The fourth part changes the topic from out-of-order matching to single selection. The accuracy of the modified test results is significantly higher than that of the original test. Students can read the questions more easily and do not need to be distracted by the distribution of questions.

2.4 Writing

2.4.1 Original Writing Test

Writing is considered a basic skill to test the ability of second language learners. At present, learners are still in the stage of knowledge accumulation. In order to successfully pass the four tests in 12 weeks, learners need to practice writing during the course to improve their specific English ability. The purpose of the writing test is to help learners acquire the ability to write academic essays. The test is conducted in the context of the students having acquired the relevant knowledge, the learners have also received writing instruction before this task, and the format of the test follows the requirements of the course. In addition, the topic introduction also provides students with relevant prompts about the topic to help them successfully understand the question. The question of the task is very clear, and students can develop their own ideas within the prescribed limits. This open writing test not only ensures that learners can use their knowledge to achieve their goals as much as possible but also learn the interactivity of the test. The original writing test is mainly to test the learners' strategic competence and grammatical competence. The item requires students to clarify their statements on a given task and provide reasons to support their opinions. This is the primary training method for students' writing ability in this course. Students can use their language knowledge in an appropriate way to achieve their communication goals, which is a manifestation of strategic competence. Bachman and Palmer (1996) state that strategic competence not only provides practice for language knowledge and the external environment but also provides an effective connection between language knowledge and subject knowledge. As for grammatical competence, learners need to effectively combine the language system and semantic system in the process of completing the essay, which can also be said to be the combination of words and grammar. Writing can assess how learners use the grammatical rules of the target language with different structures and can also assess the accuracy of learners' vocabulary usage in a specific topic. In terms of authenticity, although this test does not reflect the real world, as a writing test, we can consider it in terms of how it helps learners in social interaction. If a test is claimed to be authentic, it is likely that the test was developed in the real world (Brown & Abeywickrama, 2010) Learners are likely to need to express their views on this topic. Spack (1988) referred to the process of learning to write in an academic setting as one of "guiding ESL students into a community of academic discourse". Writing, as the written version of speaking, is almost the same. Therefore, the learner's ideas in writing can also be utilized in practical communication. The writing test is completed in one teaching period with the other three tests, which is very convenient for both students and teachers. Also, we only selected one item of the writing test, so the test time is set at 1 hour, which is quite sufficient for students. On the basis of the original scale, the scoring criteria also refers to CEFR to provide learners with accurate feedback. Therefore, testing is very useful. The results of the original test did not receive good expectations. Among the 15 articles, only 7 students answered the questions clearly, and 5 students did not complete the test. There were also some mistakes in the use of grammar and syntax, and the average score of all students was only 3 points (out of 5). According to this, we provided feedback for each test paper, clearly listing the problems that students have. We also collected feedback from students about the test: they said that they did not understand the requirements of the questions correctly, so they did not answer the questions. In addition, the students who have not finished the essay are new members of this course, and they have not learned a lot of knowledge. Therefore, the test was revised in response to student feedback and confirmation of unit content.

Figure 2.4b. Revised writing test

2.4.2 Revised Writing Test

The revision test has been revised primarily for writing questions. The genre of writing does not change, it is still assessed by learners answering questions and addressing all the points with their own acquisition since the theme of Unit 5 is “controversy”. The instruction for the writing part of this unit is about tourism and students have been trained to produce "supporting sentences". Therefore, the revised topic is also related to travel. At the end of the writing requirements, a requirement is added for students: they are asked to use the "supporting sentence", words and grammar learned in this unit as much as possible. Through such modification, it can not only ensure the students' training in the writing genre, but also reduce the difficulty to a certain extent, and allow learners to consolidate the knowledge of this unit. The writing test scale is also revised appropriately. In the original scoring form, the scores on content, communicative achievement, organization, and language are clarified. The revised topic requires students to make the best use of the knowledge in this unit, so this criterion should also be added.

3. Guidelines for Users

3.1 Overview

Listening and reading will be tested in the same class, and speaking test and the writing test will be taken separately in another two classes, the duration and number of items are in the table below:

Ability Duration Number of items Speaking Reading Listening Writing

The test will be completed in class, with the teacher acting as the invigilator and responsible for distributing test papers and playing the listening recordings to the students. Due to practical reasons, the 4 tests will be divided into different parts, listening and reading will be tested as one part in one class, and writing and speaking will be tested in two separate classes. Tests are individual, candidates are not allowed to use dictionaries or any digital devices. After receiving the test paper, candidates should fill in their names on the answer sheet. During the test period, candidates must remain quiet. They are not allowed to whisper, or peek at other people's answer sheets, are not allowed to have other behaviours that violate the rules of the test room. Once the time is up, all candidates should stop writing.

3.2 Speaking: Administering the Test

3.2.1 Part 2 of the Speaking Test

Students are invited into another classroom in pairs as they will need to interact with one another. They are asked to record the test, to transcribe it later as homework, and bring it back to class when receiving feedback. Students are allowed to choose who will be Candidate A and B. This is to reduce anxiety and allow students to feel a part of the testing environment. Candidate A will be presented with the pictures and questions for Part 2 first and will be given 60-90 seconds to answer the questions, student B will then be asked one question to respond in 30 seconds to Candidate A’s pictures (see figures 4). Candidate B will then be given their set of pictures to answer two questions about the pictures, and Candidate A will be given a question to respond in 30 seconds to Candidate B’s pictures (see figures 4).

Figure 4a. Reconstructed test item pilot test, speaker A

Figure 4b. Reconstructed test item pilot test, speaker B

Figure 4c. Part 2 of the Speaking Test, transcript for teachers

3.2.2 Parts 3 and 4 of the Speaking Test

In this part of the test, candidates need to interact with one another and show they can use turn-taking strategies, intonation stress, sustain a conversation, and use idiomatic language and figurative language (see figures 5).

Figure 5a. Reconstructed test item pilot test, Part 3, interaction

Figure 5b. Reconstructed test item pilot test, Part 4, interaction continued (teacher use only)

Figure 5c. Parts 3 and 4 of the Speaking Test, script for teachers While the candidates are speaking, the teacher needs to time and administer the errors, but also the appropriate use of test items (see figure 6). Due to the fact that every individual has their own style and manner of speaking, based on their personal experiences in life, could mean that language production is inherently variable, making it difficult to establish clear criteria for assessing speaking ability. Nonetheless, the notion of the uniqueness of speech highlights the importance of considering individual differences and characteristics when assessing speaking ability, rather than relying on a one-size-fits-all approach (Fulcher, 2003). This is why it is important to observe and administer if learners can use appropriate interaction linkers, initiate, and respond appropriately, and negotiate toward an outcome. This part of the test looks more into the sociolinguistic and pragmatic approach.

Figure 6. Administering Constructs for Speaking (used to take notes during the test)

3.2.3 Metalinguistic Awareness of Learners

In order for the learners to understand their feedback and learning outcomes, students are asked to record the speaking test and transcribe their part of the speaking. Students then have to highlight their own errors and correct them with what they should have said (see an example in figure 7). The implementation of self-assessment as a method for learners to evaluate their own language proficiency and to develop their metacognitive abilities to self-appraise and self-regulate. The most effective and meaningful self-assessment methods involve learners setting their own goals, monitoring their progress, and reflecting on their own learning experiences (Brindley, 1989). These methods have been integrated into language learning and teaching and are accompanied by feedback and guidance from the teacher. By providing learners with suggestions for improvement and helping them develop a more accurate understanding of their language proficiency, the teacher can enhance the effectiveness of self-assessment. Ultimately, self-assessment should be an ongoing process that enables learners to take ownership of their language learning and development. By engaging in self-assessment, learners can become more aware of their strengths and weaknesses, set achievable goals, and reflect on their progress (Brindley, 1989). This holistic approach allows learners to listen to themselves and enables learners to experience cognitive and socio-effective processes of listening comprehension, while writing down what they are listening to, which is a form of a dictation task, reinforcing them to write, to then read and correct their own errors while working on a speaking-related task (Goh & Burns, 2012). Figure 7. Example of student error correction.

3.3 Guidelines for Users (Revised Reading Test)

3.3.1 Administration Guidelines (Revised Reading Test)

Candidates have 1 hour to complete all four tasks of the revised reading test. Before the test starts, the invigilator will read aloud the instructions that are already written on candidates’ answer sheets and reiterate that no dictionaries or other electronic devices are allowed. The invigilator should kindly remind candidates that there are 4 parts and 26 questions in total and that they should pay attention to the respective scorings. Candidates are expected to mark all their answers on their answer sheets (see Figure 8). Candidates must not leave the examination room without permission. All forms of communication between candidates will be considered as cheating.

Figure 8. Candidates’ Answer Sheet for Revised Reading Test

3.3.2 Scoring Guidelines (Revised Reading Test)

Instructors should refer to the model answer sheet (see Figure 9) when grading candidates’ answers. There are 4 parts and 26 questions in total. Questions in parts 5, 6, and 7 count for one point each, whereas questions in part 8 count for 2 points each. The total score of the revised reading test is 36. To align with the TAFE scoring standard, candidates are expected to achieve a minimum score of 22 out of 36 (≈60%) to pass the test.

Figure 9. Model Answer Sheet for Revised Reading Test

3.4 Listening

Candidates will have 40 minutes to complete the listening test. The listening test is divided into four parts with a total of 30 questions, each of which is one point. Each part of the test will consist of one or more audio clips and corresponding text tasks. Parts 1-4 are played twice. Candidates will have a short time to read the questions before the recording begins. After the recording, candidates will have 5 minutes to fill in their answers on the answer sheet. In the audio materials, candidates will hear English from different speaking styles and accents to present a variety of real contexts and communication situations.

3.4.1 Listening Test Scale

Scoring Grade Description C1: 23-26 Candidates are usually able to understand long and complex speeches, process language and recognize grammar, and pay attention to the overall content of the speaker's expression, including unusual collocations and academic expressions.

3.5 Writing

You have one hour to complete the writing test. Read the questions carefully and complete the questions given on the topic. You need to write an academic paper on a specific topic, please make sure the format of the essay is accurate and formal. Writing test scale The test examines four areas of a candidate's writing: content, communicative competence, structure, and language. The content focuses on the candidate's task completion: whether the candidate answered the questions and did not miss any questions, and whether the content of the article is relevant. Communicative Achievement focuses on whether the test taker used writing and language styles as opposed to genre. Organization focuses on the structure of the article: whether the article is orderly and logical. Language examines appropriate and accurate use of language; errors in words and spelling. After completing the above scoring, the total score obtained by the learner will be re-evaluated according to the Cambridge English score. The following are the assessment scales used by the raters:

Figure 10. Writing scale, adapted from the Cambridge C1 Advanced Handbook for teachers

Length of responses Candidates need to make sure that they write the correct number of words. Responses that are too short may not have an adequate range of language and may not provide all the information that is required. Responses which are too long may contain irrelevant content and have a negative effect on the reader.

Use of “supporting sentence”, vocabulary, and grammar studied in this unit. Candidates are expected to use the knowledge learned in Unit 5 in this essay. "Supporting sentence", as a key sentence pattern, needs to be used accurately to support the candidate's point of view. In addition, the use of other knowledge also needs to be correct and effective.

Varieties of English Candidates are expected to use a particular variety of English with some degree of consistency in areas such as spelling, and not for example switch from using a British spelling of a word to an American spelling of the same word.

Sample Essay Today, the growth in international travel has been considerable and has had a positive impact on many factors, especially economically and culturally. The economy is the factor most positively affected by travel abroad. International tourism promotes the economic development of various countries. Tourists basically cannot avoid consumption and need to pay a series of other expenses such as accommodation and food. Employment opportunities stimulate economic growth and make an important contribution to national income and tax revenue. In addition, tourism has also become the main source of fiscal revenue for many developing countries, which is financially beneficial to economic development. International tourism can realize the circulation of international currencies and promote the growth of international foreign exchange. In addition, traveling abroad also promotes the spread of culture and protects cultural diversity. Global tourism acts as a formidable means for fostering cultural exchange and cross-cultural comprehension among nations. Tourists who travel to foreign lands are given the chance to encounter novel cultures, traditions, and ways of life, thereby acquiring a deeper appreciation and comprehension of the world's diversity. Through this type of intercultural interaction, cultural barriers can be dismantled, and greater tolerance and respect between nations can be nurtured. In conclusion, there are many positive effects of traveling abroad, such as promoting the protection of historical sites, the inheritance of traditional cultures of various countries, and the promotion of friendly exchanges between countries. However, promoting economic development and cultural dissemination are the most important and positive effects of traveling abroad, because they play an essential role in people's lives. Scale Mark Commentary Content

Communicative Achievement

Organization

Language

Comments:

4. Test Specifications

4.1 Test General Specifications

The purpose of this test is to assess learners' learning outcomes and communicative competence after studying this unit. Bachman (1990) believed that: testing is a kind of measurement and a way to evaluate teaching behaviour. Students need to pass the test of this course to prove that they have the English knowledge and abilities required for future study, life, and work. Periodic tests help students understand their learning effects and make progress through teacher feedback. The test is designed according to the current level of the students. Learners can put the learned knowledge into practice through the test of the four language skills and consolidate it. Speaking and writing will achieve effective output in real-world communication, students can express their opinions fluently and accurately and analyse them reasonably. With the help of listening and reading, learners can contact learning materials other than textbooks to expand their knowledge system. In order to make the best use of what students have learned, the test has been revised according to the theme of this unit. It is expected that learners can understand "controversy" and accurately and effectively produce the key knowledge of this unit in the process of communication. This test will be done in several classes, according to the teaching plan. In the classroom environment, students will experience less tension and maximize the learning effect.

4.2.1 Test Purpose Speaking

The students in the original test and pilot test were from two different classes and teachers. The original test was conducted between the 21st and 25th of April 2023, whereas the pilot test was redesigned and administered between the 1st of May till the 5th of May 2023.

As mentioned in the rationale, the learners are studying at TAFE to get into one of the pathway courses it offers. In order to do so, learners are expected to pass four out of twelve tests. The speaking component is one of the important tasks as learners are expected to communicate with Australian students during their pathway courses, work on group assignments, and present various topics in their field. TAFE expects the learners to have a minimum overall score of 15 out of 25, which is 60%.

4.2.2 Test Description Speaking

As illustrated in part 3.2 of this paper, the speaking test consists of three parts, each assessing different communicative competencies. In Part 1, the interlocutor engages in a brief exchange with each candidate to evaluate their use of general interactional and social language. However, this part was omitted in the pilot test. Part 2 involves an individual 'long turn' for each candidate, where they describe two out of three presented photographs, testing their ability to organize discourse, compare and describe visual information, and express opinions and speculations (Cambridge Assessment English, 2015). Part 3 requires the candidates to engage in a two-way conversation, based on spoken instructions and written stimuli, to complete a discussion and decision-making task. The conversation is divided into two phases and aims to assess the candidates' ability to sustain interaction, exchange ideas, express and justify opinions, agree or disagree, suggest, speculate, evaluate, and negotiate (Cambridge Assessment English, 2015). Lastly, Part 4 is a discussion on topics related to the collaborative task, with a focus on expressing and justifying opinions, agreeing or disagreeing, and speculating. The aim of these four parts is to assess multiple communicative competencies and simulate the mobilization of language skills in real-world contexts, contributing to the development of comprehensive English language proficiency among learners (Cambridge Assessment English, 2015). The timing of each part is predetermined, and the test is administered in separate sessions to ensure sufficient time for each section.

4.2.3 Scoring Pronunciation

Fulcher (2003) suggests that pronunciation can be tested through the use of minimal pairs, where learners are asked to distinguish between two similar sounds that are contrastive in the target language (e.g., Chip/Cheap). Other approaches could include assessing learners' ability to produce certain sounds, stress patterns, or connected speech (i.e., Don't trust him he lies/ opticians can heal eyes). Testing involves a range of tasks, such as paraphrasing sentences or words in the questions asked, describing pictures or scenes, or showing the ability to maintain interactive conversations. These test items are tested in parts 2, 3, and 4 of the Speaking Test, and we have kept the original scoring rubric or the CAE test as it is reliable. Additionally, it was recommended by the teachers at TAFE to not change it, so they could use the test in the future.

4.2.4 Scoring Interactive Communication / Fluency

Testing learners' ability to produce language smoothly and at a natural pace, without hesitations or pauses, this item can be tested through the use of timed speaking tasks, where learners are given a set amount of time to speak on a given topic (Fulcher, 2003). Add to this, parts 3 and 4 of the Speaking test are to reinforce students to have conversations where they are encouraged to interact with the interlocutor and another student to maintain a flow of conversation. The main focus of the interlocutor will be to look out for the learners’ appropriate use of interaction linkers, initiate and respond appropriately, and negotiate toward an outcome.

4.2.5 Scoring Accuracy

According to Fulcher (2003), accuracy can be tested by assessing learners' grammatical and lexical precision, as well as their ability to use appropriate discourse markers in context. Testing could involve tasks that require learners to produce grammatically and semantically correct sentences, which could involve eliciting language from learners in response to specific prompts or situations, such as describing a picture, telling a story, or interaction sequences, as demonstrated in parts 2, 3, and 4. The effective use of these strategies is important for successful communication, as these strategies help learners to overcome difficulties in expressing themselves and make use of the language resources they have available. They can also help learners to negotiate meaning, repair communication breakdowns, and manage their speech production in real-time. Effective use of communication strategies can therefore lead to more successful and confident communication, which can in turn lead to increased motivation and language accuracy in the L2 (Fulcher, 2003). Incorporating the notion of the structure of conversation into the assessment of speaking involves assessing how well the speaker is able to maintain a coherent and organized conversation. This includes the ability to initiate and respond to conversation appropriately, the ability to maintain a topic, and the ability to use appropriate discourse markers and other features of spoken language to signal the structure of the conversation. To test the ability to use discourse markers effectively, the assessment could focus on the appropriate use of these markers. Discourse markers are words or phrases that indicate the relationship between sentences or clauses in a conversation (e.g., however, whereas, so, or for example). Skilled use of discourse markers can enhance the coherence and organization of a conversation, as well as support the listener in understanding the intended meaning of the speaker. Moreover, social and pragmatic knowledge is also needed to show competence by initiating and responding in conversation and how to use appropriate register and tone. Testing interaction competence involves a range of features (e.g., turn-taking strategies, intonation stress, sustaining a conversation, and using idiomatic language and figurative language).

Figure 10. Test results from the original test compared to the pilot test. In comparing the two tests, it is imperative to consider that they were conducted on two distinct classes by different teachers (see figure 10). Upon initial inspection, it is apparent that the pilot test yielded a higher score than the original test. There could be a few reasons for this discrepancy. It is possible that the students in the second class were higher achievers than those in the first class. Alternatively, it is possible that the pilot test felt easier than the original test as the students were tested on material they have reviewed in class, which allows them to perform better than on a test with unfamiliar material as mentioned in the theory in this paper. There are two example feedback forms in section 4.2.6 on how we scored the results based on the information in 4.2. Figure 11a, shows an example of the learner being able to correct herself in the test, this was taken into consideration when marking her. Figure 11b is an example of a student who was not able correct her pronunciation, however, she was able to still get a high score overall as she did perform well in other areas.

4.2.6 Rater Feedback

Figure 11a. Adapted Speaking Scale (used to finalise student results), from the Cambridge C1 Advanced Handbook for teachers (p.85)

Figure 11b. Speaking scale, adapted from the Cambridge C1 Advanced Handbook for teachers (p.85)

4.3 Reading

4.3.1 Test Purpose (Reading Test)

Based on the learners’ prior knowledge of the testing materials, they are expected to have acquired the skills to (1) sort out the general messages of some texts after reading the title and the introduction of them, (2) look for key information which will help them answer the questions, and (3) organize texts coherently. Hence, the predominant purpose of this reading test is to test whether the candidates understand text cohesion and structure; and whether the candidates have the ability to comprehend the given information or even grasp the underlying messages of the articles.

4.3.2 Test Description (Revised Reading Test)

Reading Test Description Purpose To test candidates’ ability to sort out the general messages and key information of the text. To test candidates’ understanding of text cohesion, structure, and development. Duration Number of tasks Number of items English variety Standard English Scoring Total score: 36 *Refer to the key answer sheet

No. of Items Task Type Description / Constructs Candidates are to read the article “Using Social Networking Sites to Find a Job”. Questions are designed to test candidates’ comprehension skills, including their understanding of opinions and messages expressed within the passage. Candidates are expected to read closely to deduce contextual meanings. The first five items are sequenced according to the information in the text while the sixth item is about the general interpretation of the text. Candidates are to read four extracts from articles regarding experts’ viewpoints on climate change. Questions are designed to test candidates’ ability to distinguish similar viewpoints and match them to their corresponding citations. It is noted that the four items may be interferential. *The topic is relevant to module 5 of the textbook “Controversy”, candidates have done a similar activity in class. Candidates are to read a magazine article about rock climbing, they are expected to gain an overall understanding of text structure and development before answering the question items. This section consists of a gapped text with six jumbled extracts. Candidates are required to fit in the best extract in each gap. It is specified that each extract should only be used once. Candidates are to read an article about risk taking. The article is sectioned into four parts (A, B, C, and D). Through skimming and scanning, candidates are expected to match ten questions to their respective sections. It is specified that some options may be chosen more than once.

4.3.3 Scoring (Reading)

Instructors should refer to the model answer sheet when grading candidates’ answers. There are 4 parts and 26 questions in total. Questions in parts 5, 6, and 7 count for one point each, whereas questions in part 8 count for 2 points each. The total score of the revised reading test is 36. To align with the TAFE scoring standard, candidates are expected to achieve a minimum score of 22 out of 36 (≈60%) to pass the test. As reflected in the test score statistics, the accuracy of the modified test results is significantly higher than that of the original test, and most students are able to pass the test (≥60%). However, it is particularly crucial to note that Part 3 results of the revised reading test are beyond our expectations. The total score of Part 3 is 6, 13 out of 15 candidates failed to get half of the questions correct. Thus, the test items must be analyzed again for future pilot tests. As specified by Popham (1990), the test development process is a reiterative process, and even operational tests are subject to revision.

4.4 Listening

4.4.1 Test Purpose

After this stage of learning, students should be able to master the vocabulary learned in this stage and be able to correctly dictate and use it. At the same time, students should have a certain ability to understand a long and relatively complex speech or dialogue. Therefore, the purpose of the test is to assess the Language competence and Strategic competence of the students at this stage. Students should be proficient in using the vocabulary covered in the previous units. Ability to understand and summarize key information through conversation and presentation.

4.4.2 Test Description

This is the listening section of the test. Students are forbidden to speak during the examination. Invigilators will play audio or use headphones to receive radio waves to play audio. If there is any problem with the equipment, please raise your hand and inform the invigilator. Students will hear audio recordings of speeches or conversations between multiple people. The test lasts about 40 minutes. After the test, students will have 5 minutes to fill in their answers on the answer sheet. According to the test requirements, students should use the same variety of English (e.g., British English or American English) when filling in the answers, and cannot use multiple varieties of English at the same time. Please note the case format and Spaces in the answer.

4.4.3 Scoring

The listening test is divided into four parts with 30 questions. Each question is 1 point, and the total is 30 points. Final marks will be calculated on the Cambridge English Scale.

4.5 Writing

4.5.1 Test purpose

After studying five units, students have basically mastered the skills of academic writing. Therefore, the purpose of this writing test is to assess students' ability to write academic papers, demonstrate mastery of vocabulary and grammar, effectively contribute ideas and opinions, and analyze logically.

4.5.2 Test description

This is the writing part of the test. Candidates will be given an open topic. The test lasts for one hour. Students are required to write an essay of 220 to 260 words. The requirements are: relevant to the topic, fluent in writing, correct in expression, coherent in meaning, and no major language errors. The content of the test is linked to the writing exercise in Unit 5. Students need to express their position and reasons in the essay. In addition, the test provides students with some notes including a detailed statement of different factors. This test does not involve overly knowledgeable and highly specialized content.

4.5.3 Scoring

The test criteria are divided into 4 parts, each with a maximum score of 5 and a minimum score of 0. In addition to grading, raters are required to fill out an analysis form to ensure learners receive effective feedback. The total score will be presented as the average of the 4 abilities. The final score will be converted to a score on the Cambridge English Scale for the paper. Writing test description Purpose Accurate spelling of words and ability to use vocabulary and grammar. Ability to write using the functions of expressing opinions, proving, persuading, etc. Communication ability Duration Topic Positive effects of travelling abroad Number of items Type Essay Number of questions English variety A particular variety of English Scoring writing scales with four criteria. 5 maximum points for each criterion. The total score is the average of the four criteria.

5. Additional Materials

5.1 Reading Score Statistics

5.2 Listening Score Statistics

6.Group work – Declaration of equitable contribution to the assessment task/s

Group number (as identified on your sign-up link): Group 1 Group members Full name (Lionnie) Un Lai Shuai Xie Yi Liu

Please provide a brief description of each group member’s contributions to meetings, ideas, completion of different aspects of the project, and roles undertaken: Member 1: Arranged 4 meetings to discuss who is doing what, when, checking progress, finalising work. Printed and copied all materials and tested one class with the original unit 5 test. Then scanned all results and distributed it to group members to redesign the pilot test. Retrieved the piloted tests, revised and corrected the materials, then tested another class with the pilot test. Scanned the unit, test materials, tests and test results, then distributed all results to group members. Revised the Poster and implemented ideas. Has written the Cover Page, Table of Contents, Rationale 1.1, 1.2, 1.4, 1.5, 2.1 Speaking part, 3.2 Speaking part, 4.2 Speaking part. Created GoogleDocs for all members to contribute their work, has revised and put them in a new Word Document, and submitted. Member 2: Attended all meetings and participated in all discussions. Marked original reading test results in an Excel spreadsheet for data analysis (i.e., difficulty index); revised the testing materials; prepared answer sheets and answer keys for candidates and instructors. Recreated a more presentable version of the table: Appendix 1 Checklist for evaluating usefulness. Responsible for 2.2, 3.3, 4.3 reading parts, and 5.1 score statistics analysis. Designed project poster. Member 3: Attended all meetings and participated in all discussions. Marked original listening test results in an Excel spreadsheet for data analysis (i.e. difficulty index). Revised the listening test materials to prepare answer sheets and answers for both candidates and teachers. Responsible for the 1.3 Test Usefulness, 2.3 listening section, 3.4 listening section, 4.4 listening section, and 5.1 Listening Score Statistics. Member 4: Attended all meetings and participated in all discussions. Assessment of writing tests, both original and revised; revised the writing test materials; prepared answer sheets for candidates and instructors. Responsible for 1.3 Test Usefulness, 2.4 Writing section, 3.5 Writing section of Guidelines for Users, 4.5 Writing section of Test Specifications, 3.1 Overview of Guidelines for Users and 4.1 Test General Specifications. Revised the writing scale. Group signatures to indicate satisfaction with equitable contribution to task completion: Yes, we all agree to these terms and conditions:

7. References

Assessment, U. O. C. C. (2016). Cambridge English Advanced 2 Student’s Book with answers and Audio: Authentic Examination Papers. Cambridge English. Adetunji, E. (2017). Reading problems and Solutions at Tertiary Level. In E. E. adegbija (Ed.), Effective communication skills in learning Basic Principles (pp.52-60). Ilorin: University of Ilorin, General studies Division. Alderson, J. (2000). Assessing Reading (Cambridge Language Assessment). Cambridge: Cambridge Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford university press.Bachman, L. & Palmer, A. (1996). Language testing in practice: Designing and developing useful language tests. Oxford, England: Oxford University Press. Bell, J. & Gower, R. (2014). Advanced Expert Coursebook. Pearson Education Brindley, G. (1986). The assessment of second language proficiency: Issues and approaches. Adelaide, Australia: National Curriculum Resource Centre. Brown, H. D., & Abeywickrama, P. (2010). Language assessment: principles and classroom practices. Pearson Education. Cambridge Assessment English. (2015). C1 Advanced Handbook for Teachers. Fulcher, G. (2003). Testing second language speaking. Longman. Goh, C., & Burns, A. (2012). Teaching Speaking: A Holistic Approach. Cambridge University Press. Henning, G. (1987). A guide to language testing: Development - Evaluation - Research. Rowley, MA: Newbury House. Popham, W. (1990). Modern educational measurement. Englewood Cliffs, NJ: Prentice Spaan, M. (2006). Test and item specifications development. Language Assessment Quarterly, Savignon, R. (2017). Visual literacy and the use of advance organize in reading comprehension lessons. In A. Dada (Ed.), Communication and Education. Ibadan: Constellation Books. Spack, R. F. (1988). Initiating ESL students into the academic discourse community: How far should we go? TESOL Quarterly 22 (1),29-52.

APPENDIX 1 Test Usefulness Below is a checklist extracted from the Oxford University Press for evaluating usefulness: Questions for logical evaluation of usefulness Extend to which quality is satisfied Explanation of how quality is satisfied Reliability To what extend do characteristics of the test setting vary from one administration of the test to another? 100% The rating scale and user guide regulate the homogeneity of the test. To what extend do characteristics of the test rubric vary in an unmotivated way from one part of the test to another, or on different forms of the test? 100% The test items are adapted from the Cambridge English Test and are tailored to candidates' current learning. To what extend do characteristics of the test input vary in an unmotivated way from one part of the test to another, or on different forms of the test? 100% The test is designed according to the key content of this unit of the textbook. To what extend do characteristics of the expected response vary in an unmotivated way from one part of the test to another, or on different forms of the test? 100% According to the test results, candidates' reactions to different questions are consistent. To what extent do characteristics of the relationship between input and response vary in an unmotivated way from one part of the test to another, or on different forms of the test? 100% According to the test results, there is consistency between the textbook content and the candidate's performance. Construct validity Clarity and appropriateness of the construct definition, and the appropriateness of the task characteristics with respect to the construct definition Is the language ability construct for this test clearly and unambiguously defined? 100% The test has clear definitions for listening, speaking, reading, writing and writing. Is the language ability construct for this test relevant to the purpose of the test? 100% The construction of language ability is related to the purpose of testing. To what extend does the test task reflect the construct definition? 100% The testing structure is designed according to the construction definition. To what extend do the scoring procedures reflect the construct definition? 100% The test results reflect candidates' language proficiency and abilities. Will the scores obtained from the test help us to make the desired interpretations about test takers’ language ability? 100% Candidates will strengthen the areas with insufficient abilities based on the scores obtained. Possible sources of bias in the task characteristics What characteristics of the test setting are likely to cause different test takers to perform differently? 100% Tests are rewritten according to the guidelines and the first test guidelines to ensure test validity. What characteristics of the test rubric are likely to cause different test takers to perform differently? 100% The testing project is designed based on the current learning progress. What characteristics of the test input are likely to cause different test takers to perform differently? 100% The input for this exam strictly follows the construction design and will not result in different student performance. What characteristics of the expected response are likely to cause different test takers to perform differently? 100% The test results vary depending on the language proficiency of the candidates. What characteristics of the relationship between input and response are likely to cause different test takers to perform differently? 100% The test results vary depending on the candidate's proficiency and language proficiency. Authenticity To what extend does the description of the tasks in the TLU domain include information about setting, input, expected response, and relationship between input and response? 100% The test description is similar to CAE testing. To what extend do the characteristics of the test task correspond to those of TLU tasks? 100% This test meets the task characteristics of TLU. Interactiveness Involvement of the test takers’ topical knowledge To what extend does the task presuppose the appropriate area or level of topical knowledge, and to what extend can we expect the test takers to have this area or level of topical knowledge? 100% This test has been rewritten based on previous exams. Candidates should have a language proficiency level corresponding to the difficulty level of the current textbook. To what extend are the personal characteristics of the test takers included in design statement? 100% The learning objectives and personal factors of candidates are one of the key factors in exam design. To what extend are the characteristics of the test takers suitable for test takers with the specified personal characteristics? 100% The testing task is largely suitable for candidates with specific characteristics, and the test results can show the strengths and weaknesses of the candidates' personal abilities. Involvement of the test takers’ language knowledge Does the processing required in the test task involve a very narrow range or a wide range of areas of language knowledge? 100% The testing tasks cover four areas: listening, speaking, reading, and writing, with each section testing different abilities in the relevant field. Involvement of language functions in the test takers What language functions, other than simple demonstration of language ability, are involved in processing the input and formulating a response? 100% This test covers a variety of test items, and candidates are required to complete the exam tasks under the same input conditions. To what extend are the test tasks interdependent? 100% The testing tasks of this test are interdependent. How much opportunity for strategy involvement is provided? 100% Each task requires a corresponding strategy to complete. Is this test task likely to evoke an affective response that would make it relatively easy or difficult for the test takers to perform at their best? 100% The testing context provides real-world materials that can elicit emotional responses from candidates. Impact Impact on individuals (test takers) To what extend might the experience of taking the test or the feedback received affect characteristics of test takers that pertain to language use (such as topical knowledge, perception of the target language use situation, areas of language knowledge, and use of strategies)? 100% Candidates will improve their current learning situation through the exam results. Through exams, candidates have a clear understanding of the target language usage. What provisions are there for involving test takers directly, or for collecting and utilizing feedback from test takers, in the design and development of the test? 100% The exam design was improved from the original test results. The source of the test results is the students of the course. How relevant, complete and meaningful is the feedback that is provided to test takers? 100% The feedback provided by the test is highly correlated with the candidate's personal abilities. Are decision procedures and criteria applied uniformly to all groups of test takers? 100% The procedures and criteria are applicable to all candidates. How relevant and appropriate are the test scores to the decisions to be made? 100% Tests cores are relevant and appropriate. Assist candidates in identifying and filling in gaps. Are test takers fully informed about the procedures and criteria that will be used in making decisions? 100% Before the test, candidates will be informed of the testing procedures and standards. Are these procedures and criteria actually followed in making decisions? 100% Before the test, candidates will be informed of the testing procedures and standards to follow. Impact on individuals (teachers) How consistent are the areas of language ability to be measured with those that are included in teaching materials? 100% The test is adapted based on the difficulty and content of the textbook. How consistent are the characteristics of the test and test tasks with the characteristics of teaching and learning activities? 100% Test tasks and textbook samples, with coherence and adaptability. How consistent is the purpose of the test with the values and goals of teachers and of the instructional program? 100% The purpose of the test is closely related to the instructional program. Impact on Society and Education Systems Are the interpretations we make of the test scores consistent with the values and goals of society and the education system? 100% Consistent with values and goals, helping candidates have a comprehensive understanding of their current target language level. To what extent do the values and goals of the test developer coincide or conflict with those of society and the education system? 100% The values of developers align with the values of society and the education system. What are the potential consequences, both positive and negative, for society and the education system, of using the test in this particular way? 100% The use of this test has a positive impact on society and the education system. Can help candidates measure their language proficiency. What is the most desirable positive consequence, or the best thing that could happen, as a result of using the test in this particular way, and how likely is this to happen? 100% The most positive consequence is that it can help candidates improve their target language proficiency. There is a high possibility that candidates will have a comprehensive understanding of their own level through their grades. What is the least desirable negative consequence, or the worst thing that could happen, as a result of using the test in this particular way, and how likely is this to happen? 100% The negative consequence is that candidates may not have sufficient strategies to cope with the exam, resulting in unsatisfactory grades. Practicality 41.What type and relative amounts of resources are required for: (a) the design stage, (b) the operationalization stage, (c) the administration stage? 100% The design stage requires a significant amount of time and this material. What resources will be available for carrying out (a) (b), and (c) above? 100% Materials related to testing are required. Used to ensure successful evaluation of testing.

APPENDIX 2 Coursebook Unit 5

APPENDIX 3 Original Test Speaking Original Test Items

Original Script

Reading Original Test Items

Listening

Writing

← Back to Papers