Chris Craft
Jessalyn Smith
Research Project
AEET/EDET 780
May 30, 2007
How do classroom remotes affect student achievement?
Introduction
Objective
The
primary goal of this project is to investigate the impact of classroom remote
response units to the secondary classroom. There has been recent research pertaining to other areas of
education, however none pertaining specifically to middle school. This study will compare two middle
school classrooms (one using classroom remotes) to address the question: How do classroom remotes affect student
achievement?
Review of Relevant Literature
As
the educational community becomes more and more interested in students becoming
actively engaged, student response systems are often touted as the answer. The
question must be asked as to whether the current body of research will support
the claims often made by both the companies who produce these products as well
as the resellers who peddle their wares on campuses worldwide.
Finding
studies that deal with student achievement in a non-higher education scenario
has proven challenging. The following is the only study in this review that
deals with elementary students. Jonas & Sparks (2005) found that the use of
a particular student response system – Quizdom (www.quizdom.com) –
increased student achievement by a factor of 2.8. This is a salient finding, as
it appears that simply implementing this technology provided an immediate and
visible affect on student achievement. The study used a Texas standardized test
on which to base the difference in achievement.
Upon
further examination, it is noteworthy that the pre-test score on the Texas
Assessment of Knowledge & Skills (TAKS) scores were significantly higher
for the experimental group. The experimental group reported pre-test scores
that rarely dropped below 80, whereas the control group (roughly half the size
of the experimental group as well) had scores as low as 28.5. This reveals a
significant discord between the control and experiment group and calls into
question the integrity of the results. One might argue that students who are
able to score in the 80Õs on the pre-test are pre-disposed to learning and were
able to capture the information better than the control group, irrespective of
the technology. Another concerning factor is that this study appears to have
been commissioned by Quizdom, Inc., and can only be found through a link on the
companyÕs web site.
There
has been research conducted at institutions of higher education, Edens (2007)
implemented the Classroom Performance System by eInstruction
(www.einstruction.com). There were noteworthy pedagogical analyses done in this
study as well, in that in one section, quizzes were designed to allow students
to self-assess understanding as well as inform the professor of the same. In
another section, the quizzes amounted to one-quarter of the final grade.
The
result of the study shows a change only in motivation. In the section where
quizzes counted towards the final grade, students were more motivated to
prepare for and attend class. They also reported anxiety when using the CPS,
although this anxiety reached into the other section for students with a low
level of self-regulatory behavior. ÒStudents in the self-regulation group were
more likely to value CPS as a self-assessment toolÓ (Edens 2007, p.2). The
study also notes that students whose goal was mastery were likely to succeed no
matter which section they were in. As to the direct question as to whether the
CPS system affected student achievement, the study reports Òno significant
difference between groups existed on achievementÓ (Edens 2007, p.2).
The
next study in this review dealt with a large-scale implementation of CPS
remotes in interim sections (lasting for only three weeks, sometimes called a
Maymester) of Astronomy 101 at the University of Alabama. Byrd, Coleman &
Werneth (2004) were noticing a lack of student attendance, students studying
right before exams, and overall low levels of student engagement. They note
that prior to implementing the new technology they implemented a series of
pedagogical changes in an attempt to increase student achievement. The authors
report that prior to any changes, the final exam grades averaged 57%. After
implementing a series of cooperatively answered quizzes, the final exam grade
jumped to an average of 80%. This
increase is directly related to the pedagogical change, and the authors noted
the need for greater convenience when dealing with the cooperative quizzes.
Handing papers out and collecting them, then grading them, and handing them
back was cumbersome especially given the short duration of the course. The
authors implemented CPS remotes for automated quiz taking, grading, and
reporting.
In
sections that used the CPS system for quizzes, final exam averages increased to
83%, directly linked to the CPS system due to the fact that no other
pedagogical changes were reported. Students also reported higher satisfaction
with the course, noting that the average grade given to the course by students
rose to 4.19 (presumably out of 5), up from 3.80. Students graded the
instructor equally. This study also notes that the quizzes were non-punitive
which seems to directly and positively influence reported student enjoyment of
the course.
Guthrie
& Carlin (2004) detailed the integration of CPS remotes into a college
management course, a requisite course taken by undergraduate business majors.
The two sections into which CPS was introduced were already delivering lectures
by data projector and presentation software. Since the eInstruction program
integrates with Microsoft PowerPoint¨, there was little barrier to integration.
In one section, professors interspersed questions throughout the presentation,
providing an opportunity to check for student understanding and clarify any misinformation.
The other section gave students short quizzes at the end of a lecture. The
results of this study warrant discussion.
First,
students reported feeling like they had learned more in the section in which
questions were interspersed. The difference was significant, in that 40% of
students in the interspersed section reported having learned more as a result
of the CPS implementation compared to only 8% in the section with quizzes at
the end.
Another
noteworthy finding is that 42% of students felt that the anonymous nature of
the CPS system encouraged student participation. Again the difference between
sections is notable, in that in the interspersed section 71% of students felt
that they participated more compared with 33% in the other section. In looking
at whether students would prefer a more widespread implementation spreading to
other courses, 61% of students in the interspersed course said they would
prefer more usage, compared to 15% of students in the other section.
A
poignant disclaimer to this study is that in the section where questions were
asked at the end, there were significant technical difficulties during the
first few weeks of the course. There were reception issues in receiving student
responses that clearly frustrated students. This must be taken into
consideration when comparing student responses. It is worth noting as well that
it appears that technical ability and confidence on the part of the instruction
plays a big part in student attitude towards the system as well.
In
another study, the authors implemented CPS remotes into a biology course and a
physics course at the University of Texas at Austin (Woods & Chiu, n.d.).
The authors mention that the Òmost valuable benefit for students is an increase
in interactivity and class participationÓ (Woods & Chiu, n.d. p.3). This student involvement also directly
correlates to student attendance. The result of the study seems to indicate
that when students are involved in an active, engaged classroom they are more
likely to attend class. The
authors also note that the CPS system allows instructors to effectively gauge
student understanding at regular interval and provide appropriate feedback
and/or clarification.
Another
study (Demetry, 2005) compared student personality types (as determined by the
Myers Briggs Type Indicator – MBTI) to student response in a college
classroom. While this study dealt with student response systems only minimally,
the results of a student survey reveal that students enjoy being able to see
how their peers are thinking. They also acknowledged that the use of CPS
remotes made them less likely to skip class. Student attitudes regarding the
CPS system were varied, although the numerical scores were constant. In other
words, the majority of students reported liking the system, although for
reasons that correspond with their personality type. For example, Extroverts
often have trouble with passive 50-minute lectures and report enjoying the CPS
system due to the ability to interact. The study also revealed gender differences
when dealing with preparing for and attending class, although the CPS system
seemed to provide adequate motivation for both to overcome this and succeed.
So
far the studies have dealt with an elementary school, and then undergraduate
classrooms in higher education. Now we move on to a medical school, in a study
detailing the implementation of CPS remotes in the teaching of neuroscience
(Brann & Sloop, 2006). The course in question is for students preparing for
faculty positions at medical colleges worldwide. The study only mentioned CPS
in one paragraph, but did note the effectiveness of CPS as an Òadjunct
assessment tool in the small class graduate classroom; however, it may be even
more useful in large class settingsÓ (Brann & Sloop, 2006, p.43).
One
final study deals with the problem of laptops and students paying attention in
class. Surprisingly, this is a study from the University of Cincinnati College
of Law, and the students in question are aspiring lawyers. The authors (Caron
& Gely, 2003), along with financial help from the eInstruction Corporation,
set out to redesign a law class that would be sufficiently engaging as to
prevent students from instant messaging, emailing, or other distractions. The
results of this implementation seem to run counter to the previous studies, in
that the authors note that Òstudents need to know that they will participate in
class every day and that their participation will ÔcountÕÓ. (Caron & Gely,
2003, p. 31) This need for accountability appears to be unique to the law
school. It is possible that this need for participation to ÒcountÓ may apply to
other graduate-level studies and beyond, but as of yet that is not shown in the
research.
Further
research is needed in this area, particularly in the secondary arena. There
appears to be a strong correlation between student enjoyment of class to
integration of some form of student response technology. This needs to be
explored in a climate of declining student interest in school.
Research Questions
The
current research reports do not support one clear conclusion when it comes to
the classroom remotes and a possible affect on student achievement. While some
research shows that it increases achievement and others show no increase, the
primary research question of interest for this study is: How does the use of
classroom remotes affect student achievement?
When
implementing a research study there is often one primary question. However, with the implementation of a
new technology, there are often unintended outcomes and smaller units of
interest that stem from the use of equipment (Gredler, 1995). Therefore, in addition to the primary
question, secondary research questions are:
a. Is there an impact on studentsÕ attitudes toward the use of
technology in education? Recently
the success of an educational program tends to be measure by a quantitative
numeric quantity such as a test score; however education can be more than what
is thought of as the input/output model (Gredler, 1995). Therefore, it is also important to
document qualitative findings as well—which this information will. Also, this information is important to
collect because unintended byproducts of the remote system are important to
document. b. Is attendance affected. Previous research has suggested that
studentÕs participate more when they know it counts as part of a grade—is
this the same for attendance?
Each
of the questions identified for this study are relevant to the overall impact
of classroom remotes. Instruments
and methods to measure student achievement, attitudes, and differences in
attendance are all relatively straightforward and traditional. Therefore, these questions are
appropriate and reasonable.
Methods
Setting and Participants
The
classroom remote systems will be used in a middle school Spanish class. The school is in an urban location in
South Carolina. This school hosts
only 6th grade and serves predominantly middle-class families[J1] . This particular school has over one
thousand students, has excellent performance ratings and is consistent in
meeting the goals and objectives for education. Typically Spanish class sizes range from 13-26.
Specifically
for this study these 6th grade students are enrolled in a morning
Spanish class. Students are on a
rotating schedule (A/B schedule) where they are in the class every other
day. Therefore, two separate
classes will be studied. In this
study, only one morning class will use the CPS during the class period. However, the content and material used
in both classes will be identical.
Design
A
quasi-experimental block design will be used. The blocking variable is the class. This study will last one nine weeks
period and in this case, one class will be randomly assigned the use of
CPS. In this case, the instructor
is the same and the lectures will be presented identically with the same
questions and materials.
Therefore, there will not be teacher-class or class-instruction
interactions that need to be considered.
By using random assignment and keeping conditions the same for each
class, any differences in the outcome variables can be attributed to the use of
the classroom remotes.
Materials and Instrumentation
First traditional lectures will be used for class instruction. These methods include lectures, games, and quizzes. In both sections, PowerPoint will be used to display target vocabulary, quizzes and games. To measure classroom achievement, quizzes will be used throughout the nine weeks as well as a pre and post test given to test studentsÕ content knowledge.
To
gain more information, students will be asked to complete a pre and post survey
which will be administered to assess student knowledge and perceptions
pertaining to technology and its use in the classroom. The pre-survey will be the same for
both classes and will address the general perceptions about the use of
technology in the classroom. The
post survey will be slightly different for the two classes, yet both will have
the same items addressing perceptions.
For those using CPS the questions will deal specifically with how the
system affected their learning (these items will be rated on a Likert
scale.) For those not using CPS,
the items will be similar but they will ask whether the students felt like
their learning experience would have been enhanced if they had used CPS, and
again these items will rated on a Likert scale. To ensure that student responses to survey items are honest,
they will be kept confidential and only summary information will be reported.
Finally, to compare attendance and tardiness rates between the two classes, attendance will be taken daily, using both ClassXP and a secondary paper attendance method to ensure correctness. Tardiness will also be monitored and noted accordingly.
Procedure
For
both classes the procedure will follow as:
It is important to reiterate that these procedures will be followed
for each class. What will
differ between the classes is how attendance and quizzes are taken. Students in the CPS class will take
both using the CPS device. The
students in the other class will use traditional methods. This is summarized in Table 1
Table 1: Summary of
Daily Classroom Activities and Methods.
|
Activity/Event
|
CPS Class |
Non-CPS
Class |
|
Attendance |
Use CPS to
Òclick inÓ and ClassXP |
Raise
hand/paper documentation, and ClassXP |
|
Lecture |
Note taking |
Note taking
|
|
Review Game |
Displayed
via PowerPoint¨ |
Displayed
via PowerPoint¨ |
|
Quiz |
Displayed
on screen and use CPS to answer questions |
Paper and
pencil |
The pre and post attitudinal surveys will be given to students at the beginning and end of the nine weeks period. These surveys will be given using phpESP survey software. The pre and post content knowledge tests will be taken through the class Moodle. Moodle (www.moodle.org) is an open-source alternative to the more commonly used BlackBoard¨ software. It provides for online course management to include allowing students to access digital presentations, take online quizzes, and participate in discussion forums. [J2]
Methods of Analysis
Again the purpose of this study was to determine the differences (if any) between a classroom using CPS and one not. Therefore an experimental design was used and the statistical methods used to analyze the data will need to support causal inference.
The subsets of information to be analyzed relating to the first research question pertaining to student achievement are the pre and post content knowledge tests and the daily quizzes. To compare this information the studentsÕ mean difference between the pre and post test score will be compared between the two classes. A t-test for two independent samples will be used if the conditions of sample size and normality are met. If these assumptions are not met (e.g. the data is non-normal) a nonparametric mean difference Wilcoxin test will be conducted to determine if there is any difference in achievement. The mean differences of the daily quizzes will be assessed the same way.
To investigate the difference (if any) in student attitudes and thoughts toward technology, all Likert scale items will be given a numeric codes and summarized using mean differences. Again a t-test or Wilcoxin difference test will be used to summarize the results of this section. It would also be useful to do some additional summary analysis to look for trends in the data. This information should also be summarized.
To determine the if there were any differences in the attendance or tardiness between those who used CPS and those who did not, the number of students absent and tardy will be recorded for each day and the mean differences between the two groups will be compared.
For all pieces of data the standard deviations should be reported so the variability between the two classes can be reported. Table 2 provides a summary of the data components to be analyzed and the procedures used.
Table 2: Summary of
Data Analysis Procedures.
|
Data Component |
Statistic to Compare |
Procedure |
|
Pre and Post Test Differences |
Mean |
Mean Difference Test (t-test or Wilcoxin) |
|
Quizzes |
Mean |
Mean Difference Test (t-test or Wilcoxin) |
|
Attitudinal Surveys |
Mean Values for Each Item Noting overall trends |
Mean Difference Test (t-test or Wilcoxin) N/A |
|
Attendance/Tardiness |
Counts |
Difference Test (t-test or Wilcoxin) |
|
All Components |
Standard Deviations |
NA |
By
using an experimental design and the previously discussed statistical
procedures, it will be possible to make causal inference. Therefore, if there are
statistically significant differences between the two groups, it can be
properly inferred that they are due to the CPS usage.
Educational Importance
As stated previously, the educational community is becoming more interested in students becoming actively engaged. The use of a student response system seems to be a via option for engaging students. With the introduction of these systems into the classroom, research needs to be conducted to evaluate the effectiveness of these devices, so teachers and researchers are not misled by their potential.
Since
there is relatively little research done in this area pertaining to the use
of student response systems in secondary education, it is important for
studies to be conducted. This
study should prove useful to educational practitioners by providing
information related to the effectiveness of the response systems in the
secondary setting.
References
Brann,
D. W., & Sloop, S. (2006). Curriculum development and technology
incorporation in teaching neuroscience to graduate students in a medical
school environment. Advances in Physiology Education, 30(1), 38-45.
Byrd,
G. G., Coleman, S., & Werneth, C. (2004). Exploring the universe
together: Cooperative quizzes with and without a Classroom Performance
System in Astronomy 101. Astronomy Education Review, 3(1), 26-30.
Caron,
P. L., & Gely, R. (2004). Taking back the law school classroom: Using
technology to foster active student learning. Journal of Legal
Education, 54, 551- 584.
Demetry,
C. (2005). Use of educational technology to transform the 50-minute
lecture: Is student response dependant on learning style? In American
Society for Engineering Education Annual Conference & Exposition. Retrieved May 23, 2007, from Worchester
Polytechnic Institute Web site:
http://www.wpi.edu/Academics/ATC/Collaboratory/Stories/ASEE05demetry.pdf
Edens,
K. M. (2007, February 22). Examining the Effects of Pedagogy and Student
Response Systems Technology on Achievement and Attitudes. In South
Carolina Educators for the Practical Use of Research Annual Conference. Symposium conducted at University of South
Carolina.
Gredler,
M. E. (1995). Program Evaluation.
Englewood Cliffs, NJ: Prentice Hall.
Guthrie,
R. W., & Carlin, A. (2004). Waking the dead: Using interactive
technology to engage passive listeners in the classroom. In Tenth
Americas Conference on Information Systems.
New York, New York.
Jonas,
C., & Sparks, G. P. (n.d.). Interactive engagement as a tool for
increasing student achievement.
Quizdom, Inc. Retrieved May 26, 2007, from
http://www.qwizdom.com/download/Mercy%20Street%20research%20data.pdf
Ross, Steven M. & Morrison, Gary R. (1997). Getting
Started in Instructional Technology Research, 2nd Edition. Washington, DC: Association for Educational Communications and
Technology
Woods,
H. A., & Chiu, C. (n.d.). Wireless response technology in college
classrooms. McGraw Hill Education &
eInstruction. Retrieved May 25, 2007, from
http://www.mhhe.com/cps/docs/CPSWP_WakindDead082003.pdf