Expanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEM

Project No.
1323162
PI Name
Mark Urban-Lurain
Institution
Michigan State University
Target Discipline


IUSE-EHR/TUES/CCLI
WIDER

Abstract 1

Expanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEM

Presentation Type
Paper
Team
Mark Urban-Lurain, Michigan State University, Overall Project PI John Merrill, Michigan State University, PI Melanie Cooper, Michigan State University, co-PI Carl Lira, Michigan State University, co-PI Kevin Haudek, Michigan State University, co-PI Andrea Bierema, Michigan State University, Post-doctoral Researcher Anne-Marie Hoskinson, Michigan State University, Post-doctoral Researcher Rosa Moscarella, Michigan State University, Post-doctoral Researcher Matthew Steele, Michigan State University, Post-doctoral Researcher Alexandria Mazur, Michigan State University, Undergraduate Researcher Paula Lemons, University of Georgia, PI Jennifer Kaplan, University of Georgia, PI Mark Farmer, University of Georgia, co-PI Tessa Andrews, University of Georgia, Senior Personnel Jill McCourt, University of Georgia, Post-doctoral Researcher Kyle Jennings, University of Georgia, Graduate Researcher Treメcherie Crumbs, University of Georgia, Undergraduate Researcher Alex Lyford, University of Georgia, Undergraduate Researcher Luanna Prevost, University of South Florida, PI Kelli Hayes, University of South Florida, Graduate Researcher Michelle Smith, University of Maine, PI Karen Pelletreau, University of Maine, Post-doctoral Researcher Scott Merrill, University of Maine, Undergraduate Researcher Jenny Knight, University of Colorado, Boulder, PI Jeremy Rentsch, University of Colorado, Boulder, Post-doctoral Researcher Ross Nehm, SUNY-Stony Brook, PI Minsu Ha, SUNY-Stony Brook, Post-doctoral Researcher Mary Anne Sydlik, Western Michigan University, PI ヨ Project Evaluation Eva Ngulo, Western Michigan University, Graduate Researcher


Need

Faculty who wish to respond to, and build upon, students' existing understandings of key STEM concepts must first know what and how students think about the concepts. While multiple-choice assessments are easy to administer, they cannot measure studentsメ abilities to assemble individual bits of knowledge into a coherent and functional explanatory structure. Writing is an authentic task that can reveal student thinking, but is time-consuming to evaluate and therefore difficult to implement in large classes that are typical of many introductory STEM courses. The Automated Analysis of Constructed Response (AACR, pronounced モacerヤ) project combines educational research-based methods with computerized linguistic analysis technology to quickly evaluate student writing, generating useful and timely feedback for faculty to inform their instruction.

Goals

We are a large, multi-institutional collaboration (TUES 3 and WIDER funding) that has several connected goals: 1) create a community web portal for automated analysis of AACR assessments to expand and deepen collaborations between STEM education researchers and instructors; 2) transport AACR innovations through ongoing faculty professional development; 3) expand our question disciplinary research from biology to chemistry, chemical engineering, and statistics; 4) engage in ongoing project evaluation for continuous quality improvement; and 5) lay the foundation for sustainability.

Approach

We use a variety of computerized lexical analysis tools to create statistical models that predict expert rating of student writing with inter-rater reliability as good as expert-to-expert IRR (>.8). These models are used to generate reports for faculty that detail the correct and incorrect concepts in their studentsメ responses. Local Faculty Learning Communities (FLCs) meet to discuss the reports and create instructional interventions to improve student outcomes. We are developing a web portal that will allow any faculty to obtain questions and upload their studentsメ responses for analysis.

Outcomes

Analysis of student writing has led to new insights into how students struggle with key concepts, such as the Central Dogma of Biology, with FLC faculty collectively creating new instruction to address these challenges. Research on the FLCs show faculty are moving from asking �how many students got the right answer� to reflecting on student thinking and modifying instruction to address common learning challenges. We continue to explore a variety of lexical analysis and classification techniques to speed up and improve the development of questions and analytic resources. In the coming year, we are working on the web portal to make the analyses widely available.

Broader Impacts

The project has created a set of FLCs across multiple institutions that engage STEM faculty teaching foundational courses to administer AACR questions, reflect on the results and implement revised instruction. In this current year we have added additional faculty to each FLC. We are expanding questions development beyond biology into chemistry, statistics, thermodynamics and physics. We have presented at multiple conferences, published several papers and have created two web sites to disseminate our results.

This paper will present an overview of the AACR project and highlight some of our key findings.

Unexpected Challenges

Each institution has a PI for each grant plus co-PIs, post-doctoral researchers, graduate and undergraduate student researchers. There are three major working groups: Question Development Cycle (QDC), Faculty Learning Community / Professional Development (FLC-PD); Web Portal development; an external evaluation team, an external advisory board, and faculty participating in Faculty Learning Communities at each institution. Coordinating communication across these subgroups while maintaining engagement is challenging. The subgroups have weekly meetings as does a project management team. Feedback from the external evaluator is crucial for keeping the project on track.

Citations

Journals

ユ Kaplan, J. J. and Haudek, K. C. and Ha, M. and Rogness, N. and Fisher, D. (2014). Using Lexical Analysis Software to Assess Student Writing in Statistics. Technology Innovations in Statistics Education. 8 (1), http://escholarship.org/uc/item/57r90703

ユ Urban-Lurain, Mark and Cooper, Melanie M. and Haudek, Kevin C. and Kaplan, Jennifer J. and Knight, J. K. and Lemons, Paula P. and Lira, Carl T. and Merrill, John E. and Nehm, Ross H. and Prevost, Luanna B. and Smith, Michelle K. and Sydlik, Maryanne (2015). Expanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEM. Computers in Education Journal. 6 (1), 65-81.

ユ Weston, Michele and Prevost, L. B. and Haudek, K.C. and Merrill, J. and Urban-Lurain, M. (2014). Examining the Impact of Question Surface Features on Students' Answers to Constructed Response Questions on Photosynthesis. CBE Life Sci Educ. 14 (2). DOI: 10.1187/cbe.14-07-0110

ユ Moharreri, Kayhan and Ha, Minsu and Nehm, Ross H (2014). EvoGrader: an online formative assessment tool for automatically evaluating written evolutionary explanations. Evolution: Education and Outreach. 7 (1), 1-14. DOI: 10.1186/s12052-014-0015-2

Conference Papers and Presentations

ユ Ha, Minsu and Nehm, Ross H. (2015). Assessment item 'Cover Stories', semantic similarity and successful computerized scoring of open-ended text. NARST.

ユ Urban-Lurain, Mark and Merrill, John and Haudek, Kevin and Nehm, Ross and Moscarella, Rosa and Steele, Matthew and Park, Mihwa (2015). Automated analysis of constructed responses: What are we modeling?. Society for the Advancement of Biology Education Research.

ユ Park, Mihwa and Haudek, Kevin and Urban-Lurain, Mark (2015). Computerized lexical analysis of students? written responses for diagnosing conceptual understanding of energy. NARST.

ユ Haudek, Kevin C. and Weston, Michele M. and Moscarella, Rosa and Merrill, John and Urban-Lurain, Mark (2015). Construction of rubrics to evaluate content in students' scientific explanation using computerized text analysis. NARST.

ユ Weston, Michele and Haudek, Kevin C. and Prevost, Luanna B. and Merrill, John E. and Urban-Lurain, Mark (2014). Examining the impact of question surface features on students? answers to constructed response questions in biology. CREATE4STEM Mini Conference. East Lansing, MI.

ユ Ha, Minsu and Nehm, Ross H. (2015). Exploring students' evolutionary explanations across natural, sexual, and artificial selection secenarios. NARST.

ユ Chen, Jianfu and Ha, Minsu and Nehm, Ross H. (2015). Measuring semantic similarity in written text: Applications to learning and assessment. NARST.

ユ Moscarella, Rosa and Mazur, Alexandria and Pelletreau, Karen and Smith, Michelle and Urban-Lurain, Mark (2015). The Central Dogma of molecular biology: Investigating student challenges understanding transcription. Society for the Advancement of Biology Education Research.

ユ McCourt, Jill and Andrews, Tessa and Crumbs, Tre'cherie and Knight, Jennifer and Merrill, John and Merrill, Scott and Nehm, Ross and Pelletreau, Karen and Prevost, Luanna and Smith, Michelle and Urban-Lurain, Mark and Lemons, Paula (2015). Using faculty learning communities to promote the development of student-centered biology instructors. Society for the Advancement of Biology Education Research.

ユ Urban-Lurain, M. and Merrill, J. and Cooper, Melanie M and Lira, Carl and Haudek, Kevin C (2014). AACR III: Expanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEM. MSU CREATE4STEM mini-conference. Presented at MSU CREATE4STEM Mini-conference.

ユ Smith, Michelle K. (2013). Addressing Student Conceptual Difficulties in Undergraduate Genetics Courses. Invited talk given at Emory University. Invited talk given at Emory University; Atlanta.

ユ Prevost, L.B. (2014). Assessing students'ability to trace matter and energy using lexical analysis of written assessments. Society for Advancement of Biology Education Research annual meeting.

ユ Corley, Leah M. and Olwine, Jana and Cooper, Melanie M and Haudek, Kevin C. and Urban-Lurain, M. (2014). Automated analysis of students' constructed explanations in chemistry. 2014 Biennial Conference on Chemical Education.

ユ Prevost, L. B. and Haudek, Kevin C and Cooper, Melanie M and Urban-Lurain, M. (2014). Computerized Lexical Analysis of Students' Written Interpretations of Chemical Representations. NARST annual conference.

ユ Kaplan, J. J. and Gabrosek, John and Curtiss, Phyllis and Malone, Christopher J. (2014). Everyone Knows What a Histogram Is, or Do They?: How Non-Statisticians Read Histograms. Joint Statistical Meetings.

ユ Ha, M. and Ponnuraj, G. T. and Nehm, Ross H. (2014). EvoGrader: An Online Formative Assessment Tool for Automatically Analyzing Students' Ideas in Written Evolutionary Explanations. Society for Advancement of Biology Education Research annual meeting. .

ユ Weston, Michele and Haudek, Kevin C and Prevost, Luanna and Merrill, J. and Urban-Lurain, M. (2014). Examining the Impact of Question Surface Features on Students' Answers to Constructed Response Questions in Biology. MSU CREATE4STEM mini-conference. Poster presented at the MSU CREATE4STEM mini-conference.

ユ Weston, Michele and Haudek, Kevin C and Prevost, Luanna and Merrill, J. and Urban-Lurain, M. (2014). Examining the Impact of Question Surface Features on Students' Answers to Constructed Response Questions in Biology. MSU University Undergraduate Research and Art Forum. Poster presented at the MSU UURAF.

ユ Urban-Lurain, M. and Cooper, Melanie M. and Haudek, Kevin C and Kaplan, J. J. and Knight, J. K. and Lemons, Paula P. and Lira, Carl T. and Merrill, J. E. and Nehm, R. and Prevost, L. B. and Smith, Michelle K. and Sydlik, Maryanne (2014). Expanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEM. ASEE Annual Conference.

ユ Prevost, L.B. (2014). Exploring students' mental models of matter and energy transformation through lexical analysis of written assessments. Ecological Society of America annual meeting.

ユ Lira, Carl T. and Elliott, J. R. (2013). Facilitating Learning in Thermodynamics and Computations Using Technology. Annual Meeting of the American Institute of Chemical Engineers.

ユ Smith, Michelle K. (2013). Improving Student Learning Through Assessment and Observation. Invited talk given at Yale University. Invited talk given at Yale University; New Haven.

ユ Smith, Michelle K. (2013). Institutional Change in STEM Education: Using Content Assessment and Classroom Observations to Guide the Process. Invited talk given at University of Georgia. Invited talk given at University of Georgia; Athens.

ユ Voreis, Jill S. and Andrews, Tessa M. and Federer, Meghan R. and Knight, Jennifer K and Merrill, J. E. and Merrill, Scott and Nehm, Ross H and Prevost, L. B. and Smith, Michelle K. and Urban-Lurain, M. and Lemons, Paula P. (2014). Investigating the Impact of Faculty Learning Communities on Biology Instructors. Society for Advancement of Biology Education Research annual meeting.

ユ Adkins, T. and Voreis, Jill S. and Lemons, Paula P. (2014). Investigating the impact of faculty learning communities on biology instructors. REU Undergraduate Biology Education Research Program.

ユ Smith, Michelle K. (2014). Show me the data! How the process of science can help you teach and learn biology. Invited talk given at Salisbury University. Invited talk given at Salisbury University, Salisb.

ユ Smith, Michelle K. (2014). Using Assessment and Observation to Improve Student Learning. Invited talk given at York University. Invited talk given at York University; Toronto, ON.

ユ Smith, Michelle K. (2013). Using Student Conceptual Difficulties to Inform Teaching. Invited talk given at University of Maine. Invited talk given at University of Maine; Orono,

ユ Urban-Lurain, Mark and Prevost, Luanna B and Haudek, Kevin C and Norton-Henry, Emily and Berry, Matthew C. and Merrill, John E. (2013). Using computerized lexical analysis of student writing to support just-in-time teaching in large enrollment STEM courses. Frontiers in Education.



Project Page

Project Document


Additional Abstract:

Modeling Student Thinking in STEM: Insights from the Automated Analysis of Constructed Response (AACR) Project

Team
Mark Urban-Lurain, Michigan State University, Overall Project PI John Merrill, Michigan State University, PI Melanie Cooper, Michigan State University, co-PI Carl Lira, Michigan State University, co-PI Kevin Haudek, Michigan State University, co-PI Andrea Bierema, Michigan State University, Post-doctoral Researcher Anne-Marie Hoskinson, Michigan State University, Post-doctoral Researcher Rosa Moscarella, Michigan State University, Post-doctoral Researcher Matthew Steele, Michigan State University, Post-doctoral Researcher Alexandria Mazur, Michigan State University, Undergraduate Researcher Paula Lemons, University of Georgia, PI Jennifer Kaplan, University of Georgia, PI Mark Farmer, University of Georgia, co-PI Tessa Andrews, University of Georgia, Senior Personnel Jill McCourt, University of Georgia, Post-doctoral Researcher Kyle Jennings, University of Georgia, Graduate Researcher Treメcherie Crumbs, University of Georgia, Undergraduate Researcher Alex Lyford, University of Georgia, Undergraduate Researcher Luanna Prevost, University of South Florida, PI Kelli Hayes, University of South Florida, Graduate Researcher Michelle Smith, University of Maine, PI Karen Pelletreau, University of Maine, Post-doctoral Researcher Scott Merrill, University of Maine, Undergraduate Researcher Jenny Knight, University of Colorado, Boulder, PI Jeremy Rentsch, University of Colorado, Boulder, Post-doctoral Researcher Ross Nehm, SUNY-Stony Brook, PI Minsu Ha, SUNY-Stony Brook, Post-doctoral Researcher Mary Anne Sydlik, Western Michigan University, PI ヨ Project Evaluation Eva Ngulo, Western Michigan University, Graduate Researcher
Need

Faculty who wish to respond to, and build upon, students' existing understandings of key STEM concepts must first know what and how students think about the concepts. While multiple-choice assessments are easy to administer, they cannot measure studentsメ abilities to assemble individual bits of knowledge into a coherent and functional explanatory structure. Writing is an authentic task that can reveal student thinking, but is time-consuming to evaluate and therefore difficult to implement in large classes that are typical of many introductory STEM courses. The Automated Analysis of Constructed Response (AACR, pronounced モacerヤ) project combines educational research-based methods with computerized linguistic analysis technology to quickly evaluate student writing, generating useful and timely feedback for faculty to inform their instruction.

Goals

Throughout this project, we have explored a number of ways to represent student conceptions - and misconceptions - about key STEM ideas. We find that faculty are enthusiastic about the opportunity to have their students write in large introductory courses, but many are skeptical about how computers can analyze writing. One of our project goals is to find ways to represent student thinking that are meaningful to faculty and provide actionable information for their instruction.

Approach

We use a variety of computerized lexical analysis tools to create statistical models that predict expert rating of student writing with inter-rater reliability as good as expert-to-expert IRR (>.8). These models are used to generate reports for faculty that detail the correct and incorrect concepts in their studentsメ responses. Our research results align with scale-free, small world graph semantic network frameworks (Steyvers & Tenenbaum, 2005) and provide support for モknowledge-in-piecesヤ theories of conceptual change (diSessa, 2008).

Outcomes

In this poster, we outline the conceptual and theoretical basis for the AACR work, describe the process by which we create the models, and discuss the linguistic and cognitive structures that we are modeling. By opening the モblack boxヤ of this process, we anticipate that faculty will be more confident in the information that AACR can provide and will better understand the cognitive processes by which learners move from novice towards expertise in a discipline. We will provide examples from student writing in introductory biology courses about key student struggles related to Central Dogma and some of the learning challenges that result from the extensive vocabulary that is part of introductory biology.

Broader Impacts

We are using a number of computerized text analysis approaches that have been developed for business and web analytics. By re-purposing these tools for education research, we anticipate being able to leverage the power of these tools without having to reinvent the analytic techniques. We are expanding questions development beyond biology into chemistry, statistics, thermodynamics and physics. We have presented at multiple conferences, published several papers and have created two web sites to disseminate our results.