A Community of Enhanced Assessment Facilitates Reformed Teaching

Project No.
1347733
PI Name
Paula Lemons
Institution
University of Georgia


WIDER

Abstract 1

A Community of Enhanced Assessment Facilitates Reformed Teaching

Presentation Type
Paper
Team
Mark Urban-Lurain, Michigan State University, Overall Project PI John Merrill, Michigan State University, PI Melanie Cooper, Michigan State University, co-PI Carl Lira, Michigan State University, co-PI Kevin Haudek, Michigan State University, co-PI Andrea Bierema, Michigan State University, Post-doctoral Researcher Anne-Marie Hoskinson, Michigan State University, Post-doctoral Researcher Rosa Moscarella, Michigan State University, Post-doctoral Researcher Matthew Steele, Michigan State University, Post-doctoral Researcher Alexandria Mazur, Michigan State University, Undergraduate Researcher Paula Lemons, University of Georgia, PI Jennifer Kaplan, University of Georgia, PI Mark Farmer, University of Georgia, co-PI Allen Moore, University of Georgia, co-PI Tessa Andrews, University of Georgia, Senior Personnel Jill McCourt, University of Georgia, Post-doctoral Researcher Kyle Jennings, University of Georgia, Graduate Researcher Treメcherie Crumbs, University of Georgia, Undergraduate Researcher Ersta Ferryanto, University of Georgia, Undergraduate Researcher Alex Lyford, University of Georgia, Undergraduate Researcher Luanna Prevost, University of South Florida, PI Kelli Carter, University of South Florida, Graduate Researcher Michelle Smith, University of Maine, PI Karen Pelletreau, University of Maine, Post-doctoral Researcher Scott Merrill, University of Maine, Undergraduate Researcher Jenny Knight, University of Colorado, Boulder, PI Jeremy Rentsch, University of Colorado, Boulder, Post-doctoral Researcher Ross Nehm, SUNY-Stony Brook, PI Minsu Ha, SUNY-Stony Brook, Post-doctoral Researcher Mary Anne Sydlik, Western Michigan University, PI ヨ Project Evaluation Eva Ngulo, Western Michigan University, Graduate Researcher


Need

Changing teaching strategies remains a challenge for faculty; faculty need long-term support with the opportunity for feedback and reflection. Our project supports college biology faculty who are changing their teaching by participating in the Automated Analysis of Constructed Response (AACR) project. AACR offers constructed response questions about biological concepts that can be analyzed by computers to generate reports about student thinking. Thirty-one biology faculty at six research institutions participate in Faculty Learning Communities (FLCs) engaged in teaching reform. The project also benefits the STEM higher education community because it generates new knowledge about the impact of teaching professional development.

Goals

Project goals include: (1) identifying ways to support faculty to use AACR questions and reports, (2) identifying essential components of FLCs designed to facilitate AACR use, (3) describing the progression of teaching practices and conceptions while faculty are engaged with AACR. The following activities are meant to accomplish these goals: faculty (1) meet in local FLCs three times per semester, (2) participate in cross-institutional curriculum development teams, (3) ask AACR questions and receive reports, and (4) the research team collects data from participants, including classroom observations, surveys, and interviews.

Approach

Our FLCs, align with evidence for promoting teaching reform in university STEM. Our research is grounded in the innovation-decision model, which describes the adoption of innovations, like new teaching practices. Our overarching research question is: does participation in an FLC provide the impetus for faculty to change their teaching and sustain those changes? We use a mixed methods approach to investigate this question.

Outcomes

First, all 19 faculty who started the project have remained for two years and we added 12 new faculty this year. We hypothesize persistence is high because the FLC offers a low-cost, high-utility environment for participants. Second, we have identified FLCs as a successful strategy for promoting adoption of AACR use and for priming faculty to use AACR reports to answer questions about student learning or to modify their teaching. Facilitation of FLCs by experienced education researchers is essential. Third, we have learned that participants currently can be categorized into four quadrants in terms of their teaching practices and conceptions: (1) student-centered conceptions/practices, (2) student-centered conceptions/teacher-centered practices, (3) teacher-centered conceptions/student-centered practices, and (4) teacher-centered conceptions/practices. We hypothesize that in order to continue to improve as teachers, participants in each quadrant need unique types of support, which potentially can be offered in the FLC context.

Key deliverables include: (1) two manuscripts in preparation about the persistence of faculty in FLCs and the teaching practices and conceptions of participants, (2) an in-class activity, developed by participants, to address misconceptions about stop codons in molecular biology.

Broader Impacts

(1) faculty in diverse academic positions and varying career stages are being supported to adopt more reformed pedagogies, (2) departments in which these faculty work are being changed because of the new opportunity to review data on student thinking, (3) researchers at many stages of development and from diverse demographics are actively engaged in achieving project goals.

Unexpected Challenges

Each institution has a PI for each grant plus co-PIs, post-doctoral researchers, graduate and undergraduate student researchers. There are three major working groups: Question Development Cycle (QDC), Faculty Learning Community / Professional Development (FLC-PD); Web Portal development; an external evaluation team, an external advisory board, and faculty participating in Faculty Learning Communities at each institution. Coordinating communication across these subgroups while maintaining engagement is challenging. The subgroups have weekly meetings as does a project management team. Feedback from the external evaluator is crucial for keeping the project on track. 5. If you have publications related to this project, please list the full citation(s).

Citations

Journals
ユ Kaplan, J. J. and Haudek, K. C. and Ha, M. and Rogness, N. and Fisher, D. (2014). Using Lexical Analysis Software to Assess Student Writing in Statistics. Technology Innovations in Statistics Education. 8 (1), http://escholarship.org/uc/item/57r90703
ユ Urban-Lurain, Mark and Cooper, Melanie M. and Haudek, Kevin C. and Kaplan, Jennifer J. and Knight, J. K. and Lemons, Paula P. and Lira, Carl T. and Merrill, John E. and Nehm, Ross H. and Prevost, Luanna B. and Smith, Michelle K. and Sydlik, Maryanne (2015). Expanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEM. Computers in Education Journal. 6 (1), 65-81.
ユ Weston, Michele and Prevost, L. B. and Haudek, K.C. and Merrill, J. and Urban-Lurain, M. (2014). Examining the Impact of Question Surface Features on Students' Answers to Constructed Response Questions on Photosynthesis. CBE Life Sci Educ. 14 (2). DOI: 10.1187/cbe.14-07-0110
ユ Moharreri, Kayhan and Ha, Minsu and Nehm, Ross H (2014). EvoGrader: an online formative assessment tool for automatically evaluating written evolutionary explanations. Evolution: Education and Outreach. 7 (1), 1-14. DOI: 10.1186/s12052-014-0015-2
Conference Papers and Presentations
ユ Ha, Minsu and Nehm, Ross H. (2015). Assessment item 'Cover Stories', semantic similarity and successful computerized scoring of open-ended text. NARST.
ユ Urban-Lurain, Mark and Merrill, John and Haudek, Kevin and Nehm, Ross and Moscarella, Rosa and Steele, Matthew and Park, Mihwa (2015). Automated analysis of constructed responses: What are we modeling?. Society for the Advancement of Biology Education Research.
ユ Park, Mihwa and Haudek, Kevin and Urban-Lurain, Mark (2015). Computerized lexical analysis of students? written responses for diagnosing conceptual understanding of energy. NARST.
ユ Haudek, Kevin C. and Weston, Michele M. and Moscarella, Rosa and Merrill, John and Urban-Lurain, Mark (2015). Construction of rubrics to evaluate content in students' scientific explanation using computerized text analysis. NARST.
ユ Weston, Michele and Haudek, Kevin C. and Prevost, Luanna B. and Merrill, John E. and Urban-Lurain, Mark (2014). Examining the impact of question surface features on students? answers to constructed response questions in biology. CREATE4STEM Mini Conference. East Lansing, MI.
ユ Ha, Minsu and Nehm, Ross H. (2015). Exploring students' evolutionary explanations across natural, sexual, and artificial selection secenarios. NARST.
ユ Chen, Jianfu and Ha, Minsu and Nehm, Ross H. (2015). Measuring semantic similarity in written text: Applications to learning and assessment. NARST.
ユ Moscarella, Rosa and Mazur, Alexandria and Pelletreau, Karen and Smith, Michelle and Urban-Lurain, Mark (2015). The Central Dogma of molecular biology: Investigating student challenges understanding transcription. Society for the Advancement of Biology Education Research.
ユ McCourt, Jill and Andrews, Tessa and Crumbs, Tre'cherie and Knight, Jennifer and Merrill, John and Merrill, Scott and Nehm, Ross and Pelletreau, Karen and Prevost, Luanna and Smith, Michelle and Urban-Lurain, Mark and Lemons, Paula (2015). Using faculty learning communities to promote the development of student-centered biology instructors. Society for the Advancement of Biology Education Research.
ユ Urban-Lurain, M. and Merrill, J. and Cooper, Melanie M and Lira, Carl and Haudek, Kevin C (2014). AACR III: Expanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEM. MSU CREATE4STEM mini-conference. Presented at MSU CREATE4STEM Mini-conference.
ユ Smith, Michelle K. (2013). Addressing Student Conceptual Difficulties in Undergraduate Genetics Courses. Invited talk given at Emory University. Invited talk given at Emory University; Atlanta.
ユ Prevost, L.B. (2014). Assessing students'ability to trace matter and energy using lexical analysis of written assessments. Society for Advancement of Biology Education Research annual meeting.
ユ Corley, Leah M. and Olwine, Jana and Cooper, Melanie M and Haudek, Kevin C. and Urban-Lurain, M. (2014). Automated analysis of students' constructed explanations in chemistry. 2014 Biennial Conference on Chemical Education.
ユ Prevost, L. B. and Haudek, Kevin C and Cooper, Melanie M and Urban-Lurain, M. (2014). Computerized Lexical Analysis of Students' Written Interpretations of Chemical Representations. NARST annual conference.
ユ Kaplan, J. J. and Gabrosek, John and Curtiss, Phyllis and Malone, Christopher J. (2014). Everyone Knows What a Histogram Is, or Do They?: How Non-Statisticians Read Histograms. Joint Statistical Meetings.
ユ Ha, M. and Ponnuraj, G. T. and Nehm, Ross H. (2014). EvoGrader: An Online Formative Assessment Tool for Automatically Analyzing Students' Ideas in Written Evolutionary Explanations. Society for Advancement of Biology Education Research annual meeting. .
ユ Weston, Michele and Haudek, Kevin C and Prevost, Luanna and Merrill, J. and Urban-Lurain, M. (2014). Examining the Impact of Question Surface Features on Students' Answers to Constructed Response Questions in Biology. MSU CREATE4STEM mini-conference. Poster presented at the MSU CREATE4STEM mini-conference.
ユ Weston, Michele and Haudek, Kevin C and Prevost, Luanna and Merrill, J. and Urban-Lurain, M. (2014). Examining the Impact of Question Surface Features on Students' Answers to Constructed Response Questions in Biology. MSU University Undergraduate Research and Art Forum. Poster presented at the MSU UURAF.
ユ Urban-Lurain, M. and Cooper, Melanie M. and Haudek, Kevin C and Kaplan, J. J. and Knight, J. K. and Lemons, Paula P. and Lira, Carl T. and Merrill, J. E. and Nehm, R. and Prevost, L. B. and Smith, Michelle K. and Sydlik, Maryanne (2014). Expanding a National Network for Automated Analysis of Constructed Response Assessments to Reveal Student Thinking in STEM. ASEE Annual Conference.
ユ Prevost, L.B. (2014). Exploring students' mental models of matter and energy transformation through lexical analysis of written assessments. Ecological Society of America annual meeting.
ユ Lira, Carl T. and Elliott, J. R. (2013). Facilitating Learning in Thermodynamics and Computations Using Technology. Annual Meeting of the American Institute of Chemical Engineers.
ユ Smith, Michelle K. (2013). Improving Student Learning Through Assessment and Observation. Invited talk given at Yale University. Invited talk given at Yale University; New Haven.
ユ Smith, Michelle K. (2013). Institutional Change in STEM Education: Using Content Assessment and Classroom Observations to Guide the Process. Invited talk given at University of Georgia. Invited talk given at University of Georgia; Athens.
ユ Voreis, Jill S. and Andrews, Tessa M. and Federer, Meghan R. and Knight, Jennifer K and Merrill, J. E. and Merrill, Scott and Nehm, Ross H and Prevost, L. B. and Smith, Michelle K. and Urban-Lurain, M. and Lemons, Paula P. (2014). Investigating the Impact of Faculty Learning Communities on Biology Instructors. Society for Advancement of Biology Education Research annual meeting.
ユ Adkins, T. and Voreis, Jill S. and Lemons, Paula P. (2014). Investigating the impact of faculty learning communities on biology instructors. REU Undergraduate Biology Education Research Program.
ユ Smith, Michelle K. (2014). Show me the data! How the process of science can help you teach and learn biology. Invited talk given at Salisbury University. Invited talk given at Salisbury University, Salisb.
ユ Smith, Michelle K. (2014). Using Assessment and Observation to Improve Student Learning. Invited talk given at York University. Invited talk given at York University; Toronto, ON.
ユ Smith, Michelle K. (2013). Using Student Conceptual Difficulties to Inform Teaching. Invited talk given at University of Maine. Invited talk given at University of Maine; Orono,
ユ Urban-Lurain, Mark and Prevost, Luanna B and Haudek, Kevin C and Norton-Henry, Emily and Berry, Matthew C. and Merrill, John E. (2013). Using computerized lexical analysis of student writing to support just-in-time teaching in large enrollment STEM courses. Frontiers in Education.



Project Page

Project Document