Graduate Program Assessment Report

Assessment Report

Graduate Biennial Program Assessment Report
Mathematical Sciences

Graduate assessment reports are to be submitted biennially.  The report deadline is October 15th.

Biennial Graduate Assessment Process:

Every graduate program assessment must have the following key components:

1. Program Description:  

Depending on the program plan (A: Thesis; B: Professional, or C: Course Work) will define the nature of your PLO’s. Ideally plans would include assessment that would cover all plans, but that would depend on the nature of your Master’s program.

2. Program Learning Outcomes:  

PLOs are the accumulated knowledge, skills, and attitudes that students develop during a course of study in the program. Essentially, PLOs tell us what students will learn in the program. PLOs should be written as specific, measurable statements describing what students will be able to do upon completion of the program. Each PLO should contain an action verb and a learning statement. (For help in developing learning outcomes see “Program Assessment Overview”, under Resources on Provost Page:  https://www.montana.edu/provost/assessment/program_assessment.html)

3. Threshold Values:  

Along with program learning outcomes, program assessment reports should include threshold values to measure student achievement for learning outcomes.

4. Methods of Assessment:  

Every assessment report needs evidence to demonstrate student learning at the program level. This evidence can be in the form of a direct measure of student learning or an indirect measure of student learning. Both direct and indirect assessment data must be associated with the program’s learning outcomes, and collected within a timeframe determined by the program.

5. Timeframe for Collecting and Analyzing Data:  

Please provide a multi-year assessment schedule that will show when all program learning outcomes will be assessed, and by what criteria (data). Ideally, assessment data should be collected throughout the year on an annual basis. At the minimum, program faculty should schedule an annual meeting to review these data and discuss student progress toward the SLOs.

6. Use of Assessment Data:  

The assessment report should identify who received the analyzed assessment data, and how it was used by program faculty for program improvement (s).

7. Closing the Loop:  

Assessment reports should also be reflective on previous assessment and program improvements. Based on assessment from previous years, please include program level changes that have led to outcome improvements.

1. Program Description: 

Ph.D. Mathematics: This program provides graduates proficiency in mathematics and the opportunity to carry out independent research in the mathematical sciences as demonstrated by the completion of a doctoral dissertation.

Ph.D. Mathematics, Mathematics Education emphasis: This program is aimed at students with research interests focused on mathematics teaching and learning and includes the study of graduate-level mathematics.

Ph.D. Statistics: This program provides graduates proficiency in statistics and the opportunity to carry out independent research in statistics as demonstrated by the completion of a doctoral dissertation.

M.S. Data Science: This program provides graduate students with foundational training in data analysis, with equal emphasis on the principles of computer science, mathematics, and statistics, and the ability to apply these principles to a range of data-driven problems.

M.S. Mathematics: This program provides fundamental knowledge in core areas of pure and applied mathematics. It prepares graduates for careers in industry and for a PhD program in mathematics or applied mathematics.

M.S. Mathematics, Mathematics Education Option: This program provides fundamental knowledge for secondary mathematics teaching. The program deepens graduates’ understanding of school mathematics, increases their pedagogical content knowledge, and provides opportunities for personal reflection and professional growth.

M.S. Statistics: This program gives students a solid background in the theory of statistics and hands-on practice in the application of statistics to real problems. Students in this program prepare either for further graduate work or for academic, industrial, business, or government employment.

Graduate Certificate in Statistics: This certificate program provides additional education in statistical thinking and methodology over and above the basic coursework taken by the typical graduate student. This certificate provides a clear record of additional training in statistics for future graduate programs or employers.
Graduate Certificate for Dual Enrollment Mathematics Teachers: This certificate program provides a set of three courses in mathematics that provide foundational knowledge and study in algebra, calculus, and statistics.

2. Program Learning Outcomes, Assessment Schedule, and Methods of Assessment: 

Recruitment Outcomes

Recruitment strategies vary across the different PhD programs. Virtually all students in the PhD in Statistics program are recruits from the MS in Statistics program. In contrast, the PhD programs in Mathematics and Mathematics – Mathematics Education Emphasis recruit students from our own MS in Mathematics program or from other MS in Mathematics or Mathematics Education programs here in the US and globally. Some of the students in the PhD in Mathematics began the program having only completed a BS degree.

For all PhD programs, the department tracks the number of MS students who moved into the relevant PhD program. During the assessment period, one student moved from the MS Mathematics to the PhD Mathematics, one student moved from the MS Mathematics to the PhD Mathematics – Mathematics Education Emphasis, and two students moved from the MS Statistics to the PhD Statistics.

3. Threshold values for program learning outcomes (please include assessment rubrics)

The PhD programs in Mathematics and Statistics are assessed by measuring the completion rates of students who advance through the program using a sequence of Milestones. The number of students who complete a given milestone is measured. The time required for each to student to advance between successive milestones is also measured. The threshold values and data sources in Table 1 below incorporate both quantities into the assessment.

The MS programs in Mathematics, Mathematics Education and Statistics are assessed by measuring the completion rates of students who advance through their program by achieving satisfactory performance in coursework, by demonstrating core competencies on a written comprehensive exam and by demonstrating the ability to communicate knowledge relevant to the particular field of study. The nature of the program determines the structure of the assessment for these various programs, and the threshold values and data sources are described in Table 2 below.

Table 1: Threshold Values and Data Sources, Ph.D.

Table 2: Threshold Values and Data Sources, M.S.

Each graduate program is assessed according to the rubrics outlined in the tables below. Table 3 describes the rubric used for all three PhD programs within the department. The MS programs within the department are assessed according to a variety of rubric items, and those vary depending on what is appropriate to the discipline. Table 4 identifies the rubric items that are used to assess each MS program, with the distinction of programs noted in the first column of each row of the table.

Table 3: Assessment Rubric, Ph.D.

4. What Was Done

a) Was the completed assessment consistent with the plan provided? YES___x__ NO_____ If no, please explain why the plan was altered.

A summary of the enrollment and outcomes for each program are given below. The assessment outcomes are then summarized by program in tables.

Enrollment Summaries By Program

PhD Mathematics

Fourteen students were enrolled in the PhD Mathematics program during the assessment period. Two students completed the written comprehensive exam (Milestone 1) during the assessment period. Of those two students, one completed the requirement within two years of entering the program. The other student is enrolled with part-time status and required more than two years to complete the milestone. During the review period, three students completed the dissertation proposal/oral comprehensive exam (Milestone 2), and all three completed the milestone within two years of completing Milestone 1. Six students completed the dissertation defense (Milestone 3) during the review period, and all six completed the milestone within two years of achieving the previous milestone.

PhD Mathematics – Mathematics Education Option

Six students were enrolled in the PhD Mathematics – Mathematics Education program during the assessment period. Two students left the program for personal/familial reasons. Two of the 4 remaining students completed Milestone 1 within 2 years. In our review, we found that students’ ability to complete Milestone 1 varied due in part to when they enrolled in our program. Our Milestone 1 outcome is leading us to examine our course offering schedule, change our advising practices and revise the milestone so that all students can complete the written comprehensive exam within 3 years. Of the four students who completed the Oral Comprehensive Exam, all did so within 2 years of achieving Milestone 2. One student attempted, and successfully completed, a Defense of Dissertation within 2 years of achieving Milestone 2.

PhD Statistics

Sixteen students were enrolled in the PhD Statistics program during the assessment period. Four students completed the written comprehensive exam (Milestone 1) during the assessment period. All four of those students completed the requirement within two years of entering the program. During the review period, one student completed the dissertation proposal/oral comprehensive exam (Milestone 2), and also completed the milestone within two years of completing Milestone 1. Five students completed the dissertation defense (Milestone 3) during the review period, and four of those completed the milestone within two years of achieving the previous milestone.

MS Data Science

Ten students enrolled in the M.S. Data Science program during the review period. Of the 7 MS DS whose 3-year window for achieving milestones fell within the assessment period, one transferred to the MS Stat program and is meeting that program standard. Of the remaining 6 students, all completed M508 with a B or better, and 5 of 6 completed CSCI 531 and STAT 541 with a B or better in the expected time frame. Three of those ten students have graduated from the program.

MS Mathematics

A total of thirty three students were enrolled in the MS Mathematics program for some portion of the assessment period. Twenty three of those students were scheduled to complete the written comprehensive exam and coursework within the review period. Sixteen completed the requirements for a 70% completion rate. Of the seven students who didn’t complete the requirements during the review period, two are continuing and are scheduled to complete requirements during the next review period. The remaining five students left the program without completing some or all of the requirements. Two of those students transferred to other MS programs within the department, and three students left the university. One student was administratively dismissed from the program shortly after entering, and this student is not included in any of the data during the review period.

MSMME

Forty-eight students were enrolled in the Master of Science in Mathematics – Mathematics Education Emphasis during the assessment period. Of those enrolled, 19 had progressed to a point in their program where we would expect them to have achieved our program learning outcomes. Two students left our program, one to pursue a career path outside of education and another who was not able to manage online learning with the demands of teaching during the Covid-19 pandemic. A third student paused coursework in order to provide leadership in her teaching position during the pandemic. That student met our program learning outcomes in the areas of mathematics content and pedagogy. We expect the student to communicate between coursework and practice in an upcoming semester. 16/19 achieved the program learning outcomes.

MS Statistics

26 students were enrolled in the MS Statistics during this period. Of the 18 MS students whose 3-year window fell within the assessment period, 16/18 passed the required written comprehensive exam, 16/18 completed Stat 510 with B or better, and 15/18 presented their writing project. Two students left the program during the first year and the third student is on pace to present the writing project during the Fall 2022 semester, within 3 years of starting the program.

Graduate Certificate in Applied Statistics

Of the 5 students enrolled in the program within the assessment period, 3 completed the required coursework for the degree. One is continuing, and one left the university.

Graduate Certificate for Dual Enrollment Mathematics Teachers

No one enrolled in the Graduate Certificate for Dual Enrollment Mathematics Teachers. We attribute this to the certificate being new, teachers choosing not to take on additional responsibilities during the pandemic, and challenges getting financial aid for a certificate program.

b) Please provide a rubric that demonstrates how your data was evaluated.

PhD Assessment Results

Doctoral student progress is assessed using a set of consecutive milestones. The Graduate Program Coordinator maintains a data table which tracks each graduate student enrolled in one of the graduate programs. When a doctoral student begins the program, they are added to the data table, and the date by which they should achieve their first milestone is calculated. As they achieve a milestone, the date of their next milestone is calculated and added to the table. When the date of a milestone is in the 2 year assessment window, the student is counted in the denominator. If they achieved the milestone, they are counted in the numerator. Students who leave the program are counted in the denominator only for the assessment period during which they failed to achieve a milestone.

PhD Mathematics

PhD Mathematics (Education)

PhD Statistics

MS Programs

Students enrolled in one of the MS programs within the department are assessed based on their ability to satisfy the specific program requirements within the expected time window, depending on the specific nature of the program. For the MSMME and Statistics programs this is a 3-year window, and for the MS Mathematics program, the window is two years. When the date of a milestone is in the assessment window, the student is counted in the denominator. If they achieved the milestone, they are counted in the numerator. Students who leave the program are counted in the denominator only for the assessment period during which they failed to achieve a milestone. Assessment of the MS programs is essentially a reporting of the graduation rate of these programs within the appropriate timeframe.

MS Data Science

MS Mathematics

MSSME

Certificate Programs

Students enrolled in one of the Certificate programs within the department are assessed based on their ability to satisfy the specific program requirements, depending on the specific nature of the program. The requirements are focused on completion of courses and assignments in those courses. Assessment of the Certificate programs is essentially a reporting of the graduation rate of these programs within the review period.

5.  What was Learned

The graduate program committee, which has representatives of each program, reviewed and analyzed the data and summarized it in this report. The faculty reviewed and discussed the report at a faculty meeting on October 19, 2022.

a) Areas of strength

After data analysis, the faculty identified the following strengths.

1. Graduate students have access to faculty and their expertise, and our relatively small class sizes foster this academic closeness between faculty and students. This was made evident through conversations with faculty and with external reviewers during our recent program review; we suggest incorporating student interviews into our graduate assessment process to more formally collect these data.

2. Our program has supported non-traditional students. We give students who are pursuing a graduate degree part-time or self-funded a supportive environment in which they can excel. This philosophy may explain why sometimes we do not achieve the thresholds – we accept students who are non-traditional, and sometimes they realize after starting that completing a degree is not feasible given their life goals. They then leave our programs. Almost all cases of not meeting thresholds are cases of students who have left the program for non-academic reasons.

3. Our statistics programs excel at giving students opportunities to develop their statistical consulting skills. Incorporating student interviews into our graduate assessment process will help to formalize this finding.

b) Areas that need improvement

1. For graduate programs with small enrollments, the percentages reported do not give a satisfactory representation of the program, and there is no mechanism to account for students who disengage from the program for a temporary period of time and then reenroll at a later date. This issue applies to students who may be pursuing a degree with part-time enrollment.

2. There is no mechanism to track the reasons why students leave the programs.

3. For the MSDS program, we don’t have a mechanism for evaluating one of our program outcomes, as we discovered in this first iteration of assessing that program. One alternative assessment to consider for evaluating the PLO might be to implement exit interviews for graduating MSDS students.

4. Exit interviews for graduating students might be an option to provide feedback about the programs’ strengths and areas that need improvement.

6. How We Responded

a) Based on the faculty responses, will there any curricular or assessment changes (such as plans for measurable improvements, or realignment of learning outcomes)?

YES___x___ NO_______

If yes, when will these changes be implemented?

We will implement the changes to assessment before the next assessment period, and gather information each term that we will need for contacting graduates in order to conduct interviews.

b) When will the changes be next assessed?

In fall 2024.

7. Closing the loop

a. If there have been changes in program/curriculum to reflect concerns from previous assessments, what impact have the changes had (if any) on achieving the desired level of student learning outcomes?
In the previous review, there were five elements that needed improvement. Below we report how each of them was addressed in this document.

In the previous review, there were five elements that needed improvement. Below we report how each of them was addressed in this document.

1. The MS Mathematics data now assesses those enrolled in the MS thesis option by including it in the data source for the “solid understanding of core content.”

2. The MS Data Science program is assessed in this report. Since the last review as assessment strategy was developed, and the program outcomes are reported here.

3. The Graduate Certificate in Statistics is included in this report. Since the last review an assessment strategy was developed, and the program outcomes are reported here.

4. The Graduate Certificate for Dual Enrollment Mathematics Teachers is included in this report. Since the last review, we reorganized course offerings in the MSMME to support the certificate, and the program outcomes are reported here.

5. This report includes a summary of the number of MS students who move into the PhD programs within the department.

NOTE: Student names must not be included in data collection. Dialog on successful completions, manner of assessment (publications, thesis/dissertation, or qualifying exam) may be presented in table format if they apply to learning outcomes. In programs where numbers are very small and individual identification can be made, focus should be on programmatic improvements rather than student success. Data should be collected through the year on an annual basis.

PDF of Graduate Program Assessment Report