Diagnostic Portfolios

Introduction

I worked at the Seattle campus of Antioch University from 2004 to 2008.  Though I eventually got into Sakai training, my first two years as a graduate assistant (enrolled in the graduate management program) were focused primarily on student-learning outcomes-assessment.Understanding the ways schools know what their students are learning is critical to helping Antioch University in their various attempts to do online learning.

There are three ways to evaluate what a student is learning: 1) grades, 2) capstone projects, and 3) diagnostic portfolios.  Before we get into the virtues of each type of student evaluation we have a few things to consider first: A) the pressure on university systems, B) it’s the faculty conversation about student learning that matters, and C) if there are two evaluation processes, one always trumps the others.

Pressure

Universities are frequently under attack from “small-government” political organizations, who question if any given University is worth the resources it uses.  All Universities are dependent on federal money, regardless of how “private” they claim to be.  In order to prove that Universities are worth while, there are various accreditation organizations who spend their time making sure Universities either know what they are doing, or lack the “accreditation” that would count them as an officially “worth while school.” 

What accrediting organizations want to know is “how does this specific Universityknow what their students are learning.“  They want to know what the process is that leads faculty to pay attention to what their students are learning, and what conversations the faculty are having with each other about how to improve how well the over-all university system is teaching it’s students what it claims to teach those students.  Accreditation is not permanent, it usually has to be redone a few times a decade. This is not a joke: fail to express effectively how your university’s faculty interact together to improve their over-all student learning potency, and your institution is likely to fail it’s next accreditation attempt. 

The question of “how well are our students learning what we are trying to teach them” can only be answered once we can tell what individual students are learning over the course of their student career at our University.  The basic student evaluation tool (grades, capstone projects, or diagnostic portfolios) is central to faculty having conversations about how they can improve student learning. (Indeed, this IS the main reason why any of these tools exist at Antioch University.)  Some of these tools are more impressive than others to accrediting organizations:

Grades

As a college graduate and  citizen of the USA, it’s very unlikely that you have ever seen “grades done right.”  Grades-done-right as we know it today was evolved primarily in Asia and has not made the transition into the USA for the most part.  Let’s use Indonesia (based off of China’s system) for example: A grade below a “C” means the student does not get credit for taking the course.  A “C” grade means that the student mastered all of the required material for the course.  A “B” grade means that in addition to fulfilling the requirements of a C grade, the student is able to “synthesize” the material, using it effectively in projects not related directly to the course, and developing new ideas related to the course subject.  An “A” grade means that in addition to fulfilling the requirements for a C and B grade, the student is able to critically analyze the course material, pointing out it’s flaws and suggesting appropriate alternatives.  Notice that under this system, a GPA of 2.5 is a very respectable GPA that implies the student is solid and worthy of further learning opportunities.

In addition to these grades having important qualitative meaning not found in the USA system, there are a sort of “grade cops” that go around randomly sampling student assignments and course grades, making sure that the instructors are grading as required.  In the USA grades aren’t considered suspect unless a student complains, but in Indonesia it’s recognized that grades must mean something much more than “70%, 80%, 90%” to prove that students are mastering the course material in exceptional ways, and that this takes considerable effort on the part of the instructor.  The USA essentially has no grading quality enforcement what so ever.

(Imagine if we were to apply this to my 1st grade math grade: a “C” would mean “Benjamin can consistently do addition and subtraction, while a B would mean “Benjamin applies addition and subtraction to material from outside of class, comparing how large his favorite teams of superheroes are to each other from his Saturday morning cartoons,” while an A would mean “in addition to Benjamin applying his math to subjects outside of class, he has pointed out limitations of basic addition and subtraction, and has suggested alternatives involving estimations and recognizing abstract patterns instead.”  Further more, any grade I got would have a chance to be evaluated against my school work by someone witch-hunting for instructors who aren’t sticking to the established norms for the letter grade system.)

So to say the least, the way grades are done in the USA are highly suspect, and have begun to be frowned upon by various accreditation institutions.  Fortunately Antioch University, like the Evergreen State College, has never been into grades.  The two main alternatives are Capstone Projects and Diagnostic Portfolios.

Capstone Projects

A “Capstone Project” is a single piece of work that shows that a student has the abilities his degree title implies he should have.  A professional portfolio (which is not the same thing as a “diagnostic portfolio”) showcasing a student’s best work is an example of a Capstone Project.  The most common type of Capstone Project is a master’s thesis or doctoral dissertation.  Some capstone projects have multiple aspects to them, such as a professional portfolio including a required thesis.

There is one extraordinarily significant limitation to the Capstone Project approach: the student’s worthiness of having the degree is proved, but NOT that he actually learned anything from his school.  There is no evidence of where the student’s skills started off when he entered the program, so potentially he may have learned nothing at all from attending the school.  This makes capstone projects less than ideal for having conversations about student learning. 

A related problem is that this worthiness can easily be faked: graduate students at Antioch University have been known to hire “writing coaches” to help them write thesis, and some of their capstone “change projects” took place so far away that no faculty, staff, or even fellow students ever met anyone involved in the supposed change projects.  It seems possible at this point that some Antioch University Capstone Projects may have been completely fabricated all together, and in some cases we have no way to prove it one way or the other.

Diagnostic Portfolios

A “diagnostic portfolio” is a matrix, with various skills a student is to master on one axis, and depth of skill on the other axis, with evidences of student learning for each skill and depth listed in each empty square of the matrix:

Diagnostic Portfolio Example	Depth 1	Depth 2	Depth 3	Depth 4
Skill A	evidence of skill A, depth 1	evidence of skill A, depth 2	evidence of skill A, depth 3	evidence of skill A, depth 4
Skill B	evidence of skill B, depth 1	evidence of skill B, depth 2	evidence of skill B, depth 3	evidence of skill B, depth 4
Skill C	evidence of skill C, depth 1	evidence of skill C, depth 2	evidence of skill C, depth 3	evidence of skill C, depth 4

The Diagnostic Portfolio intentionally captures the student’s flaws and documents them, especially early on in the student’s skill depths, so that there is real evidence of skill development. Only the final depth in the Diagnostic Portfolio would be expected to be “perfect” work, and probably only the last few skill depths would contain material a student would include in a “professional portfolio” after they graduated.  This allows students to organize their work (and instructor feedback on that work) in a way that shows what they learned over the course of their studies. This is a meaningful way to evaluate students as far as “proof of student learning” is concerned, and is the hip way to do student evaluation in today’s USA. This approach was pioneered and developed first and foremost by Alverno College, over the last three decades.  For most of this time the diagnostic portfolio was kept in paper form, but they eventually developed their own web-based tool around the same time Antioch University Seattle imitated this system for their PsyD program. (I know because I was the one who imitated it for the original prototype of the PsyD ePortfolio, at the suggestion of various faculty members.)  Alverno eventually joined forces with the OSP, which has joined forces with Sakai.  Now Antioch is using Sakai.

There Can Be Only One

The key to understanding how grades, capstone projects, and diagnostic portfolios work together, is to realize simply that they do not work together: one always trumps the others.  For example a Law program may have grades in it’s classes, but the capstone project of passing the bar exam would completely trump grades in the eyes of the students, if passing the bar was required for graduation.  In my case, in my undergraduate Human Services program at Western Washington University had until then recently been a non-graded program with a capstone-style portfolio, but had attempted to convert to an Indonesian-style grading system instead (with the requirements for A & B reversed and absolutely no “grade cops.”)  The portfolio was still there, and we were encouraged to hold on to our course syllabus and key assignments and keep a binder showing our student learning, but in the face of the grades, these portfolios meant absolutely nothing to us students. In the end the portfolios had less than a one letter grade influence on only one class in our program, in the final quarter.  Regardless of our best intentions, when it comes to student evaluation tools, there can be only one master tool in the end, because one tool will eventually be seen as a means to achieve the other.

For techs working for Antioch University, it should be very good news when a program asks for a matrix-based diagnostic “ePortfolio in Sakai,” because it means that Antioch University’s accreditation has been made one step more certain in the future (and thus the funding behind the tech’s long-term career-prospects one step more stable.)  In my humble opinion, the future of adult education generally is going to be the diagnostic portfolio: I think pursuing evidences for the diagnostic portfolio will eventually completely replace having quarters and semesters, with every class being a permanently opened shop for students to pursue specific evidences in.  Semesters, school years and quarters may melt before the might that is the future of the diagnostic portfolio: behold it’s glory and wonder!