Arnold Kling  

A Means A, a bit more fleshed out

PRINT
Manufacturing and Reality... A Stanford Course Where A Mean...

Pasted in below the fold. Comments welcome.

A Means A
A Reliable Independent Grading Service for College Level Course Work

Business Proposal
Version as of August 5, 2011
Arnold Kling

Summary

A Means A solves the problem of credibility and comparability of grades in courses taught at different institutions of higher education. The innovation is to separate the grading process from other aspects of higher education. For any college-level course, A Means A will devise an appropriate exam and use independent professionals to grade the exam, according to transparent, standard criteria.

Introduction

Today, what does a good grade mean? Did the student really attain the course objectives? Or was the professor a pushover? Or were the institutional standards low? We envision a grading system that is more robust, so that everyone who reads a transcript knows what the grades mean. A Means A. The potential benefits of this are discussed in the section below called Business Opportunity.

One existing service that illustrates the level of credibility and standardization that we seek is the Advanced Placement testing undertaken by the College Board. AP tests include both multiple-choice sections, which are graded by machines, and free-response sections, which are graded by trained teachers under clear guidelines. The College Board is transparent about what sorts of student responses merit the highest score (5) or any of the other scores. A high school student can use an AP test to demonstrate to a college that the student has attained the objective of the course, regardless of the reputation of the student's high school.

A Means A will extend the reliable, independent grading model of the AP exam to a broad spectrum of college-level courses. However, while the AP program compels instructors to "teach to the test," A Means A will "test to what you teach." That is, A Means A will take course objectives as given by instructors. It will design and grade tests that align with the objectives of the course. The reasons for taking this approach are discussed in the section below called Test To What You Teach.

The organizational roles for this business include CEO, head of marketing, and manager of the grading service. I see myself as best suited to manage the grading service while the business is in its experimental and development phase. Once the business gets going, I would probably fade into an advisory role.

Business Opportunity

A reliable, independent grading service can earn revenue from educational innovators and from existing institutions. For innovators, the service can provide validation and certification. For existing institutions, it can help with performance management and, in some cases, with efficiency.

Educational innovators face barriers to acceptance. With modern communication technology, it should be possible to come up with alternatives to the traditional classroom and lecture approach to education. However, consumers are reluctant to adopt innovations that have yet to establish credibility. We want innovators to be able to say, "Our instruction methods have been shown to work, as demonstrated by student performance on tests administered by A Means A, the reliable, independent grading service."

For existing colleges, A Means A could help with performance management. As it stands now, college administrators lack data on how well instructors are performing in the classroom. By obtaining results from exams developed and graded by an independent service, administrators can open a window into what is going on in various classrooms.

For some courses that are large and well standardized across universities, A Means A could offer efficiency based on scale economies. The cost of developing and grading an exam for organic chemistry or intermediate microeconomics could be reduced by having a specialize grading service undertake the effort rather than using the resources of every college that offers such courses.

The ability for A Means A to charge a mark-up over its costs will depend on becoming an industry standard. In the best case scenario, employers will demand to see A Means A grades on transcripts, colleges will use A Means A to determine whether or not to accept credit for transfers, graduate and professional schools will use A Means A in admissions decisions, and consumers will use A Means A results to evaluate alternative educational institutions. Educational researchers will frequently say "According to data from A Means A, ..." The higher education market is so large that if a business becomes a standard essential component, the profit opportunity should be rather significant.

If A Means A falls far short of becoming an industry standard, then the costs of providing the service are likely to prove too high relative to the prices that it can charge. The College Board charges over $80 per student for an AP exam. A Means A is likely to cost much more, particularly for non-standard courses. Even though outside tests could substitute for internal testing, colleges are likely to see independent testing as an "add-on" cost. They will only pay if there is strong pressure from external constituents.

Of course, attaining the status of an industry standard represents a formidable marketing challenge. The team will have to possess outstanding skills at marketing strategy and execution.

Test to What You Teach

Centralized, standardized tests are not the best approach for a college environment. With lower-level courses, professors value their autonomy to emphasize certain units rather than others. Upper-level courses are often "custom designed." If anything, college education is likely to become more dynamic going forward, as educators experiment with new technology to deliver content. A Means A will want to have the flexibility to adapt to diversity in content and teaching methods, not become a hindrance to creativity and innovation.

The practicality of "test to what you teach" as a model for independent grading has been demonstrated by the Swarthmore College Honors program. The courses are seminars, taken by juniors and seniors in their major and minor subjects. The professors who teach the courses have control over the curriculum. The college hires outside examiners who write exams, based on syllabus material supplied by the instructors. The college administers the exams, and the outside examiners grade them. In the Swarthmore program, the exams are all free-response, without any machine-graded component. They also include an oral component. For A Means A, the mix of machine-graded,, free-response, and oral examinations is yet to be determined.

With "test to what you teach," educators will send A Means A course materials, including a syllabus, a list of course objectives, and examples of the types of questions students are expected to answer. A Means A will find a qualified professional to design an exam and to provide input into grading criteria. Some criteria, for example for grammar and organization of written work, will be standard across all A Means A tests. Other criteria will be specific to course content.

One potential drawback to "test to what you teach" is that one professor teaching, say, organic chemistry, might have course objectives that differ dramatically from those of another professor. A Means A will have to develop reporting methods that make it as easy as possible for users to understand these differences. For example, in organic chemistry, a committee might agree on a standard set of objectives. Any given professor will have the option of having students take the test that is based on the standard objectives or of taking a custom test. A Means A can note when a grade is based on a non-standard test and also include an explanation of how the test differed from the standard.


Comments and Sharing





COMMENTS (28 to date)
Neeraj Krishnan writes:

Would this mean reduced pay for graduate teaching assistants part of whose jobs it is to grade? I guess the same graduate students are also potential employees of A means A Inc.

Jon Leonard writes:

The College Board already has some exams to that end, the CLEP exams. It's an interesting question why they're less well-known than the AP tests; it may be that certifications for the relevant classes aren't all that useful.

Noah Yetter writes:

Grading destroys learning. It doesn't matter whether the grader and teacher are the same person.

roystgnr writes:
Grading destroys learning.
Definitely. For instance, I can claim to have accomplished an unlimited amount of learning about theories connecting 3rd century Roman art, sociopolitical evolution, and theology, but this claim would be thoroughly destroyed as soon as anyone tried to grade me on it.

And at least there I could try to BS! Don't even get me started on those math and science classes where the grading is objective...

Renee Marlin-Bennett writes:

Using the College Board as model is a flawed idea. The AP exams are widely criticized by faculty in my field because of the emphasis on memorizing factoids as opposed to understanding core concepts. Any standardized grading system is likely to similarly focus on easily accountable 'learning outcomes' rather than more sophisticated ones. Such standardiztion and outsourcing of grading would encourage students to learn for the grade, and it might encourage profs to teach to the preordained test. In my view, that would drain the intellectual excitement out of learning and teaching.

One last comment: Why standardize? Why shouldn't a degree from Brown be fundamentally different than a degree from Oberlin?

Steve Sailer writes:

I think the crucial question you have to focus on is that obnoxious one that students all ask: "Will this be on the test?"

Jonathan Bechtel writes:

Another issue that I haven't heard mentioned yet is the ability for such a system to differentiate people at the very top.

I believe this is a problem with the AP/SAT system now, and I'd guess it's probably inherent to the paradigm itself.

I'd guess this problem is particularly true in higher education, where the participants have a higher natural endowment of human capital, and hence upward stratification is both more important as a metric and harder to achieve.

Someone from the other side writes:

Jonathan:

Having taken GMAT a few years ago and also briefly looked at GRE, I am not sure this is truly an issue here. The SAT fails to distinguish the top end because it has a very broad applicability (because it needs to cover almost the whole population) but for higher education tests, you can adjust the way the tests are designed to calibrate for higher average (and potentially lower variance) in the people taking it.

While the correlation is not perfect, in my experience GMAT scales quite well with what it is supposed to test, the only issue I have with it is that scoring is anything BUT transparent.

Disclaimer: I had a very good GMAT score (but was always at or very near the top of the class, so it may have been justified), so may be biased

Mike writes:

I believe scale will be very important. Your model won't work if you are designing a test for Swarthmore juniors. I will likely work very well if you have 10,000 students taking the same exam.

I find the whole higher education market fascinating. There are so many efficiencies to be wrung out of the current system.

Jason writes:

I think that this is extremely important work. When in college I found the whole Professor needs control complex annoying. Given the complete lack of standards in assessment why does academia think that Professorial discretion leads to better assessment.

I think Jon Leonards comment above was important regarding CLEP exams. A firm that could give college credit for 50% to 70% based on tests, and sell instructional services for the outside tests and the remaining 50% to 30% of the ciriculum would be attractive to many students.

There has to be a huge market to avoiding the unscientific and arbitrary nature of higher ed. Even if a small portion of it must remain.

I was also thinking of the a more real test of learning. An assessment of learning 1 year or 5 years from the education course. Isn't the evidence that most students recall virtually none of the content of course not in their field of study.

I am also concerned that the A means A needing to be a standard will lead to being sucked into the current academia mercantilism. Competition is a good thing and demonstrating that your testing is as good or better than the current method, should allow any firm to enter the market.

I thank you for your great work. Clearly, this is complex and the first generation solution won't be perfect, which is why we are stuck in the status quo. I suggest focusing on demonstrating better assessment with documentation of how flawed and unscientific the current grading is.

Closing 2 questions:

Are we in the 95% confidence interval that Professorial discretion in grading leads to better long term education, as compared to a more standardized assessment. (There could be 10 to 100 micro economics exams to choose from instead of the 8,000? that exist across the US today.)

And if so, at what cost?

Troy Camplin writes:

This solves the only problem with my suggestion that we should have free-lance professors.

Mark Frazier writes:

Arnold,

I agree on the opportunity, but there may be far more affordable ways to do it.

Brainbench.com is offering 1000+ certifications confirming proficiency in software apps, language and communications skills. A topview of their offerings is at:

http://brainbench.com/xml/bb/common/testcenter/testcenter.xml

All of Brainbench's certifications are inexpensive (less than $50) and taken online. The certification tests can be taken as "open book" exams. This would seem to be a recipe for a worthless certification.

Yet Brainbench ensures that its certifications carry weight. It deters individuals from faking their true knowledge of a subject through a simple, automated system: it lets employers download, on demand, a custom test that it can administer on the spot to any Brainbench certificate holder who is applying for a job.

An independent grading venture along the lines you suggest could reach a large global audience of independent learners by applying a similar to grading knowledge in fields such as economics, philosophy, and government.

On a related note, the grading business might be launched alongside other disruptive ventures. These include microvouchers to widen access to online courses and certifications, formation of charter schools that vest students as co-owners, and peer learning circles that use "personal currencies" to engage teachers in results-based contracts. Highlights of these approaches are at http://is.gd/kidsown and http://is.gd/altfunding.

Best,

Mark Frazier
Openworld

Tim writes:

This sounds like a fantastic idea. As a teacher, it would be nice to be forced into total objectivity. It would also be nice to give up the ability to change a student's grade when he or she comes begging for mercy.

My question is, would you allow students not affiliated with any college or university to take pre-written exams for different courses? Like, say there was a motivated student who wants to save money by teaching himself a topic using textbooks and information available online rather than attending a lecture that doesn't help him understand things any better. Would that student be able to pay A means A like 150 bucks to take the exam and then show employers that he's mastered game theory or organic chemistry?

Noah Yetter writes:
Don't even get me started on those math and science classes where the grading is objective...

Grading in math & science is less objective than you think.

If you ask "2 x 6 = ?" on a test, and the child gives you 12, does that prove they know how to multiply? Nothing of the sort. They may know how to multiply, they may have memorized the particular fact that 2 times 6 is 12, or they may have guessed. You may object that testing them repeatedly and with different problems will tease out whether they know multiplication or not, but any such test will have an enormous number of false positives (though to be fair, a miniscule number of false negatives).

I urge you to read John Holt's classic "How Children Fail". He illustrates very clearly the failure of children to learn these skills, and how our system of teaching promotes that failure. The very same patterns persist into higher education.

My bias against grading has nothing to do with what you can or cannot claim to know vs. what a test will allegedly prove you know. It is about what a system of grading does to you as a learner. If you were a "successful" student and didn't feel threatened by grades you will not recognize this from experience. Ask someone who was a poor student.

Jonathan Bechtel writes:

Someone From The Other Side:

I think the stratification that goes on with the GMAT might be hard to replicate in a de-centralized education system that allowed people to learn what they they want. Since it's the only test you're allowed to take (for the most part) to get into business school, they have the benefit of large numbers.

My intuition is that if you had a bottom-up educational system the types of subjects studied would be astounding.....it'd be a real testament to human creativity.

I know you could implement Arnold's system within the current one, but if the "A Means A" approach was used extensively the test still might have a hard time capturing the tail end of the bell curve.

Tom West writes:

Having experienced multi-section courses were multiple professors were theoretically teaching the same syllabus, I'd be terrified of any course where the professor wasn't writing the exam. The variability of topic coverage between professors is just way, way too large. (And this was for first and second year courses.)

DK writes:
Given the complete lack of standards in assessment why does academia think that Professorial discretion leads to better assessment.

Why? Because that makes professors' lives easier. You see, in Academia professors set the rules for professors. It shouldn't be surprising at all that the rules end up serving interests of professors first and paying public a distant second.

Rebecca writes:
My question is, would you allow students not affiliated with any college or university to take pre-written exams for different courses? Like, say there was a motivated student who wants to save money by teaching himself a topic using textbooks and information available online rather than attending a lecture that doesn't help him understand things any better. Would that student be able to pay A means A like 150 bucks to take the exam and then show employers that he's mastered game theory or organic chemistry?

Why not? In fact I think this might be one of the BEST things an enterprise like this could provide. If someone has the capability of learning the material independently then more power too them.

Now, that doesn't mean they should be excused from learning ALL the material generally expected in a particular course. But the resources are out there, and I see no good reason why a student shouldn't be able to "test out" of an entire degree if they have developed the skills and knowledge necessary to pass the courses.

Rebecca writes:
Would this mean reduced pay for graduate teaching assistants part of whose jobs it is to grade? I guess the same graduate students are also potential employees of A means A Inc.

What it would ACTUALLY do is reduce incentive higher ed has to bring in more graduate students than there could ever be jobs, which currently results in massive un-/under-employment among new PhD graduates. That would be a good thing.

Someone from the other side writes:


[Comment removed for supplying false email address. Email the webmaster@econlib.org to request restoring this comment. A valid email address is required to post comments on EconLog and EconTalk.--Econlib Ed.]

Benjamin Tran writes:

I believe that the A means A is a good idea for only part of the education system mainly with more educated individuals at a college level but I do not believe that it would work in a high school level or lower. There are just to many students to assume that we can base decisions off a program such as this. Possibly if we were concerning only students that are in the thousands it would work better.

I also agree with Tom West, having another person write the exam besides the professor I would be terrified. Despite the fact that we were taught the criteria on the syllabus.

Arnold Kling writes:

Rebecca,
Independent learners should be able to test out of courses. And I think they should be able to test out of an entire BA program if they can manage it.

Tom,
If a course with the same set of objectives is taught differently by different professors, then I think that having an identical test would be a feature, not a bug. I would rather know which professor is helping the students achieve the course objectives than make it easy for a professor to depart from the objectives of the course.

I am willing to grant autonomy in setting objectives. But if you take the objectives as given, the professors should be accountable for achieving the objectives.

roystgnr writes:
Ask someone who was a poor student.

I knew lots of poor students in classes where grades weren't a threat. I'm talking interpretive dioramas. In upper-level high school history classes. As *group* projects. Somehow the poor students accomplished even *less* learning in those environments, a problem compounded by non-threatening grading that didn't give them the incentive to get it right nor the chance to repeat the attempt.

At least students who flunked their fractions tests eventually got to learn how to perform arithmetic on fractions. Students who should have flunked but didn't got to have their noses rubbed in it later instead, when they got passed along to algebra but didn't have a hope of learning it.

Grading is critical not just for motivation but for individualization - how else do you know which students are likely to be bored by your slow pace versus which need more help to catch up to you? "You get a D, and come see me after class to go over what you missed" isn't *nearly* as discouraging as "you get a checkmark, as well as an implicit assumption that you now know all the prerequisites for understanding the next lecture!"

And if we imagine a counterfactual world where there really was no point to grading, then there would be no point to school either! An institution whose effects can't be measured is something like Sagan's invisible dragon - if you can't tell whether or not it exists, then you can't find any reason to care. And of course, there's another parallel to Sagan's analogy: people who insist that you can't test the accuracy of their beliefs are behaving just as they would if they already held a disbelief in that accuracy...

Kendall writes:

Arnold,
I can't speak to other subjects but I know from experience in teaching math you have to have very detailed objectives if you are going to hold professors accountable for achieving objectives by giving them independent tests. A topic like "Graphing rational functions" is too large of a universe to fairly compare different professors.

For example solving 2^(5x+3) = 4^(7x-2) is fairly easy if you change the base or the right hand side. But that trick only works (or is only easy) for a narrow set of problems. So if I teach my students the more general method of taking logs of both sides and this question shows up my students would not do as well as a professor who taught them the trick, but my students would be better prepared to solve a broader range of problems.

Kendall writes:

Noah,

Do you think students learn as much from a seminar course with no tests/grades as from a graded course?

Arnold Kling writes:

Kendall,
If you were working with A Means A, you could include in the materials you submit examples of problems that students are expected to be able to solve. That would allow you to clarify the course objectives.

JTapp writes:

ETS already provides thousands of universities in the U.S. With subject-oriented "major field tests". My university requires all colleges to administer them to seniors. We use the info to benchmark our colleges and students against thousands of others in the U.S. Surprised that George Mason doesn't participate...

Dr. David Shupe writes:

I write as Chief Innovation Officer of an independent academic R&D firm whose interests include this topic. What we have learned is that the problems with course grades (which are many) are derived from a deeper problem that almost no one is talking about: students earn a college degree through the accumulation of course credits (with passing grades). Changes in grading practice (which are needed) will have positive effects if and when there are also related changes in the underlying structures, so that, even if the business side of the college or university runs on credits (which are proxies for $), the faculty can collectively: 1) define each degree as a set of expected demonstrated capabilities-- outcomes-for-graduates, 2) each outcome-for-graduate is defined as a specific pattern of of calibrated outcomes (what does a student need to show? how well? (using explicit criteria), how often? (is once enough?, 3) student work/activity is evaluated relative to these calibrated outcomes within courses, and 4) individual student achievement is tracked as the progressive completion of each of the expected outcomes-for-graduates ("eight down and four to go"). Given this, the focus is on student development of knowledge and skills across-the-curriculum, and how a student does in a specific course is an integral part of this larger process. This changes the entire discussion about course grades, doesn't it?

Comments for this entry have been closed
Return to top