Arnold Kling  

Stating The Problem: First Attempt

Cowen on College Subsidies... Ron Bailey on Confirmation Bia...

Sebastian Thrun has one scenario.

In 50 years, he says, there will be only 10 institutions in the world delivering higher education

I do not think that there will be institutions as we think of them. I think that in about 15 or 20 years there will be learning platforms. The model that I have in mind is Neal Stephenson's illustrated primer in The Diamond Age. I am actually trying to think through what it would take to produce something like that.

I think a key step would be developing software that can engage with students in a conversational way. This would provide the sort of personal instruction and coaching that I believe I provide in the classroom (perhaps my role is one of an ogre, but for now I am not thinking in Tyler's terms).

I call this The Problem. In this post, I want to attempt to articulate The Problem.

The idea is to create a program for use on a tablet computer to help instruct students in an introductory statistics course. The program would assign problems to students and then interact with them in three ways.

1. If a student is stuck, give a hint.

2. If a student goes off in a wrong direction, guide the student back to the right direction.

3. If the student asks an open-ended question, give an appropriate answer.

For example, suppose that a student has seen my lecture on failure models (number 4 on this list) and is given the following problem:

the probability of drawing a spade from a well-shuffled deck of cards is 1/4. If you draw three cards from three different, well-shuffled decks, what is the probability that you do not draw any spades?

The student might first select one of two initial responses--either "I think I know how to do this" or "I don't know where to start." If the latter is selected, the computer might prompt the student by asking "Suppose we say that one success consists of drawing a spade. Call the probability p. What is p?" Further hints can be given, as needed.

Suppose that the student writes down (on the tablet) 1 - p3, which is a wrong answer. I want the computer to recognize the student's handwriting (or be able to ask, "what does that say?" and understand the answer). Then I want the computer to say, "What probability will that give you? The probability p3 is the probability that...(the student answers) ...And one minus that is the probability that...(the student answers, and I would hope sees what is wrong with the original answer)"

I see The Problem as one that involves natural-language processing, but in a confined domain. It should be considerably less difficult than the problem of language translation or the problem that Apple's Siri attempts to address to be a generic personal assistant. The problem is more akin to writing an intelligent "help file" or customer support application.

However, I do think that The Problem is hard enough and important enough to be worth a lot of effort. To me, it is analogous to the problem of building a self-driving car. Along the way, you will encounter many sub-problems that need to be solved. Solving these sup-problems will advance the generic capabilities of AI. One capability will be self-improvement. That is, as it is put to use, the software will encounter new events (new ways for students to go wrong, new questions that they ask) and it will need to expand its knowledge base in response to these new events.

If you are interested in The Problem, let me know. I could see myself putting together a team to work on it.

Comments and Sharing

COMMENTS (18 to date)
david writes:

Wolfram Alpha can probably be built to tackle questions like the example given, although at present it only has one interpretation card-related questions, as "probability of whatever given a number of random cards from one hand" rather than a general case.

Arnold Kling writes:

David, I am not focused on getting a computer to be able to answer the probability problem. I am focused on getting the computer to be able to coax a student to find the solution. I see the coaxing objective as a more subtle problem.

Peter S. writes:

Khan Academy has put a lot of effort into elements of this problem. I don't think they're trying to cut teachers entirely out of the equation (at least not yet), but they are focused on providing hints and trying to build students' intuition with their problem sets.

Here's a recent video of theirs that describes some of this:

SWH writes:

What you seek is a realistic answer to higher education, which I would define as directed at self-motivated learners of any age.

The more difficult problem (I think) is K-12 education where social components (both inter-student and student/teacher) are a dominant factor. Memory and learning are strongly tied to the emotion involved in the events - at least in the early years. There is a strong push to automate/digitize k-12 education, for cost reasons and to "fix" schools. But automating/ digitizing the creation of emotional events is harder that the task you seek to accomplish .....IMHO

Bootvis writes:

Where can I sign-up? ;)

JeffM writes:

For over a year, I have been tutoring at a free math website. I have averaged about 4 posts a day. So I now have done over 1500 postings.

The site answers questions on almost all basic branches of math up to abstract algebra, topology, and analysis, but I am not equipped to handle such advanced topics. (My academic training was in history, and I never taught anything professionally.) I limit myself to arithmetic, algebra, differential calculus, and probability theory. The users of the site are primarily kids in late elementary school or high school or college kids taking college algebra or first year calculus, but we get adults going back to school or studying for the GED, people who have little or no training in math but have a practical problem requiring math to solve, and parents trying to help kids with homework. The point is that I have some experience in working with a wide range of students by computer.

I am confident that skilled teachers can do a great job in developing programs that will permit a computer to teach effectively most of the time. (At the site, there is a retired math teacher who is fabulous at guiding kids to answers in arithmetic and geometry without giving answers. How we are going to develop such skilled teachers if most teaching is done by computer is a conundrum I have not pondered). What I think will be crucial is for the programs to recognize somehow when the student is NOT getting it. Once a student misses a step, I suspect that human intervention will be required for a long time to come and that such intervention must happen quickly before the student gets too frustrated. Some students may never need such intervention if the program is really good. Some will need it rarely even though the program is good. And some will need it often. If the program is not very good, all will need humnan intervention fairly frequently.

In terms of design and development, that ability of the program to recognize when it is failing to instruct will be difficult but really important for two reasons. The first is obvious: such recognition must happen quickly to avoid frustrating the student. The other is that the resulting feedback loop will allow for constant refinement and improvement of the program.

One of the things I have learned since volunteering at the math site is how important it is to keep asking whether the student understands what was just explained and to try a different approach if the response is in the negative. Frequently, this requires remembering why you the teacher had trouble grasping what is being explained long ago when you were learning it or imagining what might be confusing the student. The latter in particular requires some psychological perceptivity. I am not sanguine that computers will be able to do this in anywhere close to the near future.

Daublin writes:

[Comment removed for supplying false email address. Email the to request restoring this comment and your comment privileges. A valid email address is required to post comments on EconLog and EconTalk.--Econlib Ed.]

Mike writes:

I am a high school math teacher and I am very interested in this project.

I agree with SWH that we need motivated learners before we can even think of doing something like this. There must be motivational aspects built into the system.

I also think we must be clear about our objectives. I happen to agree that signalling is one of the main purposes of education. But I don't think we can base our educational objectives on this purpose.

But we do need to know whether the objective is to teach a certain technique or fact, or if the objective is to develop some transferable problem solving ability.

If we assume that students are motivated and are generally good problem solvers and we just want to introduce them to a new technique, I think that the problem becomes much easier.

Mr. Commenter Person writes:


Since late 2008, you have remained my favorite poster on this site. Lately, one of the reasons for my interest is the way you keep digging up all of these great ideas about Neal Stephenson books. Keep it up!

Hugh writes:

Without using AI you could get your software to monitor which human-generated hints were of most use to the wayward student.

Once you have identified the best hint this could be incorporated into the software, freeing up the human teachers to concentrate on, say, the open-ended questions.

Bryan Willman writes:

There is a Second Problem.

How does "the platform" encourage the student to interact with other people, learn social graces, and how to deal with love interests, and so on?

All of the discussion of education as signal, boot-camp, etc. always miss the "meeting friends, learning to deal with powerful people, dealing with mating and romance, etc." aspects of college and high school.

Your platform will probably need to mentor people through those sorts of issues as well...

Becky Hargrove writes:

Bryan Willman,
The platform could work as a primary reference point for those who decentralize education locally with new offerings every 'semester'. Each community could hold an 'education options' fair (which could be anything) and the possibilities that generated enough interest (a certain number who would sign on) would then go into a 'semester manual' among which people would make their decisions for the semester. Eventually every community could create a very different and interesting knowledge base to compete with other communities.

Tom West writes:

Unless the AI get a *lot* better, I think we'll hit the primary problem, which is (in my experience) very few students can manage to put substantial effort into learning if there isn't the perception on the other end that there's someone putting a substantial effort into teaching.

It's the same reason that most people I know have their chess rating drop about 200 points if they're playing against a computer. Without the incentive to beat a real-life human who's putting in effort, it's really hard to summon the effort to play at one's best.

aretae writes:


I started work on a version of this software for math about 4 years ago. It worked decently, but was unmaintainable/hard to I spent a while (doing other work) trying to noodle through the problem. I've restarted a solution in Groovy/Grails (as opposed to first answer), but I haven't yet uploaded it to the web.

NukemHill writes:

There are a couple of branches of AI that are immediately applicable: Decision Trees, and Neural Networks.

Part of the key to the solution is being able to identify a large enough bounded set of "possible" wrong answers. There are obviously bounded sets of right answers to any particular problem that has answers (except when said problems have infinite answers; but we'll ignore that for now, as that's a definitional issue that can be handled separately...). But wrong answers can go off in any number of directions, many of them initially unknown and "unknowable".

Being able to identify a core set of "knowable" wrong answers will go a long ways towards being able to keep the student reigned in. This is where, I think, properly applying Decisions Trees could be of great use.

And then having a system that identifies new and interesting wrong answers, and categorizing them for future use, is where Neural Nets come in.

I spent some time on AI in grad school. It is a subject area that has always fascinated me, and if I ever get things moving with my business and have a steady enough source of income, I will be devoting my extra free time towards developing a deeper working knowledge of some of the sub-categories in the discipline.

DaveL writes:

I'd like to second what Bryan Willman said, and also to remind Kling that in The Diamond Age, the only "Young Lady's Illustrated Primer" that truly worked was the one backed by a nearly full-time tutor who bonded with her charge. (I don't consider the Mouse Army version to have worked, for example.) Even the one that worked produced a somewhat unusual "young lady" from a social perspective.

Not that I wouldn't consider such a device an awesome achievement if it could be made to work.

Mike writes:


I went and read the The Diamond Age because of your several references to the book. It helped that I had read other Neal Stephenson books, so it was natural to read another.

A Young Lady's Illustrated Primer is the goal. Details to be worked out.

I think about the possibilities available to Africa, Home Schoolers, etc. if such a tool were available. It is easy to see a number of possible organizations that could/would spring up to take advantage of such a tool.

These orgranizations would answer many of the socialization questions brought up by others. I don't think these concerns are nearly as important as just having good tools for educating in the hands of people who really care about their children. Those people will solve the other problems with ease.

Count me in on your effort. Though my effort might be limited to only being a customer of your first attempt.

Saturos writes:

Doesn't MathXL already do this?

Comments for this entry have been closed
Return to top