Optimization in Crowdsourcing - YouTube

Thank you, and it's a nice follow up on the first part of

Collin's presentation, which was on crowdsourcing.

So I was asked to speak on optimization for

crowdsourcing and education.

I'm not gonna speak on education very much,

because I haven't done that much work.

But I've done some work on crowdsourcing and

in particular in the context of sequential optimization which is

what I'm going to be talking about.

It is really nice to hear the word POMDP in the morning

session come in twice. I feel like I am in an AI setting, so

I am going to be talking more about POMDPs.

Just to start this is joint work with Dan Weld.

Most of this work has happened at the University of Washington

even after my coming back to India.

Some of the work that has been happening in this particular

field is with Dan Weld and two of our students who have

are about to get degrees at University of Washington.

So this is the 30,000 feet view.

Crowdsourcing has become huge.

And this is probably the only slide I'm gonna connect with

social good.

So crowdsourcing is social good from one vantage point.

It has, starting 2006, when the word crowdsourcing was formed,

and then Amazon Mechanical Turk and many,

many different platforms have really made crowdsourcing

grow very huge and has grown really rapidly.

And I would make the point that it democratizes labor, and

this is I think also the point that Collin was trying to make

that now we can give labor to a lot of people in the world.

At the same time there are some challenges in making

crowdsourcing as successful as we want it to be.

And it is our thesis that AI, machine learning, optimization

can really make crowdsourcing achieve its potential,

it can reduce work errors, and in our experiments we

have sometimes been able to reduce errors by up to 85%.

So we have really made and sometimes we have found

crowdsourcing to be super super successful, okay,

using the AI techniques.

So now, I don't have to go too much into how crowdsourcing

came up, Wikipedia is one of the first examples of

a crowdsourced encyclopedia, and look at the viral growth of

Wikipedia articles that happen when people got interested.

Citizen Science was being referred to by Bart earlier and

Galaxy Zoo was one such example which was getting 50 million

classifications in a year, which machine learning algorithms were

not able to do at the time.

And it was not just that, the human workers,

human citizen scientists actually ended up discovering

special form of galaxies known as the Pea Galaxies,

which the astronomers had no idea about.

Most of my talk is gonna be about labor marketplaces which

are projected to grow by $5 billion by 2018.

These are older projections and

some of the older statistics,

I couldn't find the new statistics.

But basically for example at Amazon Mechanical Turk we had

more than half a million workers three or four years ago.

And oDesk was seeing 35 million hours clocked

in their platform in 2012. Now they have become UpWork.

So there's so much going on.

There's so many very interesting strengths which got me really

excited about crowdsourcing.

For example, world becoming a unified labor force.

Like a global meritocracy of kinds.

Whether you are in India whether you are in Africa whether you

are in some Pacific island, as long as you have internet

connectivity and skill to monetize digitally, then you can probably

get work. This is such a cool thing for mankind in general.

And of course it had many other strengths, not from the social

side, social good aspect like it was perfect for

startups, it was cloud computing for human work.

Really interesting new applications could come about,

for example,

one of my colleagues thought about an app for blind people.

Where blind people stuck in a new circumstance can actually

ask a question and some crowd worker can answer their question

based on the image that they are sending.

The things that you would not have expected if such

a thing was not available and so on.

But you can find pretty much every kind of expertise on that

crowdsourcing platform.

At the same time there are lots of challenges,

which is what got us interested.

The most important challenge was high variance in worker quality.

There are challenges about how to track

the quality of output and in general broad challenges about

how do you get high quality output?

Because these are crowdworkers, some workers are awesome,

some workers are not that great, some workers need training,

how do you manage this large enterprise?

And there are many other things where AI can play a role like,

usually you divide a complex task into small micro tasks and

so how do you divide it, how do you test such work flows?

How do you optimize them?

How do you figure out who's the right worker to be working on

my kind of tasks and so on and so forth?

So we have really worked very hard on demonstrating the value

of the AI and machine learning for

crowdsourcing. At a high level it works like this.

So imagine this is an AI agent, this is your requester who is

telling you what tasks need to be solved, and

every AI system is operating in some environment and so

here my environment is a crowdsourcing platform.

And so when a new task comes along, the AI system figures out

which jobs to give out and when work comes out,

you can do your learning and you can your planning.

So you can ask questions like, what are my workflow parameters?

What are the optimal parameters?

What are individual worker's abilities?

What are they good at?

What are they may not be as good at?

How difficult is my task, and so on?

So then by learning all these parameters then you can

control the task better to figure out,

if I have multiple workflows, which workflow do I select?

Within a workflow which job do I post?

When do I know that my work has been done at a high quality?

If I have a worker who is good, but has some hole in their

understanding, when do I teach them?

When do I test their quality and so on and so forth?

So we have done a reasonably large body of work in this space, and

let me just give you a couple of quick examples of the things.

For example,

everybody starts out with these simple yes, no questions.

Is this bird an Indigo Bunting?

Now I can ask several of you and some of you will say yes,

some of you will say no, and hopefully people who know this

thing will give us answers, but they may make mistakes.

So can we do some kind of consensus on top of them?

But not just consensus.

When do we know we have a good quality answer and

when do we know that we need to ask more people?

That becomes a decision theory question and so when using POMDPs

in a cost controlled setting if you did dynamic optimization,

dynamic sequential optimization, we got much higher accuracy

than the static controller for the same cost.

So this got us interested, but

then we went to more complex tasks.

Suppose this is your doctor, he wrote this nice thing for you.

Now, how do you know what is going on?

So, you can give it to a crowdworker and

in $0.27 in this particular case, they were able to get

the full transcription using a very interesting workflow.

But they didn't know how to optimize this workflow, we did.

So when we optimize this we found that we were able to use

the same amount of money, but

get much higher quality of image transcription and so on.

We have done taxonomization of all the items.

And again, one of the graduate students who was an HCI graduate

student came up with a very creative workflow to do this

taxonomization, but she was not an optimization person.

And so, when you model this for

optimization you find that for 13% of work

you can get the same quality if you use optimization carefully.

We have many, many examples of this.

Like, if you have lots of workers with varying abilities and

if you have lots of questions with varying difficulties,

we know which particular worker to give each question to and

again red is our thing, and higher is better.

We have thought about when to test a worker,

and when not to test a worker,

and again higher is better and we are the red one.

We have thought about when to train a worker, and

what to train the worker on.

And again green is what we could eventually achieve, but

we were able to achieve red in our experiments.

And in the latest work,

we have done not just quality-cost optimization,

but, we have also done quality-cost-completion time

optimization.

If I start paying each worker more,

then more workers will take my job, my time will reduce, but

my budget will exhaust quickly.

So I will not be able to solve the whole task, so

my quality will go down.

There's a very interesting interplay that happens when I

start doing this three-way optimization,

how to manage the whole optimization.

So I'm be able to use my budget the best,

with the parameters that I have in my task.

We have a latest paper coming out.

So at a high level we have looked at intelligent

control and optimization in the context of smart crowdsourcing.

Crowdsourcing has a lot of advantages but

there are some challenges to be resolved.

For example, we can figure out what to ask,

how many times to ask, who to ask, how/when to teach,

when to test, when to stop.

All these actions can be taken by our AI agents

using POMDPs as its base, and we can do it for data quality.

In other works, we've also done it for

classifier quality. If I'm in an active learning setting,

then I can use workers very effectively.

And from our point of view,

from an AI-minded person's point of view, it's a small step

towards the vision of human intelligence and machine

intelligence coming together to achieve something bigger.

In our more recent work we are not only thinking about

democratization of workers, but

also thinking about democratization of requesters.

Suppose you as an individual want to give out a crowdsourcing

task, can I make it easy for you to crowdsource?

There are a lot of good practices, but

it's not very streamlined, still.

You don't know how to make the right task,

you may be confused about your own task.

You may not know what

the workers are getting confused about.

How do you improve the task design?

So we've done an HCI interface,

where we'll give you the interface that will allow you to

learn your task better using interactions with workers.

So this is the work that we have done in the context of

crowdsourcing.

I also wanted to add that in my other lives, I do research in

natural language processing, and recently we have started two

very interesting projects. One is on analyzing legal data sets.

So lots of court data in Indian judicial system has been

languishing.

And we have started analysis to understand which courts are

outperforming other courts, which districts are doing well, which

districts are not doing well, where are the cases getting

stuck, so that we can inform the legal department someday.

And in second project, we have started a healthcare initiative,

where we're trying to read MRI images and

CAT scans automatically to see if we can help

the radiologists do their job more effectively.

And since we're talking about social good, for

young people I also advised a dating company to help

you find your right partner.

So there is a lot of very, very interesting stuff that we can do

as an AI person in the mix, but I always feel that until we

get the right partner to partner with, who is the domain expert,

it's a non-starter for us.

So in the remaining time as we are doing these discussions,

I would love to hear more about interesting problems from domain

experts, where AI people can actually help contribute.

So I'll stop here.

>> [APPLAUSE]