Google’s New AI Puts Video Calls On Steroids! 💪 - YouTube

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.

With the increased popularity of online meetings, telepresence applications are on the rise

where we can talk to each other from afar.

Today, let’s see how these powerful, new neural network-based learning methods can

be applied to them.

It turns out, they can help us do everyone’s favorite, which is, showing up to a meeting,

and changing our background to pretend we are somewhere else.

Now that is a deceptively difficult problem.

Here, the background has been changed, that is the easier problem, but, look!

The lighting of the new environment hasn’t been applied to the subject.

And now, hold on to your papers, and check this out.

This is the result of the new technique after it recreates the image as if she was really

there.

I particularly like the fact that the result includes high-quality specular highlights

too, or in other words, the environment reflecting off of our skin.

However, of course, this is not the first method attempting this.

So let’s see how it performs compared to the competition!

These techniques are from one and two years ago, and…they don’t perform so well.

Not only did they lose a lot of detail all across the image, but, the specular highlights

are gone.

As a result, the image feels more like a video game character than a real person.

Luckily, the authors also have access to the reference information to make our job of comparing

the results easier.

Roughly speaking, the more the outputs look like this, the better.

So now, hold on to your papers, and let’s see how the new method performed.

Oh yes!

Now we’re talking!

Now, of course, not even this is perfect.

Clearly, the specularity of clothing was determined incorrectly, and the matting around the thinner

parts of the hair could be better, which is notoriously difficult to get right.

But, this is such a huge step forward in just one paper.

And we are not nearly done - there are two more things that I found to be remarkable

about this work.

One, is that the whole method was trained on still images, yet, it still works on video

too!

And we don’t have any apparent temporal coherence issues, or in other words, no flickering

arises from the fact that it processes the video as not a video, but a series of separate

images.

Very cool.

Two, if we are in a meeting with someone and we like their background, we can simply borrow

it.

Look.

This technique can take their image, get the background out, estimate its lighting, and

give the whole package to us too.

I think this will be a game changer.

People may start to become more selective with these backgrounds, not just because of

how the background looks, but, because how it makes them look.

Remember, lighting off of a well chosen background makes a great deal of a difference in our

appearance in the real world, and now, with this method in virtual worlds too.

And this will likely happen not decades from now, but in the near future.

So this new method is clearly capable of some serious magic.

But how?

What is going on under the hood to achieve this?

This method performs two important steps to accomplish this, step number one is matting.

This means separating the foreground from the background, and then, if done well, we

can now easily cut out the background and also have the subject on a separate layer

and proceed to step number two.

Which is, relighting.

In this step, the goal is to estimate the illumination of the new scene and recolor

the subject as if she were really there.

This new technique performs both, but most of the contributions lie in this step.

To be able to accomplish this, we have to be able to estimate the material properties

of the subject.

The technique has to know one, where the diffuse parts are, these are the parts that don’t

change too much as the lighting changes, and two, where the specular parts are, in other

words, shiny regions that reflect back the environment more clearly.

Putting it all together, we get really high-quality relighting for ourselves, and given that this

was developed by Google, I expect that this will supercharge our meetings quite soon.

And just imagine what we will have two more papers down the line.

My goodness, what a time

to be alive!

Thanks for watching and for your generous support,

and I'll see you next time!