What is Diminished Reality?

06 April, 2021

Reality is what you make of it? Well - this doesn’t always mean adding some stuff. Wouldn’t you sometimes like to “photoshop” things out of your life? Then we are talking about Diminished Reality already.

Unlike Augmented Reality, in which virtual objects are added to the user's environment, Diminished Reality removes components of reality from the viewer's perspective. Based on computer vision techniques, unwanted image elements are detected and replaced by other image elements, creating an overall plausible and consistent impression for the viewer.

Diminished reality (DR) is a technology that allows users to remove (or: diminish) objects from their view in real time. In a next step the objects can be replaced by other objects through Augmented Reality. The combination of both can be a powerful tool to create an outstanding shopping experience.

It is very likely that you have already seen (or maybe better: not seen) something very similar to DR in movies: During post production some unwanted elements can be removed from the images. A common example are wires that help actors and actresses to fall or hurtle a significant distance through the air safely. Typically so-called in-painting techniques are employed to remove these features within each frame. In DR one crucial challenge is added: The diminishing of the object has to happen in real-time - not post-production.

Different techniques of DR

There are different types of Diminished Reality: Observational DR and in-painting DR are two of them. The observational technique uses pre-captured images/video of a background. When new items are integrated into the space, the existing images can be used as a reference to obtain information on the background that is obstructed by the new objects. 

In-painting is a technique where parts of images or videos are reconstructed or “painted over” - using information in the source image on textures and colors around the object or area to be diminished. It is less accurate than observational DR, but if there is no prior information on a scene available - it provides an approach for accomplishing DR. In-painting isn’t really new: In art restoration the technique is used to recreate damaged areas of a painting. If you have ever used the clone stamp tool in Photoshop - you have used a simplified version of in-painting yourself.

Real time is the challenge

One of the main challenges today for achieving Diminished Reality is real-time video in-painting. Cinematic inpainting effects do not have to be performed live - but can be applied during post production. So the complexity and computing time of the algorithms used can be quite high. But when diminishing effects must be used in live video - these methods are not usable. The processing time per frame must be reduced to only a few milliseconds. 

“This is definitely one of our challenges in the ATLANTIS project”, says DI Georg Thallinger in our interview. The long-time research manager is responsible for coordinating a project that brings together AR and DR technologies in one authoring tool. It will enable users to capture a room in a 3D layout, remove objects from it (DR) and place other objects in it (AR). Even if these users have only average or very limited media skills. So the ATLANTIS authoring tool will be highly interesting for sales professionals, interior designers - or almost everybody who wants to set up or redesign a room in his/her home.

Instance Segmentation of Objets

“One of the biggest challenges around DR for us is to implement instance segmentation of objects in real time,” says Georg Thallinger. “This means that the user can move through a room and the authoring tool will automatically detect objects that can be removed. Therefore the processing time has to be decreased to an instant - so that the segmentation can be continuously updated with the user's movement and the changing camera viewing angle.” The methods that support this acceleration are based on Deep Learning. “We have to “teach” the tool to recognize a whole range of different object classes in a training process.”

So imagine: Very soon your real estate agent can place your furniture in any apartment you are visiting instantly - so you can get a better impression. Or you can remove your sofa from your living room and place different new ones there in AR to make the decision easier. Or you could redesign a whole production plant with the authoring tool on a mobile device. The number of scenarios you could think of are uncountable.

DR in e-commerce

Paired with Augmented Reality - DR can be a very powerful sales and marketing tool for companies. There are already some great tools to virtually see how a sofa, carpet or a shelf would look like in a living room or bedroom. But what if the place where the customer wants to place the new piece of furniture is not empty at all - but an old sofa/carpet/item is in the way? With AR only, he or she only gets an awkward or confusing view.

With Diminished Reality the user will be able to remove the old piece of furniture first - and then add the one he or she wants to purchase. The customer experience will be completely different and highly immersive. This is why you shouldn’t underestimate DR: It gives us the power to change the way we see the real world - and companies of many different industries a perfect tool to create a convincing shopping experience.