Stephen Waller
swift2.jpg

Swift Mixed Reality Concepts

swift3.gif
 

Client: Internal Project / Dezeen x Samsung
Agency: Swift Creatives
Role: Co-concept, Animation, Design & Creative Direction

Over the past year at Swift Creatives, my role and focus has pivoted towards exploring the myriad of possibilities offered by the dissolving boundaries between physical and digital products and spaces. My colleague, Matthew Cockerill and I were starting to feel frustrated that so many examples of augmented and mixed reality products appearing on the market lacked imagination or often fell back on replicating screen based paradigms in a physical environment.

We wanted to explore how physical and digital systems could evolve away from standardised interfaces and truly become interwoven into our lives. Almost to the point where we no longer recognise where the physical world ends, and the digital begins. Building on the work Swift undertook with HTC in concepting around kitchen-based AR, we chose four moments within our daily lives and tried to think how a truly mixed reality system could enhance that moment, whether it was playful, relaxing, practical or informative.

 
 

 

THE CONCEPTS

 

We wanted our four concepts to be beautiful, engaging and purposeful. So the humble nursery mobile (based on the classic Flensted design) expands its interactive space out of the crib and onto the wall, inviting the child to truly immersive themselves in a new world. The onerous task of making bread is made simple by the system taking all the complications out of the process. Checking a loved one’s journey home can become a delightfully tactile moment. And lastly, as the day winds down, how someone’s home environment can be subtly altered with a sparse digital layer applied over it.

 
 

Mixed Reality #1   As the sun comes up and the household wakes our mobile comes to life, ‘shadows’ reacting to people and activity in the room. Through this we explored the potential of tracking multiple interconnected physical objects and connecting them with their ‘digital twin’. It ain’t easy!  We explored how we might use our digital content to augment elements of a physical product rather than simply placing digital content within an environment. Here tracking and response times are critical to creating a convincing experience.

Mixed Reality #1

As the sun comes up and the household wakes our mobile comes to life, ‘shadows’ reacting to people and activity in the room. Through this we explored the potential of tracking multiple interconnected physical objects and connecting them with their ‘digital twin’. It ain’t easy!

We explored how we might use our digital content to augment elements of a physical product rather than simply placing digital content within an environment. Here tracking and response times are critical to creating a convincing experience.

Mixed Reality #2   How might AR assist us in daily tasks like bread making without simply replicating a screen based user experience? We wanted to enhance the simple pleasures of bread making by using technology to take complication out of the user experience, not build more in.  We could measure free poured ingredients, track progress and guide users by understanding the object in the room and their actions. In this way we can provide useful features not possible with smartphones or tablets.

Mixed Reality #2

How might AR assist us in daily tasks like bread making without simply replicating a screen based user experience? We wanted to enhance the simple pleasures of bread making by using technology to take complication out of the user experience, not build more in.

We could measure free poured ingredients, track progress and guide users by understanding the object in the room and their actions. In this way we can provide useful features not possible with smartphones or tablets.

Mixed Reality #3   Giving physical objects digital powers. The position and progress of your loved one’s flight could be mapped onto a globe in your living room. By understanding the topology of the globe and utilising current flight status API’s that give access to position (lat/long), previous positions, bearing, speed and route the graphic can always be up to date.  Your partners plane matches the movement of the globe, showing the planes future path, destination, and time of arrival when the globe is rotated.

Mixed Reality #3

Giving physical objects digital powers. The position and progress of your loved one’s flight could be mapped onto a globe in your living room. By understanding the topology of the globe and utilising current flight status API’s that give access to position (lat/long), previous positions, bearing, speed and route the graphic can always be up to date.

Your partners plane matches the movement of the globe, showing the planes future path, destination, and time of arrival when the globe is rotated.

Mixed Reality #4   Mixed reality can also allow for more subtle and ambient experiences to augment the home environment. A simple Scandinavian snowscape sets a calming scene to end your day.  Objects within the room are enhanced or excluded to create convincing simulated atmospheres, responding to temperature, lighting conditions and responding to physical interactions with people in the room.

Mixed Reality #4

Mixed reality can also allow for more subtle and ambient experiences to augment the home environment. A simple Scandinavian snowscape sets a calming scene to end your day.

Objects within the room are enhanced or excluded to create convincing simulated atmospheres, responding to temperature, lighting conditions and responding to physical interactions with people in the room.

 

 

THE PROCESS

 

We spent some time brainstorming what we felt were valuable experiences to explore, and then spent even longer stripping them back and back and back, trying to cut out as much complexity as we could. We wanted to make experiences that just were not possible with the technology available today, but seemed feasible, or even obvious in hindsight.

We storyboarded each one out and developed overly simplistic design languages to try and keep the interfaces at a minimal level. Then we began to work with some Creative Technologists to figure out how these could be made to work.

 
 
sb1.001.gif
sb2.002.gif
 

 

PROTOTYPING

 

We decided to focus on projection as our main display mechanic, which meant we had to extensively test how projectors would need to be set up within a home environment. Sometimes we discovered it just wasn’t feasible yet with the products available on the market, whether it was based on the size of the projectors, or even the environmental conditions they were being used in. Optimal lighting settings, projection angles and timing all had to be perfected to make the concepts work.

We started to see what opportunities machine learning could afford us, in mapping the spacial qualities of the objects they were projecting onto, and then used a certain amount smoke and mirrors to work around the constraints we hit up against.

 
 
IMG_6405-2.jpg
setup3.jpg
setup4.jpg
 

 
 

The final results of our project were always going to be a set of videos we could share with potential clients and the wider world. With everything that we learnt from our prototyping efforts, we set everything up again and filmed everything in a proper studio to create the best looking demos we could. Our rule was that everything in our videos had to be shot in camera, and we didn’t want to add anything in post-production.

 
 
MR process 1.jpg
MR process 2.jpg
 

Evening-Standard.jpg

RESULTS

It was an immensely interesting project to embark on and we learnt an awful lot about what could be possible with projection-assisted MR/AR systems in the next 5 to 10 years.

All our findings were shared on our social media channels, and we received a lot of interest from potential clients. One such client was Panasonic, which commissioned us to work on a large-scale Mixed Reality project with their internal team which we look forward to hopefully seeing out in the real world at some point soon.

Beyond that, it was also picked up in the press too, with The Evening Standard running an article on it in both their print and online editions.

 
dez-sam.gif
 
 

COMPETITION

A much more direct result of our Mixed Reality work was the opportunity to adapt one of our concepts for a competition being run by Dezeen, in conjunction with Samsung’s new Ambient Mode feature on their next generation televisions. Ambient Mode is their new feature that attempts to help blend your television in with the surroundings of your home. Rather than having a dark black screen take up your wall space, Ambient Mode takes this opportunity to create something beautiful that is both adaptive and responsive to your environment.

We decided to take our ‘fish mobile’ concept and redevelop it for a more reactive digital space. We wanted the piece to really reflect the room it would be living in, utilising complimentary colour palettes and backgrounds. The lighting level in the room could also determine how fast the fish spun, and then using the motion sensors in the television to trigger the shadow animation when it detected movement close to the frame.

 
 
We started by redesigning our fish, so they no longer relied on the original Flensted design.

We started by redesigning our fish, so they no longer relied on the original Flensted design.

What was originally a real physical mobile now had to be rebuilt in 3D with realistic hanging and swinging physics.

What was originally a real physical mobile now had to be rebuilt in 3D with realistic hanging and swinging physics.

As before, the shadows are digital, but now had to be more closely tracked to how the ‘real’ shadows of the new digital fish moved.

As before, the shadows are digital, but now had to be more closely tracked to how the ‘real’ shadows of the new digital fish moved.

 

 
 

RESULT

After being shortlisted in the top 5, we were invited to present our concept at Samsung’s stand at IFA 2018 in Berlin. And then to our great delight, we ended up winning the competition overall, which was judged by the likes of design legends Neville Brody and Erwan Bouroullec.

 
 

Our award-winning concept for the Dezeen x Samsung Ambient Mode competition.

 

 
trophy.jpg
39934377_2174815252781251_5202711717796118528_n.jpg
40115781_307294363160227_8231724990844633088_n.jpg

Matthew at IFA 2018, with our booth and trophy.

 

 

The judges talk through the top 5 entries, and then discuss our winning along with Matthew.