by Maris Ensing, Mad Systems
Imagine an exhibit where the media that is delivered is tailored to you. This paradigm shift creates a user experience truly unique to each visitor, and leaves a lasting memory to be shared with friends and loved ones. Introducing Facial Recognition and Color Recognition Controlled Media Delivery Systems, patent pending solutions to completely customize a user’s experience and the way that content is delivered.
These patent pending technologies are the latest announcements from Orange, California based Mad Systems – an audiovisual system designer and integrator, specializing in interactive exhibits. The prior groundbreaking announcement surrounds a low infra-structure ultra-flexible audiovisual solution that allows for delivery of customized content, known as QuickSilver™. Though it sounds similar to what was just described above, know that QuickSilver™ and the recognition based delivery systems are standalone solutions, but were specifically designed to work together to optimize the experience. Mad Systems’ Founder and Chief Engineer, Maris Ensing, explains this 21-century audiovisual and interactive solution.
The birth of QuickSilver™ as a system got us to look at just how we could provide properly personalized media delivery to our audiences. The rule-based QuickSilver™ system displays content based on a variety of pre-determined trigger methods including buttons and infrared. But when our team considered the new-age audience that essentially demands a unique experience in order to create that lasting impression, we knew that more had to be done. How could we shift the paradigm in the way we deliver exhibits? Then came the next bright flash of an idea, and the thought of using Facial Recognition technology to control media delivery in our industry.
We knew that while Facial Recognition is incredibly attractive, some would have concerns about the quality of the recognition results and about privacy. It turns out that the recognition results can be amazing. The fact that this technology has been successfully used in a number of countries to pick wanted criminals out of huge crowds is a good indication that the technology, although not yet perfect, is ready for prime time. The privacy concerns were also something we wanted to extensively research prior to developing the recognition-based media delivery systems. The interesting thing is, that although a video signal is fed into the recognition software, the video is not stored in any way, and the data that is extracted has nothing to do with video: any information relating to people in the pictures are converted to nothing but data points. Information that is kept in encrypted form includes things like the distance and angle from the left iris to the nose, the distance and angle of the right iris to the nose, a vector representing the shape of the mouth, etc. This method of collecting data points, paired with using local computers without needing an internet connection, alleviates privacy concerns. The encrypted vector data that describes people’s faces is kept under the control of the venue, and can be scrubbed at the end of the day. This, to us, provided the breakthrough that make us feel content to continue with our development work.
Being able to identify a visitor allows us to follow them around the venue and deliver content in a way that means something to them. There are different ways to generate this ID:
- -If someone logs into the venue’s website from home, they could send a picture to the system, select preferences (things like preferred language, interests, age range, level of expertise – and for example, if they have a membership, they can give consent to storing that data so that they can be personally welcomed when they arrive at the venue).
- -If someone gets to the venue and stands in front of a log-in kiosk, the same information could easily be entered
- -The venue could provide staff with portable devices to log anyone in that wants to be logged in
- -If people do not want to log in formally, that’s fine too: the first time they get to an interactive, it could ask them if they’re OK for the system to recognize them. If that is the case, the first kiosk could ask them their language preference – and from that point on we’d know what language to use for that visitor
- -If people do not want to be recognized that’s fine too – the system will allocate an ID, but the system would simply not keep any data at all other than a flag indicating that only default media is to be delivered
Other information could be used to provide statistical information to the venue owner. This information could include dwell time, level of interest shown, and even if the visitor smiled when they saw the exhibit – again, without ever storing any imagery or video.
This is the solution we needed to allow us the required freedom to design the system we had in mind. No objects for visitors to hold or carry, with nothing required but cameras around the venue, a Facial Recognition system, and a software layer that converts visitor interests into a command for tailored media delivery. By putting a camera facing the audience, we can set boundaries in the space for recognition so that we can control exactly where we want to ‘look’ for visitors.
When analyzing pictures of visitors arriving at various locations, another thought occurred that resulted in our second related patent being filed. When school groups, church groups and clubs arrive they often wear similar clothing. We often see t-shirts, scarves, jackets or hats that are the same for all group participants – so the idea of our patent pending Color/pattern Recognition was born.
Color Recognition does not provide the granularity of Facial Recognition. We cannot recognize individuals using this method, but we can know that they’re part of a group of 5-6 year olds, or part of a club that is interested in history. Not only that, but by providing simple colored or patterned badges on lanyards, we can reserve colors for special functions. We could use blue for visually impaired visitors, a bright yellow for visitors with hearing problems. We could select purple for Chinese speaking visitors, and red for adults with an interest in physics. Now we have a solution that not only allows us to provide a good level of control over providing personalized content, but it also allows us to deal with special needs properly and discretely. The technology is relatively simple, and recognition certainty is very high.
With these recognition systems, we also started looking at how interactive exhibits work. One of the solutions that this provides is when we have large monitors, especially when they’re mounted vertically. Neither kids nor visitors in wheelchairs would be able to reach touchscreen buttons that are high up on the screen (which is why ADA rules insists that they are not), but esthetically it can be nice to have them there. For example, an interactive map of California on a vertical (portrait) mounted touch screen could have buttons quite far up on the screen to allow visitors to select cities in the north of our state. Now, when the Facial Recognition system tells us that we have someone who is not tall enough to reach up, or someone with special needs, we can have the interactive lower its buttons to the bottom so that we can provide a 100% solution for all of our visitors without having to compromise the design. Similarly, if we have someone that’s 6’4” or taller, we’d not put the buttons on the bottom of the screen. This is truly liberating when designing interactive exhibits, and of course similar benefits could be realized even when looking at electro-mechanical exhibits.
These different technologies have allowed us to start thinking about developing other great options for our clients that thus far have not been possible, and we look forward to revealing other new concepts using this technology that will allow us to continue to shift the paradigm in the way we view audiovisual and interactive exhibits, and in particular the way that we welcome visitors, and provide methods to help visitors learn and create memories.
Our Facial Recognition and Color Recognition Controlled Media Delivery Systems can, in principle, be integrated with any audiovisual or interactive systems, however, our QuickSilver™ audiovisual solution was specifically designed to elevate the experience for the user, with the venue’s pocket in mind.
