AI-VActor™ technology powers new projects with Nintendo, Hasbro and the Chicago Field Museum
By Judith Rubin
SimGraphics proprietary AI-VActor™ technology will roll out in three new breakthrough interactive character experiences this year: The new and dynamic Super Mario attraction at the Nintendo Store in New York’s Rockefeller Center; a personification of Máximo, the notable Titanosaurus at the Chicago Field Museum; and Hasbro’s widely renowned Peppa Pig character for customized public spaces.
Building on modern-day artificial intelligence (AI) technologies, AI-VActor is an advanced system that can open doors for socializing with every spectator through personalized character interactions. These first three projects demonstrate its ability to move across a variety of applications and platforms, from entertainment to education, at levels never seen before, using autonomous AI characters.
This is something that has been long anticipated in the industry, now a reality. SimGraphics specialized software and engineering experts have been actively developing virtual characters since the 1980s. “Our company’s original VActor experiences were game changers when first launched decades ago and before artificial intelligence was refined for today’s personalized user experience in communal environments,” says Richard Parker, SimGraphics CEO.
Using special sensors attached to a live actor’s face, the original VActor transmitted and conveyed the actor’s words, movements, and facial expressions to an onscreen animated character, allowing the audience to interact in real-time. The new AI-VActor is able to accomplish the same real-time interaction without the need for the performer.
The confidence displayed in this new AI-VActor by its high-profile adopters is bolstered by the system’s “guardrails” programming that ensures the branded character remains on point, while responding in a realistic and responsive manner – making it an eligible match for trademarked brands and IPs (intellectual properties).
It’s-a me, Mario!
AI-VActor’s new Mario character attraction debuted in July on a 23-foot wide, curved screen at the Nintendo Store in NYC. Conversations had begun in 2018 between Nintendo and SimGraphics, about using AI to create an autonomous experience. Nintendo’s vision was to create a branded experience that could appear in multiple locations, delivering personalized content and unique adventures.
“SimGraphics was being reinvented around AI and it was the right time to rejoin forces,” says Parker. Parker’s long history with the company began in 1987 when interning with then-Vice President, Mike Fusco, SimGraphics current CTO. Parker was invited back to lead the company in 2019.
Though the pandemic interrupted the new Mario project, it also created an opportunity. “We finished the first AI-VActor system and delivered to Nintendo in late 2019 – then everything shut down for two years,” Parker says. They took the time to refocus, bring in strategic advisors and make everything better. “We improved the entire system, and in addition we significantly improved conversational interface and content – enhancements that included a vision system so Mario could detect where guests are in the space, whether leaving or approaching,” says Parker. “The method of basic object detection, enabling Mario to react to physical things in a real-time environment, was essential.”
Nintendo was an early adopter of the original VActor system for its Super Mario experience (internally known as MIRT – Mario in Real Time) featuring the voice of actor Charles Martinet, who has voiced Mario in the Nintendo games since 1991. Ralph I. Miller, Creative Executive Producer for Nintendo America since 1989, and instrumental in producing the original MIRT, recounted that Martinet showed up [late] for auditions to find an actor that would properly personify Mario for tradeshow exhibitions, using the original VActor. Miller said, “Charles just immediately took on the character that led to the creation of the first MIRT system, brought to life with his voice.”
SimGraphics created that first interactive Mario as a 3D-rendered head, prompting Martinet to spontaneously sing…“’I ain’t got no body,’” recalls Miller. In addition to appearances at tradeshows and gaming forums, Mario became a popular character for live appearances and at global events, and for many years those have been voiced live by Martinet (often remotely). Through AI-VActor, the lively persona he created for Nintendo’s beloved Mario character is being immortalized and protected, while able to be many places at once.
As Peppa Pig turns 20, a new beginning
For its first application of AI-VActor, Hasbro is showcasing Peppa Pig, just in time for the character’s big 20th anniversary in 2024. Matt Proulx, Vice President, Location Based Entertainment with Hasbro, describes it as “a perfect fit, eliminating the traditional barriers between our characters and our youngest fans. It allows our fans to interact with Peppa in a way they never could before. Simply put, it breaks down that fourth wall that exists between viewers and the content.”
Proulx expressed that “Peppa being able to hold a conversation with the audience and recognize different questions is truly groundbreaking, and creates awe-inspiring, unforgettable experiences for the fans.”
Discussing Hasbro’s objectives, Proulx says, “Our highest priority is enforcing brand consistency in whatever venue, location, or type of guest experience. All too often, we must work hard to ensure how the brand comes to life across a variety of locations and experiences. AI-VActor eliminates that issue, as fans all over the world will experience the same quality of visual and audio elements as they interact with Peppa.”
Since he began with Hasbro in 2007, Proulx has shepherded many successful international LBE activations of Hasbro brands including Transformers, My Little Pony, Monopoly and more. “We’re always on the lookout for innovative technologies with the potential to unlock more efficient processes to save our licensees time and money,” says Proulx.
Bay Laurel Advisors
And it looks like there’s more to come soon. “This interactive Peppa experience is only a starting point for our partnership with SimGraphics,” says Proulx. “We intend to deliver even more interactive experiences for a host of Hasbro brands and characters, each with their own fantastical and passionate fan base. AI-VActor technology is meant to be used across as many attractions and experiences, due to the flexibility of its design and footprint. There’s so much potential left to unlock, and this is only the beginning!”
“Matt is a visionary,” says Eric Rodli, SimGraphics Chair and former CEO. “The most challenging part is creating the first character. Once tailored, the cost of implementation in different locations is relatively low. It supports a strong business model that works for the licensee but also protects the brand.” From 1988-1994, Rodli was President and CFO of Iwerks Entertainment (now SimEx-Iwerks), which partnered with SimGraphics to bring the original VActor to market.
The Chicago Field Museum: Máximo appeal
Máximo is a fully constructed Titanosaurus skeleton, revealed in 2018 and centrally located in the museum’s Stanley Field Hall. In this hub of the Museum, Máximo will spring to new life as a guest interplay animated attraction this year, utilizing AI-VActor technology.
Rob Zschernitz, CTO, The Field Museum explains, “Máximo will be now able to autonomously interact with visitors, to answer questions about himself and the Museum. We’re building a repertoire of questions and answers that will expand as needed – he already has a personality and dialogue, programmed by SimGraphics to meet the Museum’s specifications,” says Zschernitz.
The Field has a reputation for technical leadership, and since 2018 visitors have been able to engage with and learn from Máximo via a text-based chatbot interface. But after a couple of years, the chatbot was starting to feel dated. With a new strategic plan instigated by CEO Julian Siggers, Ph.D., who joined the Field in September 2020, the museum began exploring possibilities for a new Máximo interactive. The synergy started to crackle when Zschernitz saw AI-VActor demos after being introduced to the company via a Field Museum trustee.
It’s all about bringing unique characters to life, whatever the venue or setting. As George Wade, President of Bay Laurel Advisors says, “Unique characters are a perfect fit for AI-VActor technology and there are so many different aspects of consumer interaction available. This is truly a different experience in our industry and has tremendous potential for operators in all LBE forms.” Wade is a leading consultant on branded LBE and was recently named to the board of Licensing International. Working alongside Eric Rodli during their Iwerks Entertainment days, Wade was a driving force behind the development of original content, as well as IP blockbuster movies simulation experiences (think Robocop the Ride). Iwerks was also pioneering new exhibition models for media-based attractions into nontraditional and educational markets.
The Field Museum’s example begins to unlock another potential for AI-VActor. Zschernitz portrays it by saying, “The new Máximo experience is likely only the beginning of how the Field will use SimGraphics technology. It’s rather a proof of concept, to see how audiences resonate with it.” If Máximo works out well, Zschernitz indicated that possible future applications could include using AI-VActor to create a virtual museum docent, and to utilize AI characters onto the museum’s other online platforms.
The added capacity to reach out in more directions and on more platforms is empowering for operators and educators alike. Zschernitz, who’s been with the Field Museum nearly 13 years and leads a team of 19 people, says, “A comprehensive visitor immersion strategy gives the ability to address a museum’s overall digital presence, both inside and outside the facility.” He adds, “There’s always this balance of being a physical space but wanting to reach as many as we can. Someone who may never experience Chicago, can still be our audience with online programming and engagement. It’s transforming technology from something we do, to something we are.”
AI-VActor character development
Parker emphasizes, “In the world of autonomous experiences, with these new projects, we’re focused on bringing well-known brands to life while preserving brand integrity. AI on guardrails – also known as ‘directable AI,’ doesn’t learn on its own, doesn’t pick up bad words….is not a parrot. It has parameters that ensure it remains in character and can’t be led astray or go off brand – it’s always under control.”
Parker explained that drawing on rich material and with the right programming, an autonomous character can be well developed and engaging while maintaining guardrails and brand consistency in the presentation. “It’s a respectful process,” Parker says. “There are lots of nuances in every branded character that’s been built up over time. Generations grow up with them. Companies have greatly invested in them. It’s therefore essential to understand the personality of the character in what they would know or talk about, and things relevant to their world and location. There’s the conversational aptitude, which includes the ability to handle things from left field, while staying in the universe of the character.”
“The stakes are huge,” says Rodli. “You must protect the IP – but there are other needs to be met as well. There might be four or five key players on a project – the IP licensor, operator, financial investor, creative, and core tech professionals. That’s four or five people to say ‘yes’ and to keep saying ‘yes’ as the project evolves. That’s the complexity of branded implementation.”
Once the basic character is established, it can be updated and refreshed with new material, and refined based on operations data. Guest privacy is protected. Rodli reinforces that, “We don’t track personally identifiable information, we only track that there is an individual in the space while they are interacting with the character, we log all the interactions. That’s information we can mine for expansion of the system to better understand consumer behaviors, and marketing insight – all valuable for IP holders when the consumer interacts directly with their brand.”
The benefits of experience
It helps that these three new projects involve some of the key players who helped introduce the original VActor, bringing passion, creativity, and decades of experience to help guide their clients to success.
“We have plenty of real-world experiences in actual environments, in dealing with the public, and creating attractions where people engage with virtual characters,” says Rodli. “This team can bring together creative and tech. We’re experienced pioneers in LBE and branded entertainment that allows us to leverage our networks and relationships. We speak the language of the project process and have a real understanding of operations and ROI.”
“As a pioneer in the virtual character arena over the past 30 years, SimGraphics has worked with most major players including Fox, Disney, the BBC, Dreamscape Immersive, T.V. Globo and Nintendo,” says Parker. “As trailblazers in virtual character development, using performance animation and digital puppetry, SimGraphics knows how to manage guests and crowds in all sorts of environments, whether large or small.”
Summing up, Parker says, “We love what we’re doing, and want to keep moving in this direction, combining high-quality brands and IP in all LBE spaces from museums, science centers, themed attractions, cinemas, retail, and educational markets. Part of our vision is to also take those characters into the digital in-home and in-school marketplace. That’s the big pitch!”
SimGraphics will have a presence at the IAAPA Expo in Orlando in November 2023. To set up a meeting or for more information, email [email protected].
