As expenditure into consumer analytics tools continues to grow, with research leaning heavily on new technology like artificial intelligence (AI) and machine learning (ML), companies are able to understand their consumers and their behaviour at completely new levels. From minute changes in facial expressions when dealing with sales staff, to heat maps in retail stores indicating which products are catching consumers’ attention, emerging new technology is blowing the lid on what companies thought they understood about consumers.
We had the opportunity to speak with Karen Marquez, CEO and Co-founder of Robbie AI, one of the companies at the forefront of this technological revolution, in an extensive two-part interview. In this part, we will go more in-depth into the kind of data and insights Robbie AI is able to collect, as well the privacy implications involved.
In the first part, we learned that the company’s offering to the sector is purely on the digital side: all software, zero hardware, designed in a way to be easily integrateable into existing systems used by businesses. You can find the first part here.
What are the key metrics that can be derived from your technology and how does this data help your clients produce actionable business plans?
Some of the examples:
– Robbie concierge allows for the automated check-in and check-out of customers (like hotels, airlines, airports, stores). This allows for time reduction in operations, instant registration of customers, differentiation between new versus returning customers, analyzing foot traffic at entrances and exits, duration of interactions with staff, output and outcome of those interactions, time spent interacting with products or services, waiting times, time since last visit, etc.
Other examples also include accurately measuring consumer traffic (aggregated per hour, days of the week, etc.) and rush hours (to optimize human resources).
– Net Promoter Scores and CSAT (Customer satisfaction score) can also be discerned live, for a 100% of the customers of the business, allowing to identify detractors and promoters and strategize on that.
– Behavior per product or service
– Loyalty participation (to manage loyal customers in specific programs, without needing to identify themselves).
– Customer effort score, to know how difficult is for customers to solve their problems when interacting with staff or brands. These scores are industry benchmarked and specific per area. Imagine you are a store with different areas to track, or showrooms – you can get scores for each of these areas.
– Emotional performance of the business across time, with insights on customers: total number of customers, new versus returning, satisfaction per female/male, number of male/female, age satisfaction distribution and ethnicity (geographic origin).
– Graphics illustrating customer satisfaction per product/zone/etc.
– Heatmaps combining foot traffic with any other indicator.
– Customer retention (by measuring how many customers are lost during a specific period of time).
– Brand performance and industry benchmark
– Employee performance and happiness index when interacting with customers, etc.
– Customer acquisitions (new customers per day, week, etc.), as well as price elasticity (how customers react to changes in price), etc.
Regulators and the general public often pose concerns regarding facial technology misuse and privacy infringement. How does Robbie AI address these concerns and protect consumer data?
I think this is a great question and I think we all can understand why. The first thing I’d like to note is that Robbie AI has always been guided by a strict code of conduct, and in fact, one strong value of the company is advancing knowledge while protecting everyone in the physical and virtual world.
Right after being founded during December 2018, Robbie AI was one of the 3 foundational signatories of the Safe Face Pledge (SFP), drafted by the Algorithmic Justice League (AJL, MIT) and the Center on Technology & Privacy at Georgetown Law.
The pledge is a response to the concerns raised by private companies, legislators, and members of the public regarding how the technology of some of the most powerful tech companies in the world has been using personal data.
The SFP pledge was launched with the aim that all companies providing facial recognition services or facial analysis -especially big and large tech companies – would sign and commit to responsible, ethic and lawful deployment of these products.
The pledge is a commitment to the following objectives:
Show Value for Human Life, Dignity, and Rights
Address Harmful Bias
Embed Commitments into Business Practices
I can really understand the reason behind the public’s concerns. It’s common knowledge that organizations store personal data and create highly detailed profiling, like the case with Facebook and Cambridge Analytica, or China using Face++ and Sense Time and to screen citizens in private and public spaces.
Also, state-owned biometric databases are subject to hacks and leaks. In 2018, India’s government ID database suffered breaches that potentially compromised the records of 1.1 billion registered citizens. There is also the case of the US Customs and Border Protection database breach of 2019, where the photos and license plates of travelers were obtained by unknown hackers. Just a few examples of why keeping images is prone to misuse.
The second big problem is that facial recognition/identification of gender could be a source for discrimination opportunities when used for automated decision-making and racial profiling. The majority of commercial facial recognition systems have been shown to misidentify people of color, women, seniors and young people.
Examples are showing ads excluding genders from job opportunities or to exclude African-Americans from housing listings. A while ago, a software developer embarrassed Google by tweeting that the company’s Photos service had labeled photos of him with a black friend as “gorillas.” AI is not biased itself, but the data you feed in the system is, by nature. The people involved in the development of that AI might have an implicit bias in social constructs that affect the design and benchmark of the system.
Facial recognition has been around for years. If done correctly, it can have enormous potential and impact. Facial technology can prevent fraud, protect communication and access to financial and personal information, and avoid identity theft.
But to do it right, you need to include privacy by design, and ethics in training data by design. AI is nothing more than a model – it needs to learn to think about complex contexts and about people faces. So you need to feed it with a lot of diverse, large, and reliable data. But unfortunately, there has not been an effort to create this ground truth, and there’s no such data out there that is ready to use. With the exception of just a few, there are no datasets with diversity in gender, age, and ethnicity. At the beginning of the Robbie AI research journey, we found some datasets like the Japanese female database, or The Bogazici face database (Turkish race/ethnic ancestry), and that’s pretty much it. This is a problem we encountered ourselves. And this is a complex problem, because as your network grows, so does the need for data.
So that’s why it’s so important to train on real-world data, as diverse and rich and complex as our society is, in order to minimize these kinds of outcomes. It’s important to stand for all the commitment the Safe Face Pledge stands for.
From an AI perspective, we don’t do biometric data in the standard way. We apply a novel and proprietary processing system that removes any identifiable elements of the face. Some of the data points, like demographics and emotion, are reported as statistics in the KPIs with one goal in mind: providing categories of data in reports, which do not include personal information, but statistical information. The only metric facial recognition traffic aims to measure is recurrence. There is no kind of authentication or verification and therefore, no template created, stored, or encrypted, and it’s very similar to the way Apple masks it’s device MAC addresses. Our system has a randomization process, in which each person might change parameterization at any time, and multiple times during the same day, so there’s no tracking or risk of stolen personal data ever being traced back to individuals.
We have the technical means of de-identification to prevent a connection between the person and their statistical information from being established. Robbie AI is based on privacy by design by applying these techniques at the very moment of image acquisition.
The process starts at the moment of image acquisition (scanning, not recording), is processed in less than 1 second, and directly stores hashed information for statistical purposes, and transforms those pixels into a Minkowski Space: there is no standard coding/hashing, there is no digital representation the characteristics of an individual’s face, and while acting as an isolated number, is not susceptible to comparison with public photos, nor third-party representations, or template databases – not even following different mathematical techniques for extracting facial features in standard facial recognition.
What are some future applications of your technology that you foresee in coming years?
I think Robbie’s very large diversity in data, and therefore accurate results in terms of skin tone, gender, or age, and the efficiency to compute data in real time, is a first step in a new toolset of holistic human biometric solutions that can protect us all, in the physical and digital world, against spoofing, impersonation, fraud, cybercrime, etc. Facial recognition and facial expression recognition are not new, and they’ve been around for a long time.
What’s new, from an AI point of view, it’s the training data that allows Robbie to perform fast, with minimum failure, for everyone, which is a unique opportunity to improve benchmarking and performance in digital-physical environment in a wide set of scenarios.
Robbie AI believes in this vision of supporting consumers’ privacy anytime and anywhere. This includes replacing logins and passwords and registration codes, to responding to customer issues the moment they occur, alerting of fraud in real time, or providing real-time authentication to vehicles, IoT devices and your own smart home.