As someone once said, technology companies are quick to focus on what they can do, but not so quick to examine “the how they do it”. Predictive technology may be speeding up digital transformation and be an integral part of our digital lives, but it comes with implications for data privacy and the potential to do great harm as well as good. This article aims to demonstrate the risks associated with predictive tech, and to give product leaders and product managers some food for thought and potential routes forward as they consider how to deal with the ethical issues associated with designing products.
We can all easily think of many instances where predictive tech has performed poorly, the flawed algorithm used for UK’s public exam results this summer,
facial recognition systems that fail to recognise women of colour, where data bias has resulted in discrimination. This first-rate #mtpcon London talk,
Can AI Create a Fairer World? from artificial intelligence technologist and founder of
AI For Good Kriti Sharma highlights some of the many ways that AI can go wrong. As Kriti says: “Algorithms are being used all the time to make decisions about who we are and what we want.” As she points out, these decisions are often based on historical or inaccurate data, and “we have an opportunity to start embedding a more conscious mindset, more ethical decisions in the way we look at the problems we’re trying to solve”.
[caption id="attachment_21683" align="aligncenter" width="1024"]
According to a BBC investigation, women with darker skin are more than twice as likely to be told their photos fail UK passport rules when they submit them online than lighter-skinned men (Image: shutterstock)[/caption]
The
techlash that began a few years ago as the tech industry was seen to be unable or unwilling to govern itself is still very much with us, people like
Caroline Criado Perez author of
Invisible Women: Exposing Data Bias in a World Designed for Men, and
Laura Bates continue to call out product flaws and data gaps, and the products we use fall short of our expectations. Recent research shows that public sentiment is against the industry - according to
Pew Research, while tech companies were widely seen as having a positive impact on the US four years ago, the share of Americans who now hold this view has dropped 21%, from 71% to 50%. Negative views of technology companies’ have doubled in the same period, from 17% to 33%. A
2020 Doteveryone report also shows that trust in tech companies is wavering, with only 19% of people in the UK believing that tech companies are designing their products and services with citizens’ best interests in mind.
It’s all About Trust
Nathan Kinch, a data ethics and experience design expert and co-founder of
Greater Than Learning, comments that there’s a robust body of evidence - the
Edelman Trust Barometer and the
Accenture Competitive Agility Index among them - to suggest that, of all business variables, trust has the most disproportionate impact on the bottom line. He adds: “It's not just that trust is at an all-time low, but active distrust is at an all-time high, meaning people feel that they have valid reasons to actively distrust organisations.”
Everyone wants their brand to be trusted, and no one wants their brand to become toxic through the ill-advised use of predictive tech. It’s well known, for example, that Facebook pays its staff above the market rate, and designer and futurist
Cennydd Bowles posits that the company will be forced to sustain the cost in the long term because its brand is now less attractive to candidates as a result of the
Cambridge Analytica data scandal and the company’s perceived failure to address bias and data misuse, as summed up in The Verge article:
The conservative audit of bias on Facebook is long on feelings and short on facts.
As Cennydd points out, the research studies on attitudes to tech companies are lagging indicators - he feels the pressure for businesses to address the ethical issues associated with predictive technology and the use of data has increased. Certainly there are more voices now calling for a change in approach, the major economies of the world are introducing guidelines for producing trustworthy artificial intelligence (AI), and public bodies like the World Economic Forum are researching and advising on how business should approach the responsible and ethical use of AI.
Product people need to put themselves front and centre in this search for trustworthiness, say observers. As Nathan comments, product managers can be an immensely powerful and influential community in the ethics debate because “they get to decide what a product ends up looking like”. In a blog post published at the start of this year,
Product Consequences and a Code of Product Ethics?, Rich Mironov wrote that, “as product professionals, we’re often not taking time to anticipate bad outcomes”, a comment that must surely resonate with the Mind the Product community. Heather Bussing, a US attorney who teaches advanced legal writing, social media and internet law, says: “I think it's very easy to get focused on what can we do and never ask, ‘what should we do? Should we be doing this? What could possibly go wrong? What are we missing? How could this be misused? What else do we need to know? And if this is combined with something else, what could happen?’.”
[caption id="attachment_21679" align="aligncenter" width="809"]
Whatever its purpose, be it identifying people or cars, for example, all machine learning programs are based on data (Image: Shutterstock)[/caption]
A few of the Issues With Data
Deepak Paramanand is product lead at
Hitachi Europe, and was part of the team that built
Swiftkey, now owned by
Microsoft. He observes that ethics issues are magnified in AI because of the application of heavy-duty maths that lay people don’t understand and the use of data as an approximation for the real world. Heather adds that it’s important to remember that artificial intelligence is a misnomer. All machine learning programs are based on data, she says, and what you can learn from them is primarily determined by the data they have and how that data has been cleaned and sorted. She says: “Data is incredibly useful because we can see things that we wouldn't otherwise see, we can recognise patterns, we can see trends, we can see when things start to change. But data is just a representation of something larger, and so the best we get when we use these tools is a better question, not an answer. The information we're getting from our machines has fundamentally changed, and we need to change our approach to it.”
Peter Stadlinger, Head of Product for
Freshworks and former Director of Product for
Salesforce Einstein, finds that people often skip over training data when developing predictive tech, saving their focus for the algorithmic model. “People like to tinker around with the model,” he says, observing that these models have typically been around for years and that we “don’t need another person to try to prove to the world that they’re exceptional”. But, “to do good AI you need multidimensional data, ample and diverse data”, he says. He adds that the best machine learning models are being built in non-tech fields, because these fields tend to be far less regulated, so there are no restrictions on how training data is collected. “No one's going to argue that you can't take pictures of lettuce,” he says, but if you're taking pictures of people walking down the street, then you’ll upset an awful lot of people and break some laws. “People think that machine learning is a really hard thing, but it is, in my opinion, all about collecting the right training data and picking one of the many models that have been established. People always get it the wrong way. Get the basics right, and then the rest will follow. Focus on the training data.”
Heather also says we should question whether our past data remains valuable in the light of the way life has changed this year. “The world just turned upside down. Should we continue to rely on past data, which may no longer be relevant? For example, some flight risk tools take commute time into account to determine when we might want to leave a job. But what if a huge percentage of your workforce has gone remote and has zero commute time?” One of the ethical issues for predictive tech is explainability and transparency, she adds, so how do you rely on a prediction, a percentage or a recommendation, if you don't know what goes into it, how it works, and how it applies to your situation?
So how do you navigate through the minefield that is predictive tech, anticipate bad outcomes, look for bias, and ensure that your product behaves in a way that is trustworthy and fair? What can product managers do to ensure that they’re mindful and responsible? There are some big issues for the product community to contemplate when considering predictive tech and the data used to power it.
Developing an Ethical Product Practice
Cennydd and others say they’re seeing an increase in interest from tech companies in developing codes of ethics or ethics frameworks. Companies want to be developing products in an ethical and trustworthy way. “There's a lot of grassroots appetite for this,” says Cennydd, “People don’t want to build unethical technology.” The very biggest tech companies tend to have already developed ethics guidelines - Deepak Paramanand says that at Microsoft he wasn’t allowed to ship a product until he had demonstrated he had done everything he could to adhere to its AI ethics principles - but typically most organisations have a lacklustre approach to ethics, according to Nathan Kinch, and currently their approach tends to be driven by law. “But law and ethics aren't the same thing,” he says. “We define an ethics framework as the consistent process that an organisation executes to decide, to document, and to verify how its business activities are socially preferable.”
Ishraf Ahmad, Director of Product experience and design at
United Health Group, comments it would be naive to think that the ethics of predictive tech don’t need to be tackled. He says that while his organisation’s thinking in this area is still nascent, “ethical algorithmic design is the future in healthcare”. “There have already been cases where we’ve seen that predictive algorithms can go from saving lives, to saving and cutting costs, bringing some sort of understanding to chaos and making things more efficient. I don't think we can deny the benefits of predictive tech when it's done correctly.”
But he sees lots of challenges. “The obvious one is data sets,” he says, “because without having substantial relevant data from all sets of people, whether it be race, gender class, whatever, we inherently create bias in our models. For example, historically, black patient data is limited and it means that when we try to apply care recommendations, we may be missing genetic research, cultural cues, all sorts of specific markers that help us understand what is truly relevant for the patient.” Data scientists and product people also have to recognise their bias, he says. The company has bias training, and also tries to make sure that there’s diversity among its data scientists: “Unless you can push towards a more diverse set of people who are training the models, you're not going to get a breakthrough,” he says. Balancing transparency and privacy are also challenges: “As individuals, we’re starting to understand that algorithms are making decisions for us in the background, so companies need to make their algorithms more transparent. As a business we understand that there may be bias within our models and our algorithms, so what are the right steps to take? How can we audit our algorithms to see if we’ve created bias?.”
Don’t Start From Scratch
Cennydd always urges the companies he consults with to listen to the liberal arts and humanities and not do what the tech industry so often does and try to reinvent everything from first principles. “Academics, science fiction, authors, writers, artists have been asking questions about the future of technology and the social consequences of it for decades. So listen to it, heed their work.”
Product people should also look at incentives, Cennydd says, because ethics can sometimes be compromised when difficult decisions need to be made. He explains: “Dark patterns, for example, are usually caused by faulty incentives, when a team is incentivised to push one particular red line up a chart, and not given any counter instructions or incentives to recognise the potential harm that it does. It then becomes a target to be gamed.”
Tech companies also need to get better at anticipation, Cennydd says. An ethicist will talk about moral imagination - the idea that you should consider the moral impact of various future states - but the agile and lean ideologies which predominate in business today don’t leave much room for this kind of thinking. “Agile and lean are based on an empirical way of thinking - we build, measure, learn, and then we repeat. It means you push ethical harm out into the market, and then you think, ‘but it's fine, we'll fix it in version 2’. That's a damaging way to approach ethics,” says Cennydd. He suggests that product teams try to use elements of the agile process and twist them towards ethics goals. “You might have a team that runs on an iteration zero or sprint zero, and that's the perfect opportunity to inject some anticipatory questions. Ask what happens if this is a wild success? Who could use this technology? Or, how would we cope if this is so successful, that it starts to be addictive and harmful to our user base? Find some of those potential problems and ask how you can mitigate them.”
Nathan Kinch advocates that product people become the ethics evangelists for their organisations, committing to operationalise a formal ethics framework. “You have an immensely influential role to play,” he says.
How You Can Get Started
This is a big and often overwhelming topic but there are some simple ways to get started. Here, we'll outline some of things you can do right now, if you're not already.
Don’t reinvent the wheel
Look at what philosophers and ethicists have said about the use of technology and its role in society. There are a number of very useful books on the subject -
Cathy O’Neil’s Weapons of Math Destruction,
James Bridle’s New Dark Age, and
Cennydd Bowles’ Future Ethics among them.
Marshal grassroots support
In general, people want to behave in an ethical way, and to do good, so getting support should not be difficult. Product people, as Nathan says, are ideally placed to become the ethics evangelists for an organisation. But beware of ethics washing, if ethics guidelines are developed for the benefit of public relations, they will have no real value. Ethics can act as a positive, constructive force to clarify what you stand for in a crowded market, rather than an expensive drag on innovation.
Review how you handle data
Look at what you do with your data, where it comes from, what can be done to improve team diversity, improve transparency. Look for potential problems, ask if and how a product could be exploited, whether it could become addictive. Chances are you will see opportunities to do things in a more ethical and inclusive way.
Codify your standards
There are a number of published frameworks that companies can use to help put their ethics guidelines into practice. In Cennydd’s experience, some of the best ones are written by industry bodies - he cites the
IEEE as one example - as well as nation states and some of the largest tech companies. The EU, for instance, has put forward
seven key requirements for trustworthy AI, namely:
- Human agency and oversight
- Technical robustness and safety
- Privacy and data governance
- Transparency
- Diversity, non-discrimination and fairness
- Societal and environmental wellbeing
- Accountability
Microsoft has lots of published resources on trustworthy and ethical AI, as does
Google, and others.
Part 2: It’s Time To Operationalise Data Ethics
Next week, exclusively for Mind the Product members we'll release a follow-up to this article:
Predictive Tech and Data Ethics: Part 2 - It’s Time To Operationalise Data Ethics by Nathan Kinch, Mathew Mytka, and Mike Haley. In 12 simple steps, Nathan, Matthew and Mike explain why it's time to go beyond feel-good statements, and help us to better understand their rationale behind why data ethics matters now and how data ethics can be done better. You'll find it on your membership dashboard on
Wednesday 14th October.
Further Reading
