BestsellerMagazine.com - CATEGORY Headline News: TITLE
Bloomberg. JPMorgan now has computers doing what used to take 360,000 man-hours of legal work every year." data-reactid="39">It's not just minimum-wage customer service and warehouse staff -- machines are taking over the work of highly paid bankers and lawyers, too.
Goldman Sachs has automated half of the tasks involved in filing an IPO, according to Bloomberg.
JPMorgan now has computers doing what used to take 360,000 man-hours of legal work every year.
AI is benefiting patients, consumers, and businesses alike. But it's also beginning to creep into our commercial and personal lives in other, less innocuous ways.
You are the product
"Data mining" is an appropriately invasive metaphor for the kinds of privacy risks we increasingly face. AI-powered digital platforms like social media open up new opportunities for extracting data from users. AI also makes existing data more valuable for understanding and manipulating consumers. And data-hungry techniques like machine learning will motivate even more data collection.
You can learn a lot from even seemingly anonymous and unrelated scraps of information. Eighty-five percent of Americans can be identified from their zip code, birthdate, and gender alone. Corporate giants know a lot more about us than that, and there are few rules in place limiting how companies use all the information they collect.
The insights gained from this golden age of corporate reconnaissance allow companies to classify, cluster, and target people with incredible precision. Facebook segments its 2 billion users not just by zip code, birthdate, and gender, but also by relationship, employer, job title, education, ethnic group, homeownership, important life events, parents' children's age group, politics, fitness and wellness, hobbies, technology use, and an unbelievable list of behaviors from taking casino vacations to buying pet products.
We're just now becoming more aware of the implications this level of microtargeting has for society. Now that two-thirds of Americans get news from social media, public discourse is increasingly exposed to all manner of epistemic failures including confirmation bias, groupthink, and wishful thinking -- with visible social and political consequences.
Recall from AlphaGo that AI tries to accomplish what we ask it to do, blind to irrelevant concerns. If boosting engagement means serving up emotionally charged material readers already agree with rather than content that helps people learn new things, that's what it'll do.
Your overconfidence is your weakness
The incredible sophistication of AI could tempt us to think its algorithms are infallible. But that would be a mistake. In fact, brand-new technology, "hard" numbers, and functional obscurity make it easy to overrate computers' abilities.
One of the most notoriously destructive cases of technological hubris was that of Therac-25, a therapy device for cancer patients that delivered massive overdoses of radiation and injured or killed six people over the course of two years in the 1980s.
Therac-25 was a convoluted kludge of code and hardware thrown together from earlier models. Since the old versions seemed to work fine, the manufacturer did away with important hardware safety features, relying instead on (buggy) software. It was a disaster waiting to happen.
autopsy of the deadly device published in IEEE Computer details an absurd 25-car pileup of software and hardware catastrophes. But two facts stand out: No one really understood how the thing worked, and Therac-25's engineers were wildly overconfident that it did." data-reactid="54">A 1995 academic autopsy of the deadly device published in IEEE Computer details an absurd 25-car pileup of software and hardware catastrophes. But two facts stand out: No one really understood how the thing worked, and Therac-25's engineers were wildly overconfident that it did.
The first sign of trouble occurred in 1985 at a hospital in Marietta, Georgia, where a patient was was injured by 80 times the lethal dose of radiation. When the hospital telephoned Therac-25's manufacturer to find out if there had been some malfunction, engineers took a look, and responded three days later that a scanning error was impossible.
If engineers were oblivious, imagine the helplessness of hospital staff. Each of us can empathize with the Tyler, Texas, technician whose machine read "Malfunction 54" and "dose input 2." No one knew what "Malfunction 54" meant. As for "dose input 2," a manual explained -- unhelpfully -- that it meant the dose delivered had been either too high or too low.
By 1986, company engineers still hadn't put the pieces together and told a Yakima, Washington, hospital that it was impossible for Therac-25 to overdose. The hospital recounted:
In a letter from the manufacturer dated 16-Sep-85, it is stated that "Analysis of the hazard rate resulting from these modifications indicates an improvement of at least five orders of magnitude"! With such an improvement in safety (10,000,000 percent) we did not believe that there could have been any accelerator malfunction.
Therac-25 was an extreme case of overconfidence in hardware, software, and the supposed infallibility of exact figures, but it won't be the last. Its breakdown underscores why human common sense will remain indispensable for the foreseeable future.
Computers are growing more capable, but we're also asking them to do more. Many of the machine-learning techniques we'll be called upon to trust in the coming years -- notably deep learning -- are known for their intrinsic inscrutability. The "act rationally" approach to AI, of which they belong, is less transparent than the logical reasoning and heuristics of the "think logically" and "think humanly" approaches. Just like the Therac-25 device, neural networks are a black box.
And, of course, not all Silicon Valley companies are known for their intellectual humility and cautiousness where opportunity for disruption is concerned.
Then there's the overconfidence AI users, frequently embodied by something called the paradox of automation: Automation feeds our reliance upon it, degrading our skills and readiness.
An intriguing 2015 study looked at how automated early warning systems affect how well we drive. Researchers measured reaction times and facial expressions of drivers who had to react quickly to avoid getting T-boned by foam cubes. When the warning system gave accurate warnings, response times improved. But when warning systems failed to go off or gave misleading or incomplete information, response times to danger were worse than if there had been no system.
Blind faith in technology caused literal blindness to danger, too. Even with a full two seconds to react, several drivers didn't manage hit their brakes at all:
When asked what happened in this situation, [three] drivers reported that they: "had not seen the obstacle."
Fully fledged self-driving cars are expected to reduce traffic deaths. But they'll also cause drivers to commit a smaller number of other accidents that wouldn't have occurred before (and which will be blamed on "human error").
Our overconfidence in AI is a much more general problem than how it relates to transportation. It's one we'll have to keep watching.
Justice truly blinded
Recall how that initial version of AlphaGo learned to mimic the orthodox approach of 20th-century Japanese players: Because that style was overrepresented in AlphaGo's training examples.
There's no such thing as unbiased AI software, because every AI program relies upon a model of the world. And just like human mental models -- scientific theories, political ideologies, and first principles -- no single AI model works perfectly across all problems. The choices we make are guided by the model we chose.
Nowhere is that more the case than in the courts. The criminal justice system is beginning to use AI at every stage, from choosing which neighborhoods to police, to sentencing, to predicting who is likely to commit another crime. On its face, more accurate justice could be a good thing. It might mean fewer people going on to commit violent crimes, and fewer people in jail who don't need to be there. Several studies have examined how effective AI is at reducing crime, and the jury is still out.
ProPublica investigated a criminology AI software named COMPAS. Of the 7,000 arrested people included in ProPublica's study, just 1 out of 5 that COMPAS predicted would commit a violent crime over the next two years actually did. The system also miscategorized black people as likely reoffenders more often than it did white people." data-reactid="76">But we need to be wary. In 2016, ProPublica investigated a criminology AI software named COMPAS. Of the 7,000 arrested people included in ProPublica's study, just 1 out of 5 that COMPAS predicted would commit a violent crime over the next two years actually did. The system also miscategorized black people as likely reoffenders more often than it did white people.
COMPAS wasn't programmed with race in mind. But its racial bias shouldn't surprise us, as machine learners are easily contaminated by it. If there's a particular bias a developer wants to avoid (like race), it's not enough to exclude it from the data set. Unless you're very careful, other attributes accidentally end up serving as proxies. One counterintuitive solution might be to explicitly teach the AI software to avoid specific biases by training it on those attributes -- similar to what researchers who study implicit bias aim to do with humans.
Each classification AI software faces a trade-off between being too lax and being too sensitive. The way you calibrate its sensitivity is by assigning a higher "cost" to the mistake you're more concerned about. If you're a radiologist, you want your cancer-screening AI to be more forgiving of false positives than false negatives; it's more important to catch as many cancers as you can, even if it means some patients will turn out not to be sick after all. So you'll assign a higher cost to missing cancers.
Every legal system struggles with a similar problem. Wrongful convictions are unfair to convicts and their families; wrongful acquittals are dangerous for the public. When Sir William Blackstone, the 18th-century English jurist asserted, "It is better that 10 guilty persons escape, than that one innocent party suffer," he was assigning to wrongful conviction a higher cost than to wrongful acquittal. (Notice that that's just the opposite of our cancer-screening AI.)
Costs are choices the programmer has to input. AI can't do it for you. You can avoid making painful trade-offs only with a better classifier, and human behavior is far too unpredictable to classify perfectly. And you can tell the makers of COMPAS had to make these kinds of choices.
The main thing you want to look at is something called an AUC score. If COMPAS had scored 1.0, that would indicate perfection: Every predicted recidivist will go on to commit a crime, and every predicted non-recidivist would not. A score of 0.5 would mean it's as good as throwing darts: If 3 out of 10 people go on to commit a crime, the classifier would perform as well as picking three people at random.
COMPAS' AUC score clocked in around 0.7 -- meaning it is moderately predictive, but there are substantial trade-offs. Whether COMPAS is beneficial depends on whether the costs its programmers chose reflect our legal system's values and on how people in the legal system use it. On both fronts, we should be skeptical.
Page 31 of the COMPAS handbook warns, "Our risk scales are able to identify groups of high-risk offenders -- not a particular high-risk individual." Therefore, when staff disagrees with an assessment, they should "use professional judgment and override the computed risk."