“The most important goal I had in mind was to convince people to stop blindly trusting algorithms and assuming that they are inherently fair and objective.” PeopleMindImportantGoalFairsAssumingObjectivesConvinceAlgorithms Author:Cathy O'Neil
“I wanted to prevent people from giving them too much power. I see that as a pattern. I wanted that to come to an end as soon as possible.” PeopleGivingEndsWantedToo MuchPatternsToo Much Power Author:Cathy O'Neil
“There's less of a connection for a lot of people between the technical decisions we make and the ethical ramifications we are responsible for.” PeopleDecisionConnectionsResponsibleEthicalRamificationsDecisions We Make Author:Cathy O'Neil
“Most people don't have any association in their minds with what they do and with ethics. They think they somehow moved past the questions of morality or values or ethics, and that's something that I've never imagined to be true.” PeopleThinkingMindPastValuesMoralityEthicsMovedBeing TrueAssociation Author:Cathy O'Neil
“It's a standard thing you hear from startup people - that their product is somehow improving the world. And if you follow the reasoning, you will get somewhere, and I'll tell you where you get: You'll get to the description of what happens to the winners under the system that they're building.” PeopleIfsWorldHappensBuildingProductsStandardsWinnerReasoningDescriptionImproving Author:Cathy O'Neil
“The disconnect I was experiencing was that people hated Wall Street, but they loved tech.” PeopleStreetsWallHated Author:Cathy O'Neil
“People are starting to be very skeptical of the Facebook algorithm and all kinds of data surveillance.” PeopleKindStartingAll KindsDataSkepticalSurveillanceAlgorithms Author:Cathy O'Neil
“People felt like they were friends with Google, and they believed in the "Do No Evil" thing that Google said. They trusted Google more than they trusted the government, and I never understood that.” PeopleSaidGovernmentEvilFeltUnderstoodTrustedGoogleEvil Things Author:Cathy O'Neil
“By construction, the world of big data is siloed and segmented and segregated so that successful people, like myself - technologists, well-educated white people, for the most part - benefit from big data, and it's the people on the other side of the economic spectrum, especially people of color, who suffer from it. They suffer from it individually, at different times, at different moments. They never get a clear explanation of what actually happened to them because all these scores are secret and sometimes they don't even know they're being scored.” PeopleKnowsWorldWellsDifferentSometimesMomentsBigsSufferingSidesWhiteSecretSuccessfulClearHappenedEconomicColorBenefitsEducatedDataExplanationScoreConstructionSuccessful PeopleSpectrumDifferent TimesWell Educated Author:Cathy O'Neil
“When people are not given an option by some secret scoring system, it's very hard to complain, so they often don't even know that they've been victimized.” PeopleKnowsHardGivenSecretComplaining Author:Cathy O'Neil
“I don't think anybody's ever notified that they were sentenced to an extra two years because their recidivism score had been high, or notified that this beat cop happened to be in their neighborhood checking people's pockets for pot because of a predictive policing algorithm. That's just not how it works.” PeopleThinkingYearsTwoHappenedBeatsExtrasScorePocketsNeighborhoodTwo YearsPotCopAlgorithmsRecidivism Author:Cathy O'Neil
“Because of my experience in Occupy, instead of asking the question, "Who will benefit from this system I'm implementing with the data?" I started to ask the question, "What will happen to the most vulnerable?" Or "Who is going to lose under this system? How will this affect the worst-off person?" Which is a very different question from "How does this improve certain people's lives?"” PeoplePersonsDoeDifferentHappensCertainAsksLosesWorstBenefitsAskingVulnerableDataImplementing Author:Cathy O'Neil
“With recidivism algorithms, for example, I worry about racist outcomes. With personality tests [for hiring], I worry about filtering out people with mental health problems from jobs. And with a teacher value-added model algorithm [used in New York City to score teachers], I worry literally that it's not meaningful. That it's almost a random number generator.” PeopleProblemJobsUsedValuesNumbersCitiesWorryTeacherNew YorkExamplePersonalityModelsTestsMental HealthMeaningfulOutcomesScoreRacistNew York CityHiringAlgorithmsHealth ProblemsGeneratorRecidivism Author:Cathy O'Neil
“The Facebook algorithm designers chose to let us see what our friends are talking about. They chose to show us, in some sense, more of the same. And that is the design decision that they could have decided differently. They could have said, "We're going to show you stuff that you've probably never seen before." I think they probably optimized their algorithm to make the most amount of money, and that probably meant showing people stuff that they already sort of agreed with, or were more likely to agree with.” PeopleThinkingSaidShowsStuffDecisionTalkingDesignAmountDecidedAgreeDesignerAlgorithms Author:Cathy O'Neil
