Former author of one of the top 5 facial recognition servers in the world for multiple years running, here's what's going on: the industry has solved this issue, but the potential clients are seeking the lowest bidder, and picking the newer companies, the nepostically created not really players but well connected, and those companies have terrible implementations. This is not a case of the technology not there yet, we solved all these racial bias issues 10 years ago. But new companies with new training sets and new ML engineers that do not know any of the industry's history are now landing contracts with terrible quality models, but well connected sales channels.
This study finds a higher rate of correct identification for black people than for other ethnic groups, whereas a few years ago the problem seemed to be that the software was less effective at identifying black people.
Do you have some insight about why this reversal might have occurred?
To have a high quality facial recognition system it needs to include every possible combination of ethnicity, in addition to all of those they each need to include variations of daylight, of dappled light, of partial obscuring, night time illumination, across every variation of season, variations of expression and face angle, across variations of weather, variations of distance, across variations of things placed on a person's face, and then across all kinds of variations of video compression. All these face image variations in the training set enable the trained model to find and track the features that persist through all these variations. In truth it requires hundreds of millions of facial images to create an accurate facial recognition system. Most new companies and many that have been around for respectable periods are not realizing how much data collection, annotation and additional variation creation it requires for a high quality FR training set. The company I worked at spent 20 years collecting laser scans of real people to then create the augmented real person data set with several hundred million faces.
Your problem is evident in your question. What does “black” mean? It’s entirely subjective. Dwayne Johnson? Liv Tyler? Nelson Mandela? Barack Obama? Mariah Carey?
This is a semantic issue. Ethnic groups are constructs. A system which misidentifies people identifies all people poorly.
It doesn’t track across regions either. People labelled “white” by law in some countries (Brazil, South Africa, etc) would be classified as “black” elsewhere.
In England, the example here, we do not classify people the way the Us does, with its history of “one drop” politics. Many British people considered “white” are “black” in the US.
There is no scientifically valid way of defining who counts as “black” so any discussion of tuning a system based on this definition is a disaster.
Even the people commenting are talking about different groups based on their own culture and prejudices.
> Many British people considered “white” are “black” in the US.
I'm also British, can you give an example of that? A minor celebrity/TV personality say?
To the extent you have a point though I think it's irrelevant anyway—they paused the programme because they found, according to whatever definition they measured with, it had that skew.
How recently? We had a home security camera and every time our (Black) son walked up to the door, the camera would classify him as an “animal”. This was as recently as 2022
In the other direction, my camera regularly identifies cats, crows, and shadows as people. I think recognition in security cameras has a very long way to go.
This is actually more (socially/ethically/philosophically) interesting than one might assume from the headline: it's not false positives, it's that it's more effective (correctly identifies someone is on a watch-list) for one group than another within a protected characteristic.
So essentially they're pausing the use of it because it works too well for group A / not well enough for group B, potentially leading to disproportionate (albeit correct) arrests of group A.
Absolutly impossible to condone further structural bias against a minority, and just ignore the free "white pass" built into the software, and esspecialy troubling that it passes white women, the most.
The only possible action is to reject and dissable any system with a racial bias, investigate how such a thing happened, with a very pointy look for intent on the part of the vendors, who would then qualify for bieng housed in one of his majestys facilities for persons such as these.
If you start with hypothetical demographic groups A and B that are for all intents and purposes exactly identical, but you implement a system such that if A commits a crime they have a 10% chance of being caught and if B commits a crime they have a 50% chance of being caught, you will achieve the following:
1. More short-term crime prevention than a system catching 10% of A's crimes and 10% of B's crimes (good!)
2. Enforce a societal belief that A is intrinsically better than B (bad!)
3. Disproportionately burden children, families, and communities in B than A, causing them to indeed perform worse in everything than those in A (bad!)
4. As a result of 2 & 3 it is not a stretch to say simply causing B to do more actual crime (potentially negating point 1 entirely)
If you believe that crime enforcement is not for the sake of vengeance but instead something done to improve the well-being, safety, and happiness of citizens, you may see that inequality=bad just as crime=bad. How to best solve this trolley problem is complicated but it's important that people are aware that it is complicated before firing off an answer.
Most crime is intra-racial. Group A will do better over time, with fewer people becoming victims of crime, if it is subject to better policing, because more of the malevolent actors within it will be incarcerated or deterred from engaging in (mostly intra-racial) predatory behavior.
That is only part of the equation. You may be removing more malevolent actors in the immediate short term, but depending on how that policing is done, you might also be creating more malevolent actors too. Overpolicing a group can create distrust between the community and the police. Once you feel the system does not care about you or treats you unfairly, there is little reason for why you should care about it. And if P(Caught|Group X) != P(Caught), the system is treating you unfairly.
I would argue we as society don't want crime to stop simply for the sake of crime stopping (or for prison labor), but ultimately because we want to feel happy and safe from harm and unjust treatment. The systems we design need to factor in the humanness of the police and their communities and make sure they are not set up in a way that loses sight of that bigger picture.
See, what you've said is precisely "structural bias against a minority", or "systemic injustice". Then again, the elites are, technically, also a minority as well, and we all know how well letting their crimes slide works out for the rest of the society.
it is FALSELY unidentifying people, which makes the harware, software, sales, implimentation of the whole system a criminal enterprise, which it is. Kudos to the police for rejecting this racist biggoted unjust criminal software implimentation.
> the system was more likely to correctly identify men than women and it was “statistically significantly more likely to correctly identify black participants than participants from other ethnic groups”
Technology has moved on a lot no doubt, however, studies were finding the opposite (and with order of magnitude errors) as recently as 2020 with a lazy google literature search
> these algorithms were found to be between 10 and 100 times more likely to misidentify a Black or East Asian face than a white face
Given that these are machine learning algorithms their performance will very much depend on the training dataset. So it is probably not (just) that “technology has moved on a lot”, but that the engineers working on it curated new training sets. It is not entirely unreasonable to think that they too read the paper you are talking about and made measures in an attempt to correct for the effect.
Cameras are much better now than they were before, all the same everytime you get a "have you seen this person" its a grainy pixelated flat image from CCTV that could easily be one of thousands of people.
They are concerned about a higher rate of false positives (therefore a higher rate of incorrect arrests etc.) of white people (and probably Asians etc.) and women. This is also discriminatory.
People forget equality law runs both says. it is illegal to discriminate against men, whites, or heterosexuals just as it is to discriminate against women, non-whites or gays.
> the system was more likely to correctly identify men than women and it was “statistically significantly more likely to correctly identify black participants than participants from other ethnic groups”.
I am genuinely unsure what's going on.
My understanding of the article is that the system is problematic because it is more likely to correctly identify black people than "other ethnic groups". Is that right?
It's problematic for use in Essex as it works best for a small minority of the Essex population and has a much higher error rate for a typical sample of the Essex community.
Adendum: Essex Ethnicity breakdown- 85.1% White British · 5.2% Other White · 3.7% Asian · 2.5% Black · 2.4% Mixed · 1.1% Other · (2021).
Essentially (with made up numbers): 100 men on a high street, 4 of which are on a watch-list; 2 of which are black. Both black guys get identified, only one of the others does.
As a case study, the Trump admin has done all those things (except the littering I guess) so I would say less likely since none of them have gone to prison.
The poor and marginalized tend to be incarcerated at much higher rates for lesser crimes than the richer and/or powerful whose crimes are much broader and more impactful on society.
The system in question in Essex is broken because it penalizes one race at higher rates than another race which commits the same crimes.
The problem is that a likely outcome is that they will arrest two white men who are not the ones on the white list. That is discriminatory, at least if it keeps happening so that you get a higher rate of wrongful arrests of one group.
“statistically significantly more likely to correctly identify black participants than participants from other ethnic groups”.
Great. Wasn’t the problem before always that it couldn’t correctly identify non-white people?
It does it accurately now. That is somehow also a problem? It should make more mistakes?
Former author of one of the top 5 facial recognition servers in the world for multiple years running, here's what's going on: the industry has solved this issue, but the potential clients are seeking the lowest bidder, and picking the newer companies, the nepostically created not really players but well connected, and those companies have terrible implementations. This is not a case of the technology not there yet, we solved all these racial bias issues 10 years ago. But new companies with new training sets and new ML engineers that do not know any of the industry's history are now landing contracts with terrible quality models, but well connected sales channels.
This study finds a higher rate of correct identification for black people than for other ethnic groups, whereas a few years ago the problem seemed to be that the software was less effective at identifying black people.
Do you have some insight about why this reversal might have occurred?
To have a high quality facial recognition system it needs to include every possible combination of ethnicity, in addition to all of those they each need to include variations of daylight, of dappled light, of partial obscuring, night time illumination, across every variation of season, variations of expression and face angle, across variations of weather, variations of distance, across variations of things placed on a person's face, and then across all kinds of variations of video compression. All these face image variations in the training set enable the trained model to find and track the features that persist through all these variations. In truth it requires hundreds of millions of facial images to create an accurate facial recognition system. Most new companies and many that have been around for respectable periods are not realizing how much data collection, annotation and additional variation creation it requires for a high quality FR training set. The company I worked at spent 20 years collecting laser scans of real people to then create the augmented real person data set with several hundred million faces.
Your problem is evident in your question. What does “black” mean? It’s entirely subjective. Dwayne Johnson? Liv Tyler? Nelson Mandela? Barack Obama? Mariah Carey?
This is a semantic issue. Ethnic groups are constructs. A system which misidentifies people identifies all people poorly.
It doesn’t track across regions either. People labelled “white” by law in some countries (Brazil, South Africa, etc) would be classified as “black” elsewhere.
In England, the example here, we do not classify people the way the Us does, with its history of “one drop” politics. Many British people considered “white” are “black” in the US.
There is no scientifically valid way of defining who counts as “black” so any discussion of tuning a system based on this definition is a disaster.
Even the people commenting are talking about different groups based on their own culture and prejudices.
> Many British people considered “white” are “black” in the US.
I'm also British, can you give an example of that? A minor celebrity/TV personality say?
To the extent you have a point though I think it's irrelevant anyway—they paused the programme because they found, according to whatever definition they measured with, it had that skew.
How recently? We had a home security camera and every time our (Black) son walked up to the door, the camera would classify him as an “animal”. This was as recently as 2022
In the other direction, my camera regularly identifies cats, crows, and shadows as people. I think recognition in security cameras has a very long way to go.
My friend recently sent me a screenshot of her Arlo detecting her backside as a 'vehicle'.
[dead]
So just like the rest of government IT then.
Can you link the peer-reviewed citations for having solved the racial bias issues, in anything but specific bespoke cases?
Frankly, I'm skeptical, but I'm willing to be convinced by reputable evidence.
This is actually more (socially/ethically/philosophically) interesting than one might assume from the headline: it's not false positives, it's that it's more effective (correctly identifies someone is on a watch-list) for one group than another within a protected characteristic.
So essentially they're pausing the use of it because it works too well for group A / not well enough for group B, potentially leading to disproportionate (albeit correct) arrests of group A.
Absolutly impossible to condone further structural bias against a minority, and just ignore the free "white pass" built into the software, and esspecialy troubling that it passes white women, the most. The only possible action is to reject and dissable any system with a racial bias, investigate how such a thing happened, with a very pointy look for intent on the part of the vendors, who would then qualify for bieng housed in one of his majestys facilities for persons such as these.
If it’s not falsely identifying people I don’t see a problem at all. If it’s identifying criminals every criminal should be caught
If you start with hypothetical demographic groups A and B that are for all intents and purposes exactly identical, but you implement a system such that if A commits a crime they have a 10% chance of being caught and if B commits a crime they have a 50% chance of being caught, you will achieve the following:
1. More short-term crime prevention than a system catching 10% of A's crimes and 10% of B's crimes (good!)
2. Enforce a societal belief that A is intrinsically better than B (bad!)
3. Disproportionately burden children, families, and communities in B than A, causing them to indeed perform worse in everything than those in A (bad!)
4. As a result of 2 & 3 it is not a stretch to say simply causing B to do more actual crime (potentially negating point 1 entirely)
If you believe that crime enforcement is not for the sake of vengeance but instead something done to improve the well-being, safety, and happiness of citizens, you may see that inequality=bad just as crime=bad. How to best solve this trolley problem is complicated but it's important that people are aware that it is complicated before firing off an answer.
Most crime is intra-racial. Group A will do better over time, with fewer people becoming victims of crime, if it is subject to better policing, because more of the malevolent actors within it will be incarcerated or deterred from engaging in (mostly intra-racial) predatory behavior.
That is only part of the equation. You may be removing more malevolent actors in the immediate short term, but depending on how that policing is done, you might also be creating more malevolent actors too. Overpolicing a group can create distrust between the community and the police. Once you feel the system does not care about you or treats you unfairly, there is little reason for why you should care about it. And if P(Caught|Group X) != P(Caught), the system is treating you unfairly.
I would argue we as society don't want crime to stop simply for the sake of crime stopping (or for prison labor), but ultimately because we want to feel happy and safe from harm and unjust treatment. The systems we design need to factor in the humanness of the police and their communities and make sure they are not set up in a way that loses sight of that bigger picture.
If there was a cop that only every arrested black people and always let off any white person, would you agree that there is a problem with that cop?
Selective enforcement has been used historically to persecute certain minorities and that's not acceptable in a functioning society.
See, what you've said is precisely "structural bias against a minority", or "systemic injustice". Then again, the elites are, technically, also a minority as well, and we all know how well letting their crimes slide works out for the rest of the society.
it is FALSELY unidentifying people, which makes the harware, software, sales, implimentation of the whole system a criminal enterprise, which it is. Kudos to the police for rejecting this racist biggoted unjust criminal software implimentation.
> the system was more likely to correctly identify men than women and it was “statistically significantly more likely to correctly identify black participants than participants from other ethnic groups”
Technology has moved on a lot no doubt, however, studies were finding the opposite (and with order of magnitude errors) as recently as 2020 with a lazy google literature search
> these algorithms were found to be between 10 and 100 times more likely to misidentify a Black or East Asian face than a white face
https://jolt.law.harvard.edu/digest/why-racial-bias-is-preva...
Given that these are machine learning algorithms their performance will very much depend on the training dataset. So it is probably not (just) that “technology has moved on a lot”, but that the engineers working on it curated new training sets. It is not entirely unreasonable to think that they too read the paper you are talking about and made measures in an attempt to correct for the effect.
Maybe, or there might be qualities around say contrast and the physical cameras themselves that build this in.
But it doesn't seem like these physical factors would have reversed the effect between 2020 and now. Unless you can think of some?
Cameras are much better now than they were before, all the same everytime you get a "have you seen this person" its a grainy pixelated flat image from CCTV that could easily be one of thousands of people.
then in theory, the dataset can be changed to make model error rates "fair" for all intersections of race, gender, age etc.
> more likely to correctly identify men than women.
> more likely to correctly identify black participants than participants from other ethnic groups.
> AI surveillance that is experimental, untested, inaccurate or potentially biased has no place on our streets.
I wonder if they're more worried about putting too many men in prison or too many black people.
They are concerned about a higher rate of false positives (therefore a higher rate of incorrect arrests etc.) of white people (and probably Asians etc.) and women. This is also discriminatory.
People forget equality law runs both says. it is illegal to discriminate against men, whites, or heterosexuals just as it is to discriminate against women, non-whites or gays.
Neither, they're worried about bad rep.
> the system was more likely to correctly identify men than women and it was “statistically significantly more likely to correctly identify black participants than participants from other ethnic groups”.
I am genuinely unsure what's going on.
My understanding of the article is that the system is problematic because it is more likely to correctly identify black people than "other ethnic groups". Is that right?
It's problematic for use in Essex as it works best for a small minority of the Essex population and has a much higher error rate for a typical sample of the Essex community.
Adendum: Essex Ethnicity breakdown- 85.1% White British · 5.2% Other White · 3.7% Asian · 2.5% Black · 2.4% Mixed · 1.1% Other · (2021).
from: https://en.wikipedia.org/wiki/Essex
ie: most accurate (however acccurate that is) for the men of 2.5% of the regions population
Not so accurate for 98.75% of the regions population.
Essentially (with made up numbers): 100 men on a high street, 4 of which are on a watch-list; 2 of which are black. Both black guys get identified, only one of the others does.
Ditto men vs. women, mutatis mutandis.
So it should be improved but sounds like it’s just catching criminals who need to be caught no?
https://en.wikipedia.org/wiki/Selective_enforcement
Right so the answer is to improve enforcement not disable it and let more crime go unsanswred
[flagged]
As a case study, the Trump admin has done all those things (except the littering I guess) so I would say less likely since none of them have gone to prison.
The poor and marginalized tend to be incarcerated at much higher rates for lesser crimes than the richer and/or powerful whose crimes are much broader and more impactful on society.
The system in question in Essex is broken because it penalizes one race at higher rates than another race which commits the same crimes.
This is just a ridiculous attempt to bring Trump into an unrelated conversation.
No, it was to show how unreliable those criteria are as OP pulled the conversation into racial/cultural determinism.
Feel free to tackle the substance of any of my points.
The problem is that a likely outcome is that they will arrest two white men who are not the ones on the white list. That is discriminatory, at least if it keeps happening so that you get a higher rate of wrongful arrests of one group.
This seems like an easy problem to solve - when the system informs you of a black criminal, just roll dice to ignore them and let them get away.
What do you propose is done when the system doesn't inform you of a white criminal?
If the suspect is Black, the software should automatically return zero matches in 30% of cases. Problem solved.
antimemetics (look one more time)
“statistically significantly more likely to correctly identify black participants than participants from other ethnic groups”.
Great. Wasn’t the problem before always that it couldn’t correctly identify non-white people? It does it accurately now. That is somehow also a problem? It should make more mistakes?
Correlation does not indicate causation
Alternative headlines:
Essex police, well aware of all the issues before using it, pause use until expected bad publicity dies down
Or
Essex police chosen as force to take some flack for the issues while other forces steam ahead