Client side scanning may cost more than it delivers

On May 11, 2022, the EU will publicize a proposal for a law on mandatory chat control. The European Commission wants all providers of email, chat and messaging services to search for suspicious messages in a fully automated way and forward them to the police in the fight against child pornography.

History

In 2020, the European Commission initiated temporary legislation which allows the searching of all private chats, messages, and emails for illegal depictions of minors and attempted initiation of contact with minors. This allows the providers of Facebook Messenger, Gmail, et al, to scan every message for suspicious text and images.

A majority of the Members of the European Parliament adopted the chat control regulation on July 6, 2021, allowing providers to scan communications voluntarily. So far, only some unencrypted US services such as Gmail, Meta/Facebook Messenger, and X-Box apply chat control voluntarily.

The European Commission announced that it will propose follow-up legislation that will make the use of chat control mandatory for all email and messenger providers. This legislation will be presented tomorrow, May 11, 2022 and would also apply to communications services that are end-to-end (E2E) encrypted.

It is important to note that the European Parliament has already pointed out that even voluntary scanning, which is currently permitted by the short-term law, lacks a legal basis and would probably be invalidated if it were taken to court.

Privacy advocates

Needless to say that many privacy advocates are ready to storm the barricades to prevent this law from being approved. Not only does this violate our basic human right to privacy, but encrypted messaging has been a boon to activists, dissidents, journalists, whistleblowers, and marginalized groups around the world.

Privacy advocates argue it brings the EU closer to the surveillance state that many see in other countries and that is a frightful image. It is also a step back when it comes to cybersecurity. What do we call software that eavesdrops on what we are doing on our devices and sends it to a third party? Spyware! And what happens to servers that accumulate large amounts of private data? They become targets for cybercriminals.

The goal

Similar developments are taking place in the US and the supporting narrative has expanded from domestic terrorism to other illegal content and activity, such as child sexual exploitation and abuse, terrorism, foreign adversaries‚ and attempts to undermine democratic values and institutions.

What most, if not all, of these activities have in common is that you usually won’t see the criminals using the same platforms as those of us that want to stay in touch with friends and relatives. They are already conducting their “business” in illegal marketplaces on the Dark Web, or they are using encrypted phone services.

Client side scanning

What does client side scanning mean exactly, some may wonder. Client side scanning broadly refers to systems that scan message contents for matches against a database of objectionable content before the message is sent to the intended recipient.

In this case, it means that the EU wants to force all providers of email, messaging, and chat services to comprehensively search all private messages, even in the absence of any suspicion. That makes the contents of messages no longer private between the sender and receiver, and client-side scanning breaks the E2E encryption trust model.

Pitfalls

As we have seen in the US, once the trend has been set, the number of targets can quickly expand from child abuse to other areas. As some of the privacy advocates noted, it’s a slippery slope.

It’s building a database of objectionable content. Given the amount of data you will need something to make a first selection. Machine Learning and Artificial Intelligence will undoubtedly be put to use. These systems can be manipulated and led astray, where static databases are too easy to circumvent.

False positives are a risk to keep in mind. What happens to a sender, or receiver for that matter, that gets tied to several flagged messages? I’m asking for me. Once an interest in cybercrime, vulnerabilities, and other related areas get added to the areas of government interest, my search queries alone would be enough to get me in trouble. On a lighter note, how hard will it be to explain that autocorrect is responsible for your message getting flagged? And will my reputation accompany me on my travels? In other words, will the US know if the EU thinks I’m involved in something shady?

The complexity of breaking the chain of E2E encryption could also limit the reliability of a communications system, and potentially stop legitimate messages from reaching their intended destinations.

So far, for every method that has been devised to limit the amount of private data that gets shared and scrutinized after the first selection, a downside has been brought up. And the stage in which these messages are unencrypted to be reviewed offers a target area where criminals can exfiltrate a lot of valuable information.

Since client-side scanning technologies may represent the most powerful surveillance system ever imagined, it is imperative that we find a way to make them abuse-resistant and auditable before we decide to start using them. Failures from the past have taught us that it’s often the other way around. We learn from our mistakes, but how costly are they?

It is also important to realize that the criminals we are trying to catch will simply move away from the platforms we decide to subject to client side scanning. So in the end, we are monitoring the communications of innocent citizens, for what exactly?