Consultation of the Dutch Data Protection Authority to acquire insight into prohibited AI systems in The Netherlands

• The Dutch Data Protection Authority (AP) launched a consultation on banned AI systems despite the lack of a formal supervisory role.
• There are concerns about possible misuse of sensitive information by the AP and risk of "naming and shaming."
• Opinion: the AP should have postponed its consultation until the European Commission publishes the guidelines and its role has become more clear.

With the enactment of the AI Act[1] on 2 August 2024, the legal landscape around artificial intelligence (“AI”) has been identified a little further, although questions about the Act’s actual application keep popping up. Clearly, the Dutch Data Protection Authority (“AP”) is also still searching, as appears from its “first call for input” made – in its capacity of Department for the Coordination of Algorithmic Oversight (“DCA”) – on 27 September 2024 to stakeholders (“citizens, governments, companies and other organisations”) in connection with prohibited AI systems.[2]

Article 5 of the AI Act contains an exhaustive list of AI systems that have been declared prohibited – given the unacceptable risk they entail – and must be taken off the market and/or out of service ultimately on 2 February 2025. One of these bans concerns: 

The AP’s consultation focuses on this category of systems. Under the AI Act, the national market surveillance authority is charged with enforcing the law against this category. It cannot simply take action against all AI systems that have 'something fishy about them’; the AI Act requires that the national market surveillance authority must have “sufficient reason” “to consider an AI system to present a risk”. Only in that case can the system be evaluated to assess whether it meets the requirements from the AI Act (and is therefore not prohibited). It follows expressly from the AI Act whether there is a risk.[3] The national market surveillance authority is explicitly not free to establish the criteria for this. If the authority establishes that the risky system does not meet the requirements, and no appropriate corrective measures are taken to “bring” the AI system “into compliance”, it is up to the national market surveillance authority to take the system off the market or to recall it. 

Remarkably, in its consultation the AP seems to anticipate a designation as national market surveillance authority under the AI Act for prohibited AI systems, whereas no such designation exists yet; the Ministry of Economic Affairs and the Ministry of Legal Protection still have time to decide on this until 2 August 2025. Although it is imaginable that the AP will be assigned this task, it cannot be completely excluded that another authority or an authority to be newly established will be designated for this purpose. 

Another remarkable aspect is the short turnaround time the AP has attached to its call. The stakeholders must submit their input by 17 November 2024. This means that as of that date, the AP is expected to have very sensitive (personal) data in its possession, whereas neither this task nor this power arises from the AI Act and/or from the AP’s role as DCA[4] As the supervisory authority for data processing, the AP in particular could have been expected to act with greater care in the processing of data.

This is even more important now that a risk of naming and shaming is looming (for example by an angry customer, interest group, or competitor) as a result of the consultation. The AP requests stakeholders to report concrete examples of existing AI systems that may qualify as prohibited, by requiring answers to questions like “Can you give examples of systems that are AI-enabled and that (may) lead to manipulative or deceptive and/or exploitative practices?” and “Do you know of examples of AI systems that use subliminal techniques?”.

The question arises what the AP intends to do with the insights acquired. Will they be stored in a secure, closed environment, until there is clarity on the AP’s role as a national market surveillance authority? And will they then be used to detect concrete, possibly prohibited AI systems and investigate these (in other words, is this a fishing expedition in which the AP hopes to ‘catch’ something, using its consultation as a metaphorical fishing rod)? Even in a scenario where insights will only be used when the AP will have been designated as supervisory authority under the AI Act, the legitimacy of the current call is doubtful. Is this really the most appropriate way to identify prohibited AI systems that are being operated in a Member State? In the AI Act, not a word is mentioned on this method for enforcing the law against prohibited AI systems at a national level.

These questions are not answered in the call. The AP does indicate that it will publish a “summary and appreciation of the input” through generic terms, after which the input will be deleted. However, it is unknown when the AP intends to issue this publication: in the near future, or only after its possible designation as a supervisory authority? And how will function creep be prevented – a situation in which the insights acquired are (Heaven forbid) used for enforcement purposes too? This question is even more pressing since the AP emphasizes in its call that organisations or groups may be mentioned explicitly in its report if desired.

Moreover, the AP may share its “acquired insights” with other (European) AI supervisory authorities. Will this only be done to share knowledge, or will this information be used on a wider scale for enforcement purposes? The call does not answer this question. 

The AP also states in its call that it will use the input in any case to lay a preparatory foundation for further explanation of the prohibitions in the AI Act. Although the AP’s attempt at transparency about its outlook on the enforcement of prohibited AI systems is laudable (partly from a legal certainty perspective), the AP – should it be designated as a market surveillance authority for prohibited AI systems – is not entitled to the task of establishing a general review framework; the AI Act assigns the task of drafting guidelines on AI systems (or their qualification) for prohibited practices expressly to the European Commission. Indeed this is not illogical, because it creates a strong extent of harmonisation for all Member States on this point, and limits the risk of deviating standards, due to which AI systems must be presented differently across Member States in order to avoid being qualified as prohibited.

All things considered, the AP would have done better not to issue its “first call for input”, but instead to have sat back and waited for the process of being designated as market surveillance authority for prohibited AI systems, while giving the European Commission the chance to prepare guidelines on the interpretation of the qualification of prohibited AI systems. By issuing this consultation, the AP has triggered more questions than it intended to have answered.

[1] Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act).

[2] https://www.autoriteitpersoonsgegevens.nl/documenten/oproep-voor-input-verbod-op-manipulatieve-en-uitbuitende-ai-systemen

[3] Reference is made to Article 3 paragraph 19 of Regulation (EU) 2019/1020. In brief, this concerns risks of health, safety or fundamental rights of persons.

[4] Annex with Parliamentary Paper 35 788, no. 77; Parliamentary Paper 26 643, no. 953.