AI in your own data centre or an EU cloud: how to keep control of your data

“We would like to use AI, but our data must not leave the building.” Medium-sized companies say this often, and they are right to. Personal data is subject to the GDPR, which allows transfers to countries outside the EU only under the conditions of its Chapter V. On top of that come trade secrets and the commitments you have made to your own customers.

That is no reason to go without AI. It is a reason to decide exactly where it runs.

1. 1Your own cloud account in an EU region: the language models come from your cloud, for example through AWS Bedrock or Azure, and run in your account, in a region you choose, with your keys and your logging. Which models are available in which region is up to each provider; this is checked beforehand.
2. 2Your own data centre or hardware: open models are self-hosted, with model serving for example through Triton and, where several applications share one GPU, with separated compute and memory, for example through NVIDIA MIG. The data never leaves your network.
3. 3A mixed setup: documents, search index and permission check stay with you, and only the wording of the answer is done by a model in your EU cloud. It sees nothing but the passages the permission check has released for that request.

1. 1The location: everything runs in your environment, and you decide before any data moves which services are connected.
2. 2The choice of model: routing across several models lets you switch the language model without rebuilding the system. You are not tied to one vendor.
3. 3The permissions: they are taken over from your existing systems instead of being reinvented, and checked in front of the model.
4. 4The evidence: every query is logged traceably, and the log stays with you.
5. 5The code: it lives in your repository with its documentation, so you can also run the system yourself.

The right setup depends on a few questions. What data is searched: personal, confidential or public? What have you promised your customers? What infrastructure exists, and who runs it? How fast must an answer arrive, and what may it cost?

The last two questions can only be answered honestly with measurement. That is why latency, cost per answer and quality are measured for every request from the start. Only then can you tell whether a self-hosted model pays off or whether a model from the EU cloud is the better choice.

Technology can make data sovereignty possible; it does not replace a legal review. Involve your data protection officer early. AI systems are also subject to the EU AI Act (Regulation (EU) 2024/1689). It entered into force in August 2024; its obligations apply in stages and depend on the risk of the use case.

This article describes technical options and is not legal advice.