European Space Agency: Data Factory, the Single Source of Truth

With the Data Factory, the European Space Agency (ESA) has created a single source of truth that makes its data and project situation transparent, its processes more efficient and decisions more sustainable.

Initial request was to migrate existing applications to modern technologies – to the cloud 

Complete new development using current technologies in addition to migration 

Analysis and migration of large data sets from a heterogeneous landscape 

Standardization of the data for a common „language“

Entire project is based on Azure technologies 

Creation of a central Data Factory using Microsoft Azure Cloud 

Azure Synapse for the database and Azure Web Apps for the frontend 

Integration of Power BI dashboards with Azure Web Apps

Creation of a single, reliable source of truth through the Data Factory 

Establishment of a „common language“ across all directorates and projects by identifying the object model and ensuring transparent data labeling for stakeholders 

Faster project decisions and improved performance with real-time Power BI dashboards 

Foundation for further automation and AI projects

The European Space Agency (ESA) intends to take a significant step towards the digitalization of its data by replacing its exis ting on-premise databases. To find the right partner for this en deavor, the ESA turned to Microsoft, as it was immediately clear that the technologies should be centered around the Microsoft ecosystem. Out of ten service providers from across Europe, the ESA selected novaCapta as the most promising partner. Initial ly, the plan was to simply migrate the existing data to the cloud to make the current database more future-oriented. However, after thorough consideration, ESA and novaCapta decided on a more comprehensive approach: alongside the introduction of the Azure Cloud and the associated data migration, the Data Factory was created. 

„The approach and expertise of novaCapta convinced us. We jointly decided to abandon the original plan of a ‚simple‘ mi gration and opted to completely overhaul our data structure. Our goal was to enhance our performance and accelerate our processes. This was reflected in novaCapta‘s concept,“ explains Bernhard Isemann, Senior Data Manager & Head of ESA Data Factory, ESA.

The first step involved identifying how many different data warehouses the ESA had across Europe and analyzing this he terogeneous landscape. To better understand and analyze the processes, data, and objects, and to define the stakeholders‘ requirements, two-day workshops were held. Understanding the project approach and the user journey – what data is nee ded – was essential not only for defining a common language in the Data Factory, but also for engaging all relevant stakehol ders and establishing a transparent process.

With a solid understanding of the data, the new hybrid struc ture (on-premises data servers and Azure Cloud) was imple mented. Then, using Azure Synapse – an analytics service that combines data warehousing with big data analytics – the data was retrieved and processed using Power BI. Naming the data in a traceable manner was a key aspect to establishing a unified language across all directorates and projects. Next, the first real-time dashboards were created with Power BI, using commercial data such as financial and contract figu res. These dashboards were integrated into Azure Web Apps through Power BI embedded and made available to all relevant stakeholders. The advantage of Power Platform components like Power BI is that the ESA can independently create reports in the long term, deploy them through the established mecha nism, and keep the dashboard alive without relying on external service providers.

“With the Data Factory, we no longer need to question where the data comes from or whether the figures are correct. It gives us a single source of truth on which we can directly proceed to project planning and discussions. This saves us time and ensu res that all project participants share the same understanding,” explains Bernhard Isemann. The overall system, consisting of databases, Microsoft Azure, and Power BI, is called the ESA Data Factory. With this future oriented data factory, the ESA can now take further steps to wards artificial intelligence, enhancing its performance and the scope of data analysis. In the long term, dynamic dashboards could be created that, with AI integration, operate independ ently, interpreting and answering prompts. Further projects around the Data Factory are already in planning, including pro fessionalizing the Factory’s services and conducting security hardening.

Download Case Study European Space Agency als PDF (DE)

Download Case Study European Space Agency als PDF (EN)