AI Factories represent the European Union’s key strategic initiative to position Europe at the forefront of AI innovation. They aim to establish advanced ecosystems that integrate high-performance computing, top talent, cutting-edge skills, and advanced services—all specifically tailored to the development and training of large AI models[1]. Additionally, it is widely recognized that access to vast amounts of high-quality, purpose-specific data is essential to ensure the correctness, accuracy, and effectiveness of AI models. Consequently, seamless access to data stands out as a fundamental requirement for AI Factories

And this is precisely one of the primary goals of European common data spaces: to provide access to private industrial data under well-defined conditions and usage policies, offering a major advantage to AI innovators and enhancing the mission of AI Factories

While having both instruments in place is essential, it is not enough. AI Factories and data spaces operate as distinct ecosystems, each with its own governance rules, strategic objectives, business models, and technical requirements. Although they share some commonalities, seamlessly connecting them remains a complex challenge

At BDVA, we strongly believe we can support address this challenge, as we are at the heart of a vibrant ecosystem that brings together data, data spaces, industrial AI, and HPC. To this end, we have assembled a group of experts within our community to discuss the issue and explore potential solutions. And we have compiled insights from these discussions into the document: ‘AI Factories and the Data Challenge: Access, Acquisition, and Usage of Data – Connection to Data Spaces.’ This guide is designed to support those working on bridging AI Factories and data spaces.

Figure 1. Mutual benefits from AI Factories and Data Spaces collaboration

The document explores the mutual benefit both for AI Factories and data spaces resulting from this collaboration. EU Data spaces bring together large industrial communities, that have already joined forces and established collaborations to provide and share data in order to generate value for their businesses. AI Factories can benefit from this on the onboarding of industrial actors that otherwise are not so much aware of the availability of HPC services and infrastructures. Additionally, data spaces make available to AI Factories sector-specific industrial and high-quality data and controlled and well-governed environments to ensure proper governance and compliance mechanisms for these data.  Finally, data spaces can also provide the flexibility needed to have the data prepared to the level required by AI practitioners. On the other way around, AI Factories provide a well-defined purpose for EU Data Spaces, positioning themselves as powerful and relevant use cases, and potentially becoming some of their first major customers. AI Factories can also complement the offerings of data spaces by providing access to HPC infrastructures when required by the business model or use cases of the data space, along with the necessary HPC services and support.

A whole section is devoted to outline the different scenarios under which AI Factories and data spaces can collaborate, considering an increasing level of integration, which would result in higher value bit also additional challenges. Starting by the simplest scenario with AI Factory and data spaces as separate entities collaborating in an “ad-hoc” fashion, the inclusion in the picture of intermediaries that support the AI Factory to find and contact the most appropriate data space and developing critical capabilities such as data connection, trusted data transactions, and governance frameworks, would be key at early stages. AI Factory can become a participant in the Data Space subject to the same rules, governance, and protocols as any other participant. But the data space can also define a new specific role for the AI Factory, with unique responsibilities and privileges tailored to its function. A much more elaborated scenario would consist of a federation between the AI Factory and the data space, where they operate in a collaborative relationship while maintaining a degree of autonomy. And eventually this may lead to a fully automated and integrated model, where they operate as a seamlessly connected system, leveraging technology to enable real-time data exchange, resource sharing, and automated workflows.

Figure 2. Different scenarios for AI Factories and Data Spaces connection

The document also provides guidelines for a “quick start” in the connection between AI Factories and data spaces, starting from governance authorities agree on common rules under a common minimum governance framework, identify technical enablers for data exchange and retrieval under an optimal connectivity scheme, and establish clear business conditions for this collaboration (including pricing models, such as regular fees, pay-per-use, or revenue-sharing arrangements for data). This last point is extended to a dedicated section with reflections on potential cost structures and revenue models to ensure a sustainable partnership.

Additional aspects considered in the paper are how the entire data lifecycle is shared by the AI Factory and the data space, hence shared data governance schemes should be in place that consider data preparation, data quality, data provenance and data liability, also according to current regulations. Connectivity (guarantying low latency) and interoperability emerge as key aspects to guarantee seamless integration. And the collaborative scheme should also provide, on an end-to-end basis, integrity, inviolability and privacy of the acquired data, when going through the different processing and computational processes, and when delivered to end users

The paper concludes with some recommendations on defining a unified approach for AI Factories and EU Data Spaces, realize a federation of data spaces to provide a single-entry point to the whole EU data space ecosystem and leverage on the HPC Federation to align, connect, and integrate AI Factories with other ecosystems, continue learning based on real hands-on experiences of the first AI Factories, consider the inclusion of experienced third parties that can serve as trusted intermediaries, establish demonstrator or lighthouse projects showcasing practical implementations of AI Factory connections to Data Spaces, and Explore new and innovative business models and opportunities between involving all stakeholders from both sides. Everything monitored by a Working Group that includes representatives from the European Commission, EuroHPC JU, AI experts from supercomputing centers, and data spaces utilizing HPC.

BDVA plans to continue this work with in-depth discussions on all these topics with experts from our community and relevant stakeholders, leveraging the work of the first AI Factories already running and paving the way towards a seamless connection between AI Factories and Data Spaces.

[1] The first two calls for the selection of entities to establish AI Factories were published by the EuroHPC JU[1] last 10^th September 2024, and the first seven proposals to establish and operate those AI Factories have already been selected

The post AI Factories and the data challenge: access, acquisition and usage of data. Connection to data spaces appeared first on BDV Big Data Value Association.