SoBigData

Studying social phenomena through large datasets requires researchers to consider several factors simultaneously: data, analytical methods, computing capacity and appropriate privacy and data protection arrangements. If these are assembled individually for each project, it can create substantial barriers for researchers who are not data scientists. SoBigData treats this process as an infrastructure service. Its virtual research environments combine catalogued datasets and methods with computational services and environments, such as Jupyter, RStudio, and Galaxy. Researchers can import or select analytical methods, prototype analyses in notebooks, or construct and execute workflows using shared computing resources. The Social Mining Analytics Engine supports the repeated and scheduled execution of analyses and automatically records provenance information. Researchers retain control of the research design and interpretation, while the infrastructure provides an environment in which computational social science analyses can be developed, run, modified and repeated.

SoBigData is a distributed research infrastructure connecting more than 25 institutions across six core countries and serving over 15,000 users. Its strength lies in the network as much as in its computing capacity: participating institutions contribute data, methods, expertise and services that researchers can access through a shared platform. Developed with support from the European Union’s Horizon 2020 programme, SoBigData was included in the 2021 ESFRI Roadmap. It brings together expertise in computer science, social science, law and ethics. The infrastructure’s hub provides 14,772 CPU cores, 24 NVIDIA A100 GPUs, 3,572 TB of storage and 102 TB of RAM, and connects to national nodes that contribute further resources. Its datasets draw on institutional holdings and data collected for specific research purposes using dedicated tools. A structured integration process enables contributors to share datasets and methods across the network. Metadata, submission materials and model documentation make these outputs discoverable and, where fully integrated, executable by researchers beyond the teams that developed them.
SoBigData is both a provider of digital research services and a community organised around thematic Research Spaces: Demography; Economy and Finance; Disaster Response and Recovery; Health Studies; Pervasive Intelligence; Social Impact of AI and Explainable Machine Learning; Societal Debates and Misinformation; and Sustainable Cities for Citizens. Through its Gateway and virtual research environments, users can form working groups, connect with one another and seek support from the infrastructure team. In addition, mobility programmes (TNA) help researchers collaborate across institutions, a Board of Operational Ethics and Legality (BOEL) advises projects and communities on legal, ethical and societal questions, and its SoBigData Academy offers digital training resources for responsible data analysis and AI. Together, these services bring together a wide range of stakeholders and sustain the multidisciplinary community at the heart of SoBigData RI.

SoBigData reduces the technical and organisational barriers to computational social science research by providing access to data, analytical methods and computing resources via shared infrastructure. Methods developed by individual research groups can be documented in the shared catalogue and, at a higher level of integration, made available for experiment prototyping or execution by other researchers. This promotes the reuse of existing computational work and reduces the need for each project to assemble its own analytical environment. Persistent identifiers, metadata, standard formats and implementation guidance are requested from contributors and provided to users, and SoBigData components can be used across other infrastructures.

The infrastructure also facilitates more transparent and reproducible computational workflows. The Social Mining Analytics Engine automatically records provenance information and supports repeated and scheduled execution. Integrated workflow environments allow analytical steps and their outputs to be combined and revisited.

Responsible use is supported by an ongoing ethical and legal service. Users are first requested to take a MOOC on ethics and legislation, and the ethics board evaluates requests to use the infrastructure. Researchers can seek guidance on methodological, legal, and societal issues, and cases requiring formal ethical review can be identified and referred accordingly. This embeds responsible data considerations into the infrastructure available to researchers, providing a model that can be transferred to other research infrastructures that combine AI methods, sensitive data, and shared computational services.

Official Website: http://www.sobigdata.eu

Research Infrastructure Catalogue & Workspace: https://sobigdata.d4science.org/