Updated at: 25 August 2026 (Javier Garcia-Bernardo, Laura Boeschoten, and Thijs Carrière)

Group project guidelines

The group project is a central part of the course. The project is designed to give you hands-on experience with digital trace data, and to apply the knowledge you have gained in the course to a real-world problem. In the practical you will work in groups to collect text data, label it using Natural Language Processing models, discuss the errors and biases that you encounter in the data, and interpret the results in light of the errors.

The slides used in the first lab are available here.

Feedback moments during workshops

Feedback on the group project will take place during the regular workshop sessions. These moments combine peer review, short teacher check-ins, and time to continue working on the project.

During peer review, groups will present their progress to another group. Each peer-review round will take about 30 minutes. Come prepared to explain your research question, data collection strategy, expected representation and measurement errors, and the main decisions you still need feedback on.

Before the session, each group should prepare 2-3 specific questions they want feedback on. Prepare a few (fewer than 5) slides showing your progress and indicating the topics each group member worked on; every slide should list its contributors. If there are problems with collaboration or task division, come early and talk to the teacher before the peer-review round starts.

After receiving feedback, each group will spend 15 minutes documenting:

• what feedback they received;
• how it relates to the questions they brought;
• what they will change or keep based on that feedback;
• who will take responsibility for the next tasks.

Keep track of that document. We will ask for it in case of disagreements about the individual grade or in case of disagreements about contribution or task distribution.

The rest of the workshop is working on the practical. The teacher will circulate and ask questions about the group project, helping groups sharpen their research question, data collection plan, and error framework.

Practical information

• The project consists of two assignments, each worth 30% of the final grade.
• The groups will be formed in the first practical. If you miss the first practical, you will not be able to participate in the group project and will fail the course.
• Progress will be tracked through peer review and teacher check-ins during the regular workshop sessions. Attendance and active participation in all four feedback moments are mandatory.
• The same course-wide individual process grade is used for both assignments. Each feedback session in weeks 3, 4, 6 and 7 is worth 20 points. Students receive 20 points if they attend and participate at the expected level, and 0 points if they are absent, do not participate or participate very little. Students who consistently do more than expected can receive 20 extra points (all or nothing), based on their answers to the practical instructors and the task distribution and contributors indicated in the progress slides.
• Disagreements about contribution or task distribution must be brought up as soon as they arise, no later than the next feedback session and before the relevant submission.
• After Assignment 1, groups first receive the group grade for the report. The final grades for both assignments are determined after the fourth feedback session, when the course-wide individual process grade is complete.
• Upload the written report and the presentation through the link on the home page. Please name your file groupX_report.pdf, groupX_presentation.pdf.

Assignment 1: Errors in data collection (30% of the grade)

In the first assignment you will develop a research question, collect text data using one of the methods explained in the lectures/labs (data donation, plug-ins, scraping, APIs), and identify and discuss the errors that you anticipate and encounter when collecting data to answer it.

The outcome is a short report (<1,000 words excluding references and potential figures) that you will submit in week 5 (see weekly schedule). You will receive feedback from peers and lecturers during the workshop sessions (weeks 3 and 4).

Please find the template and the rubric of assignment 1 here.

Before you start, make sure your research question is broader than one platform, dataset, or source. For example, avoid framing the project only as “what happens in BBC comments?” and instead define the broader target population or phenomenon you want to study, then explain how your collected data approximates it.

In the report, explicitly distinguish representation errors from measurement errors. This distinction may look different depending on the data collection method: it is often more direct in data donation projects, while scraping projects require extra care in defining the target population, the observed population, and what each collected variable actually measures. For example, think carefully about whether deleted comments, missing users, platform moderation, or unavailable metadata create representation problems, measurement problems, or both.

Steps:

• During the final 20 minutes of the first lab meeting, discuss as a group how you envision working together. Discuss each person’s responsibilities and make agreements on how to collaborate. You can use the group contract for this, which can be found here.
• Decide on a specific research question and a target dataset. Since you will use the data in the second assignment, make sure that the data is suitable for the analysis you have in mind. In the second assignment, you will label the data using text classification models using a machine learning model. These models classify text, for example detecting the presence of hate speech, political leaning, personality traits or different emotions. We recommend you to use one of the proposed themes: political polarization, hate speech, personality traits, or mental health. Some of the research questions from previous years are:
  • How does the listening behavior of Utrecht University students towards music change over the course of the academic year?
  • Do supporters of Donald Trump became more polarized after the assassination attempt of July 13th, 2024?
  • What are the attitudes towards the actions of radical climate activists?
• Take a quick look at the PRIDE Questionnaire for approval Single Study, which helps conduct the project in an ethically responsible manner and in accordance with legislation. You will only need to complete it for the second assignment, but it is a good idea to start thinking about the ethical implications of your project from the beginning.
• Start collecting the data using the methods explained in the lectures/labs: either data donation/plug-ins (lab 2), or scraping/APIs (lab 3). In the unlikely case that you are using participants, you will need to create an informed consent form. For that, talk with your project supervisor (Thijs).
• Participate in the first peer-review workshop (week 3) to discuss your progress. Prepare a few (<5) slides with your progress which indicates on what topics each group member worked (each slide should contain the contributors). Bring already some of the collected data and 2-3 questions you want feedback on. The teacher may ask any group member the checkpoint questions below, and we expect everyone in the group to be able to answer them:
  • What broader phenomenon or target population are you studying, and what observed population or trace data do you actually have?
  • Why is this digital trace data a plausible proxy for your research question, and what does it fail to capture because of the platform, tool, or data source?
  • What exact collection route are you using (scraping, API, data donation, plug-in, or another method), and what are your inclusion rules, unit of analysis, and first signs of missing or distorted data?
• Finish collecting the data and discuss the errors (representation and measurement) that you anticipate when answering the research question with the proposed data.
• Participate in the second peer-review workshop (week 4) to discuss your progress. Prepare a few slides with your progress and 2-3 questions you want feedback on. The teacher may ask any group member the checkpoint questions below, and we expect everyone in the group to be able to answer them:
  • Representation: Draw the path: target population → data-source population → accessible or recruited units → scraped or donated data → analytic dataset. At each step, who or what might be missing or overrepresented?
  • Measurement: For each key concept, draw the path: concept → observable behaviour or content → recorded digital trace → scraped or donated data → processed variable. At each step, what could distort the measurement?
  • Evaluation: Which errors most threaten your conclusions, and how could you assess or reduce them?
• Finish assessing the errors in the data, ideally by comparing the data with other sources of information (e.g. representative surveys).
• Submit a short report (<1,000 words excluding references and potential figures) in week 5 (see weekly schedule). The link for handing in the report can be found on the course’s homepage. The report should include links to the conversation or conversations with GenAI and briefly identify where using GenAI resulted in cognitive offloading and where and how GenAI supported learning.

Grading:

• Process: 30%. This is the course-wide individual process grade described above.
• Report: 70% This is a group grade.
• Error framework: Students should clearly separate representation and measurement errors, explain each step in the data collection process where these errors may enter, and apply the total error framework to their specific method rather than using it generically.

Assignment 2: Errors in data labeling and moving past errors

In the second assignment you will further analyze the data collected in the first assignment to answer the RQ. For this you will use a Natural Language Processing model of your choice to label the data, which you will learn on week 5. These type of models can predict things from text. For example, they can predict if the text contains hate speech, polarized content, negative language, or specific personality traits. You will discuss the biases that you encounter in the labeling process, and how you will move past these errors in data collection and data labeling.

The outcome is a final presentation (12 minutes + 5 minutes of Q&A) that you will present in week 8 (see weekly schedule). Please submit the final presentation using the upload link before 23:59 in the day of the deadline (see weekly schedule). Name your slides groupX_presentation.pdf/pptx.

Please find the template and the rubric of assignment 2 here.

You will receive feedback from peers and lecturers during the workshop sessions (weeks 6 and 7).

Steps:

• Run a text classification model of your choice from Hugging Face on the data collected in the first assignment. These models classify text, for example detecting the presence of hate speech, political leaning, personality traits or different emotions. The model should be appropriate to answer your research question. If you would like to use LLMs (e.g., GPT-5) for labeling, you can look at the materials here. Please be aware that using LLMs costs money (for a typical project in the course, usually less than one euro).
• Reflect on the biases you expect during the labeling process, and the origin of those biases.
• Participate in the third peer-review workshop (week 6) to discuss your progress. Prepare a few slides with your progress, make sure the slides indicate who worked on what topics, and bring 2-3 questions you want feedback on. The teacher may ask any group member the checkpoint questions below, and we expect everyone in the group to be able to answer them:
  • What construct is your model trying to detect, what data was the model trained on, and how well does that match your own data, language, platform, and research question?
  • How will you check model quality and bias?
  • What are the main ethical and PRIDE issues in your project (consent, legal basis, sensitive data, vulnerable groups, re-identification, contextual integrity, data minimization, or possible harm), and what concrete decisions reduce those risks?
• Reflect on how you would expand the study to tackle these issues. What type of data would you need to collect?
• Complete the PRIDE Questionnaire for approval Single Study, which helps conduct the project in an ethically responsible manner and in accordance with legislation.
• Reflect on the ethical principles discussed through the course. Is your project aligned with them?
• Draft the final presentation.
• Participate in the fourth peer-review workshop (week 7) to discuss your progress. Prepare a few slides with your progress and bring the completed PRIDE questionnaire. The teacher may ask any group member the checkpoint questions below, and we expect everyone in the group to be able to answer them:
  • What is your answer to the research question, and how do your representation errors, measurement errors, and model errors limit or qualify that answer?
  • If you redesigned or expanded the study, what would most improve it: another platform, data donation, survey linkage, designed big data, manual labels, better sampling, external benchmarks, or stronger model validation?
  • Does your final presentation make the whole logic visible: research question, target and observed population, collection method, error framework, model or analysis, findings, ethics and PRIDE, limitations, and next steps?
• Submit the slides and the PRIDE form before the deadline (see weekly schedule) through the link on the home page. Please name your file groupX_PRIDE.pdf, groupX_presentation.pdf.
• Present the group project in week 8 (see weekly schedule). We expect all groups to actively engage with questions and discussion.
• During the presentations, we require all other groups to active participate in the discussion and ask questions.

Grading:

• Process: 30%. This is the course-wide individual process grade described above.
• Presentation: 70%. This is a group grade.