Practical Data Collection with Python

Economic research increasingly relies on data that is not readily available in standard databases. Policy institutions, central banks, and researchers now regularly work with online sources such as institutional websites, statistical portals, news outlets, and administrative pages.

Web scraping has become a core skill for economists who need timely, flexible, and reproducible access to such data. When done properly, it allows researchers to build custom datasets, update indicators automatically, and complement traditional data sources with new forms of information.

This course focuses on practical, responsible web scraping using Python, with applications directly relevant to economic research and policy work.

​

About The Better Policy Project Courses

The Better Policy Project delivers applied training at the intersection of economics, data, and policy. Our courses are designed for professionals working in central banks, public institutions, international organisations, and universities.

We focus on skills that economists actually use in their day-to-day work—combining methodological rigour with hands-on implementation.

​

What Makes This Course Different

• Built for Economists: Web scraping is taught as a research tool, not as a generic programming exercise.

• Policy-Relevant Use Cases: Examples include institutional websites, economic indicators, announcements, and structured online data sources.

• Modern, Reproducible Workflows: Emphasis on clean code, documentation, and data pipelines that can be updated and maintained over time.

• Responsible Data Collection: Best practices for ethical scraping, legal considerations, and website-friendly approaches are integrated throughout the course.

​

Why Python for Web Scraping?

• Widely Used Across Institutions: Python is already standard in many research and policy environments.

• Strong Ecosystem: Libraries such as requests, BeautifulSoup, and browser automation tools support a wide range of scraping tasks.

• End-to-End Analysis: Scraped data can be cleaned, analysed, and visualised within the same environment.

• Readable and Maintainable Code: Clear syntax supports collaboration and long-term project sustainability.

​

Course Overview

This 5-day online course provides a structured introduction to web scraping for economic analysis using Python.

Participants will learn how to:

• collect data from websites in a reliable and ethical manner,

• automate data updates,

• clean and structure scraped data for analysis,

• integrate web data into economic research workflows.

By the end of the course, participants will be able to independently build and maintain web-based data collection pipelines.

​

Course Structure

Day 1 – Introduction to Web Scraping and Python Setup

• Types of web data relevant for economics

• HTML basics and how websites work

• Python environment and core libraries

Day 2 – Scraping Static Websites

• Using requests and BeautifulSoup

• Extracting tables, text, and links

• Structuring scraped data

Day 3 – Working with Dynamic Websites

• Understanding JavaScript-rendered content

• Browser automation tools

• Handling pagination and complex page structures

Day 4 – Automation, Data Quality, and Storage

• Building reusable scraping scripts

• Error handling and logging

• Storing data in files and databases

Day 5 – Applications and Integration

• Case studies using economic and institutional websites

• Updating datasets automatically

• Integrating scraped data into analysis and reporting workflows

​

Practical Details

📅 Course Dates: March 16–20, 2026
⏰ Time: To be announced
📌 Application Deadline: March 9, 2026