IPUMS IHGIS: Unlocking International Population and Agricultural Census Data

By Tracy Kugler

Nearly all countries throughout the world conduct population and housing censuses at least every ten years, and most also conduct agricultural censuses or surveys regularly. These censuses collect information on demographics, education, employment, housing characteristics, migration, agricultural land ownership, agricultural workforce, livestock, crops, and more. The resulting data can be used to study a wide range of questions, from the character of demographic transitions within and across countries, to utilization of irrigation, to educational trends among women. 

Unfortunately, this wealth of data has remained largely inaccessible to researchers. The data are typically published in reports as tables summarizing population characteristics. In recent decades, many of these reports have been published as PDF documents and made available on national statistical office websites. While the reports are available, data from a PDF document cannot be easily imported into a statistical or GIS package. Furthermore, the table structures are highly heterogeneous, both across countries and even within the same report.

The International Historical Geographic Information System (IPUMS IHGIS) is designed to provide easy access to these data in a way that researchers can easily use for analysis. In the early phases, IHGIS was known internally as “Project Mako,” named after the Mako shark, which has a global range, voracious appetite, and a reputation for a broad-ranging diet. Like the shark, IHGIS (née Project Mako) will encompass the world and ingest all kinds of data tables.

IHGIS Data

The initial version of IHGIS includes 270 tables from 9 population and housing censuses and 4 agricultural censuses. We plan to release new datasets several times a year. Our next release will include tables for an additional 12 datasets and is planned for early 2021. We have acquired over 30,000 data tables from 150 population and 107 agricultural censuses from 132 countries, which we will move through the processing pipeline over the next few years.

Datasets present/planned in the first two IHGIS data releases.
Datasets present/planned in the first two IHGIS data releases.

Our data collection efforts for population data have focused primarily on countries for which microdata are not yet available in IPUMS International. The geographic detail available with microdata is often limited due to confidentiality concerns associated with individual-level data. For several countries, notably Canada, Russia, and much of northern Europe, IPUMS International is only able to release first-level (e.g., province) identifiers. Confidentiality concerns are mitigated in summary tables. IHGIS may therefore be able to provide much more geographic detail, and we will focus on acquiring such data in future collection efforts.

You can explore the current collection through the IHIGIS data finder, where you can filter by dataset, browse available tables, select the tables you are interested in, and download the data. Your extract will include consistently structured data tables in CSV format, ready for use in your analysis. You will also receive comprehensive metadata in both human- and machine-readable formats. For more information about how to use the data finder and interpret your extract, check out our User Guide.

IHGIS also provides GIS shapefiles delineating the boundaries of the geographic units described in the data tables. Each unit is identified with a unique code in both the data tables and shapefiles, allowing you to easily join them in a GIS package.

IHGIS Under the Hood

Transforming data tables from the myriad structures in which they are published to the standardized IHGIS structure is no small task. Clearly, it would be impossible without substantial software infrastructure. But it is equally infeasible to completely automate the task of interpreting the contents of any given table. Therefore, the overarching philosophy of IHGIS data processing is to have computers do what computers are good at and have humans do what humans are good at. For example, it is relatively easy for a person to determine whether row headers identify geographic units or categories of marital status or educational attainment. Developing software to make that determination would be a significant challenge. On the other hand, having humans extract state-level totals from a table by copying and pasting is tedious, time-consuming, and error-prone.

The heart of the IHGIS data processing workflow is a table markup framework. Table markup uses Excel as an interface for a lightweight process through which researchers (mostly undergraduate research assistants) indicate the location of key structural elements within each table. For each table, students extract information such as the universe, time frame, and geographic extent. They then add keyword tags indicating the location of geographic unit headers, headers describing the characteristics summarized in the table, the table title, the extent of the data, and other structural elements.

Example of markup for a relatively simple table
Example of markup for a relatively simple table

The markup serves as a guide for our software, enabling ingest into a metadata database. The database organizes all row and column headers, titles, universes, and other metadata elements and their relationships in a consistent way. The database, in turn, enables automated restructuring of the data tables to generate the consistently structured tables in IHGIS extracts. For example, many source tables include nested geographic units at two or more levels (e.g., states and counties). IHGIS pulls the appropriate rows apart to create separate files for each level, enabling easier data linkages in GIS packages.

We hope you enjoy using IHGIS, and please send us a note at ipums@umn.edu if you have any questions, comments, or suggestions.

Cite us! Seriously though…

By Renae Rodgers and Kari Williams

Hi there IPUMS users! Let’s talk about citations. When using our datasets in your insightful, groundbreaking, interesting work, please cite us! 

Seriously though. 

Cite us. 

You wouldn’t steal a car, you wouldn’t rob a little old lady of her handbag, you wouldn’t base work on that of a colleague and not put their paper(s) in your reference section, right?!? Then don’t use IPUMS data and fail to mention it! 

To help you on your way, here are some answers to frequently asked questions:

Q:   Do I have to though? 

A:   Yes. Properly citing IPUMS data is part of the user agreement. Before you ever submitted your first extract, you agreed to do this!

Screenshot of citation agreement

Q:   I’ve mentioned IPUMS in the caption of my figures and tables, so I am good to go, right?

A:   Nope. Putting our URL in a footnote, endnote, or caption is insufficient. Name-checking IPUMS in your “Data and Methods” section is not enough. Just for good measure, we will mention that, citing a paper by IPUMS staff about IPUMS data is not the same thing as citing a dataset.

 

Q:   What about talking about how much I love IPUMS on Twitter or naming my firstborn after this amazing data provider? Is that an appropriate substitute?

A:   [public radio voice] If you appreciate the resources that IPUMS provides, using the data and citing it is the best way to support us. Our ability to continue to provide this service is dependent on capable and intelligent users like you citing our datasets! Seriously, a core part of our funding depends on our ability to prove that the data infrastructure IPUMS offers is being used. If you want IPUMS to keep offering the latest data and developing new tools, we need you to cite us so we can demonstrate to our funders that IPUMS is useful. Citing us is the best way to support us (though we are keen to hear about your children with middle names based on your favorite IPUMS variables).

 

Q:   How do I cite IPUMS properly?

A:   We are so glad you asked! When you receive an email notification that your custom dataset from IPUMS is ready to download, it includes the citation! Each IPUMS data product has its own citation – be sure to use the citation associated with the IPUMS data that you used. If you use more than one IPUMS data product, cite all of them!

Q:   Okay, wait! I have one more. 

A:   Go for it.

Q:   What if…I deleted the extract email and didn’t make note of the citation? 

A:   Not a problem! For your convenience, we just happen to have this handy link of all the current IPUMS dataset citations with DOIs. You can also find each IPUMS dataset’s citation on the left menu of the homepage.

To those users who are diligent about citing IPUMS datasets, we thank you! If you have used IPUMS without citing it or committed one of the other faux pas above in the past, we hope you now have the instruction and incentive to do better going forward!

In the Archive: “25 Years of IPUMS Data”

“25 Years of IPUMS Data,” the current IPUMS/MPC archive exhibit, highlights a dynamic quarter center history of data innovation at the University of Minnesota. In the late 1980s, the Social History Research Laboratory at the University of Minnesota’s History Department proposed “the creation of a single integrated microdata series composed of public use samples for every year … with the exception of the 1890 census, which was destroyed by fire.”  The primary aim was to make the U.S. census microdata “as compatible over time as possible while losing little, if any, of the detail in the original datasets” (Integrated Public Use Microdata Series: A Prospectus).

Continue reading…