Most people de-identify a dataset by deleting the name column and the email column, then calling it done. That works right up until someone notices there's only one 67-year-old female associate dean in the file, and she's from a campus with 900 students.
De-identification is a process, not a single step. This guide walks through it: what counts as an identifier, which rules might apply to you, a pass you can run on almost any tabular dataset, how to check whether it worked, and what to do when it can't work well enough.
Direct and indirect identifiers
It helps to sort every variable into one of three groups before you touch anything.
- Direct identifiers point to a person on their own: names, email addresses, phone numbers, student or employee IDs, Social Security numbers, street addresses, photos of faces, IP addresses.
- Indirect (quasi-) identifiers don't identify anyone alone, but can in combination: exact birth date, ZIP code, job title, department, race and ethnicity, gender, rare diagnoses, dates of events, the name of a small school.
- Everything else: the survey answers, test scores, and measurements you actually want to analyze. Even these can identify people when they're unusual (a salary of $410,000 at a small college narrows things down fast).
Removing the first group is the easy part. The real work is the second group, because every indirect identifier you keep makes the next one more dangerous. A well-known study by Latanya Sweeney found that ZIP code, birth date, and sex together were enough to single out a large majority of Americans.
Which rules apply?
"De-identified" means different things under different rules, and it's worth knowing which ones cover your data before you start.
- HIPAA applies only to health information held by covered entities (hospitals, clinics, health plans) and their business associates. It offers two routes. Safe Harbor requires removing 18 listed identifiers, including all geography smaller than a state (the first three digits of a ZIP code can stay if that area has more than 20,000 people), all dates except the year, and ages over 89. Expert Determination lets a qualified statistician certify that the risk of re-identification is very small. The HHS de-identification guidance is the source to read.
- FERPA covers student education records. Records can be released without consent once all personally identifiable information is removed and the institution has made a reasonable determination that a student's identity isn't personally identifiable, taking other available information into account. Small cells matter here, which is why institutional reports often suppress counts under 5 or 10.
- The Common Rule (federal human subjects regulations) turns on whether the investigator can readily ascertain identity. Your IRB decides how that applies to your study, and your consent form sets limits on what you promised participants.
Even when no law applies, your consent form does. If you told people their answers would be "kept confidential and reported only in aggregate," sharing row-level data may break that promise no matter how well you scrub it. The data ethics guide covers consent in more detail, and the consent language guide shows how to word the form so sharing stays possible.
A de-identification pass, step by step
This works for most survey, administrative, and program data. Do it in code, not by hand in a spreadsheet, so you can rerun it and show exactly what changed.
- Keep the original untouched. Work on a copy, and store the original somewhere access-controlled. You'll need it if a reviewer asks a question or you find a mistake.
- List every variable and classify it as direct, indirect, or analytic. Your codebook is the natural place to do this; add a column for it.
- Remove direct identifiers. If you need to link records across files, replace the real ID with a new random study ID and keep the crosswalk (real ID to study ID) separate and private. Don't use a plain hash of the real ID; short IDs like student numbers can be reversed by trying every possibility.
- Coarsen dates. Birth date becomes age or birth year. Event dates become the year, the term, or days since a reference point for each person.
- Coarsen geography. Street address becomes county, state, or region. For multi-site studies, consider replacing site names with codes, and ask whether the site itself is identifying.
- Group rare categories. Collapse small job titles into broader ones, combine small racial and ethnic categories where your analysis allows, and top-code extreme values (every salary above a threshold becomes "above threshold").
- Deal with free text. Open-ended answers are where people write "as the only Arabic-speaking librarian in the building..." Review them line by line, redact or generalize, or leave them out of the shared file.
- Check the result (next section), then document every change in a short de-identification log that goes with the data.
Every step here trades detail for privacy. Coarsen only as far as you need to; turning every age into "adult" protects people but also makes the data useless. The goal is the least change that gets the risk down to an acceptable level.
How to check whether it worked
The simplest test is to count. Take the indirect identifiers you kept, group the records by every combination of them, and look for small groups.
- If any combination describes only one or two people, those people are at risk. A common rule of thumb is that every combination should match at least 5 records (this is the idea behind k-anonymity, with k = 5).
- Fix small groups by coarsening one of the variables further, or by suppressing that value for those few records.
- Then ask the harder question: what else could someone combine this with? A public staff directory, a news story, a LinkedIn profile, the institution's own IPEDS numbers. For most education data the realistic threat is an insider: a colleague who already knows most of the story.
In R, a grouped count on the quasi-identifiers (count(age_group, gender, dept) with dplyr) shows the small cells in seconds. The sdcMicro package goes further, with risk measures and tools to fix what it finds. In Stata, contract or duplicates report on those variables does the same job. For health data under Expert Determination, this is where you bring in a statistician.
When de-identification isn't enough
Some data can't be de-identified without losing what makes them worth sharing: detailed interview transcripts, longitudinal records that follow the same students for years, data from very small populations, geocoded data, and anything with rich free text. For these, the better answer is controlled access rather than more scrubbing.
- Restricted-use deposits. Archives such as ICPSR offer restricted access, where approved researchers sign a data use agreement before getting the files. The repository guide covers the options.
- A public version plus a restricted version. Share a heavily coarsened public file for teaching and quick checks, and a detailed file under agreement.
- Share the documentation even when you can't share the data. A public codebook and a clear data availability statement tell people what exists and how to apply for it.
Plan this early. The data management plan is where you tell the funder how sensitive data will be shared, and the consent form is where you get permission. Fixing either after collection is much harder.
A note on AI tools
Pasting an identified dataset or transcripts into a public chatbot to "help remove the names" discloses the data to a third party before it's de-identified. If you want AI help with redaction, use a tool your institution has approved for sensitive data, or a model that runs locally. The private AI guide explains what that setup looks like.
If you support researchers
De-identification questions usually arrive at the worst time: the week before a journal wants the data, or right after a funder asks where it's deposited. A short checklist, a worked example in R or Stata, and a named person to ask can save a lot of those weeks. If your campus is building that kind of support, the guide to starting research data services is a place to begin, and I help institutions set it up. Get in touch if that would help.
Common questions
Is removing names enough to de-identify a dataset? No. Indirect identifiers such as dates, small geographic areas, job titles, and rare characteristics can identify people in combination. You need to remove direct identifiers and then reduce or coarsen the indirect ones until no small group stands out.
What is the difference between de-identified and anonymous data? Anonymous usually means the data were never linked to identities, for example a survey that never collected names or emails. De-identified means identifiers existed and were removed or altered. Many regulations treat de-identified data as lower risk, but not zero risk.
Does HIPAA apply to my research data? Only if the data came from, or are held by, a HIPAA covered entity or its business associate, such as a hospital or health plan. Most education and social science data fall under the Common Rule, FERPA, or your consent forms instead. Ask your IRB if you are unsure.
Can I share qualitative interview data? Often, with care. Transcripts need names, places, employers, and distinctive stories edited or generalized, and some data are only suitable for restricted access. Consent forms need to allow sharing in the first place.