analyze the social security administration public use microdata files (ssapumf) with r

the social security administration (ssa) must be overflowing with quiet heroes, because their public-use microdata files are as inconspicuous as they are thorough.  sure, ssa publishes enough great statistical research of their own that outside researchers rarely find ourselves wanting more and finer data that this agency can provide, but does that stop them from releasing detailed microdata as well?  why no.  no it does not.  if you wake up one morning with a hankerin' to study the person-level lifetime cash-flows of fdr's legacy, roll up your sleeves and start right here.

compared to the other data sets on asdfree.com, the social security administration public use microdata files (ssapumf) are as straightforward as it gets.  you won't find complex sample survey data here, so just review the short-and-to-the-point data descriptions then calculate your statistics the way you would with other non-survey data.  each of these files contain either one record per person or one record per person per year, and effortlessly generalize to the entire population of either social security number holders (most of the country) or social security recipients (just beneficiaries).  the one-percent samples should be multiplied by 100 to get accurate nationwide count statistics and the five-percent samples by 20, but ykta (my new urban dictionary entry).  this new github repository contains one script:


download all microdata.R

• download each zipped file directly onto your local computer
• load each file into a data.frame using a mixture of both fancery and schmantzery
• reproduce the overall count statistics provided in each respective data dictionary
• save each file as an R data file (.rda) for ultra-fast future use

click here to view this lonely script

for more detail about the social security administration public use microdata files (ssapumf), visit:

• the social security administration home page
• the social security administration open data initiative
• the national archives' history of social security

notes:

i skipped importing these new beneficiary data system (nbds) files because i broadly distrust data older than i am and you probably want these easy-to-use, far more current files anyway. 


confidential to sas, spss, stata, and sudaan users: no doubt they were very impressive when they originally became available.  but so was the bone flute.  time to transition to r.  :D

analyze the medical large claims experience study (mlces) with r

not a survey, not even remotely current, the society of actuaries' medical large claims experience study (mlces) might be the best private health insurance claims data available to the public.  this data should be used to calibrate other data sets, and probably nothing more.

researchers interested in studying healthcare patterns among our elderly, disabled, or poor can go to the centers for medicare and medicaid services for all sorts of up-to-date utilization data.  but what if you want to study the behavior and spending of everyone else?  you could look at the medical expenditure panel survey (meps), the consumer expenditure survey (ce) or the national health interview survey (nhis), but there's an attrition problem with those - anyone who suddenly falls expensively-ill also starts slamming the door on follow-up survey interviews.  and that's understandable - who wants to respond to a government questionnaire when you're struggling with a serious health condition?  american healthcare surveys are biased at the tail - they don't capture our sickest very well.

think about it some more: we have single-payer healthcare for our elderly (medicare), disabled (medicare again), and poor (medicaid), meaning there's a government agency that's got all that data in one place.  every claim paid by the government is just hanging out in baltimore, waiting for you to come a knockin'.  and there's no non-response bias with government healthcare claims data - the united states government knows exactly how much the united states government paid on your behalf, whether or not you agreed to respond to somesuch survey.  doctors submit bills pretty consistently, after all.  so the utilization patterns of medicare and medicaid beneficiaries are stored in a central location, standardized, and available for purchase or (with limitations) for immediate download.  but in a heavily-privatized medical industry like ours, what do you do when you want to explore the purchasing patterns of everyone else?  well, you still probably look at meps or ce.  but if your research question is hyper-focused on the dist-ri-bu-tion of medical claims among the privately-insured, well hey, the distribution of medical claims in mlces is much more realistic than what you'll find in survey data.  yes, it's old.  yes, it's only composed of claims from seven insurers and not every private insurer covering every covered life in the united states.  and yes, it might even have a y2k bug or two.  but for publicly-available medical claims for the privately insured in the united states of america, well, take it or leave it.  this new github repository contains two scripts:

1997-1999 mlces - download.R

• download each zipped year of data onto your local computer
• load the entire table into RAM
• save the condensed file as an R data file (.rda)

replicate soa publications.R

• produce the control counts and totals at the bottom of this document
• replicate the 1997 statistics shown in table iv-a of this excel package

click here to view these two scripts


 for more detail about the medical large claims experience study (mlces), visit:

• the mlces homepage
• the society of actuaries (soa) homepage

notes:

this data set is not generalizable to any recent population of americans.  its chief value is its relationship to itself - the distribution of medical spending, especially at the extreme values.  in caveman speak: percentages good, totals bad.


confidential to sas, spss, stata, sudaan users: the best things in life are free.  time to transition to r.  :D

analyze the pesquisa nacional por amostra de domicilios (pnad) with r

think of the pesquisa nacional por amostra de domicilios (pnad) as the brazilian census for off-years - the ones that don't end in zero.  the principal household survey for the nation of brazil, pnad measures general education, labor, income, and housing characteristics of the population.  pesquisa nacional por amostra de domicilios translates to 'national household sample survey' and, like most things, sounds better in the native portuguese (exhibit a).  my first foray into survey data for a country outside of the united states, this comes at the request of djalma pessoa who co-authored both this post and the published scripts.  djalma works as a consultant for the office in charge of statistical survey methodology at the brazilian institute of geography and statistics (ibge), just like a united states census bureau headquartered in rio de janeiro.

pnad has been on the shelves for forty-four years, investigating all the good stuff: migration, fertility, marriage, health, and food security.  microdata are available back to 2001 and starting in 2004, pnad started including the rural north (the amazon state, among others).  the sample design is self-weighted with three selection stages: primary sampling units are municipalities stratified by population size, selected systematically with pps. secondary and tertiary sampling units are enumeration areas, then households.  the weights also need to be post-stratified to the 2010 official brazilian census.  all in all, a pretty straightforward methodology.  let the code do all the setup for you so you can worry about the more exciting questions and then clock out for the day.  by the way, in brazil, do they call happy hour cappy hour?  this new github repository contains four scripts:


2001-2011 - download all microdata.R

• download the fixed-width file containing household and person records
• merge 'em together into a rectangular file at the person-level
• create an adjusted weight and a new variable - one - in the data table

2011 single-year - analysis examples.R

• connect to the sql database created by the 'download all microdata' program
• create the complex sample survey object, post-stratifying using a custom-built function
• perform a boatload of analysis examples

2011 single-year - variable recode example.R

• connect to the sql database created by the 'download all microdata' program
• recode some numeric variables into a broader categorical variable
• re-create the complex sample survey object, post-stratifying using a custom-built function

replicate IBGE estimates - 2011.R

• connect to the sql database created by the 'download all microdata' program
• create the complex sample survey object, post-stratifying using a custom-built function
• precisely match the sas-sudaan output provided by analysts at ibge (as seen in the script directory)

click here to view these four scripts



for more detail about the annual pesquisa naciona por amostra de domicilios, visit:

• the official brazilian statistics agency's homepage (english) (portuguese) (even espanhol)
• the ibge ftp site with all your favorite microdatos
• the translated-into-english pnad wikipedia entry

notes:

to accommodate smaller computer workstations (with only 4gb of ram), these scripts perform all manipulations inside sqlite and rely on database-backed survey objects.  the post-stratification function in the current implementation of the r survey package does not work on database-backed survey design objects.  therefore, with little fanfare, i've written one that does.  you'll find it getting pulled in at the source_url() line.  exciting.


confidential to sas, spss, stata, sudaan users: yes, and bicycles with training wheels might be easier to ride, but that doesn't make them a long-term solution.  time to transition to r.  :D