Fips/Zip crosswalk has more zip codes than zip population file

[chinandrew]

In [2]: f_z = pd.read_csv("delphi_utils/data/fips_zip_table.csv", dtype = str)                                                                                       

In [3]: zpop = pd.read_csv("delphi_utils/data/zip_pop.csv", dtype = str)                                                                                             

In [4]: len(set(f_z.zip))                                                                                                                                            
Out[4]: 32976

In [5]: len(set(zpop.zip))                                                                                                                                           
Out[5]: 32959

In [6]: set(f_z.zip) - set(zpop.zip)                                                                                                                                 
Out[6]: 
{'57756',
 '57764',
 '57770',
 '57772',
 '57794',
 '99554',
 '99563',
 '99581',
 '99585',
 '99604',
 '99620',
 '99632',
 '99650',
 '99657',
 '99658',
 '99662',
 '99666'}

In [7]: set(zpop.zip) - set(f_z.zip)                                                                                                                                 
Out[7]: set()

Those appear to be real zips, unsure why discrepancy exists.

[krivard]

Possibilities:

• old/retired zip codes
• zip codes that receive mail but have no residents

?

[krivard]

Work needed on this:

• Figure out why these lists don't match (possibly useful source table)
• Figure out whether we can fix it
• Figure out whether we should fix it

[rafaelcatoia]

Hi @krivard, can I start working on this one?

[rafaelcatoia]

@krivard , I looked up into those Zips and found some information about them.

• old/retired zip codes?

I made search on discontinued Post Offices from 01/01/2010 to 01/09/2021 and it returned 698 discontinued Post Offices and the zip codes they covered in that interval. None of them are in our list.

• Are they zip codes that receive mail but have no residents?

The population size is usually low (One with ~6k, one ~2k, and others with less than 1k) and majorly composed by American Indian Or Alaskan Native.

• Figure out why these lists don't
  00_Issue_0648.pdf
  match (possibly useful source table)

• Figure out why these lists don't match (possibly useful source table)

I looked into the 2018 Dartmouth Atlas Project tables but coudn't find those zip codes there.

• Figure out whether we can fix it

Not sure there is much to fix

• Figure out whether we should fix it

Code and more detailed information is attached here

[krivard]

To sum up:

1. There are 17 zips that we can place in a county, but for which we do not have population information.
2. They are still active/have not been discontinued. All are PO box zip codes.
3. Population information is available (somewhere?) for these zips.
4. Our zip->fips list contains some zips that were discontinued by the USPS, but we have population for all of them.

(1) + (3) implies there may be a fix -- where did you find population information for these zips? How much work would it be to add code to data_proc to parse that source into zip_pop.csv?

(4) is good for peace of mind but i think we don't need to do anything -- for any of our data sources with outdated zips, we still want to map those zips to counties.

[rafaelcatoia]

I found population size information in these websites: (zip 57770 to exemplify) unitedstateszipcodes, zipdatamaps, zip-codes. They usually match regarding population size of 2010 but may differ on updated information.

Some of those sites have API but they are demo version and also have limited Lookups per month. Since that there are only 17 zips, I think it would be easy to manually create a new csv file and aggregate it to data_proc

What do you think?

[dshemetov]

While this source's methods of determining population are proprietary, the downside of trusting it (reporting possibly inaccurate prop signals) don't seem large compared to the upside (increasing our prop signal coverage). So I think we should do it 🚀

[krivard]

Let's do it manually for this repair; there's an upcoming task for ingesting the 2020 census population information in detail (like, census-block level, so smaller than zips) but we're not sure when that will be scheduled.

(do document what how you create the manual csv in the README)

[rafaelcatoia]

@krivard, I ran the analysis again and now we have 7 new zipcodes without population size assigned to them.
I already have the population numbers for all zipcodes and made the changes in geo_data_proc.py. I'am going to document the numbers in the README and then open the PR.

Zipcode	State	New	Pop
57756	SD		1126
57764	SD		1923
57770	SD		5271
57772	SD		2048
57794	SD		644
99554	AK		677
99563	AK		938
99566	AK	x	192
99573	AK	x	1115
99574	AK	x	2348
99581	AK		762
99585	AK		417
99586	AK	x	605
99604	AK		1093
99620	AK		577
99632	AK		813
99650	AK		568
99657	AK		329
99658	AK		329
99662	AK		480
99666	AK		189
99677	AK	x	88
99686	AK	x	4005
99693	AK	x	248