gridpin

Geocode a polars column and check the match rate

A column of addresses in polars, and two questions: where is each one, and how much of the column can be trusted. Both are answered on your machine, with no API key and no request per row.

New Geocode addresses Python Research

What you have

pip install polars gridpin and a country file. A table of customers and a script:

customers.csvfile
id,address
1,"233 S Wacker Dr, Chicago, IL 60606"
2,"1600 Pennsylvania Ave NW, Washington, DC 20500"
3,"350 Fifth Avenue, New York, NY 10118"
4,"Main Street, Springfield"
5,"Nowhere Street 99999, Atlantis"
check.pyfile
import polars as pl
import gridpin.polars  # registers df.gridpin
import gridpin.dq

pl.Config(tbl_hide_dataframe_shape=True, tbl_hide_column_data_types=True, fmt_str_lengths=40)
df = pl.read_csv("customers.csv").gridpin.geocode("address", "usa.bin")
print(df.select("id", "gp_match_level", "gp_confidence", "gp_weak"))

house = gridpin.dq.reaches(df["gp_match_level"], "house")
print(f"to the house: {sum(house)} of {len(house)}")

print(df.filter(pl.col("gp_weak").is_null() | pl.col("gp_weak")).select("id", "address"))

Steps

1. Run it:

terminalCLI
$ python check.py
┌────┬────────────────┬───────────────┬─────────┐
│ id ┆ gp_match_level ┆ gp_confidence ┆ gp_weak │
╞════╪════════════════╪═══════════════╪═════════╡
│ 1  ┆ PointAddress   ┆ 0.95          ┆ false   │
│ 2  ┆ PointAddress   ┆ 0.95          ┆ false   │
│ 3  ┆ PointAddress   ┆ 0.95          ┆ false   │
│ 4  ┆ StreetName     ┆ 0.2           ┆ true    │
│ 5  ┆ null           ┆ null          ┆ null    │
└────┴────────────────┴───────────────┴─────────┘
to the house: 3 of 5
┌────┬────────────────────────────────┐
│ id ┆ address                        │
╞════╪════════════════════════════════╡
│ 4  ┆ Main Street, Springfield       │
│ 5  ┆ Nowhere Street 99999, Atlantis │
└────┴────────────────────────────────┘

What you get

df.gridpin.geocode(column, sheet) keeps every row and its order and adds the columns of gridpin batch: gp_lat, gp_lon, gp_match_level, gp_confidence, gp_weak and the parts of the address. A row with no answer has nulls there. gridpin.dq.reaches turns the match levels into true or false for a level: house, street or locality.

Why it works

The rows go to the engine in one batch, spread over the cores of the same process: the file is memory-mapped once, and nothing leaves the machine. A weak answer (gp_weak) is one the engine itself does not vouch for: row 4 names a street and a town that many states share, with no state and no number.

Limits

  • One country per call; split the frame by country first.
  • A row that reaches the house can still be the wrong house if the address itself is wrong: the check measures how far the answer reaches, not whether the customer lives there.

The function: polars and data-quality checks. The same in pandas and GeoPandas: Geocoding in GeoPandas.

More