Geocode in Spark and Databricks
The engine and the file sit on the workers, and no address leaves the cluster. No API key and no rate limit: the cost is the cluster you already run.
One call
$ pip install "gridpin-spark @ git+https://github.com/gridpin/gridpin@v0.9.0#subdirectory=integrations/spark"
The package is not on PyPI: pip builds it from the folder integrations/spark of the repository.
from gridpin_spark import geocode
out = geocode(df, "address", "/Volumes/geo/sheets/usa.bin")
out.select("address", "gp_lat", "gp_lon", "gp_precision", "gp_match_level").show()
Every row stays; one without an address or an answer has empty columns and gp_match_status U. The file opens once per Python worker and a batch is answered in one call. Checked on local Spark 4.3.
Databricks
| Compute | Works |
|---|---|
| Classic cluster, DBR 14.3 LTS or later, file in a Unity Catalog volume | yes, by Databricks' documentation; not yet run on a live workspace |
| Serverless | no: custom code has 1 GB of memory, less than the United States file, and a Unity Catalog UDF has no file system |