A focused playbook for engineering teams running spatial data at scale. Migrate legacy Shapefiles and GeoJSON to columnar, cloud-native formats like GeoParquet and FlatGeobuf, and tune the storage layer for predictable cost and query latency.
Practical guides for GIS data engineers, Python backend developers, and cloud architects — covering ZSTD level selection, row-group sizing, space-filling-curve partitioning, schema mapping, metadata preservation, CI/CD validation, and querying GeoParquet in place with DuckDB, Athena, and Trino under a predictable storage budget.
Four connected tracks. Start with format trade-offs, layer compression and indexing on top, automate the migration end-to-end with production-grade pipelines, then query the result in place with cloud SQL engines.
Modern geospatial storage architecture: WKB, CRS handling, columnar vs streaming formats, and the trade-offs between GeoParquet, FlatGeobuf, GeoJSON, and Shapefile.
Tune ZSTD levels, size row groups, apply dictionary encoding, and partition with quadtree / Hilbert curves so spatial queries stay fast at scale.
Batch convert legacy Shapefile / GeoJSON to GeoParquet with Python: schema mapping, metadata preservation, parallel execution, validation, and fallback routing.
Query GeoParquet in place with the DuckDB spatial extension, AWS Athena, and Trino / Presto — predicate pushdown, partitioning, and cost control for cloud-scale spatial SQL.
New to cloud-native geospatial storage? These are the most-referenced guides — one strong entry point into each track. Read them in order to go from format choice to a tuned, automated migration.
Pick the right cloud-native format: columnar GeoParquet for analytics vs streaming FlatGeobuf for feature-by-feature reads.
Choose a ZSTD level that balances file size, write cost, and query latency for vector and raster workloads.
Automate Shapefile and GeoJSON migration to GeoParquet with schema mapping, parallel execution, and validation.
Read GeoParquet straight from object storage with predicate pushdown — no database load step, no data movement.