Skip to content

Fast and hackable importer for Openstreetmap PBF files into the PostgreSQL database.

License

Notifications You must be signed in to change notification settings

exatel/topo_import

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

31 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

OSM Routing Importer

Imports OpenStreetMap data from a binary PBF file format into PostgreSQL in a way which allows to do a topological routing.

Program was created because osm2pgrouting (C++) was choking on a full country import. For a single Masovian voivodeship: osm2pgrouting uses 10 minutes and 14GB of RAM. This project needs 2 minutes and 1GB of RAM - with a similar end result (same length of migrated ways, less unnecessary splits).

This later turned out to be easy to extend and allowed us to easily modify the import to add a way splitting.

Model

Two tables are used:

Ways table:

 Column |           Type
--------|---------------------------
 id     | bigint  # Generated ID, as OSM ways are split at intersections
 id_osm | bigint  # Original (duplicated) osm_id
 type   | integer # Type based on highway tag
 source | bigint  # source node_id of intersection 
 target | bigint  # target node_id
 lon1   | double precision # start/target coords
 lat1   | double precision
 lon2   | double precision
 lat2   | double precision
 name   | text             # Way name - if any
 length | double precision # Length in meters
 geom   | geometry(LineString,4326) # Geometry

Nodes table, contains only nodes on intersections of the ways.

 Column |         Type
--------|----------------------
 id     | bigint  # OSM ID of a node
 lon    | double precision
 lat    | double precision
 geom   | geometry(Point,4326)

If using the way splitter the new, artificial, nodes are given IDs equal to the original node ID * 10000 + internal counter.

Address exporter

Address exporter exports address data from OSM PBF file using pyosmium library.

Example call: ./topo_import.py --address-import --pbf ~/poland-latest.osm.pbf --cache-mem once exported, the IPython embed console will show and can store the data with: extractor.save_to_csv("output.csv")

With --cache-mem will require around 12GB of free RAM to extract from Poland. Without this option uses a node-cache in file that can take around 30GB of space.

License & Credits

This program is licensed under GNU GPLv3 license.

The PBF file parsing is based on a GPLv3 parsepbf project, found originally at: http://pbf.raggedred.net/parsepbf.tar.bz2 Original readme is located in osmpbf/readme.txt.

Library was heavily rewritten to use a callback approach during reading to ease the RAM usage and refresh the code a bit.

About

Fast and hackable importer for Openstreetmap PBF files into the PostgreSQL database.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published