State of the Gazetteer in 2023 This is a blog post by nvkelso that was published on Jun 07, 2023 and tagged whosonfirst, wof, data and analysis

The Who’s On First (WOF) gazetteer is a big list of places, each with a stable identifier, and a set of common and optional descriptive properties about that location. Like any product, Who’s On First is made by people, and we’ve collectively spent about “20 years” building WOF as an original work and as an aggregation of many other open data projects.

Since we launched in 2015, the project has grown in coverage, complexity, and supported applications. In this this post I will summarize Who’s On First’s key advantages, offer a comparative analysis of WOF and other open gazetteers, quantify our global coverage by placetype, offer score cards by country, dive into name localization, look at internationalization through the lens of disputed territories, and quantify geometry types and sources of those polygon and points, hold hands with and thank our sources, and invite collaboration.

WOF gazetteer’s focus on localities and unique identifiers sets us apart from many other projects. We choose to approach geography at a personal level – starting with the locality (or populated place) where people play, study, work, and sleep – and country for when people need a passport to cross a frontier and experience something new and exciting.

Between those “top” and “bottom” placetypes in the administrative hierarchy, countries choose to order their geography in many varied and nested subdivisions. Who’s On First doesn’t judge, but we do align them into common region (1st order subdivisions, tracked by the ISO), “county” (2nd order subdivisions), and localadmin (municipalities) placetypes. Most localities aren’t legally incorporated, and may or may not be coincidental with a parent localadmin. Many urban localities are further subdivided into borough and neighbourhood areas. There are even more optional placetypes to allow for intermediate levels and other edge cases.

This common structure allows us to track all the places in the world. Or at least those places we’ve learned about thus far. We hope it provides you an easier mental model to integrate the data into your own application.

Eager to play with the data yourself? Jump to the downloads section below…

Who’s On First gazetteer at a glance

• A single source of truth across a minimalist common set of placetypes, spanning the entire globe, and supplemented as-needed on a regional level
• Global coverage for 5M (million) administrative places, including 4.5M localities
• More than 25M additional places are available, including: 3.9M postal codes, 7K (thousand) constituencies, and 21M venues
• 452K features have detailed polygons, with complete coverage at country, region, and county placetypes and exhaustive coverage for localadmin and large localities in major industrialized economies, with exhaustive coverage as point geometries for remaining features. We also store 447K alternate geometries.
• Names are localized into hundreds of languages, using internet standard language codes
• Population values, min_zoom ranking, and polygon label centroids enable sophisticated map designs with beautiful and informative cartography
• Internationalized with disputed territory boundaries for global audiences and compliance with local regulations
• Open license (CC-BY attribution) means you’re free to use the data for commercial purposes, including derivative rights, as long as you credit the project and our sources
• Includes original work and data aggregated from 360+ authoritative sources. Our source data licenses have been vetted by multiple corporate attorneys at large, publicly traded companies.
• Holds hands with 60+ datasets via linked unique identifiers to allow crosswalk across 6.5M feature-level concordances.
• Spelunker web app for viewing formatted data with a map 
(browse New York city record)
• Write Field web editor for quick property edits 
(edit New York city record) with human review
• View raw data on Github (view New York city example)
• Downloadable SQLite and Shapefile distributions from geocode.earth (thanks!)
• Collaborative project started in 2015 and remains active, including contributions from Snapchat
• The Who’s On First project is hosted by the Linux Foundation

Comparative Analysis of Open Gazetteers

We recognize our WOF gazetteer is one of many open gazetteers, let’s look at five, including Who’s On First:

• Who’s On First: Open data project with both administrative and non-administrative data like postal codes, constituencies, and venues. The CC-BY license allows commercial use. Includes polygons, fully localized names, disputed territory handling for reverse geocoding, navigation centroids, and unique IDs to capture value from metrics. The data is used in applications with over 300 million monthly active users.
• GeoNames: Commercial project across a range of administrative and non-administrative placetypes including postal codes and venues. Premium polygons can be licensed for a fee. Great locality coverage!
• geoBoundaries: Academic project focused exclusively on administrative hierarchy. A complicated license that mixes in ODbL and other non-commercial data sources with CC-BY data. Names are not localized, no disputed territories, no label centroids, and no unique IDs.
• GADM: Academic project focused exclusively on administrative hierarchy. Non-commercial use license although a custom commercial license is available on request. Names include latin, variants, and local script but not in a machine readable way. No disputed territories, no label centroids, and no unique IDs.
• All The Places: Open data project with CC-0 license focused exclusively on venues.

LEGEND for tables below

• ✅ Great coverage and usability
• ☑️ Basic coverage or usability
• ❌ No coverage or usability (unless noted)
• Unique IDs must also be stable
• L10n = Localized names
• i18n = Internationalized features
• Links = Concordance 🔗 with other gazetteer projects

Gazetteer data competitive analysis table

* WARNING: geoBoundaries contains multiple sources, many of which are ODbL and some of which are CC-BY. Discretion is advised!

DISCLAIMER: The above table represents a good faith effort to compare open gazetteer projects with nominally CC-BY and more permissive licenses in May 2023. Each project has its own motivations, use cases, and update frequencies. Major respect for everyone involved (and to OpenStreetMap and their ODbL effort).

Gazetteer applications competitive analysis table

* While Who’s On First does includes an optional API it is not currently hosted online.

NOTE: geoBoundaries has a great web viewer to compare data for countries and their subdivisions across projects – including those here in WOF and from the lower geometry precision Natural Earth.

Who’s On First global coverage deep dive

Who’s On First has 5M administrative places, including 4.5M localities. Detailed breakdowns by placetype, geometry type, country & etc are available farther down in the “By Geometry Type” section.

Additional coverage is available in the WOF gazetteer for 3.9M postal codes, 7K constituencies, and 21M venues, but they are largely excluded from the “administrative” tables below, although descriptions and coverage maps are provided.

Coverage by placetype

Locality coverage

Localities form the bulk of place records in the WOF gazetteer. Of those 4.5 million records, 95% have point geometries and 5% have polygons. Distribution of our localities follows population density. See the “Scorecards” section below for a description of which countries have locality polygon coverage.

Primary administrative hierarchy coverage

Gazetteer projects must decide how to set up 1st order administrative divisions around the world, and this decision has repercussions in calculating the depth of administrative levels in a country.

Take for example the United Kingdom of Great Britain and Northern Ireland. Are the 4 “constituent countries” of the U.K. the 1st order subdivisions or are their 334 subdivisions? Are the 101 provinces of France the 1st order subdivisions, or the 29 regions? Are the “overseas” regions and provinces of France part of the main country or different countries? What about the United Kingdom, France, and other country’s dependencies? What about Belgium and its federal system? Or the United States and it’s dependencies?

Who’s On First tracks countries and dependencies as equal top-level placetypes and group them into empires as appropriate. WOF chooses to represent all ISO 1st order subdivisions as “region” placetype features in a continuous fabric of global coverage. When the ISO also provides groupings of those, as reported by the country itself for administrative (versus statistical) use, we import those as an optional macroregion with complete in-country coverage but sparse global coverage. This situation (and others like in Belgium) impact over a dozen countries globally.

“What are…” in WOF: the United Kingdom has 4 macroregion and 334 region records; France has 29 macroregion and 101 region records; the United States has 51 region records (including the District of Columbia which is marked in label:eng_x_preferred_placetype as a “federal district”). Puerto Rico is a dependency of the United States empire. Naming placetypes in inherently controversial so we also provide an optional and localized “label:{lang}_x_preferred_placetype” property on records that carries the specific word used for that place in its country and translated to English (including the region of Riyadh in Arabic script, latinized Arabic, and English).

(below) Feature counts for each placetype are included in the legend. Every feature on the map below has a polygon geometry, but is represented as a point for visualization purposes.

Localadmin coverage

Most of the 230K localadmin placetype features have polygon geometries. See the “Depth of administrative levels”, “By geometry type”, and “Improving WOF data coverage and quality (a more detailed look)” sections below for more information.

Campus coverage

Most of the 24K campus placetype features are airports, but we include national park polygons in select countries, and in the USA there are ~ 20K mobile home parks.

Other core “administrative” placetypes coverage

Additional placetypes in the “admin” repos are included for convenience to name and/or reverse geocode for continents, oceans, marine areas, time zones, and more exotic features like empires and market areas. We’re also wondering what those 2 unknown features are!

Postalcode coverage

Who’s On First has 3.4M postalcodes with a range spatial aggregations, geometry types, and polygon accuracies:

1. Polygons for fully qualified postalcodes (e.g. “95501” in the United States sourced from the US Census Zip code tabulation areas, and also available in Australia, Australia, Finland, France, and Switzerland)
2. Polygons for postal aggregation areas (Canada, Netherlands). Think “955” in the United States which is useful for calculating regional statistics.
3. Point locations for specific delivery routes, like “SW1A 1AA” in the United Kingdom (think ZIP+4 in the United States) which have a precise location (the same situation is true for Japan) or an approximate location mapped to the nearest available polygon centroid (like in Canada which has a combination of precise and official 3-character “forward sorting area” aggregation polygons and approximate centroids for the 6-character postal codes).
4. In the Netherlands we also have a 4th type of polygon created from the alphashape of the postalcode attributes on WOF venue point locations.

The majority of WOF postalcode records exist in other countries – but their locations are approximate (or are visiting Null Island) and are not shown in the map below based on the shapefile distribution – but they are included in the SQLite distribution. If you’re interested in mapping postalcodes to the post office / locality names, we provide that for the United States in the mz:postal_locality property. If you need global mapping, see Pelias’ postal cities project.

NOTE: The United Kingdom and Australia receive regular updates, the rest are an older vintage and could use a refresh.

Several detailed map views are provided farther below for the more unusual cases.

Vancouver, Canada postalcodes

When precise locations for postalcodes (like the “V5K 1A2” delivery route) are not available they are sometimes mapped to the polygon centroids of their parent postal area (e.g. “V5K” postal aggregation area) in Vancouver, Canada shown below.

London postalcodes

Detailed point locations for specific delivery routes like “SW1A 1AA” are available in the United Kingdom, updated approximately quarterly, with the London metro area shown below.

Tokyo, Japan postalcodes

Similar point geometries are available in Japan, with the Tokyo metro area shown below.

Amsterdam, Netherlands postalcodes

Several experiments have created alphashapes from WOF venues address data, including in the Netherlands. These approximate polygons are marked “mz:is_approximate” and capture the bounding box extent of the zipcode and the label centroid in search and even reverse geocoding, but are not suitable for display in a thematic map.

Constituency coverage

For national and regional legislative bodies, including upper and lower chamber indication via “wof:association”. These will need a refresh for the 2023 redistricting in the United States. Canada includes data in British Columbia only.

Venue coverage

Back in 2015 and 2016 we imported 21 million venues in 63 countries from the SimpleGeo CC0 venues dataset. We added place hierarchy by reverse geocoding these points against the WOF administrative polygons, and otherwise normalized them into the WOF schema. But venues come and go – they are more ephemeral than administrative data – especially the last few years during the COVID pandemic. So the WOF venue coverage will lack newer businesses and still include businesses that have closed IRL. A separate analysis of WOF venues is available in an earlier blog post.

By Geometry Type

Nearly 10% of Who’s On First’s 5M administrative records have polygons, with complete global coverage at the country and dependency (collectively the “top administrative level”), and region (1st order administrative subdivision) levels. Nearly continuous global coverage is provided at the county (2nd order administrative subdivision) level. Many countries also include complete polygon coverage at the localadmin level.

Countries with any coloring in the “Country scorecards” map in the next section include complete locality polygon coverage. Other “gray” colored countries in that map will only include approximate locality polygons for the country capital and a handful of other regionally important localities.

The WOF gazetteer also stores nearly 0.5M alternate geometries. This can occur when we upgrade point geometries to polygons, have multiple polygons to choose from but we designate just one the default and store the others as alternates, deal with coastline clipping for display versus territorial waters for reverse geocoding, or need to manually curate a custom polygon (and store the original as an alternate geometry).

Every default polygon feature includes a label centroid geometry suitable for beautiful cartographic labeling derived either from MapShaper or manual curation. Sometimes navigation, reverse geocoding, and other specialized centroids are also provided.

(below) The count of WOF features grouped by placetype, how many of those have default point or polygon geometry, how many have alternate geometries (and the ratio of alternate geometries to the count of features in that placetype), a point area indicates an WOF error mostly from us add or remove neighbourhood polygon geometries, the polygon area in equal area square kilometers, and the ratio of that area compared with the scope of the globe the placetype should cover.

NOTE: Postalcodes are not tracked in this table. See Postalcode coverage section above.

By Geometry Source

Geometries directly imported by Who’s On First from source

Who’s On First both makes our own data and imports data directly from governments and other reputable sources. WOF tracks sources for each geometry in the “src:geom” property, with values being a source’s “name” in the sources repo, while imported property’s from that source will have their name prepended with the “prefix” in the source’s JSON file. Hyperlinks are provided in the table below directly to that source’s JSON file (while the sources link above is human readable formatted text).

Quattroshapes geometries imported to WOF from aggregator

These are the OG polygons that WOF launched with in 2015 thanks to Foursquare’s open data Quattroshapes gazetteer. They range across all administrative placetypes from country to locality to neighbourhood.

Who’s On First indicates if a geometry is from an aggregator in the main “src:geom” property, and lists out those sources the sources repo. Per record fidelity is available by dipping into the “qs:source” property; we plan to promote that information into a new and generic top-level WOF property.

Mesoshapes geometries imported to WOF from aggregator

These are mostly county placetype default geometries.

Who’s On First indicates if a geometry is from an aggregator in the main “src:geom” property, and lists out those sources the sources repo. Per record fidelity is available by dipping into the “meso:source” property; we plan to promote that information into a new and generic top-level WOF property.

WOF Gazetteer Data Scorecards

Country scorecards

We rate 87 countries with good to great polygon coverage at both locality and parent administrative subdivisions, including country, region (state/province), and county, in the A-AA-AAA-AAAA range. Our disputed territory polygon coverage also falls in this “great” data grouping.

We rate 177 countries that are mostly limited to point locality coverage and basic polygon administrative subdivisions in the B-BB-BBB-BBBB range. We almost always have “complete” coverage in these countries, however, often tens of thousands and sometimes with hundreds of thousands of features are point geometries only.

Breakdowns of each of these classes are available below the map…

A range coverage by country

AAAA: Starting in 2017, we have rebuilt priority countries by importing authoritative, polygon-based records at all placetypes from region down to locality to provide AAAA data quality and coverage in: Australia, Austria, Canada, Estonia, France, Germany, Hong Kong, Italy, Netherlands, New Zealand, Poland, Romania, Singapore, Spain, Switzerland, United Kingdom, and United States.

AAA: Other countries in North America, Europe, Middle East, Asia, and elsewhere have had selective rebuilds at a mix of placetypes to provide AAA data quality and coverage using national mapping agency data – though possibly of an older vintage that may not reflect all recent administrative updates. These include: Akrotiri Sovereign Base Area, Aland, Albania, Bahrain, Belgium, Brazil, Bulgaria, Croatia, Cyprus, Czechia, Denmark, Dhekelia Sovereign Base Area, Faroe Islands, Finland, Greece, Greenland, Guernsey, Hungary, Iceland, Ireland, Isle of Man, Kuwait, Latvia, Liechtenstein, Lithuania, Luxembourg, Moldova, North Macedonia, Northern Cyprus, Norway, Portugal, Puerto Rico, Qatar, San Marino, Saudi Arabia, Slovakia, Slovenia, South Africa, Sweden, Turkey, Ukraine, United Arab Emirates, and Vatican.

AA: WOF provides AA data quality with extensive (but not universal) polygon coverage at locality level in Chile, India, Indonesia, Iraq, Israel, Mexico, Morocco, Nigeria, Pakistan, Palestine, Philipeans, South Korea, and Taiwan. These polygons often date back to 2013 and represent the largest urban settlements.

A: WOF provides A data quality with approximate locality polygons in China, Japan, Malaysia, Russia, and Thailand. Several countries with extensive and higher quality point locality coverage, including: Columbia and Honduras.

B range coverage by country

BBBB: WOF provides BBBB quality data with dense point locality coverage in Afghanistan, Argentina, Bangladesh, Baykonur, Bolivia, Iran, Kazakhstan, Mozambique, Myanmar, Nepal, Paraguay, Peru, Tanzania, Venezuela, and Vietnam.

BBB: WOF provides BBB quality data with dense point locality coverage in Angola, Belarus, Bosnia and Herzegovina, Congo (DRC), Costa Rica, Gabon, Madagascar, Mongolia, North Korea, Rwanda, Uganda, and Yemen.

BB: WOF provides BB quality date with medium density point locality coverage in Armenia, Burkina Faso, Burundi, Cambodia, Cameroon, Central African Republic, Congo (ROC), Cuba, Dominican Republic, Ecuador, Egypt, Ethiopia, Ghana, Guatemala, Haiti, Ivory Coast, Kosovo, Kyrgyzstan, Laos, Mali, Montenegro, Namibia, Panama, Papua New Guinea, Serbia, Sierra Leone, South Sudan, Sri Lanka, Sudan, Syria, Tunisia, Uruguay, and Zambia.

B: WOF provides B quality date with basic density point locality coverage in_ Algeria, Azerbaijan, Belize, Benin, Bhutan, Botswana, Chad, Djibouti, El Salvador, Equatorial Guinea, Eritrea, eSwatini, Gambia, Georgia, Guinea, Guinea-Bissau, Guyana, Jamaica, Jordan, Kenya, Lesotho, Liberia, Libya, Malawi, Mauritania, Nicaragua, Niger, Oman, Senegal, Somalia, Somaliland, Suriname, Tajikistan, Togo, Turkmenistan, Uzbekistan, and Zimbabwe. This category also includes several smaller island countries._

NOTE: We still provide detailed region and county polygon coverage for all these to generate full administrative hierarchies, except counties in Namibia, Botswana, Belarus, Bulgaria, Turkmenistan, and Uzbekistan. Greenland, Iceland, and Western Sahara do not have county subdivisions IRL.

Data size by country

NOTE: Methodology: https://data.geocode.earth/wof/dist/shapefile/inventory.json to get sizes per country.

Even though most countries are limited to locality points past the county polygon level, those features (and their properties including name localizations) can add up to sizable downloads, especially when mixed in with the polygon features.

Features by country

Looking at “administrative” placetype features by country (country code), the trend mostly follows population density.

Depth of administrative levels by country

Who’s On First tracks 1st and 2nd order administrative divisions for most countries and dependencies (so three levels deep including country (and/or dependency and empire), region, and county). A handful only have 1st order divisions (2 levels). Around 39 countries include 4 to 7 administrative levels. Around 24 of those include features at the municipality (localadmin) level. We have point records only for localadmin in another 19 countries, and anticipate polygon features for this placetype can be imported for another 53 countries (see “Future Directions” section below).

NOTE: All the countries in the map above also include locality points and polygons but because those don’t form a continuous in-country or global fabric this level is excluded from the counts above.

Vintage of data by country

While 2013-era Quattroshapes is the skeleton for Who’s On First, we’ve substantially updated the gazetteer in the last decade. In 2015, we imported the original Quattroshapes polygon features to normalize them and otherwise clean them up. In 2016, we added polygons for 2nd order administrative divisions (“county”) for almost every country and dependency in the world. In 2017, we imported Quattroshapes point gazetteer and GeoNames “locality” points to provide global coverage at that placetype in all countries. Most country and dependency capitals and major metropolitan centers were updated with approximate locality polygons in 2021. Select countries have been rebuilt in whole or partially since 2017, including 42 countries in North America, Europe, and elsewhere.

Most of the AAAA and AAA countries (see section above) have a 5- or 10-year census, and we attempt to incorporate polygon data from their most recent census &/or INSPIRE-related open data efforts. Some eastern European countries have earlier 2013 vintage locality polygons. Generally region and county level changes less per year, with the localadmin level experiencing more consolidation and occasional splits on an annual basis (~1% change per year).

When Who’s On First observes a change in a country’s administrative subdivisions we add the new or changed features with an inception date, and mark the old ones with a cessation date using the expressive U.S. Library of Congress’ Extended Date/Time Format (EDTF) syntax. When possible, we also point back and forth between new and old with supersedes and supersedes_by properties. This allows older administrative units to be queried within a date range in the SQLite distribution (while the Shapefile distribution excludes non-current records). We’ve also experimented with the same for countries, like the former Yugoslavia, which we discussed in an earlier blog post, and for the 2016 French “region” level consolidation (for example the WOF macroregion of Rhone-Alpes was consolidated into the new Auvergne-Rhone-Alpes WOF macroregion).

We catalog changes to our place data in the CHANGELOG by year, month, and country.

Select data properties

Population by placetype

By tracking population (and the related population_rank), similarly named places can be ranked and disambiguated in search and visually distinguished in map display. For example, there are many places named San Francisco in the world, but only one with an exact name match also with the largest population located in the California region of the United States.

For global coverage placetypes like continent, country (and dependency), region, county, and locality the population should eventually sum to the planet’s population – modulo import vintage and population growth and the “per global” provides an coverage &/or accuracy scoring. For other placetypes like macroregion and localadmin they should sum to the population in the covered countries (not included in the table). Other placetypes are not available globally so the “per global” indicates the global significance of that placetype when viewed by population instead of feature count.

(below) This chart sums available population values on WOF records grouped by placetype. Some placetypes that have global coverage should sum to the current “per global” population estimate of 7.888 billion people but currently may not because (a) the vintage of per feature population estimates either lags behind actual population growth or may over-estimate population e.g. because of disputed territories and (b) some features lack population entirely.

Zoom ranges by placetype

Some records should be displayed earlier on a map and some later, and not all features within the same placetype are equal. For example, region subdivisions in the United States are large geographically compared regions in Albania so the former should be shown earlier and the latter shown later. This is especially true of localities where most localities can be labeled on a zoom 12 map – but only a few shown on a zoom 2 map (especially considering name translations within a limited vector tile budget).

A few areas stand out for future work: county zooms should be spread out (mostly increasing them from zoom 6 to later zooms and as an offset for their parent region’s zoom), some localadmin are visible too early, some macrohood are visible too late, and some neighbourhood and microhood are visible too early – this can be mitigated in application logic when using the data but we should clean it up at the source.

(below) This chart counts available min_zoom values on WOF records grouped by placetype. Course placetypes group features that are suitable for labeling on a low-zoom world map as “countries” or their 1st and 2nd order subdivisions, generally from zooms 0 thru 8 (though some counties should label later). Settlement placetypes include both locality and localadmin and are rarely labeled at low zooms, typically coming in the mid-zooms of 9, 10, 11, 12, and 13. Neighbourhood placetypes are subdivisions of settlements and generally should only be displayed at high-zooms of 11 thru 17.

NOTE: The zoom ranges in the table above are for standard 256 px sized tiles, which map to the following natural scales and match Leaflet and Google Maps. When viewed in the Mapbox or MapLibre rendering library the zooms will be offset by 1 due to their assumption about 512 px size.

Improving WOF data coverage and quality

As human population grows (and declines) so the world’s heart beats, and so our work as gazetteer maintainers carries on – by both repainting and “touching up” the data.

Our top two priorities in 2023 are:

1. Improve locality coverage in India, and
2. expand localadmin coverage globally

We expect to land the India locality coverage by June this year with polygons for most localities and points for smaller rural villages.

Our localadmin efforts are largely focused on achieving complete coverage in Europe, introducing localadmin coverage in Africa, and expand localadmin coverage in Asia and South America.

Priority countries (blue outlines below) include: Indonesia, Mexico, Ireland, Portugal, Belgium, and Luxembourg. An additional 26 countries (black outlines below) have been prioritized for minor edits or more complete imports &/or rebuilds. Please reach out to help :)

Detailed plan for 2023

We have sourced for import in 2023 new data from national mapping agencies in 12 countries, potential NMA data in another 7 countries, 39 countries at the localadmin level via the geoBoundaries project, and need to review licenses on another 17 localadmin geoBoundaries countries. We’re still searching for data in several priority countries, including Costa Rica, Egypt, Peru, and Thailand.

For densely populated countries with complex administrative subdivisions (like in China and India), we may consider adding an optional microcounty placetype between county and localadmin.

While we don’t have concrete plans for expanding postalcode or constituency coverage, if those are important to you please reach out. Two tractable problems are importing newer open polygon data for postalcode features in Europe and rescuing point geometry postalcode records that are visiting Null Island.

Localization

Who’s On First includes name localizations (L10n) into 494 languages. Individual language coverage ranges from rich to sparse. Major cities like New York are translated into most of the languages, but a rural locality is probably only provided in the local language and translated into English and/or the latin character set.

While the source GeoJSON and SQLite distribution include all languages, for Shapefiles we provide a more ergonomic experience by pre-joining the SPR to the names table to include the default and 25 more localized names, when available, for:

Arabic, Bengali, Chinese (simplified and/or traditional), Dutch, English, Farsi, French, German, Greek, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Polish, Portuguese, Russian, Spanish, Swedish, Turkish, Ukrainian, Urdu, and Vietnamese

Who’s On First uses the RFC 5646 / BCP-47 language indications for names to specify a 3-character code for the name translations, like name:{language}_x_preferred in the GeoJSON and name_{locale} in the Shapefiles. For example, English is stored as name:eng_x_preferred in the GeoJSON and name_eng in the Shapefiles.

The list of supported Shapefile languages is adapted from Natural Earth and Tilezen’s list of core languages. Arabic, Chinese, English, French, Russian and Spanish are used by the United Nations for meetings and official documents. The other languages listed are either proposed as an official language of the United Nations (Bengali, Hindi, Portuguese, and Turkish) or frequently used in OpenStreetMap, Who’s On First, or Wikipedia.

If you need to discern the “local” name for a given place record, the parent country record indicates official and/or spoken language codes. In some countries the parent region (or macroregion) record will also indicate language preferences (like Catalonia in Spain), which can allow for local variations.

By current WOF convention, a feature’s default name is stored in ASCII-7 English. For latin script based languages you can maximize localized name coverage by coalescing name:{lang}_x_preferred, name:eng_x_preferred, and wof:name. Specific coalesce logic depends on your application and the locale(s) used by your audience.

Localized name coverage by placetype

By convention, Who’s On First tracks a single “preferred” name spelling per language for a place. Because of the many different ways to translate and/or transliterate a place’s name from one language into another WOF also allows multiple “variant” names per language. Some languages have formalized rules for transliterating into another, but those rules can evolve over longer spans of time (e.g. romanization of Chinese) and generate time and system variants. Languages without formalized transliteration rules can organically generate even more variants.

The large variation in names is one of the reasons gazetteers like Who’s On First assign unique IDs for our features and, when we discern a match in another gazetteer, provide ID crosswalk between the “this is the same as that” features.

Localized name coverage by language

The most common language in Who’s On First is English, followed by Chinese, and French. Rounding out the top 10 in ranked order are Russian, German, Dutch, Swedish, Spanish, Italian, and Polish.

The table below only includes the top 37 languages. A total of 63 languages have more than 100,000 preferred names. Another 45 languages have more than 50,000 preferred names.

NOTE: The default language for wof:name is English, so almost all records include an implicit default English name and many also include an explicit localized English name. The Cebuano ceb localizations are largely from a Wikipedia bot and are suspect so not included in the stats above. The Norwegian language uses multiple codes, including: nno, nob, and nor.

Internationalization

For internationalization (i18n), we track 104 “disputed” territories (polygons between countries for contested areas) around the world, and Who’s On First hold hands with Natural Earth allowing you to make use of their extensive catalog of pairwise points-of-view.

Disputed placetype records include an optional mz:hieararchy_label property (and related properties for each ancestor level in the placetype hierarchy) to indicate if place records that reverse geocode within the disputed area should include full, any, or partial text strings for that record’s ancestors. For example, Western Sahara which, assuming a reverse geocoder like Pelias has implemented the flag, won’t include text for country or region ancestors but will allow showing locality names. We recommend using the Natural Earth point-of-views to localize the hierarchy label properties (e.g. to allow viewers within Morocco to also see country and region ancestor details).

NOTE: The Hans Island disputed between Canada and Denmark has been resolved and needs to be updated. Huzzah diplomacy!

Open Data

Who’s On First’s permissive, attribution required (CC-BY equivalent) open license means you can use the data however you want (including for commercial purposes), as long as you provide credit somewhere in your map or app.

Crediting the Who’s On First project is required because some of the open data projects that WOF aggregates require attribution. Linking back to the License is required in hypermedia projects.

For example, on a website or in an app’s about page, you should also include the License hyperlink:

Data from Who’s On First. License.

For example, in a web map’s sources (displayed on the map), hyperlinked to the License:

Who’s On First

Where the “License” text (or project name) hyperlinks either to the whosonfirst.org hosted license page, or to a detailed data licenses page (or section of your terms of service) on your web site or in your app. At that location, you must credit the Who’s On First sources that require attribution. It may be more practical to credit both Who’s On First and all of our sources – it’s the kind thing to do.

For stand alone print maps, a simple text credit to the project will suffice:

Who’s On First

If the map is included in a book or atlas publication then the data credits section should also repeat the Who’s On First credit and the web link (https://whosonfirst.org/docs/licenses/).

Sources

The Who’s On First dataset is both an original work and a modification of existing open data. The WOF gazetteer includes data from 360 sources. There are 139 primary sources, 188 additional sources via Quattroshapes, Mesoshapes, and other aggregators, and 33 concordance-only sources.

Sources include national mapping agencies in: United States (US Census), Australia (PSMA and ABS), Austria, Belgium (Geopunt), Brazil (IBGE via Quattroshapes), Canada (multiple, including Census Canada and Statistics Canada), Denmark, Estonia (Land Board), Finland (National Land Survey), France (IGN), Germany (BKG), Ireland (Ordnance Survey), Italy (IGN via Quattroshapes), Japan (GSI), Mexico (INEGI and IGN), Netherlands (CBS and NLD Kadaster), New Zealand (LINZ), Norway (Geonorge), Poland (GUGIK), Portugal (DG Territorio), Romania (ANCPI), Spain (IGN via Quattroshapes), Sweden (Lantmateriet), Singapore, Slovenia (EPG), South Africa (NBC and Municipal Demarcation Board), Switzerland (SwissTopo), and United Kingdom (Ordnance Survey).

Other global sources include: GeoNames, Yahoo’s GeoPlanet, Foursquare’s Quattroshapes, and Mapzen’s Mesoshapes. Several of our founders and core contributors were instrumental in these other projects.

We detail all sources and their specific license, usage, and vintage metadata in the full sources list.

Concordances

WOF includes a large number of concordances to allow our project to “hold hands” with other datasets – individual WOF records link to specific features in other open and closed gazetteer projects.

Not all sources include concordances, but when they do we indicate in the condances section the dataset prefix (like gn for GeoNames and wd for Wikidata) and the name of the unique identifier that project uses (mostly id, so put together gn:id and wd:id fully qualified property names).

Who’s On First tracks our sources in the sources repo. Besides the concordances, WOF also notes the source for each geometry in the “src:geom” property (using the JSON “name” of the source) and imported properties from that source will have their name prepended with the “prefix” in the source’s JSON file. This is mostly machine readable today and we’re working to make it fully machine readable soon.

(below) Use the sources list to decode the source project prefixes used in the following table.

Applications

When the Who’s On First (WOF) gazetteer started in 2015 we partnered closely with Pelias, another Mapzen project, to ensure our big list of places could power:

• Search for forward geocoding based on full name string input and type-ahead “autocomplete” based on partial string matching per keystroke
• Hierarchy lookup allowing a place’s parent and other ancestors to be reverse geocoding from a given latitude, longitude input. Specific attention was paid to disputed territories and which hierarchy components should be masked in a reverse geocoding lookup.
• Single feature display with label centroids for polygons, label bounding boxes, detailed polygons.
• Metrics logging is possible because of WOF’s stable unique IDs – to turn data exhaust into insights about customer usage patterns.

The results of a search can then be pipped to Valhalla, another Mapzen project for:

• Routing with polygon navigation and label “centroids” that are close to a place’s urban center rather than the math centroid of the polygon, and guaranteed to be on land.

Places can also be added to vector tiles Tilezen, another Mapzen project for:

• Basemap with label “centroids” for neighbourhood polygons.

All these Mapzen projects (Who’s On First, Pelias, Valhalla, and Tilezen) are now part of the Linux Foundation and are free for anyone to use and adapt.

Future applications looking for collaboration and funding

Map display for label points

• Label min zoom and label max zooms exist but can to be fine tuned for global, multi-zoom basemaps (especially for county and locality placetypes)
• Population estimate coverage expansion to determine relative label zoom and townspot size grading (for latest census counts)

Map display for thematic polygons

• Min zoom and max zoom variable per placetype and per country with area grading for a composite global, multi-zoom basemap
• Vector tilesets (global and per placetype) with newer tools like Tippicanoe and Planetiler delivered as serverless PMTiles archives
• Edge matchingof boundaries between countries
• Land clipping of boundaries inside countries for terrestrial land mass. This is a partially addressed problem in the USA for data from the US Census, and sometimes occurs in Europe
• Quality checks to ensure no feature dupes between placetypes (e.g. localadmin paired with locality, and neighbourhood paired with localadmin) and that historical features are marked not current

Single feature display

• Preview polygons with pre-cached, small “200 kb” display ready alternate geometries as existing default polygon geometries vary widely in size from a few KB to more than 50 MB. These are useful for displaying search results with a polygon (not just with a centroid zoomed to the bounding box).

Hierarchy

• Disputed point-of-view: For administrative hierarchy, further work to power “point-of-view” on the hierarchy masks, can be adapted from Natural Earth’s efforts for automated reverse geocoding so users in various countries get locally adjusted polygon data instead of globally masked data

Names and Concordances

• U.S. Board on Geographic Names: Adding concordance IDs and toponymic information from the Geographic Names Database, containing official standard names approved by the United States Board on Geographic Names (BGN) and maintained by the National Geospatial-Intelligence Agency.

Routing

• Intersect with urbanized landcover to ensure feature point geometries are inside urban areas and polygon feature’s navigation points are in the place’s urban center (ideally downtown)

Placetypes

• Expand localadmin placetypes globally (see section above)
• Update United States placetypes for Census 2020 data releases
• Update European placetypes for newer data releases, including in Germany, France, and Netherlands.
• Add Indonesia placetypes for newer data releases
• Add Vietnam placetypes for newer data releases
• Update Brazil placetypes for newer data releases
• Update Mexico placetypes for newer data releases
• Expand postalcode polygon coverage using latest open map data, especially in Europe
• Expand constituency coverage outside of the USA using latest open map data
• Add statistical aggregations like NUTS and US Census
• Add metropolitan/micropolitan/rural area globally using original research
• Add colloquial areas (like Outer Banks) via crowdsourcing

Example applications by placetype

When decomposing address data into placetypes (using libPostal, an OpenAI prompt, or another tool), it can be helpful to think how those strings can geo match to Who’s On First records.

Decomposing Data…

…into Data Sources

Who’s On First is a “coarse” geocoder meaning it doesn’t provide street level features. But WOF data can be complemented (as Pelias does) with data from other open data projects, including: OpenAddresses, All The Places, Natural Earth, and even the ODbL licensed OpenStreetMap for a complete geocoding solution.

If you need to process and deduplicate venue data using latitude, longitude, name, and address information we also recommend Lieu (code repo and blog post).

_NOTE: While our WOF gazetteer does include 21M venue records they are of an older vintage and do not reflect closed venues or recently opened venues. Depending on your needs, it may be better to use All The Places (scraper project for international, national, and regional chains), OpenStreetMap (caveat ODbL license restrictions), or license data from proprietary sources including Foursquare_, SafeGraph, TripAdvisor or others. Postalcode polygon country coverage can be augmented with proprietary sources like MBI.

Collaboration

Most people engage with the project by downloading and “reading” the data. In the course of using the data, you might find a few tools helpful both for finding specific records and for editing them to fix small problems in the data (like a misspelled name).

View Who’s On First records

The Spelunker is a web tool for browsing Who’s On First (shown in the screenshot immediately above) and includes search functionality. For example: view New York city.

You can also View raw data on Github. Because navigating the many whosonfirst-data repos can be overwhelming, each individual WOF record self describes in which repo it can be found. For example, view New York city as raw GeoJSON.

Editing of Who’s On First records

We use Write Field for quick property data edits. For example, load New York city for editing. This requires a Github account to create a pull request which can be reviewed by the WOF team.

For more complex and bulk imports we use QGIS and scripts to iterate over files managed in git. This workflow is described in more detail in our recent Shapefiles blog post.

Downloads

Accessibility facilitates access for all and Who’s On First data is available to download in several formats, including:

• Shapefiles - for GIS mappers in applications like QGIS and ArcGIS
• SQLite - for database driven applications

Thanks to Geocode Earth for sponsoring processing and hosting of the downloads.

While we manage Who’s On First place records as individual text files in git repos hosted on Github, from the early days we’ve made “bundled” distributions available. Those distributions were more geared for software engineers than for a more general audience of map makers and cartographers.

Shapefile downloads for the Who’s On First gazetteer were added in April 2023 as per-country ZIP archives including admin (country, region, county, locality, neighbourhood & more), postalcode, and constituency placetypes. They only contain basic properties, names, and geometries.

SQLite databases contain the full firehose of Who’s On First data, including mixed geometry types and full set of nested GeoJSON properties. Unlike the raw GeoJSON files in the git repos, data in the SQLite databases are organized into several tables, including: spr, names, concordances, ancestors, and geojson. Geometries (default and alternate) are stored as properties in the geojson table. The field layout of the spr table is explained in the Standard Place Response section of the Shapefiles blog post.

Sponsor Who’s On First

A large project like Who’s On First relies on good will to function, some amount of organization to sustain it year to year, and funding to pay data curation, engineering development, and internet server bills.

If you or your organization relies on Who’s On First please considering sponsoring us with a recurring or one-time donation. Please reach out to Nathaniel at wofgazetteer@gmail.com to discuss options.

Contact Who’s On First

If you have feedback, please submit a new issue or discussion topic via Github.

Subscribe to WOF Announcements for “big news”, via Mailchimp. Frequency is a few times per year. Subscribe now and you’ll be first to know about our big India locality data drop!

Join our new WOF Gazetteer discussion group on Google to ask questions and share updates on your country’s latest changes to its internal administrative subdivisions. If you’re a geography geek, this one’s for you.

Need to reach out privately? Email wofgazetteer@gmail.com.

Conclusion

Open gazetteers, including Who’s On First, have come a long way in the last 10 years – largely because of the success of the global open data movement. To the many people who have advocated with your local or national government to “free” this bounty of data, thank you!

Administrative borders are rarely “visible on the ground” and gazetteers that aggregate this data remain relevant by providing important and broad access to this type of map data. Each open gazetteer project has its particular license that reflects origin stories and day-to-day workflows.

At Who’s On First we’re proud to offer our over 5M administrative places and another 25M supplemental places – all with CC-BY attribution – allowing you to both read access and commercial use – with polygons, localization, and internationalization.

Who’s On First holds hands with 360 sources, has great name translations, with unique IDs, and has been confirmed to work in a wide range of applications, several of which have more than 300M monthly active users globally.

While we can and will add even more data, we think there’s enough in Who’s On First for you to get started making a map or building an app today. We’re excited to see what you build!