Travel Mapping Manual: Deal with data errors

The developer tools execute different kind of data checks to indicate potential errors. Highway data managers need to check for these errors to ensure consistent and correct highway data. Some of the data checks are available in the waypoint editor before submitting changes, but many need attention after the next site update.

Data errors
Highway data check list
Marking errors false positive (FP)
Concurrency check
Near-miss points
Marking NMPs false positive FP

When working with the WPT file editor, a data check for the loaded route is executed with every modification. The code of a data error is indicated in the last column of the waypoint table. Use the tool tip for additional info. After manual changes within the editor field, the data check is done on the next editor action, e.g. when pressing the Load button.

Check the table for errors before saving the wpt file. Fix unintended errors or mark intended errors false positive.

Note: Not all data errors are detected by the WPT file editor, see below: supported, proposed but not yet supported, not possbile to be detected in WPT file editor.

When changes to highway data have been submitted and are live on the site, all routes of the project have been checked during the site update process. They are reported on the data check list. The list is sorted by the system categories active, preview and in-development.

Check the table for errors. It is possible to filter the list by system or by region.
Click on the link in the Route column to load the route into the HB.

Fix unintended errors by using the WPT file editor.
Mark intended errors false positive.
Some of the errors are likely fixable from the information in the Log of Near-Match FPs from datacheckfps.csv. These are FP entries which have previously been added but due to minor changes, e.g. repositioning a waypoint, they no longer match the actual FP entry.

Note: It is possible to edit the url in the address bar of the browser to filter for more than one region or system. For instance, it is possible to create a link to all regions a highway data manager maintains. Save it to your browser bookmark and load it when needed for the check.

The last column of the data check list contains the FP entry to Submit.

If no FP Entry is specified, it is always a true error and cannot be marked false positive.

If a FP entry is specified, select and copy it.

To quickly mark the whole FP entry, some web browsers allow to triple-click the entry.

Paste it into your copy of the datacheck FP list.

The list is sorted by region codes. Insert your entry at the right position.

Submit the change. The data error will disappear from the data check list with the next site update.

Do not forget checking for unmatched FPs. Remove them from the datacheck FP list.

Concurrent segments of different routes are detected and considered for calculating user stats. If a concurrency is broken by accident, stats are affected. Highway data managers need to pay attention to check broken concurrencies in their regions. The check can not be done before submitting changes but is currently only possible with the data of the last site update.

The best practise to do this, is as follows:

Open the Highway Data Examiner (HDX).
Go to Option 1 on the left and enter the name of the region you want to check.
Press enter.
Wait. The graph is automatically loaded after a few seconds but it might take longer with large graphs.

You might open up a small graph first, e.g. the District of Columbia graph, then deselect Show Markers and select whatever region graph you are actually interested in to speed up loading.

Press Done on the left to close the AV table.
Deselect Show Data Tables to close the table to the right.
You see a map of the selected region with yellow dots.
If you hardly see anything but only yellow dots, deselect Show Markers.
You should see a map of the selected region with funny colors now.
Select Show Markers again, it was just to show you what happened.
Zoom-in where you want to check concurrencies.
The colors are created by lines connecting the waypoints.

Blue line = one route only
Green line = two concurrent routes
Magenta line = three concurrent routes
Yellow line = four concurrent routes
Red line = five concurrent routes
Aqua line = six concurrent routes
Purple line = seven concurrent routes
Olive line = eight concurrent routes
Orange line = nine concurrent routes
Grey line = ten or more concurrent routes

When there is a green line - 2 routes concurrent - but interrupted by a dark blue line at visible or hidden waypoints, it's a sign that concurrencies are broken

The dark blue is created when two normal blue lines are overlapping.
The funny colors are automatically created when different lines are overlapping. Zoom-in to distinguish them.

Non-standard colors can potentially indicate broken concurrencies.
- When there is a green line (2 concurrent routes) interrupted by a dark blue line at visible or hidden waypoints, it is a sign of potentially broken concurrencies. The dark blue is created when two normal blue lines are overlapping.
- Other non-standard colors are caused by multiple lines overlapping in the same place.
When you spot errors, click on the lines or markers to get info about the route and waypoint labels. It is sometimes difficult to click all routes when they are mostly overlapped.
Fix the coordinates in the corresponding wpt files.
Load the changed wpt files into the WPT file editor to avoid causing unintended data errors.
Broken concurrencies of short segments can hardly be found this way. It is recommended to check NMPs to find these errors.

Where two or more routes intersect, the routes must have a waypoint. If the coordinates of the waypoints are identical, the graph is connected and the Highway Browser can indicate intersecting routes to ease navigation through the routes when mapping travels. Near-miss points (NMPs) are waypoints very close together. They should be checked whether they are candidates to merge to fix broken intersecting links, and broken concurrencies.
NMP files can also be loaded into HDX to visualize their positions on a map. The desired NMP file cannot be selected directly but needs to be downloaded first.

The best practise to check NMPs, is as follows:

Open the NMP files filtered by region directory.
Select the nmp file for the region you want to check.
Open the file.

If it is blank, the region has no NMPs.

If there are entries, download the file.
Open the Highway Data Examiner (HDX).
Go to Option 3 on the left and select the downloaded nmp file.

You can see your region with some colored dots on the map now. These are NMP hotspots.

Zoom-in to investigate the points. Use the table on the right to go through the NMP hotspots.
Since all pairs of all involved routes are reported, very often more than just one line of the table on the right corresponds to a NMP hotspot.

Green dots are those which are already marked FP.
Yellow dots are off by exactly 0.000001°. This is likely intentional to brake concurrencies and the waypoints are candidates to be marked FP.
Red dots must be checked more detailed. These are most likely broken concurrencies or intersecting routes where the waypoints do not match.

Click on the NMP hotspot lines or their endpoints to get info about the involved routes and waypoint labels.
Since you only see the points but not the whole network graph, you might need to open another HDX instance on load the region graph from Option 1 where you can get the whole picture. To figure out which routes should intersect, what's going on there etc. For instance, it's possible that concurrent routes are only broken on a very short segment you don't see (or missed) with the Option 1 view style.
Fix the coordinates in the corresponding wpt files.
Load the changed wpt files into the WPT file editor to avoid causing unintended data errors.

Marking NMPs false positive is trickier than for simple data errors but works the same way.

The FP Entry to Submit can only be found in nearmisspoints.log which contains all NMP entries including those which are already marked FP. The list is sorted by region code.
- Best practise is to use the browser search function to find lat or lon coordinates (or route code) which one can copy from the HDX table.
- There are minimum two FP entries per hotspot but very often even more, please refer to the issue with multiple lines on the HDX table mentioned above.
The entry must be added to nmpfps.log.

The list is sorted by region codes. Insert your entry at the right position.
If the entry ends with [LOOKS INTENTIONAL], this part of the entry must be removed.
If not all entries for a NMP hotspot have previously been marked FP, only the missing entries need to be added. Entries ending with [MARKED FP] are already entered as FP.

Do not forget checking for unmatched NMP FPs. Remove them from nmpfps.log.

Link	Code	Description	WPT File Editor	FP possible
🔗	BAD_ANGLE	Angles cannot be computed for two adjacent points at the same coordinates. Instead, use `AltLabels` or fix coordinates of one point or both.	NO	NO
🔗	BUS_WITH_I	Label looks like an Interstate with `Bus` banner instead of `BL` or `BS`.	YES	YES
🔗	DISCONNECTED_ROUTE	The 3 potential underlying causes are: Routes that are not connected in the first place. Roots are out of order within the `_con.csv` line. Just plain mismatched coordinates at a border, whether NMP or too far apart to be flagged. How to fix: Make sure all roots in the `_con.csv` line are in fact connected. If not, split into multiple `_con.csv` lines. Make sure the roots are in sequential order within the `_con.csv` line. Make sure coordinates match at regional boundaries. If this means changing a `.wpt` file in a region maintained by somebody else, coordinate as needed with that person on making the changes.	NO	NO
🔗	DUPLICATE_COORDS	Duplicated coordinates for two or more waypoints	YES	YES
🔗	DUPLICATE_LABEL	Duplicated labels for more than one waypoint. Labels must be unique for each route.	YES	NO
🔗	HIDDEN_JUNCTION	Concurrent route splits off at a hidden waypoint. The concurrency is most likely broken by accident, or the waypoint needs to be visible.	NO	YES
🔗	HIDDEN_TERMINUS	Hidden point at the beginning or end of a route. Routes must begin and end with visible waypoints so that users can mark all of them as traveled.	NO	NO
🔗	INVALID_FINAL_CHAR	Disallowed character at end of label	NO	NO
🔗	INVALID_FIRST_CHAR	Disallowed character at beginning (after any leading `*` characters) of label	NO	NO
🔗	LABEL_INVALID_CHAR	Label contains characters not allowed in labels, e.g. non-ASCII.	NO	NO
🔗	LABEL_LOOKS_HIDDEN	`X123456` style label without a leading `+`.	NO	YES
🔗	LABEL_PARENS	Number of parentheses do not match. Opened `(` must be closed with `)`.	YES	NO
🔗	LABEL_SELFREF	Label appears to reference own route.	NO	YES
🔗	LABEL_SLASHES	Too many slashes in label (`> 1`).	NO	NO
🔗	LABEL_TOO_LONG	Label is too long to fit in the space allocated for the DB field.	NO	NO
🔗	LABEL_UNDERSCORES	Too many underscored suffixes (`> 1`)	YES	NO
🔗	LONG_SEGMENT	Long segment (distance `> 10 mi`, `16 km`) between this and the previous hidden point.	NO	YES
🔗	LONG_UNDERSCORE	Label has long underscore suffix (`> 4` characters after underscore)	YES	NO
🔗	MALFORMED_LAT	Invalid argument after `lat=` in URL which cannot be converted to a numeric value	NO	NO
🔗	MALFORMED_LON	Invalid argument after `lon=` in URL which cannot be converted to a numeric value	NO	NO
🔗	MALFORMED_URL	URL is missing `lat=` and/or `lon=` argument(s)	NO	NO
🔗	NONTERMINAL_UNDERSCORE	Label has underscore suffix before slash	YES	NO
🔗	OUT_OF_BOUNDS	Latitude `> 90°` or `< -90°`, or longitude `> 180°` or `< -180°`. False positives are allowed for the rare potential case of routes spanning the international date line.	NO	YES
🔗	SHARP_ANGLE	Sharp angle (`> 135°`) with previous and next waypoint.	YES	YES
🔗	VISIBLE_DISTANCE	Long distance (Visible distance `> 10 mi`, `16 km`) between this and the previous visible point. Not reported for active routes!	YES	YES
🔗	VISIBLE_HIDDEN_COLOC	The visisble waypoint is hidden on concurrent route(s).	NO	YES

Travel Mapping Manual: Deal with data errors - Draft