Coverage resolution (precision)
Resolution refers to the ability to recognize and distinguish features. One important characteristic of a coverage is its resolution or the precision at which the location and shape of geographic features are stored. For example, a resolution of 1 meter means that the true ground position of a coordinate is within 1 meter of the stored coordinate values.
 |
Coverage resolution is influenced by three factors: the coordinate precision specified; the precision of the input device (for example, a digitizing tablet); and the scale of input documents and coverage processing tolerances, particularly the fuzzy tolerance used in many geoprocessing commands.
Coordinate precision
Coordinate precision refers to the mathematical exactness of a coordinate and is based on the possible number of significant digits that can be stored for each coordinate. Coverage datasets are stored in either single- or double-precision coordinates. Single precision stores up to seven significant digits for each coordinate. This means a value of 4,999,999.6 units or 5,000,000.4 units will be rounded off to 5,000,000 units when stored in single precision. On the other hand, a value of 5,000,000.6 will be rounded off to 5,000,001 units when stored in single precision. This can be easily avoided by using double precision. Double precision stores up to 15 significant digits (typically, 13 to 14) and, therefore, retains a mathematical precision of much less than 1 meter at a global extent. Note that although mathematical precision does not define accuracy, it is a major factor in coverage resolution.
The computer can only discern a limited number of decimal places, depending on the precision being used. For single-precision numbers, values such as 1.2345678 and 1.23456789 are assumed to be equal by the computer because numbers beyond the seventh digit are ignored. You might imagine a very small halo surrounding each coordinate that is equal to the resolution of coordinate storage on the computer. Coordinates with overlapping halos are seen by the computer to represent the same location.
 |
Example of coordinate precision |
The computer equates two coordinates when their halos overlap.
You should use double-precision coordinates for coverages whenever possible. This allows you to set extremely small fuzzy tolerances for all processing (for example, 0.00000001), thereby minimizing the effects of fuzzy processing on coordinate shifting.
If you are converting a shapefile to a coverage, use the Feature Class To Coverage tool and set the precision to Double (on the Environments dialog box under Coverage Settings). A shapefile is always a double; therefore, the coverage should be set to Double to avoid losing accuracy in the conversion process.
Device precision and scale of input documents
Resolution is also limited by the resolution of the input device and the scale of the input document. For example, suppose your digitizing tablet has a resolution of .002 inches—the minimum distance between points on the digitizer is .002 inches—and you are digitizing a map at a scale of 1:63,360. The map scale can be stated in many ways:
- 1 inch on the map = 63,360 inches on the ground
- 1 inch on the map = 5,280 feet on the ground (63,360/12)
- 1 inch on the map = 1,609 meters on the ground
- 0.002 inches on the map = 10.56 feet on the ground
The last line equates the digitizer resolution to the distance on the ground. Since it is not possible for a coverage to be more accurate than the digitizer’s resolution, the minimum distance separating coordinates digitized from a 1:63,360-scale map represents 10.56 feet on the ground. If the coverage coordinates are stored in feet, the coverage resolution cannot be better than 10.56 feet. If this resolution is not acceptable, a larger-scale map must be digitized.