Evaluating defensive talent is a very difficult thing to do. That’s why companies like STATS, Inc. and Baseball Info Solutions have dozens of people around the country tracking where each ball was hit, how hard it was hit, and whether it was a ground ball, fly ball, line drive, or a combination of the last two, called a “fliner.” They then take all this data back to the lab and figure out how many runs each fielder saved (or allowed) compared to what an average fielder would have done in the same number of chances. Now there is no physical person out there representing the average player in every game situation. In other words, when Jose Reyes ranges deep into the hold, we don’t have a guy standing out there representing average to compare Reyes to. Defensive statistics fall under the category of “Inferential Statistics.” For a good explanation of what we can learn about defensive stats, I suggest reading this, from Dave Cameron. UZR is probably the best publicly available fielding stat out there, and can also be found on FanGraphs. For an explanation on what these new stats mean, click here.
As Dave noted in that post about inferential statistics, we should keep in mind the uncertainty associated with this. That is why I will often use Chone’s defensive projections (bottom of the page). These take multiple years of data from multiple sources, regress the data, and then apply an aging factor. These are more reliable than defensive stats from any one source.