The Poor Man’s Analyst

Replacement Level Pitching Part II

with 9 comments

In part one of this, I talked about why we compare players versus the baseline of a replacement player instead of a bench player. Then, I showed what a sub-replacement level pitcher looks like in Kei Igawa, and gave an example of a replacement level pitcher in Sidney Ponson, who fits the definition perfectly. Now we’ll get on with the rest of it, talking about how to value pitchers in differing roles and situations, like Joba Chamberlain, Chien Ming Wang, and Andy Pettitte.

"Yea, that's right, it's Armani"

No, the Yankees do not have new alternate road uniforms

Andy Pettitte threw 204 innings with a 4.54 ERA this past season. Chien Ming Wang threw only 95 innings in an injury-shortened season, putting up a 4.07 ERA. Wang was better, but pitched in fewer innings, so who was more valuable to the Yankees? The same question can be asked about Joba Chamberlain and Mike Mussina. If you remember from part one, I said that replacement level for relievers is lower than it is for starters. So we also need to look at the time Joba spent in the bullpen, and also account for the fact that the 8th inning is more important than the first inning before saying how valuable he was. It might sound a little complicated, but I promise you it’s not, once it’s all spelled out in plain English. More after the jump…

The full-time starter: Andy Pettitte

Andy Pettitte’s 2008 was decidedly average. His 4.54 ERA was slightly below league average, but other stats like strikeout to walk ratio were above average. Some people might say that Pettitte hurt the Yankees last year with his performance. “If he wasn’t above average then how could he be helping?” So how about we compare him to a replacement level pitcher (or as we saw in part one, Sidney Ponson). There are two things to remember here: 1) replacement level starter is projected to have a 5.50 ERA, and 2) that starter will not be projected to throw more than 160 innings, since a pitcher that bad wouldn’t be allowed to throw that many innings. But Pettitte threw 204 innings last season, so we fill in the remaining 44 innings with the relief replacement level ERA of 4.50. So in 204 innings of replacement level pitching (starter and relief combined), the tandem would give up about 120 runs (98 from the starter, 22 from the reliever). Pettitte, in those 204 innings, gave up 103 runs. The difference between the two numbers is 17 runs, and that is the value that Pettitte provided above what replacement level pitchers would provide in equal playing time.

The injury-shortened season: Chien Ming Wang

So we see that Pettitte was worth 17 runs above replacement last season, but Wang pitched better in the time that he was on the mound. In 95 innings out of the starting rotation, Wang put up a 4.07 ERA, allowing 43 runs. A replacement level starter would have allowed about 53 runs in the same amount of time. By taking the difference, we see that Wang was worth 10 runs above replacement last season. There’s nothing groundbreaking in this paragraph, I just wanted to get the answer to my Wang/Pettitte question from before.

The next section will be a little more cumbersome.

The Swingman: Joba Chamberlain

While not a swingman in the truest sense of the word (and it’s not even a word according to spell check), Joba Chamberlain shuffled back and forth between the bullpen and rotation last season. As we saw with Andy Pettitte above, there is a different baseline for relievers than for starters. So first, we must compare Joba’s numbers as a starter to replacement level, and get a value for that. Then, we take his relief numbers and compare those to the 4.50 baseline. But the later innings have a greater importance than the innings before them, and we must account for that. Luckily, there is a stat called Leverage Index that does just that.

As a starter, Joba threw 65 and 1/3 innings with a 2.76 ERA, giving up 20 earned runs. A replacement level starter would be expected to give up 36 runs in that span, so Joba was worth 16 runs above replacement as a starter. That’s in just over 65 innings. Wow. Hold on to that “16 runs” for later when we add this up.

As a reliever, it’s more complicated. The first part is still the same… compare his 2.31 ERA in 35 innings to the 4.50 replacement level, and get an 8.5 run difference (9 runs in 35 innings, subtracted from 17.5 rep. level. runs gets 8.5). Now, we account for leverage. If you don’t know what Leverage Index is, you can read a basic introduction here. Chamberlain’s LI while in relief last season was 1.35. To adjust his value above replacement (9 runs in relief) to account for this, we take the half-way point between 1.35 and 1.0 (1.175), and multiply them together. This comes out to  1.175 x 8.5 runs = 10 runs above replacement, accounting for leverage. Add the starter and relief versions of Joba Chamberlain together, and you get 26 runs above replacement.


Written by dcn29

January 2, 2009 at 4:08 AM

Posted in Player Value

Tagged with

9 Responses

Subscribe to comments with RSS.

  1. I’m extremely interested in what ‘leverage’ is actually made up of. I find this ‘statistic’ very questionable. What makes the 10 runs scored in the first inning not as good as 10 runs scored in the 8th? What data does this index of importance come from? I’m not saying it doesn’t come from something important and useful, but I really don’t agree with putting significant weight in certain situations.

    People argue that the team has ‘less time to catch up’ later in the game. However, if they scored runs earlier, they wouldn’t have to. Of course, some people react to pressure situations better than others, but I doubt there would be significant evidence finding this in statistical data. With that said, any player PAID to be playing a sport should be able to handle pressure. Period. I think the emphasis on LI is somewhat silly and overanalyzed.

    Your explanation of LI in the link doesn’t tell me anything. Do I just need to take your word for it? I just found this site and enjoy reading all sorts of analysis, but this has been a pet peeve of mine for quite a while. I just have serious doubts that the premium for closers (and it may not be that at all, but premium over middle relievers that very well could have similar abilities as closers) is insane. These guys are often in that position because they are an injury risk and may not be able to throw a significant number of innings. On top of this, to call a late-inning guy ‘spectacular’ because he allows less than 3 runs in LESS THAN ONE INNING is absurd (K-Rod, for example). So we seem to have an interesting dichotomy where excellent middle relievers are paid squat, while K-Rod and other closers are paid upwards of $12 million a year. My argument is: hire 2 solid middle relievers to take that role and get a big bat. Or, assuming your closer isn’t an injury risk, let him pitch more innings if he’s sooooo good. If you’re up by 6 before the 9th…you don’t even need to worry about LI.


    January 7, 2009 at 2:10 PM

  2. There is a much more thorough explanation of how Leverage Index was created, I just didn’t want to link to it in the article. I think you’ll believe in Leverage much more after reading them. Those articles can be found here:

    Read those first, and I’ll be happy to answer any questions you have afterwards. LI has nothing to do with whether a player can perform in pressure situations, it’s only meant to account for the importance of the situation. Consider these two situations:

    1) Bottom of the second, home team is down 2-1 with a runner on first base. Let’s say, for instance, the home team will win that game 35% of the time. The batter hits a home run to take the lead, 3-2. Now, the home team has a 65% chance of winning the game (percentages are for illustration purposes).

    2) Bottom of the 9th, same situation with the home team down 2-1 with a man on. The home team has a 20% chance of winning the game. Batter hits a home run to go up 3-2 and win the game (100% chance of winning).

    See how at a later point in the game the SAME situation and SAME events can have a greater impact on the outcome of the game. That impact is what LI captures.


    January 7, 2009 at 3:21 PM

  3. I understand that part. I think the bottom of the ninth situation does have some difference there. I understand the differences in the change in probability of certain outcomes given an out or a hit. However, as I said before, why not spend the money to prevent the consistent occurrence of these situations in the first place? If you spend money somewhere else, you reduce the number of times that situation takes place. It comes down to the fact that each team gets the same number of outs. Period. These ‘high leverage’ situations are simply a mainfestation of the fact that each team doesn’t yet have the same opportunity to score the same amount of runs. That’s all it is, in my opinion.

    I would argue that the psychological part DOES come into play in leverage, but that’s difficult to find evidence for. But, under that assumption, as a former player, pitcher, hitter, coach, I can say that the demoralization of being down 4, 5, 6 runs in the early to mid innings has a significant impact on the game. Do I think you need a good pitcher at the end of your bullpen? Absolutely. Do I think money can be spent better elsewhere, when replacements should do a sufficient job in similar late-inning situations on a consistent basis? Yes. The idea is to prevent LI situations in the first place by having a better team than the other…one that can rely on scoring more runs in less outs than their opponent.

    LI situations will take place no matter what. Even with great lineups. But, as in one of your references, the difference between ‘average’ and ‘great’ probabilities of not giving up a run in that bases loaded situation is about 5% (the titles seem arbitrary…this is a small range, though not negligible). This is the difference between average and great…but what does that mean? There could be plenty of guys very close to that ‘great’ percentage where the difference IS negligible. These are the guys that are pitching in the 7th inning, etc. They’re being paid, as I said before, squat. My question is: what does the distribution of these pitchers look like? With such a small sample of innings, there absolutely has to be reasonable alternatives and better optimization of salary distribution on an individual team (possibly saving money AND making the team win more games due to less LI situations).

    Honestly, I think there is something to win probabilities in different situations. It helps with decision making immensely. Where are we in the batting order? You always want the best possible pitcher you can have in a certain situation, hands down. I’m just not convinced that the impact K-Rod has on 1 or 2 outs in a game is 1000% better than the next best option (expected production, salary-wise) in a bullpen or on the market. Using past data is helpful in guaging the best move in certain situations, but it is not necessarily a causation based on the individual pitcher. This is simply a trend. Teams that are down by more runs later in the game lose more. It doesn’t account for the fact that they had lots of chances to go ahead earlier in the game.

    What historical data do these probabilities come from? The closer is a fairly new innovation in baseball. Different teams and different types of matchups are not taken into account here. I find it problematic lumping all of these ‘situations’ together, as well.

    My point is this: it does not seem to be that difficult to find guys in the system or on the team that can play the closer role when needed. Everyone thought Brad Lidge was finished. One home run given up, to the best player in the game, and writers and analysts said his career was in the gutter. It happens often (Brian Fuentes…flashes of greatness then thrown off to the side…but I’m not convinced that (assuming the Red Sox didn’t already have a fantastic rotation) keeping Papelbon as a closer, rather than having him pitch 200 innings was better for the Red Sox (unless, of course, there are injury concerns).


    January 7, 2009 at 7:29 PM

  4. You must really like to write, haha. I’ll address what I can:

    -“why not spend the money to prevent the consistent occurrence of these situations in the first place?”

    This isn’t really relevant to the evaluation of relief value. It’s a roster construction-related point that is way outside the scope of this post. But there is leverage associated with every event. In blowouts, the LI will be close to zero, for instance. The reason relievers should have leverage incorporated into their stat lines is because, whether they should be or shouldn’t be, they are often pitching in games where the leverage is greater than one. The events in a high leverage situation cause a larger “swing” in game state, and applying LI to a stat line simply acknowledges that fact.

    Based on your third paragraph, it seems you are misunderstanding exactly what LI captures. We care about contribution to winning. It just so happens that the overall population of hitters and starting pitchers have an average LI of 1.00, so we ignore leverage in their stat lines (it’d be like multiplying every stat by 1). Check out this chart:

    -“I’m just not convinced that the impact K-Rod has on 1 or 2 outs in a game is 1000% better than the next best option (expected production, salary-wise) in a bullpen or on the market.”

    -You’re exactly right. Closers in general seem to be overvalued, if you buy into the “sabermetric” analysis that’s out there. Having a proven closer seems to have more of a psychological benefit than anything else.

    -The data this comes from can come from a variety of sources. You can find win expectancy for yourself based on Retrosheet data, but I wouldn’t know how to do that. There used to be a site that no longer exists that calculated it based on data from like 2000-2002. I’m not sure what the time frame is for the data Tom Tango used, but it’s the same run environment that MLB plays in today, so it works.

    Regarding your last point…I’m NOT saying that Leverage Index tells us that a reliever is more important than a starter. Most sane baseball people will agree with me on that. And the role of “closer” is overrated as well, in my opinion. It’s not hard to find guys that will succeed in that role–even some guys considered the cream of the crop aren’t that much better than the league average closer.


    January 8, 2009 at 3:17 AM

  5. Thanks for the info. I’ll continue to read, as I am pretty statistically knowledgeable, but have not delved into the sabermetric community very much. I’m just curious about the metrics, how they’re used, and they’re actual significance in player valuing and signings. I appreciate that you defend the idea without taking it as a personal attack, as it was simply curiosity. You do seem to have a good understanding of actual application of LI, and I appreciate that you don’t defend the ‘closer’ role to the death.

    Good reads here. Yes, I enjoy commenting and writing things. I’m a curious person and have time on my hands during certain parts of the day.


    January 8, 2009 at 9:58 AM

  6. What fantastic timing you have with that closer issue. Dave Cameron wrote today about how closers are overvalued because of their reputations as “proven” closers.


    January 8, 2009 at 6:06 PM

  7. Just some more thoughts on the LI and closer controversies. I concede that LI can be useful in seeing a players contribution to his current team. I don’t deny that (the significance level of LI could be argued). As I said, it is extremely useful in game decision making: When do we need to bring in our best pitcher?

    However, I think the misuse of it comes when valuing a player on free agency. Despite the help that player provided in LI situations for his team the year before, it has absolutely nothing to do with the skill level of that player. His skills are his skills…if his skills aren’t that impressive in the first place, another middle reliever can come in and do the same job. In that case, we would go ahead and suggest THAT reliever is worth more on the FA market now. I guess it’s simply a misunderstanding of those trying to value free agents…when really it is a separate, on-field condition. This blown-out-of-proportion love for saves and misunderstanding of LI on the FA market must be fueling these astronomical salaries for closers.

    I’d refer ANYONE to Pizza’s quick little correlation analysis on your Fan Graphs link.


    January 10, 2009 at 4:41 PM

  8. “Despite the help that player provided in LI situations for his team the year before, it has absolutely nothing to do with the skill level of that player.”

    I don’t think this is 100% right, although some people will agree with you. Think about it this way. The reason guys like Rivera and K-Rod have a high LI every year is because their skill level dictates that they should be pitching in high leverage situations. Some of this is because of skill level, and some of it is other factors (comfort level, “proven closer” label, etc.). So that’s why we use the half-and-half method that I linked to in the post (and again below). You add the reliever’s leverage with the average leverage (1.0) and divide by two. Half talent, half “other.” If you want to make it 2/3 “other” or something like that, feel free to adjust the numbers accordingly. The difference will come out to only a few runs here and there as long as you are giving at some legitimate weight to the actual LI.


    January 10, 2009 at 5:58 PM

  9. Interesting…I had missed that part before (I saw ‘divide by two’ but there wasn’t much explanation there). Thanks for the extra info on that…like I said, I’m new to the ‘saber’ language so anything is helpful. As you said, it seems there could be significant disagreement with that, given it is somewhat arbitrary. Thanks.


    January 11, 2009 at 12:36 PM

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: