Penalty Harm Rating: Take Two

Over in this thread I laid out my early intentions of assessing the "harm" done by each penalty and coming up with a rating system.

I didn't explain it well there, but the intent is to fill a huge gap in penalty discussions. Current data supports some nice objective metrics like yards/count and the like, but the gap is one of context. In many discussions on penalties I keep seeing the same thought process play out over and over with no good answer...

Are the Bills shooting themselves in the foot worse than other teams?

This is usually some concept of "sure they have a lot of penalties, but which ones are backbreakers and which ones are boo-boos?"

Now then, I CAN go through the game logs at, and give narrative reasoning and assessments of each penalties. However, this means a paragraph or so on EVERY penalty with the reasoning explained. Reasoning which might become repetitive. How many times does it take for me to say "that holding call on 1st and 10 didn't really hurt that much because they were already backed up on their own 2" before it becomes a waste of words?

Ideally, I could rate penalties, assign them a number and then do game recaps with a nice easy format rating penalties with this new metric. Narratives then could be saved to look at anomalies, or especially interesting penalties.

I proposed three types of harm and lean toward a cumulative approach of adding all three types together for each penalty.

For any newcomers, the three areas were:

  1. Points affected or changed due to penalty
  2. Distance lost (measured from spot before penalty compared to spot after)
  3. Opportunities given. What amounts to allowing an opponent's drive to continue by giving up a first down, or negating a takeaway by your team
The first one originally was meant to also encompass likely or predicted points. Comments mostly surrounded this as a sticking point and there appear to be many legitimate dilemmas when trying to assess points that were affected by penalty. You can read the full dialogue in the comments in the other thread.

There are a few things I'm still considering, but in my impatience to go "live" and test the metric out with a game or few from last year, I'm going to establish my list of criteria (for now, refinements are still probable) and then start off testing a game and see how things shake out in reality. Note the poll below...

Here's my revised scoring system:
  • 0.1 units for every yard given up. This creates a maximum 9.9 points for yardage (a 99 yard DPI penalty for instance)
  • 1 unit for every point on the scoreboard affected. After debate from the other thread, I have decided that this will ONLY be assessed during the play in which the penalty happened. I used a holding by Urbik against the Giants as an example. There was an actual TD on the play and it cleanly follows then that the penalty actually removed the points from the board. I am currently defaulting TDs to 7
  • 1 unit for every down changed via penalty. So for instance, an automatic first down penalty is calculated based off the difference between the pre and post penalty downs. So giving a first on 4th down would be 4-1 or 3 units. Giving up a first down (DPI for instance) on a first down yields 0 units but would still calculate the yardage assessed. This will also be calculated for offense despite not mattering too often. Intentional grounding for instance though does yield a down change and would be assessed the 1 down difference of 1 unit
  • Penalties that negate a takeaway will be assessed with a similar thought process as the last bullet. As they "cost" your team a full set of downs, it will be considered as a 4 unit assessment.
Using some examples from last time, but with the revised metric:
  • E. Manning throws incomplete pass from NYG 38 on 2nd down (I had this wrong in the comments last time). Gilmore assessed DPI for 8 yards. 8 yards penalty + 0 yards affected by penalty + 1 down given + 0 points affected. This is 0.8 + 1 + 0 = 1.8 Units
  • E. Manning incomplete pass at NYG 6 on 3rd down. L. Donnell false start assessed for 2 yards. Down replayed. 2 yards penalty + 0 yards affected by penalty + 0 Downs + 0 points = 0.2 Units
  • Manning pass to D. Harris for 13 yard gain on 1st and 10 (NYG 44 to BUF 43). S. Vereen called for OPI for 10 yards. 10 penalty yards + 13 yards affected + 0 downs + 0 points = 2.3 Units
  • Here's a fun one... J. Brown kicks 44 yard FG which is good on 4th and 2 from BUF 25. K. Williams called for illegal formation for 5 yards. 5 penalty yards + 0 yards affected by penalty + 3 downs - 3 points (yes that's a minus, as he erased 3 points). That's 0.5 +0 +3 -3 = 0.5 Units of harm
  • T. Taylor from NYG 31 runs for TD on 1st down. K. Urbik called for holding and assessed for 10 yards. 10 Penalty yards + 31 yards affected by penalty + 0 downs + 7 points = 11.1 Units of harm
This model very strongly calls out any instance a player costs his team points directly or gives an opponent more opportunities.

The Kyle Williams penalty seems odd as it considers it nearly a wash. After consideration, I can stand behind this assessment 100% as the penalty did not significantly alter field position. While he did allow more opportunities for the opponent he did effectively negate 3 points as well. Had the D made a stop or got a turnover we'd look back on this event and say "we really got lucky with that penalty." Note also that, had the Giants MISSED the FG, he'd have had a harm rating of 3.5 which looks to be far more significant than say the holding call that merited a 0.2 in my examples.

Now then, I'd like to test this using a game or two and see if there's some utility to the measures and go from there.

Which game sounds like a good starting point:

A. Week 2 - NE at Buffalo. This game resulted in 25 total penalties with a fairly even split between teams. Would be a good look at measuring the "harm differences" due to seemingly similar penalty outcomes using traditional metrics (Buf 14 for 140, NE 11 for 119)

B. Week 4 - NYG at Buffalo. A game many people groaned about "shooting ourselves in the foot." The infamous Tom Coughlin telling his team to rile up the Bills so they would make mistakes game

C. Week 14 - Phi at Buffalo. Bills couldn't get out of their own way with 15 penalties while the relatively clean playing Eagles (5 penalties) barely squeaked out a win. This is the game to look at for any masochists I would imagine

D. Week 13 - Houston at Buffalo. A rare game where the opponent out-penaltied us by a comfortable margin (Bills assessed 5, Houston at 10). This is a vote for "I'd rather not relive how bad the Bills were with penalties last year"

Just another great fan opinion shared on the pages of