On September 27th, 1998, ESPN introduced a new feature requiring a new technology innovation in football broadcasting that is nowadays an integral part of the game: the “1st and Ten Line”.
During the Sunday Night Football telecast of the Baltimore Ravens and the Cincinnati Bengals, the TV viewer's experience was enhanced by a virtual “Yellow Line” - that indicates where the first-down marker is thanks to its colour- that stretches the entire width of the field as if it were literally painted on it like the white yardage lines.
Indicating where the offense has to reach to get the next first down, this line gave viewers a live and intuitive guide to the state of play without distracting from the game at all.
Behind this augmented reality experiment, a small startup company named Sportvision built by Bill Squadron and Stan Honey, who left their jobs at News Corporation, with one goal in mind: To leverage the technology's rapid growth to improve the sports-viewing experience for fans at home.
But before diving into the details, let's rewind a little because the story of the 1st and Ten in NFL football actually starts two years before with NHL hockey.
Back in 1995, most Americans knew little about ice hockey which was less popular than football, baseball, and basketball. Fox had just won the broadcast rights to professional ice hockey and was trying to figure out how to make the game understandable and easier to follow.
A tough challenge when you consider the size of a hockey puck and the high speeds at which it travels from one side to another of the ice rink.
David Hill, the head of Fox Sports, believed that if the spectators could easily follow the puck, the game would seem less confusing to newcomers, and hence become more appealing to a broader audience.
Hill asked Stan Honey (hello again 👋🏼), the Executive VP of Technology for News Corporation, which owned Fox and Fox Sports if it would be possible to make the puck easy to follow on the television screen.
His answer was positive and with several engineers, he invented a puck tracking system called FoxTrax, most know by the popular name of “Glow Puck”, first aired in the 1996 NHL All-Star game. In the final product, a blue glow was superimposed on the puck as it moved over the ice, and the color of the streak was indicating speed, from blue to red if the puck traveled over 110km/h (70 mph).
To implement this enhancement it required 3D tracking technology as sophisticated as any advanced military system (we wrote an article on this topic 😉) and new technology to match the tracking data to the image field produced by the television camera. Curious to know more, you should read this interesting article from the Engineering and Technology History Wiki.
Ok, but what's the link with the NFL yellow line? 🤔
Well, the Glow Puck was a technological success, if not a business one, and the engineering team behind this invention wanted to develop other sports broadcasting applications. Unfortunately, News Corporation's business wasn't doing well at the time so the team left to start Sportvision. The separation went well because News Corporation obtained an equity share in Sportvision in return for exclusive licensing rights to prior patents.
And guess what... one of the first technology was the 1st and Ten line.
Adding graphical enhancement to video streams was not new to football. John Madden, a former NFL coach and sportscaster had popularized the use of the “telestrator” in televised games.
During the Super Bowl XVII (1982), and for the first time, Madden analyzed the game by drawing freehand sketches over video images so viewers can get visual information on their screen and better understand his explanations.
While useful in analyzing replays, telestrator sketches could not be used during the actual televised play because they obscured parts of the underlying video image. A limitation the Sportvision team decided to eliminate to enhance the viewing experience without detracting the viewers’ attention away from the flow of the game.
The idea was very attractive. The massive engineering challenge behind too 🤯 as it's well explained in this archive video from ESPN.
As you can see in the video, the first challenge in making the yellow line is that the scene is constantly changing, which means the yellow line has to constantly change.
Sportvision's team had to create a 3D mathematical model of the football field using laser surveying tools before a game. During the game, they gathered data from 3 different cameras used for the wide shots of the fields about their positions (pans, tilt, and zoom) for every single frame. So when the operator specifies that the first down is at the 18th-yard line, for example, the computers combine the camera data with their own model of the field to draw the yellow line in the proper perspective and to redraw it, for every frame being broadcast to viewers.
The second challenge is removing any part of the line obstructed by players, refs, or the ball so that the line looks like it's underneath them, almost painted on the field.
The way the computers know which pixels to remove is by sampling the colors -think of the field as a giant green screen. Sportvision's team had to identify in advance which shades of green and brown are in the field given the weather and lighting conditions -those are the colors to be covered by the yellow line. And they also identified which colors are in the players' uniforms and should never be covered by yellow.
Nowadays, the whole yellow line process delays the live broadcast by less than a second. A pretty impressive result considering the complexity behind this technology.
More technical information on the 1st and Ten line can be found in Method and Apparatus for Adding a Graphic Indication of a First Down to a Live Video of a Football Game, U.S. Patent #6,141,060 filed on 5 March 1999, issued on 31 October 2000.
Football broadcasts have since added more graphics like the line of scrimmage and more recently, tons of augmented reality animations like during the New Orleans Saints and Chicago Bears game on January 10, 2021.
The first down line for each play gained some animations that were on-brand for Nickelodeon. More fantastically, when touchdowns were scored, the end zone transformed into a slime zone, with virtual geysers of slime bursting from the field.