Much of the debate centered around the recent, highly charged case of Wisconsin v. Kyle Rittenhouse[1] emerged from an unlikely source: the ‘pinch-to-zoom’ feature of the Apple iPhone and iPad. On November 10th, during Kenosha County prosecutor Thomas Binger’s cross-examination of the defendant, Binger asked Rittenhouse to respond to questions based on a video clip the prosecutor displayed on an iPad. When Rittenhouse replied that he could not see the footage clearly enough to answer, Binger zoomed in on the video using Apple’s pinch to zoom technology. Defense counsel Mark Richards objected with a statement which sparked the subsequent controversy: “Your Honor… iPads have artificial intelligence in them that allow things to be viewed through three dimensions and logarithms… to create what they believe is happening. So this isn’t actually enhanced video, this is Apple’s iPad programming creating what it thinks is there, not necessarily what is there.”
Judge Schroeder sustained Richards’ objection, and Binger was forced to pivot to a video connected to the courtroom television from a Windows PC. Though little evidence suggests that this change had a substantial impact on the result of the trial- the jury’s decision to acquit Rittenhouse on all charges- it did serve to exacerbate certain misconceptions about Apple’s products and software.
The history of pinch to zoom began in 2007, when Apple Co-Founder and former CEO Steve Jobs introduced the original iPhone. There, Jobs told an excited audience, “You can do multi-finger gestures on it,” simulating the pinching motion with his fingers, “and boy have we patented it!”
Therein lies the first, if perhaps somewhat insignificant, misconception. Though it is true that Apple was awarded a limited patent on pinch-to-zoom in 2010, the US Patent Office ultimately narrowed it to such a degree that it lost a great deal of potency. Roughly, the patent they own is on pinching to zoom sequentially and having the product pick up on the fact that the user is continuing the initial pinch. In other words, when one begins to zoom in, stops, and continues shortly after, Apple technology recognizes the process as one continuous zoom. But I digress.
For our purposes, it is more important to consider how exactly pinch-to-zoom works. First, note that it is not true that Apple’s algorithms alter any existing pixels within the image. Though it is the case that pinch-to-zoom uses algorithms (what Richards referred to as ‘logarithms’) to add pixels, they are used only to enhance the photo. This process is known as interpolation, and it can take several forms. The three most common forms of interpolation are presented below in ascending order of the quality of the finished product.
The simplest version of zooming requires no interpolation, involving only copying the color of the neighboring pixel. It is the fastest method but is susceptible to creating rough looking photos.
Then there is linear interpolation, in which the color of each pixel is “computed as the average color of the four closest pictures in the original image.” Though it takes longer than not using interpolation, it results in a higher quality image.
Cubic interpolation takes more time to render than its linear counterpart but yields the highest quality product. It is computed as the average color of the eight nearest pixels, as opposed to four.
Though the precise algorithm which Apple utilizes in its software is protected as a trade secret, we can extrapolate from its rapidly improving image quality and constantly refined software that Apple does not merely duplicate each individual pixel. When Rittenhouse’s counsel argued that the zoomed in image consists of what “Apple’s iPad programming what it thinks is there,” he referred to the process of interpolation, or the use of AI to estimate the color of enhanced pixels.
Supporters of Judge Schroeder’s decision to bar the evidence could set forth the following hypothetical: imagine two competing software companies each ran their individual algorithms to enhance a photo. The companies are in competition to present a higher quality photo. It stands to reason, therefore, that the two products will not be pixel-to-pixel identical copies. This, they’d argue, is evidence that the photo is not simply a representation of the object in the camera’s frame but is rather inextricably linked to the program’s input. Critics of the decision would reply that pinching-to-zoom is performed incredibly frequently, is basically synonymous with using a magnifying glass, and the algorithm, if anything, improves the quality of the image, and by extension its usefulness to a judge or jury.
Though Judge Schroeder’s decision has been much maligned, both on social media and in technology blogs, the question is not quite as clear as it appears. At that point in the trial, the drone evidence had already been admitted, but the “zoomed in version” had not. In the most literal sense, it is a different image, but whether it practically differs is up for debate.
During a case in which the prosecution’s case relied in large part on when precisely the defendant lifted his rifle, one could make a case that every pixel counts. At the same time, though, the rule of law requires that both sides of the adversarial system be given a fair shot at pleading their case. By ostensibly granting Binger only a handful of minutes to come up with an expert witness to testify to the validity of the image, Judge Schroeder may have denied the prosecution that opportunity.
The prosecution was ultimately able to present the unaltered evidence but its ramifications on evidentiary proceedings is unknown. More likely than not, the services of an expert witness can be enlisted to verify the reliability of pinch-to-zoom, but only time will tell if courts find the argument sufficiently compelling.
[1] This highly publicized case brought into question notions about the right to self-defense, protest, and equitability in the criminal justice system.