How Machines Restore Archival Film — or, at least, are trying to

posted March 3, 2020

By Peter Monaghan

Film restoration is a painstaking endeavor. It involves much careful observation of archival film, repair of any damage, and preservation from future ravages of time.

The tools for doing all that are increasingly sophisticated. Specialists certainly can relate as much, but it may interest the general film enthusiast to hear a little about what the modern-day process of digital restoration entails.

Before the age of digitization, restoration was a matter of carefully and often exhaustingly cleaning, repairing, and patching together the best surviving bits and pieces of a film. Those might come from original reels or later copies, with the best of each copied to a new, optimal preserved version.

Archived films might have come to exhibit any of a variety of damages. Scratches are common. Or, over time, the film stock on which older moving images were captured may have faded, shrunk, warped, been badly spliced, or been splotched by mold, all of which can obscure or distort the original images.

These days, of course, films increasingly — even generally — are preserved via digitization. And as that technology advances, many film archivists long for the day when films not only are preserved but also when digital tools can near-automatically repair the many problems that arise in film images.

Just pottering away at the task, using current methods, won’t suffice. Preserving all the thousands of collections large and small, with their hundreds of millions of feet of film and innumerable video recordings, is too gargantuan an undertaking. So, the ultimate dream, still a good ways off but at least now imaginable, is that some large proportion of damaged or deteriorating film could be automatically, inexpensively, and rapidly repaired, at least in digital form. Ease, speed, and affordability are crucial because without them, much of the world’s historic film may simply molder away.

Time is of the essence.

Several companies have been carving out a niche in automated or at least semi-automated digital restoration of moving-image media. Technicians have for two decades and more been slowly refining and innovating processes for doing that.

“We want to do it in an absolutely automatic way, and we believe we are very close to this,” says Alexander Petukhov, the director of one such company, Algosoft, based near Atlanta, Georgia.

“We want to make it affordable for every movie to be restored,” agrees Inna Kozlov, who founded the company in 2005. Algosoft’s digital platform, Viva-Pro Software, is among now-several increasingly sophisticated software suites for restoring and preserving film images.

For Algosoft, says Petukhov, “the dream is very simple. It’s to take old film like the Lumière Brothers or Thomas Edison, scan it, and run it through our computer and get modern Hollywood quality.”

Above: an example of how software like Algosoft’s works. In this example from Leo Tolstoy Funeral 1910, from The Russian State Documentary Film Archive, Algosoft software removes flicker, dust, dirt, and “noise.” Other parts of the software suite work on other damage and related issues.

The dream extends beyond professional restoration labs to homes, Kozlov says: “We have a dream that not only archivists but also people at home who want to watch their own films could restore them.”

To appreciate how remarkable the progress has been in such efforts, but also what its current limitations are, it’s worth delving back into restoration history, including its pre-digital phase.

The history, in brief

Restoration was originally what is called “film-to-film” work. If a sequence of images was missing or damaged in the best remaining copy of a film — or any copy that was being used as a preservation copy — it was often possible to source the missing or damaged footage from that second copy, and then that better version of the section of the film could be copied and become part of an existing or new best archival copy.

The second, source print might be of lesser quality, overall, but the primary consideration was selecting and copying the best elements could help to create an optimal restoration print.

That approach could work well, or not, depending on various factors, beginning of course with whether additional source prints even existed.

Often only damaged copies of sequences of frames have existed. Film restorers then have had to fix damage as well as possible in the best copy of each sequence. Damage can take numerous forms including scratches, dirt, chemical stains, and tears. Or, during film production, the film stock might, for example, have been exposed to light at a slightly uneven frame rate, resulting in flickering in the filmed images.

Dirt and other foreign substances that had attached to the film during exhibition, transportation, or storage could often be gently removed without damaging the film further. But scratches and tears, let alone whole missing frames, posed greater challenges.

One partial remedy was to develop a method of dodging scratches. From the 1970s, in “wet-gate printing,” restoration technicians have immersed film in a fluid that has the same spectral qualities as the developed film stock. Because the solution has the same density as the film stock, light projected through it and the film does not transmit an image of a scratch. In the process of transferring the image to new film stock, the fluid permits light to project through the film without being impeded by the blemish. The projected image can then be captured and incorporated into an archival best copy.

The wet-gate printing approach improved between the 1970s and 1990s, but it has drawbacks, starting with its difficulty and expense. It also generally works only when scratches are on the so-called “base side” of the film, and not the so-called “emulsion” side.

In the original filming process, the “emulsion” side (a thin layer of a light-sensitive compound, silver halide) captures images as the film is exposed to light through the camera lens. Any scratch to the “emulsion” side tears away some of the light-sensitive compound and the image captured on it.

By contrast, on the base side (which holds no image), scratches are in effect tiny channels dug into plastic which then fill with dirt. When light from a projector passes through the scratches, the light distorts through the no-longer-flat and -clean surface.

Fortunately most scratches tend to be on the base rather than emulsion side. Less fortunately, wet-gate printing still can deal only with relatively minor scratching.

New hope: digitization

From the 1990s, and intensively during the 2010s, digitization has offered new hope to restoration technicians. It has provided a new, non-physical medium in which to make repairs. A landmark came in 1993 with Kodak’s digital restoration in 4K resolution of the 1938 Disney feature film, Snow White.

Digitization is not a straightforward process. Digitization captures, via a scanner, whatever is on the film being transferred from an analog medium to digital form. Hence, all damage in films that are scanned and digitized appears in the digitized version unless steps are taken to mitigate the blemishes.

This is where the complex process of algorithmic correction of digitized sequences comes into play.

How exactly that is done… Well, go and complete a PhD in mathematics or one of many other highly technical fields, such as biological imaging, and then algorithms and how they work will all make sense, presumably. Put most simply, the algorithms used in film restoration are instructions that computers implement. The instructions specify that when certain conditions occur (i.e., are encountered) – for example, the patterns of digital code that tend to describe scratches — then the algorithm “engine” should alter those patterns to resemble ones that tend to describe an absence of scratches. A next-step algorithm might additionally instruct its own visual-image software code to take into account the patterns (the images) in the immediately surrounding imagery (patterns) in each frame or other frames, and to mimic those.

Already, one’s brain hurts.

Let’s just say that algorithms, neural networks, etc., are means of representing and modeling complex phenomena, and then manipulating them to achieve desired changes.

To gain some sense of what is involved in creating and refining these strange digital engines, consider that Algosoft’s Alexander Petukhov, a professor of mathematics at the University of Georgia (U.S.A.), works in such areas as neural networks, deep learning, approximation theory, image and video compression, signal processing, error correction codes, and artificial neural networks as well as other specialties with even more black-box names such as wavelets and sparse representation in redundant systems.

You need quite an armory when confronting many of the problems algorithm developers confront. They can be enormously complicated — for example, developers might need to construct algorithmic instructions to reduce the amount of jerkiness in the movement of an object. That might be needed when, for example, a camera was not sufficiently stable during filming, or when some frames are missing altogether from a sequence of frames of film.

Even a moment’s reflection makes clear that the instructions required to make such corrections can be mindbogglingly complex. Developers of automated film-restoration suites adapt to film restoration a variety of findings from the field of “big data” — which is to say, techniques for computationally analyzing large data sets to reveal patterns, trends, and associations. The patterns created by pixels captured in exposed film are one variety of such data sets.

To anyone without that PhD in mathematics, or similar learning, the algorithms used in the film-restoration process will likely remain black boxes. The proof that something useful is happening in that black box will be being able to observe that images become clearer after algorithmic processing.

The benefits of redundancy

The ultimate goal of some companies in the automated-restoration market is to create “one click” restoration. They would like users to be able simply to activate their restoration software with a click of a computer, to set the software running on the film sequence that is to be improved (restored). They want their software suites to analyze numerous features of the film, such as the levels of grain, flicker, and other imperfections in every frame’s images and then automatically adjust the digital images according to dozens of parameters. The output could then be fine-tuned by an archivist operator.

For all the mathematical complexity involved in such processes, some aspects are relatively comprehensible to a lay person. For example, consider that one feature of film sequences is that they contain a large amount of redundancy, in the sense that often much of the image in one frame is repeated in the next and even several others. (That’s because film is generally shot at the rate of 24 frames per second.) Algorithms that digitally correct anomalies in scanned and digitized film make use of that characteristic.

Thanks to that redundancy, if damage exists in one frame, technicians (and their algorithm-driven programs) can look to another frame for related but undamaged or less-damaged imagery. The algorithms are designed, in effect, to automate observations and corrections that humans used to make when film was analog, but could not hope to recognize in digital film — all those myriad ones and zeroes!

[An aside: A pioneer of digital film cleanup was Lowry Digital whose Canadian head, John Lowry, got access to a military algorithm used for satellites. When a satellite captures images as it flies over the same place several times a day from 200 miles up, you can compare the images and detect which parts are persistent over all of say 50,000 images — you can detect underlying images stripped of random events and other “noise.”]

Comparing images at anything from full-frame to part-frame or even microscopic level is obviously not a simple process, even with an algorithmic leg-up. Say, for example, that a sequence of images depicts an object in motion, and there’s a scratch or some other kind of damage in one or more frames of the physical film print that affects the image of the object. Repairing that damage requires that the technician and/or algorithm track the moving object in order to find undamaged imagery with which to make the repair.

In the process of computerized inspection of images, there’s unsurprisingly a danger of misidentifying damage and material to correct it with.

But the larger, over-all challenge in digital correction is to make the algorithms instantly as proficient as the human eye is when assessing projected images.

There are various aspects to this challenge.

Take this example of how algorithms can at once overperform and underperform, and require human intervention. Say you have a five-second close-up of an eye, and for one-quarter of a second there’s a glint in the eye. It’s a meaningful, moving glint – the filmmaker meant it to be there, as an ephemeral moment. If an algorithm removes it, the over-correction might go unnoticed, and the glimmer will be lost. Perhaps no one will even notice, at least consciously.

Just as unfortunate would be a case where, for example, overcorrection made an object — say, a car’s slender radio antenna — appear, disappear, and reappear, all within one short sequence, jolting viewers’ attention.

And how about correctly restoring images of really complex events, such as explosions?

This is all to say that, systems based on detection by algorithms can misinterpret images if the algorithm is insufficiently nuanced in its design.

Machines have virtually perfect acuity: they see what is there. But that still leaves the problem, the challenge, of training algorithms to make human-like judgments about what to remove from the scanned image — to judge whether particular elements of the image are intended features or defects. That’s no small feat — if it’s ultimately possible, at all.

Another way to view the challenge is that developers want to design algorithms and “artificial intelligence” approaches that emulate “metamerism.” The phenomenon of metamerism has to do with the way humans see — in combinations of color permitted by the three kinds of color-registering biological cone cells in their eyes. Metamerism occurs when what humans see when they look at an image of an object is matched by what they see when they look at the actual object. It’s when they experience functionally equivalent visual sensations. (They don’t necessarily see perfectly matching combinations of lightwaves, but rather combinations — spectral distributions — that their eyes interpret as equivalent.)

You want machines to act like humans in so far as the machines emulate the human biological predisposition to see slightly divergent but perceptually equivalent spectral distributions of light. (Here’s a simple visual explanation that summarizes the concept; and here’s an explanation from the point of view of color matching in industrial and other settings; or, if the technicalities don’t daunt you, try this.)

Setting side the slippery but crucial issue of accomplishing metamerism with machines, developers of automated digital technologies have permitted archivists to correct many kinds of damage with considerable faithfulness to the original film images. Say, for example, an algorithm-based scan encounters a film sequence where a ball flies through the air, but the image of the ball is scratched or otherwise obscured in some frames. If an algorithm can track an arc of movement over the course of several frames, the algorithm (usually a combination of algorithms) can also be written so that it goes to unblemished frames to retrieve unblemished images with which to repair the blemished ones.

Labor-Intensive Touch Ups

How close technicians can come to the original is the crucial issue. Perhaps upon watching the repaired film sequence, they will notice small discrepancies. One approach, then, also involving digitization, is to use an off-the-shelf or custom computer program that permits them to go in and digitally “paint in” details to replace unintended or otherwise problematic ones.

Two drawbacks of that approach are: first, it’s very time-consuming, and time is money, and second, it can make for an inconsistent look in sequences.

Producers of film restorations — and of new feature films, for that matter — frequently try to handle the cost issue by working with studios in countries with comparatively low wages. India, for example. So, for instance, at post-production studios like Reliance and Prasad in India, whole roomfulls of technicians can be found at work on a few dozen frames of film each, digitally fixing imperfections using a range of proprietary software packages or modifications of commercial ones such as Final Cut.

They work under contract to film and restoration producers, most in the West. The film companies and Indian post-production houses trade restored sections of film back and forth until desired quality and features are attained. That, too, is a painstaking process, and often a matter of doing the best one can.

Technicians, while making the corrections, eyeball their work and gauge how well they are doing — how faithful they are to an outcome that is necessarily vaguely defined. The human eye does quite well at gauging how much correction is needed before the results look out of keeping with the original footage, or surrounding footage, but human beings also tire, or their attention wanders, or they simply cannot make all the needed corrections consistent with each other. In short: the subjectivity and proficiency of their decision-making and technical skill waver.

Producers of restorations and re-touched new films, alike, report that they can even tell which images were repaired at the beginnings versus ends of shifts, and which have been done by workers on day versus night shifts. They can spot telltale variance in the look of the outputted sequences.

Restoring in Bulk

Unsurprisingly, when restoration is as expensive, time-consuming, and laborious as processes like these make it, hard choices are required about what gets restored first, or at all. To the top of the list go the most remunerative restorations along with those that most compel archivists on artistic or historic grounds.

Increasingly, in the digital age, the dream has become that one day it will be possible for public and private archives, as well as broadcasters and film companies, to digitally capture and restore whole large libraries of material.

How to do so, absent enough human input? Around the world, moving-image archivists are taking care of millions of films, most of them of little or no commercial value, regardless of the aesthetic or historical significance of many.

The obvious question then is: how could the enormous task be done less expensively, and more quickly? Could an algorithm-based system identify and correct damage accurately and consistently? It would have to be able to find or construct replacement material, and place it seamlessly into a section of digital film.

That is where companies increasingly are coming into play that are developing automated processes that, if sufficiently sophisticated and “well trained,” will become able to clean up those digital copies of pre-digital artifacts, and to do so on the fly.

That is helping archives and other institutions around the world where mass-scanning projects are under way. Their minimum aim is to run films through a digital scanner to create files of them. Then, finances permitting, project technicians can undertake restoration of damage. There, you get what you pay for, and a matrix of considerations comes into play. How much will return on investment be? How high will the quality of restoration be? For example, what quality of restoration will the intended audience expect? How quickly must the work be done?

For companies that offer the service of automated scanning and algorithm-assisted correction, the challenge is to find a sweet spot of output that clients both desire and can afford. As automated algorithm-driven approaches increase in use that are able to whip through batches of film without tiring like humans, sticking to unaltering processing protocols, human quality control will come afterwards to pick up any blatant problems.

It is the quickly growing number of companies like Algosoft — another is Mathematical Technologies Incorporated (MTI, whose Larry Chernoff, below, discusses its digital-restoration suite, DRS Nova) — that offer the mathematically intensive approaches to such tasks. Improvement in the capabilities of the processes is being driven by both restorers of old films and makers of new ones.

Unsurprisingly, major film studios, in quest of ensuring that their new releases are as impeccable as possible when they are sent out to cinemas — digitally, these days, rather than physically in film canisters — demand very high-quality postproduction. Perhaps a particle of dust got into a camera and created a blemish in a sequence of captured images.

Some blemishes, including scratches, can still be very hard to repair. But industry insiders have no doubt that restoration capabilities will continue to improve.

Intelligent or not, here comes AI

The future may be in AI, “artificial intelligence.” It’s the buzzword of the moment. What exactly AI is – is it really intelligence, or is the term “artificial intelligence” a metaphor? – is disputed; but that’s a whole other topic. In practical terms, in digital film restoration or processing, AI, machine learning, and similar capabilities promise to permit the much faster training of algorithms so they can become more sophisticated in what they detect and correct. How fast, depends on a company’s restoration resources, particularly in terms of mathematicians and software coders. Their task is to refine algorithms mathematically — to find ways to expand their capabilities and mitigate their shortcomings. That’s to say that they must create new code that achieves solutions to problems that arise and needs that become apparent. Unsurprisingly, that can be a slow, intensive feedback loop.

The value of AI, in this context, is that computers can speed up the detection and reformulation of problematic algorithms. In a sense, the algorithms train themselves, in part by sampling a wide range of approaches to see which offer improvements in performance.

That AI approach has some profound applications in film, beyond restoration. For example, it can be applied to learning and recreating particular film “looks,” such as the characteristic look of film noir with its distinctive colors, lighting, and spaces. A simpler variant of that capability can be used in restoration: if, for example, a film is damaged in scenes featuring jewelry, AI-driven restoration approach can include canvassing a large library of images to see what jewelry should look like when, for example, seen in particular light conditions.

The key to such an approach is to “train” algorithms with as many examples of images as possible. (This is why some companies — not to mention the creators of “deep fake” videos — are so keen to have images of your face and those of millions upon millions of other people; from those, they may create realistic likenesses of you or other humans engaged in various activities.)

One pivot point of deep fakes is that they can be created with proprietary software that only its owners possess, so only they can assess its capabilities and extent of use. More benignly, developers of algorithm-based film-restoration software suites have reasons not to share their software — at least, not their latest or fullest versions. Their motivation is that they also often provide restoration of films as a service.

In Hollywood and elsewhere, post-production companies — “post houses” — make use of various software suites, and various providers of restoration services. Restoration producers have a good idea about which products and services best serve their particular needs on particular projects — which are best at, say, removing certain kinds of blemishes, or at reducing graininess.

Conversely, because developers of software suites may have improved them as a function of working on particular film projects, the advances in the suites’ capabilities may not translate well to other kinds of film projects – they may lack generic solutions.

Any such complications, aside, one thing is certain, says Inna Kozlov, who founded Algosoft in 2005: automation is the key to affordability. Her company and others have restored many popular titles, and are making progress delving into public and private archives, says Kozlov, who is a professor at the Holon Institute of Technology (Israel) and also works with Alexander Petukhov at Algosoft’s U.S. base near Atlanta.

Petukhov is just as bullish about the company’s advances. He and Kozlov say they have achieved a steady stream of advances in moving-image processing: in dust and scratch removal, for example, and also in replacing — that’s to say, recreating — missing or badly damaged frames.

Restoration-studio technicians and bosses caution that vendors of algorithmic restoration suites will of course put the best light on their approaches and outcomes. The technicians and bosses aren’t keen to say much on the record, as they have to work with, and to varying degrees rely on the wave of automated-restoration companies. The most likely reality, they say, is that the companies’ claims, like any commercial ones, need to be taken cautiously. They note that getting the first 95 percent of the way to figuring out filmmakers’ original intention is the “easy” part.

The proof of Algosoft’s approach, Michael Inchalik, an advocate for it, said at a session of The Reel Thing Technical Symposium in August 2019 at the Academy of Motion Picture Arts and Sciences in Hollywood, California, is that now, when viewing with just the naked eye, “it’s hard to differentiate between original and synthesized frames.”

Inchalik, who while working with Kodak developed one of the first 4K digital scanning systems, said other notable advances had been in “optical flow,” the tracking from frame to frame to determine which images, at even a microscopic level, belong. That is in part an exercise in tracking the boundaries of images of objects — not as straightforward as it sounds.

Yet another area of marked advance, Inchalik said, is in making images as sharp as they were in original prints. Softer edges result from duplication after duplication from original prints to copies to other copies. Whether correcting that softness, or variations of film “grain” and “noise,” Inchalik said, the goal is to achieve historical accuracy — to figure out what the filmmakers’ original intentions were, and peel back to them, while also imparting a consistent appearance to each restored film.

One of Algosoft’s focuses has been on “video de-interlacing” — retraining its software’s “deep neural networks” to convert squiggly, wavering video images into smooth, high-definition digital images.

Algosoft has worked with such clients as the Library of Congress, as well as others around the U.S. and in India, Europe, China, Japan, and South Korea. Algosoft’s restoration suite, Viva, has played roles in the restoration of footage of Fritz Lang’s Metropolis, discovered in Argentina in 2010, as well of as many classic Russian films.

Image caption: Algosoft’s before and after (above) clips. Also, somewhat confusing, after & before (below) clips.

The home-movie market

Kozlov says Viva, by automatically recognizing and correcting various problems like those, cuts the need for manual retouching. That is where she sees an entrée into the home-movie market. Restoration of those, carried out by everyday owners, becomes more feasible with the “one button” approach, she believes. Software suites like Viva could have, say, separate professional and home-movie automatic-restoration tracks. The latter would leave archivists to adjust anything they might like to, while the former would be more automated so that the home user would have little to do other than press that “one button.”

Among encouraging signs, she and others working in the field say, is that not only is the technology improving, but the power and speed of computers is increasing. The bad news is of the usual variety: cost.

As Inchalik noted at the Reel Thing session, old films account for a tiny part of the billions of dollars being invested in such other areas of AI-driven imaging as military, security, industrial, and medical imaging. “That’s where the money is going,” he said. So, he told his audience of film archivists, “we have to kind of make this happen on our own.”

Algosoft’s approach to that issue is twofold.

First, it aims to build an automated cost-estimation system for potential customers. That would entail fashioning the Viva software suite so that potential customers could submit samples of films they would like repaired, and rapidly receive a system-generated quote of the likely cost of various kinds of restoration.

Second, Kozlov and her colleagues propose that archives should provide each other with access to their restoration projects through cloud-based data banks. That would provide them with “training sets” of data — with examples of challenges and even remedies from which any participating lab could shape approaches to particular film-restoration challenges.

Generally that sort of creation of very large data sets with which to “train neural networks” — with which to shape software suites’ detection of patterns to work on and respond to — is very expensive. Large companies like Google may be able to afford to invest in the labor-intensive work of building such data sets, but film archives have, of course, far more modest budgets. So collaboration makes sense, Kozlov and her colleagues suggest.

She says the development of self-driving cars offers a model. There, performance reports are fed into databases so that algorithmic analysis can work on improvements.

To reach the goal of satisfactory automated film restoration, multiple challenges of cost, cooperation, computer power, and other elements remain, but still, she declares, “it’s very close.”

Of course, moving-image archivists have their own measure of successful preservation: it begins before any digitization, let alone fancy algorithm-driven magic. It’s the work that happens before the expensive, painstaking work of getting a reel of film to digital form. It’s taking care of the original with proper storage and basic film-preservation techniques.

The analog part of the job.

But automated digital repair is clearly somewhere at film preservation’s leading edge.

---

Printed from Moving Image Archive News: http://www.movingimagearchivenews.org

URL to article: http://www.movingimagearchivenews.org/how-machines-restore-archival-film-or-at-least-are-trying-to/

Click here to print