Let’s enhance —using AI to magnify and restore images
AI tools for image enhancement have become increasingly popular, as they can upscale images without significant quality loss, reduce image noise, restore old photos, and even upscale videos. But can they do the CSI-type tricks? Let’s dive into them and let’s enhance!
The motif of magically zooming in on cameras, reading faces by their reflections on glasses, has been an evergreen of crime movies since computers came along. This video from Duncan Robson is a hilarious compilation of this trend. Whether it’s the FBI, CSI or MacGyver, they always seem to have magical enhancement technology that puts a smile on my face. But just as sci-fi movies often predicted the development of technology, these smile-inducing magical tools are not so far from reality thanks to the development of AI. Let’s check out what they can do.
AI Image enhancement use cases
I chose a few photos to test the different capabilities of AI.
1. A very low-resolution portrait (128px) by thispersondoesnotexist
2. AI art with face generated with Midjourney (512px) by futureMe app
3. AI art without face generated with Midjourney (512px) by futureMe app
4. A noisy low-res night landscape photo (512 px) by me
5. Another noisy low-res night photo with a lot of detail (512 px) by me
6. A night shot taken by phone with lots of artifacts (512 px) by me
The tests do not reflect scenes from the movies, and I can reassure you right away, the AI can’t yet recognize a face from a two-pixel refraction on an eyeball. However, it can zoom in and sharpen a very blurry photo of a face, enlarge a Midjourney image for printing, or remove excess noise and artifacts from an image.
The AI Image enhancement tools
I’ve tested quite a few apps for this experiment. There are some decent online services you can try, namely letsenhance.io, https://vanceai.com/, or canva.com. They seem to produce similar results to the upscalers you can get with Stable Diffusion Automatic1111 (I presume most use the same algorithms anyway). In the end, I narrowed the test to three apps I believe to be the most advanced on the market right now:
- Topaz Labs AI Gigapixel: Topaz tools are popular among professional photographers. The Gigapixel is specifically designed for upscaling images, but it also comes with some extras for denoising and face restoration. It costs 99$ (one-time license)
- Photoshop Neural filters: Adobe is betting big on AI and the new Neural Filters are just a start. But are they a match for other solutions that have been on the market for years? A Photoshop subscription is 19.99 $
- R-ESR-GAN+: Looking for a free alternative, I did some research on Reddit. The upscalers in Automatic1111 seem to be getting the best feedback and I believe are the best open-source solutions out there. Setting up Automatic1111 on Google Colab takes only a few minutes with a decent tutorial on youtube. It’s also the best GUI for Stable Diffusion, the ultimate open-source AI Image Generation software.
All tools were used to upscale images by 600 % and compared to regular upscale in Photoshop for reference.
Test One — Portrait
Keep in mind that this photo has very little detail, to be precise it has a resolution 40 times smaller than a popular 4K TV. The Gigapixel AI that has a Facial Restore feature gave superb results, generating a near-perfect photo. Look at the details in the eyes. Photoshop and ESRGan did a decent job in general upscaling, but the eyes tell they are fake.
Test Two — Midjourney Portrait
If you’re using Midjourney, you know it struggles with generating high-resolution images ie. for printing purposes. The tested tools really shined here and all did a great job. Gigapixel is once again my favorite, recreating a lot of detail but keeping the digital painting texture. ESR-GAN Anime version of the algorithm used here produced a super sharp image — a little overdone for me, but still very impressive.
Test Three — Midjourney Art
A very tricky sample, with thin lines and facial hair. I’d say Photoshop did the best job here, retaining the original painting details. Gigapixel did pretty well, ESR-GAN again overdid the sharpening effect.
Test Four — Noisy night scene
Although all tools are focused on upscaling, they do a really good job in noise reduction in this high-ISO photo.
Test Five — Noisy night scene 2
Another tricky image with high contrast and people in it. Really impressed with the Gigapixel here, removing the noise, and retaining details in textures.
Test Six — Phone camera with artifacts
A very blurred photo of a road salt truck on a snowy night with lots of noise and artifacts. Really impressed by how Gigapixel kept the snow in the wind while removing noise. The ESR-GAN did a great job too and delivered the most natural-looking image in my opinion.
Test summary
Topaz Gigapixel AI is definitely the best tool for upscaling and literally has the “magic” factor when it comes to portraits. If you are a photographer or want to print AI art, it’s worth the money for a lifetime license. For other purposes, all tools give good results. I’d say Photoshop AI has a lot to improve considering its price and in most situations, you will get decent results with free ESR-GAN or services like letsenhance.io.
The future of AI enhancement
As far as the film director’s creativity is concerned, we’re not there yet, and probably never will be. We can’t create high-resolution images or alternative camera angles out of thin air. But some of the tools are already working like magic, allowing us to create high quality prints from low resolution images. And that is just the beginning. What’s next? I believe it’s only a matter of time before AI enables our phone cameras to produce images with the quality of professional camera equipment. We can already see that camera manufacturers are no longer betting on the resolution of the sensor, and an 18-megapixel sensor in a semi-professional camera is the norm these days.
In the near future, the pictures we take with our camera phones will be just a “composition sketch” for AI to take over and render an image to our liking, making it look like a photo taken at dusk with a ten thousand dollar lens. Or like an old Zenith film pocket camera, the first one I had when I was a kid. Sounds like another funny idea from the movies? I’m afraid not this time.
Here’s a very simple proof of concept. On the left is a photo I took with my phone on a bike. I uploaded it as a reference to Midjourney and asked it to render it with a softer autumn sunset light. The composition is not the same, but this will probably change in the future. In fact this can already be done in Stable Diffusion.
I’ve just found a diffusion model trained with GPT-3, that can alter images with the understanding of natural language. You simply write a prompt ie. what to change in the image and how. Looks like I will need a followup AI image enhancement article.
If you don’t want to miss it and enjoyed this one, please follow me here on Medium for more stories about AI in creative advertising.
