Enhancing Photos with AI

[aricooperdavis]

I stumbled across an interesting piece of work this morning - a deep learning approach to "enhancing" photographs. This might just sound like an Instagram filter, but there's quite a bit more to it that that!

This is a technique which has used identical photographs taken on different cameras to teach a machine learner the kind of mistakes that low-res phone cameras make. After a lot of training that machine learner is now able to start with a photo taken on a low-res phone and spit out its best guess at what the photo would look like if taken on a DSLR. This includes lighting and colour balancing adjustments, but goes as far as de-blurring and de-noising.

My first thought when I saw this was to try it on some cave photos. The machine learner obviously hasn't been trained in caves, so it's likely to make some colour mistakes (most of its training data is from sunlit photos which are likely to be far warmer than underground shots) amongst other things, but it gave it a good shot.

This is a photo that I happened to have on my phone of a little bit of phreatic development between pitches in a cave being pushed on the Dachstein expedition. It was taken on my Huawei P10 Lite, which has an alright camera, but it's nothing special, and it makes a good test subject.





You can't see much improvement in detail, but it's done a good job at warming the colours up and removing some noise. It's also lifted some detail out of the shadows, although I quite liked the contrast that these shadows produced in the original. It's also stuck a weird red stripe at the top, which I can only assume is a bit of a bug.

I would be interested to see the applications of this kind of technology to help combat some of the issues often faced underground with lighting, colour, and noise. I don't think anyone will ever get the appropriate training data (as it would be a complete nightmare), but I'm certainly going to play with sticking some of my crappy cave photos through it to see how they turn out.

You can try your own photos online or have a bit more of a play with what's going on under the hood by running it yourself.

[2xw]

I've been reading a lot about this technology recently and it's going to change the world we live in. These AI processes can completely create an artificial realistic video of world leaders saying anything we like (and paste celebrities on to pr0nography if you're into that..), with just a few photos and videos taken from the web. It's going to create amazing movies and video games, but in turn I'd give it about 5 years before photographs, videos and audio recordings are completely inadmissible in court.


Pretty cool to use them on cave photos though I think I'll have a go...

[Topimo]

I'd like to play around with style transfer properly at some point:
http://genekogan.com/works/style-transfer/

I did find one website that would do it and made my face look like a Van Gogh, but it was impractically slow. 2xw, do you have time to find something more workable, perhaps a download?

There is Python code out there to do it, I seem to remember you having a play with Python Ari?

On the subject of faceswaps:

[The Old Ruminator]

This photo taken today. I have only used the Auto Haze Removal tool on Photoshop 2018. Makes a considerable difference.



With Auto Haze Removal.

[Chocolate fireguard]

Never thought I would contribute to a Film & Photography thread!!!

Ari, I read the article you referred to, but didn't understand most of it. I like to think that was because the authors made no attempt to explain many of the abbreviations and technical terms they used, but it's more likely to be that it's all beyond me anyway.

It seems to me that there is a certain amount of information in a digital image, and while it is easy to lose some of this it is not possible to increase it - any process that "enhances" the image in some way must do so at the expense of degrading it in some other way.
 
The authors of the article don't seem to mention this, but I am reluctant to accept that it's possible to get something for nothing.
Otherwise, where would it all end? A single pixel?

This is a serious contribution and I would genuinely be interested to hear from people who understand more about this sort of thing than I do - probably all the regular contributors.

Going on to your cave pictures, so far as I can see the improvement in the warmth (reminds me a bit of when I used carbide!) has been obtained at the expense of some resolution, but it's a marginal call.

[ZombieCake]

Danger is it turns into more digital fakery.

[Mike Hopley]

any process that "enhances" the image in some way must do so at the expense of degrading it in some other way.

Generally yes, once you output the new image.

This is why non-destructive workflows are useful in when working with media (photos, videos, audio...). It's handy if you can maintain flexibility as long as possible during the editing process. For example: using adjustment layers in Photoshop, rather than directly changing the colours of an image.

In any case, it's wise to preserve the original files.

[2xw]

Actually, in the case of AI, machine learning and neural network approaches, it is adding information to the point of making a whole new photograph.

To do this, the AI has been trained. It is presented with low resolution versions of high resolution photos, and attempts to make them high res again by adding pixels, upscaling the photo, removing JPEG artefacts and other details. It is then presented with the original high res images so that it can learn from its mistakes.

These are very powerful processes. Deep learning algorithms have been used by Google to make professional looking photography from its street view panoramas
https://research.googleblog.com/2017/07/using-deep-learning-to-create.html?m=1

And a team in Washington used neural networks to entirely synthesize a photorealistic speech by Obama

http://grail.cs.washington.edu/projects/AudioToObama/

[2xw]

You may also be interested in neural doodles, which can turn doodles done in paint into "high art" using semantic style transfer

https://github.com/alexjc/neural-doodle

[ZombieCake]

Nothing at all to do with photography. Just using some algorithm or database to make stupid snowflakes think they can take photos.  All so artificial and utterly pointless.  If you want to take a photo, take a photo. Might be OK, might not, all part of the process.
The computer has taken the photo, you haven't.  An analogy: You are not a racing driver the Play Station is.
RAW, Photoshop, Lightroom, has a lot to do with imaging: photography is less clear.

[The Old Ruminator]

Certainly cameras will continue to be more "intuitive" possibly having just a one pixel image and a massive amount of resolution. The "art" thereafter will be in the editing on a computer. We have to accept that folk want things to be as good as possible as easy as possible. In a way mobile phones are driving camera technology as it is one way that producers can continue to make existing products obsolete. There are now loads of editing programmes on the internet like Aurora . They are becoming very common place. Personally I dont like the garish colours that that some produce. We are shifting further from reality in the search for effect. Even some caving photos are now exhibiting this trend and when they appear on Facebook we get " Wow. What a wonderful photo. "It used to be said that " a camera never lies ". Well it does now.

[The Old Ruminator]

This is " Skylum ". Some think it is wonderful but I don't.

[Bob Mehew]

"It used to be said that " a camera never lies ". Well it does now.

Techniques were quickly introduced soon after photography was invented to produce 'lies'.  See https://www.coolantarctica.com/Shop/ansel_adams/moon_and_half_dome.php for an example of an 'acceptable' lie.

[mudman]

"It used to be said that " a camera never lies ". Well it does now.

Techniques were quickly introduced soon after photography was invented to produce 'lies'.  See https://www.coolantarctica.com/Shop/ansel_adams/moon_and_half_dome.php for an example of an 'acceptable' lie.

This is a good point. Photographers have always used various techniques to enhance, fix or manipulate their images. With the advent of digital, it just meant that photographers used different tools to do the same things although the toolbox has been expanded to allow photographers to do other things too.

[paul]

"It used to be said that " a camera never lies ". Well it does now.

Techniques were quickly introduced soon after photography was invented to produce 'lies'.  See https://www.coolantarctica.com/Shop/ansel_adams/moon_and_half_dome.php for an example of an 'acceptable' lie.

I was just thinking the same thing, especially WRT Ansel Adams. Before digital photography many photographers used to manipulate their work in the dark room (see https://en.wikipedia.org/wiki/Zone_System for more examples and details especially the "dodging and burning" section.)

[Fulk]

I've done quite a lot of darkroom work and know that it is possible to alter photographs; the difference between digital manipulation and silver-based darkroom manipulation is that the former is relatively easy, the latter very difficult.

[aricooperdavis]

Old Ruminator, I spotted an earlier post of yours demonstrating the de-hazing tool in photoshop, and thought that this was a nice continuation on that theme. I'd love if there was an open-source implementation of that, as I can't really warrant buying photoshop, and would quite enjoy looking at how that works in a bit more detail.

Topimo, I have done a little bit of machine learning work, python has a great tensorflow library that lays a lot of the groundwork, but the training data would be a hassle and would probably require a lot of fiddling. If I came up with a good idea for something to do with it then I'd be up for having a play around.

Chocolate fireguard, I just used the online example tool, so it's possible that the loss of resolution is something that they've done to make it achievable to do all that processing for people - I see no reason why doing it yourself should change the resolution at all really.

Zombie, I'm no photographer, I just take photos when I'm caving sometimes. If there's something that I can do to easily to make those photos look clearer then I'll use it, much like I'll take a nice bright headlight in to make my photos clearer, or I'll fiddle with brightness and contrast in GIMP. Not trying to cheat, just to have some fun.

[Fulk]

I don't know if this is relevant, aricooperdavis, but I use Photoshop Elements (I can't afford the full-blown product), which was ~10% the price of the complete package when I bought it. My version doesn't have the anti-haze tool – I don't know whether the latest vetrsion does.

[paul]

If you want to use Gimp for removing haze (an open source and fee photo manipulation package) see http://osp.wikidot.com/haze-removal

[aricooperdavis]

Ah that's a fantastic tutorial, thanks Paul.

[mudman]

Yes, thanks for that Paul. A surprisingly easy process.
Thought I'd give it a go so tried it on this picture from North West Inlet in Ogof Craig a Ffynnon.
Not a very good photo anyway but it would be a good test.
Definitely an improvement. I didn't follow the tutorial exactly because my levels are the inverse of the surfing picture but definitely an improvement and although it doesn't replace avoiding the haze in the first place, it is great for rescuing pictures for uploading to places like forums and club websites.

[Topimo]

Adjusting the levels like that also has the effect of boosting the saturation of colours (as can be seen in your image mudman), if you go to "Colors">"Hue-Saturation", and bring the saturation level down a little, you can season to taste...as it were.

[mudman]

Adjusting the levels like that also has the effect of boosting the saturation of colours (as can be seen in your image mudman), if you go to "Colors">"Hue-Saturation", and bring the saturation level down a little, you can season to taste...as it were.

Thanks, I've tried that and it does tone down the garishness a bit.
How's this?

[Topimo]

Looks cool, there's another tool called the Clone Tool that you could use to remove the tape with some care, but all this is subjective as to what you consider important in the shot.



https://docs.gimp.org/en/gimp-tool-clone.html

[Topimo]

Hope you don't mind, I had a little go myself.