Enhancing Photos with AI

[aricooperdavis]

I stumbled across an interesting piece of work this morning - a deep learning approach to "enhancing" photographs. This might just sound like an Instagram filter, but there's quite a bit more to it that that!

This is a technique which has used identical photographs taken on different cameras to teach a machine learner the kind of mistakes that low-res phone cameras make. After a lot of training that machine learner is now able to start with a photo taken on a low-res phone and spit out its best guess at what the photo would look like if taken on a DSLR. This includes lighting and colour balancing adjustments, but goes as far as de-blurring and de-noising.

My first thought when I saw this was to try it on some cave photos. The machine learner obviously hasn't been trained in caves, so it's likely to make some colour mistakes (most of its training data is from sunlit photos which are likely to be far warmer than underground shots) amongst other things, but it gave it a good shot.

This is a photo that I happened to have on my phone of a little bit of phreatic development between pitches in a cave being pushed on the Dachstein expedition. It was taken on my Huawei P10 Lite, which has an alright camera, but it's nothing special, and it makes a good test subject.





You can't see much improvement in detail, but it's done a good job at warming the colours up and removing some noise. It's also lifted some detail out of the shadows, although I quite liked the contrast that these shadows produced in the original. It's also stuck a weird red stripe at the top, which I can only assume is a bit of a bug.

I would be interested to see the applications of this kind of technology to help combat some of the issues often faced underground with lighting, colour, and noise. I don't think anyone will ever get the appropriate training data (as it would be a complete nightmare), but I'm certainly going to play with sticking some of my crappy cave photos through it to see how they turn out.

You can try your own photos online or have a bit more of a play with what's going on under the hood by running it yourself.

[2xw]

I've been reading a lot about this technology recently and it's going to change the world we live in. These AI processes can completely create an artificial realistic video of world leaders saying anything we like (and paste celebrities on to pr0nography if you're into that..), with just a few photos and videos taken from the web. It's going to create amazing movies and video games, but in turn I'd give it about 5 years before photographs, videos and audio recordings are completely inadmissible in court.


Pretty cool to use them on cave photos though I think I'll have a go...

[Topimo]

I'd like to play around with style transfer properly at some point:
http://genekogan.com/works/style-transfer/

I did find one website that would do it and made my face look like a Van Gogh, but it was impractically slow. 2xw, do you have time to find something more workable, perhaps a download?

There is Python code out there to do it, I seem to remember you having a play with Python Ari?

On the subject of faceswaps:

[The Old Ruminator]

This photo taken today. I have only used the Auto Haze Removal tool on Photoshop 2018. Makes a considerable difference.



With Auto Haze Removal.

[Chocolate fireguard]

Never thought I would contribute to a Film & Photography thread!!!

Ari, I read the article you referred to, but didn't understand most of it. I like to think that was because the authors made no attempt to explain many of the abbreviations and technical terms they used, but it's more likely to be that it's all beyond me anyway.

It seems to me that there is a certain amount of information in a digital image, and while it is easy to lose some of this it is not possible to increase it - any process that "enhances" the image in some way must do so at the expense of degrading it in some other way.
 
The authors of the article don't seem to mention this, but I am reluctant to accept that it's possible to get something for nothing.
Otherwise, where would it all end? A single pixel?

This is a serious contribution and I would genuinely be interested to hear from people who understand more about this sort of thing than I do - probably all the regular contributors.

Going on to your cave pictures, so far as I can see the improvement in the warmth (reminds me a bit of when I used carbide!) has been obtained at the expense of some resolution, but it's a marginal call.

[ZombieCake]

Danger is it turns into more digital fakery.

[Mike Hopley]

any process that "enhances" the image in some way must do so at the expense of degrading it in some other way.

Generally yes, once you output the new image.

This is why non-destructive workflows are useful in when working with media (photos, videos, audio...). It's handy if you can maintain flexibility as long as possible during the editing process. For example: using adjustment layers in Photoshop, rather than directly changing the colours of an image.

In any case, it's wise to preserve the original files.

[2xw]

Actually, in the case of AI, machine learning and neural network approaches, it is adding information to the point of making a whole new photograph.

To do this, the AI has been trained. It is presented with low resolution versions of high resolution photos, and attempts to make them high res again by adding pixels, upscaling the photo, removing JPEG artefacts and other details. It is then presented with the original high res images so that it can learn from its mistakes.

These are very powerful processes. Deep learning algorithms have been used by Google to make professional looking photography from its street view panoramas
https://research.googleblog.com/2017/07/using-deep-learning-to-create.html?m=1

And a team in Washington used neural networks to entirely synthesize a photorealistic speech by Obama

http://grail.cs.washington.edu/projects/AudioToObama/

[2xw]

You may also be interested in neural doodles, which can turn doodles done in paint into "high art" using semantic style transfer

https://github.com/alexjc/neural-doodle

[ZombieCake]

Nothing at all to do with photography. Just using some algorithm or database to make stupid snowflakes think they can take photos.  All so artificial and utterly pointless.  If you want to take a photo, take a photo. Might be OK, might not, all part of the process.
The computer has taken the photo, you haven't.  An analogy: You are not a racing driver the Play Station is.
RAW, Photoshop, Lightroom, has a lot to do with imaging: photography is less clear.

[The Old Ruminator]

Certainly cameras will continue to be more "intuitive" possibly having just a one pixel image and a massive amount of resolution. The "art" thereafter will be in the editing on a computer. We have to accept that folk want things to be as good as possible as easy as possible. In a way mobile phones are driving camera technology as it is one way that producers can continue to make existing products obsolete. There are now loads of editing programmes on the internet like Aurora . They are becoming very common place. Personally I dont like the garish colours that that some produce. We are shifting further from reality in the search for effect. Even some caving photos are now exhibiting this trend and when they appear on Facebook we get " Wow. What a wonderful photo. "It used to be said that " a camera never lies ". Well it does now.

[The Old Ruminator]

This is " Skylum ". Some think it is wonderful but I don't.

[Bob Mehew]

"It used to be said that " a camera never lies ". Well it does now.

Techniques were quickly introduced soon after photography was invented to produce 'lies'.  See https://www.coolantarctica.com/Shop/ansel_adams/moon_and_half_dome.php for an example of an 'acceptable' lie.

[mudman]

"It used to be said that " a camera never lies ". Well it does now.

Techniques were quickly introduced soon after photography was invented to produce 'lies'.  See https://www.coolantarctica.com/Shop/ansel_adams/moon_and_half_dome.php for an example of an 'acceptable' lie.

This is a good point. Photographers have always used various techniques to enhance, fix or manipulate their images. With the advent of digital, it just meant that photographers used different tools to do the same things although the toolbox has been expanded to allow photographers to do other things too.

[paul]

"It used to be said that " a camera never lies ". Well it does now.

Techniques were quickly introduced soon after photography was invented to produce 'lies'.  See https://www.coolantarctica.com/Shop/ansel_adams/moon_and_half_dome.php for an example of an 'acceptable' lie.

I was just thinking the same thing, especially WRT Ansel Adams. Before digital photography many photographers used to manipulate their work in the dark room (see https://en.wikipedia.org/wiki/Zone_System for more examples and details especially the "dodging and burning" section.)

[Fulk]

I've done quite a lot of darkroom work and know that it is possible to alter photographs; the difference between digital manipulation and silver-based darkroom manipulation is that the former is relatively easy, the latter very difficult.

[aricooperdavis]

Old Ruminator, I spotted an earlier post of yours demonstrating the de-hazing tool in photoshop, and thought that this was a nice continuation on that theme. I'd love if there was an open-source implementation of that, as I can't really warrant buying photoshop, and would quite enjoy looking at how that works in a bit more detail.

Topimo, I have done a little bit of machine learning work, python has a great tensorflow library that lays a lot of the groundwork, but the training data would be a hassle and would probably require a lot of fiddling. If I came up with a good idea for something to do with it then I'd be up for having a play around.

Chocolate fireguard, I just used the online example tool, so it's possible that the loss of resolution is something that they've done to make it achievable to do all that processing for people - I see no reason why doing it yourself should change the resolution at all really.

Zombie, I'm no photographer, I just take photos when I'm caving sometimes. If there's something that I can do to easily to make those photos look clearer then I'll use it, much like I'll take a nice bright headlight in to make my photos clearer, or I'll fiddle with brightness and contrast in GIMP. Not trying to cheat, just to have some fun.

[Fulk]

I don't know if this is relevant, aricooperdavis, but I use Photoshop Elements (I can't afford the full-blown product), which was ~10% the price of the complete package when I bought it. My version doesn't have the anti-haze tool – I don't know whether the latest vetrsion does.

[paul]

If you want to use Gimp for removing haze (an open source and fee photo manipulation package) see http://osp.wikidot.com/haze-removal

[aricooperdavis]

Ah that's a fantastic tutorial, thanks Paul.

[mudman]

Yes, thanks for that Paul. A surprisingly easy process.
Thought I'd give it a go so tried it on this picture from North West Inlet in Ogof Craig a Ffynnon.
Not a very good photo anyway but it would be a good test.
Definitely an improvement. I didn't follow the tutorial exactly because my levels are the inverse of the surfing picture but definitely an improvement and although it doesn't replace avoiding the haze in the first place, it is great for rescuing pictures for uploading to places like forums and club websites.

[Topimo]

Adjusting the levels like that also has the effect of boosting the saturation of colours (as can be seen in your image mudman), if you go to "Colors">"Hue-Saturation", and bring the saturation level down a little, you can season to taste...as it were.

[mudman]

Adjusting the levels like that also has the effect of boosting the saturation of colours (as can be seen in your image mudman), if you go to "Colors">"Hue-Saturation", and bring the saturation level down a little, you can season to taste...as it were.

Thanks, I've tried that and it does tone down the garishness a bit.
How's this?

[Topimo]

Looks cool, there's another tool called the Clone Tool that you could use to remove the tape with some care, but all this is subjective as to what you consider important in the shot.



https://docs.gimp.org/en/gimp-tool-clone.html

[Topimo]

Hope you don't mind, I had a little go myself.

[Topimo]

Some quick fakery.

[mudman]

Hope you don't mind, I had a little go myself.

Not at all. I like to see what others can do with stuff I've had a go at.

I like the clone tool and have used it myself although it can be a long process.

Here's a before and after of a picture of Straw Chamber in OCAF that I messed around with a while ago.

Both pictures have had to be scaled to fit in the forum limits:

[Chocolate fireguard]

From Chocolate Fireguard:
“It seems to me that there is a certain amount of information in a digital image, and while it is easy to lose some of this it is not possible to increase it - any process that "enhances" the image in some way must do so at the expense of degrading it in some other way.”

From 2xw:
“Actually, in the case of AI, machine learning and neural network approaches, it is adding information to the point of making a whole new photograph.”

OK, I give in.
People seem to understand different things when it comes to “information”.

[tony from suffolk]

A good example of adding to existing images is video technology, where sophisticated algorithms interpolate the existing images to add “missing” information by looking at the surrounding pixels. This works extremely well in the latest 4K displays when upscaling standard and high definition video.

[RichardB1983]

In the late 1990s and early 2000s, some astronomers started to use a technique where they used cheap webcams to take short-ish videos of planets or the Moon through amateur telescopes, and then there was a free program called Registax which would align and stack the video frames together. By tweaking a few settings, it was often possible to produce resultant photos that looked like they'd been taken with a professional telescope, and far beyond the normal capabilities of the telescope or camera for a single image.

The idea was that you could vastly improve the signal to noise ratio by stacking a few thousand video frames of the same object and hence bring out much more detail than was possible by taking a single photograph.

It makes me wonder whether you could achieve quite good results just by taking a mobile phone or point-and-shoot camera video of a cave or mine scene, appropriately lit up etc. and use similar techniques to bring out the details. Part of the need in astronomy is that the atmosphere is unsteady at the magnifications being used, which is not as big an issue when photographing cave passages, but it might filter out dust / water droplets on the images if that's an issue in certain areas. Might give it a go next time I'm underground with a camera.

It that the technique Tony was referring to above?

[ZombieCake]

Some quick fakery.

Fakery is a disease that is most unwarranted.  It discredits everyone and calls everything into disrepute. 

[NewStuff]

Nothing at all to do with photography. Just using some algorithm or database to make stupid snowflakes think they can take photos.  All so artificial and utterly pointless.  If you want to take a photo, take a photo. Might be OK, might not, all part of the process.
The computer has taken the photo, you haven't.  An analogy: You are not a racing driver the Play Station is.
RAW, Photoshop, Lightroom, has a lot to do with imaging: photography is less clear.

No, sorry, have to disagree.

It's a tool, in the same way dodging and burning was when printing your own pictures from film you developed. You could (and still can) make a small tweak to exposure, or you can make a composite image with no basis in reality. Only the tools are changing.

In the same way Levels, Unsharp mask, and the clone tool are in most decent editing packages.

Photography is clearer than it ever was.

[ZombieCake]

We'll actually probably sort of to agree to disagree a bit. Load up what you want but make original negatives, JPEGS, RAW accessible too. I'm sure everyone can live with that.

[NewStuff]

I think you would have to be a bit carefree to be working from originals. My workflow has a script that checks SD cards on insertion and pulls new images to a backup drive, then does the same again from the backup drive to a working drive. But I saw a lot of people lose masses of work, so all of that is before my normal backups/snapshots/disk imaging.

Now, chances of me ever handing someone without a warrant an untouched original? Slim to none.

[tony from suffolk]

Quite right NewStuff. Surely people cannot believe any published photo nowadays isn't extensively digitally manipulated? I see no problem with what can be a very creative editing process. It's also damn good fun! What the camera captures is pretty detached from reality anyway - try printing out a camera's RAW image. Not particularly interesting, is it?

[andrewmc]

The camera is not the eye.

[pwhole]

Shooting in RAW (if your camera permits it) is the single most important in-camera setting to use, as it's a 12-bit format, as opposed to 8-bit, which is the limit most graphics cards and monitors can display. This means that instead of 256 levels of brightness between black and white (as defined in the three RGB channels), there are 4096 (I think!). So instead of 16.7 million potential colours available there are 68.7 billion. Obviously a monitor has a fixed number of pixels it can display, but it's a much bigger palette to choose from, so much better quality in terms of tonal and colour rendition, especially in high-contrast scenes. My Dell monitor claims to be 10-bit, but I think it needs a much posher graphics card to make much obvious difference.

When I did B/W photography, I used to under-rate Kodak Tri-X to 320 ISO instead of 400, thereby over-exposing it, and then developing in Rodinal at a much lower dilution and for much longer than the standard. It produced an amazing improvement in tonal rendition, especially in the shadows and highlights. Developing prints in very, very weak developer for much longer times and then finishing off in a normal bath for a (very) short time then improved the tonal rendition in the print as well. If it was large-format, on good-quality fibre paper, the results were stunning, and even 35mm was brilliant, as Rodinal produces very sharp edges on the silver crystals, so it looks 'sharp' even when it's obviously grainy. Those were the days.

But shooting in RAW and then processing via a proper RAW application into 8-bit is pretty much the digital equivalent of the above.

[Topimo]

I just found these style transfer shots I generated a little while ago.

[pwhole]

Nice. Like 1920s Russian posters. Is that those big limekilns?

[Topimo]

Sheffield

[pwhole]

Ah right - of course. I caught some big perch once just downstream of those arches

[aricooperdavis]

Digging up an old thread, I know, but I've stumbled across another nice use of AI to help improve low-light photography. Again, capturing the training data would be a bit of a pain (take two photos with different exposure durations without changing anything else in the scene), but the results are remarkable.