Opinion - DXOMark's camera scoring makes ZERO sense!

[kidzrevil]

this is exactly why I don't care for this stuff. DXOmark has too many variables as to what qualifies a camera as a good one and none of it translates to real world performance and more importantly : aesthetic.

[tugela]

Differences in scores between two cameras with the same sensor probably relate to performance differences between individual cameras. When it comes to silicon, performance is not necessarily the same across the entire batch, some chips will perform better than others.

Because the tests use a variety of parameters, the firmware and supporting hardware used in different models may have an impact as well, so even though two cameras may use the same sensor, they might not get the same score.

[rak_heri]

The biggest issue with DXO is giving too much weight to resolution.

Cameras with higher resolution always rank higher than lower-res cameras with similar or better performance in ISO or dynamic range or color accuracy.

Thus a RED Helium with better resolution (8K) will blow away other comparable cameras. This is how their DxO camera rank so high in their own ranking even though it's only a 1 inch sensor.

A good way to proove is to take all their cameras and graph resolution vs DxO ranking. I am sure you will find linearity.

[Don Kotlos]

5 minutes ago, rak_heri said:

A good way to proove is to take all their cameras and graph resolution vs DxO ranking. I am sure you will find linearity.

Yes cause correlation implies causation in some perceptual realities I guess. 

Why don't we just start by reading a scientific paper they wrote on this matter:

Does resolution really increase image quality? Christel-Loïc Tisse, Frédéric Guichard, Frédéric Cao DxO Labs, 3 Rue Nationale, 92100 Boulogne, France

[gethin]

 I bought a nikon 5300 because the dynamic range was close to my d800.  This was how I discovered dxo marks were a big fat waste of time.  In fact I'm sure there is a better metric that could show the how noisy the shadows are when the highlights are just blowing out.

 

[Thobias]

I always wonder why they have so underrated Canon cameras generally... Because of this I never trust them.

[rak_heri]

3 hours ago, Don Kotlos said:

Why don't we just start by reading a scientific paper they wrote on this matter:

Does resolution really increase image quality? Christel-Loïc Tisse, Frédéric Guichard, Frédéric Cao DxO Labs, 3 Rue Nationale, 92100 Boulogne, France

DxOMark is a for-profit company. Such papers are written to produce more clicks or more sales and there's a good probability they are biased. You can link to research done by independent researchers though.

[noone]

4 hours ago, Thobias said:

I always wonder why they have so underrated Canon cameras generally... Because of this I never trust them.

They down rated Canon because the cameras didn't have the highest DR at base ISO.      They can not give a camera a higher rating than they can measure.

Once you got to higher ISOs, those Canon cameras would often be as good (if not better) which is why you don't just go off the headline figure.

The latest Canon cameras are as good as the competition at base ISO now so they get higher scores for example, they measured the 5Diii at 11.7 stops of DR but the 5Div gets 13.6 stops (nearly 2 full stops).     At higher ISOs the improvement is less than a stop.    That plus the other improvements is why the 5Div gets a much higher score.

 

Many of the situations where  a much lesser camera gets a higher score, that is what they measure but if you look further, you will find the lesser cameras fall away much quicker.

Sometimes the lesser cameras have higher pixel counts too.

With Nikon for instance, if you want 24mp and only shoot with top class manual focus lenses like an Otus at base ISO, you may well be better off with a D5## than getting a better more expensive camera.

Besides DXO, Bill Claff has a pretty good site for sensors.

http://photonstophotos.net/

His chart (not all cameras are there).

http://www.photonstophotos.net/Charts/PDR.htm

[tomekk]

On 1/12/2017 at 1:53 AM, Andrew Reid said:

The Arri Alexa for example has dual gain architecture to boost dynamic range by reading two signals per pixel out on a low and high gain. Does this count as RAW sensor performance or image processing? 

I don't see why this wouldn't count as sensor's performance? From what I understand it's done at the sensor level and it doesn't degrade data (quite opposite) so it's logically sensor's performance, right? I think, there has to be a distinction between a hardware denoising and software denoising done on an already processed and compressed/converted output which degrades data (which is what they suspect in their article I believe). By already processed I mean after conversion from ADC to a target colour space and therefore already compressed from sensor's output (if I get the science right). Even though people call it raw data, it's not a raw output from the sensor and denoising can be done in post in this case (which is cheating in my dictionary). It's basically like denoising a raw file in lightroom. You've got details extinction but it looks cleaner. Quite a few DSLRs do it getting better reviews in noise performance even though they shouldn't. The trend is worrying so I think it's important to always single them out if possible, at least in semi-pro and pro gear. Consumers are a lost case. They've got no clue and don't want/don't have time to find out. All marketing BS buzz word gets them to buy things as we can see by TVs' menus ;). 

[BucksterUK]

I've always found the DXO Mark scores for sensors (although I only tend to look at the Dynamic Range figures) - has matched my real-world relative experience with cameras

so I think its an excellent site and glad it exists

 

 

[Eric Calabros]

Ignore their overal score, its BS. but their DR measurements are accurate and reliable. RED should be honest with their customers, 15.2ev means only 12 stop of "usable" DR (check here yourself: www.photonstophotos.net/Charts/PDR.htm), and its only true at base ISO.

Where did those 18 stop claims come from? 

[Andrew Reid]

13 hours ago, noone said:

They down rated Canon because the cameras didn't have the highest DR at base ISO.      They can not give a camera a higher rating than they can measure.

Once you got to higher ISOs, those Canon cameras would often be as good (if not better) which is why you don't just go off the headline figure.

The latest Canon cameras are as good as the competition at base ISO now so they get higher scores for example, they measured the 5Diii at 11.7 stops of DR but the 5Div gets 13.6 stops (nearly 2 full stops).     At higher ISOs the improvement is less than a stop.    That plus the other improvements is why the 5Div gets a much higher score.

 

Many of the situations where  a much lesser camera gets a higher score, that is what they measure but if you look further, you will find the lesser cameras fall away much quicker.

Sometimes the lesser cameras have higher pixel counts too.

With Nikon for instance, if you want 24mp and only shoot with top class manual focus lenses like an Otus at base ISO, you may well be better off with a D5## than getting a better more expensive camera.

Besides DXO, Bill Claff has a pretty good site for sensors.

http://photonstophotos.net/

His chart (not all cameras are there).

http://www.photonstophotos.net/Charts/PDR.htm

Here's what I don't get though - the 5D Mk IV is a noisy as hell camera with 30MP, yet have you seen the ISO score on Bill's chart and at DXOMark?

Makes no sense for it to be so high.

[squig]

On 13/01/2017 at 2:04 AM, Andrew Reid said:

Here's what I don't get though - the 5D Mk IV is a noisy as hell camera with 30MP, yet have you seen the ISO score on Bill's chart and at DXOMark?

Makes no sense for it to be so high.

Most Nikons have a lot less noise than Canons at base ISO. DP Review calls it "ISO invariance". In real world shooting you can get away with underexposing a Nikon (I'm talking about raw photos, not video), whereas you have to expose to the right with a Canon to achieve the best results (raw photo and raw video). The 5D MK4 isn't as clean as a D750 or D810 at base ISO, but it's a lot cleaner than a 5D MK3. Interestingly, although Nikon uses a lot of Sony sensors, Sony cameras don't like to be underexposed, especially if you're shooting sLog.

From my experience shooting with a bunch of cameras, DXOmark raw dynamic range results appear to be accurate. Noise measurements aren't as accurate a guide for video shooters because they don't take into account problematic things like highly compressed codecs, fixed pattern noise, or blue splotches (A7s).

Doing your own tests and finding a sensor's sweet spot is crucial to getting the highest dynamic range and least noise out of any camera. Proper post processing of raw files is equally important.  Every camera has some quirks that lab tests don't pick up on.

[rooney111]

On 13/01/2017 at 2:48 PM, noone said:

.

Besides DXO, Bill Claff has a pretty good site for sensors.

http://photonstophotos.net/

His chart (not all cameras are there).

http://www.photonstophotos.net/Charts/PDR.htm

From that second link NEX 5N and D3300 appears closely identical to NX1 in the graph/chart " ISO SettingPhotographic Dynamic Range (log2)Photographic Dynamic Range versus ISO Setting" 

[dahlfors]

On 13/01/2017 at 4:04 PM, Andrew Reid said:

Here's what I don't get though - the 5D Mk IV is a noisy as hell camera with 30MP, yet have you seen the ISO score on Bill's chart and at DXOMark?

Makes no sense for it to be so high.

Because of their methodology; They do the noise measurements from prints (if I remember correctly it's around 8-12 Megapixel prints).

Downsize ANY noisy image in Photoshop - and you'll reduce the perceived noise of the image - especially with such a hefty downsizing as 30 MP -> 10 MP. High resolution sensors with fine-grained noise will gain the most in DXO's test.

DXOmarks sensor testing is very well done in an unbiased way. But you need to understand their testing methodology and how it applies to what you do with your photography to understand how the sensor will perform for your use case - or if the use case is covered enough by the DXOmark testing. You shouldn't spend much on the single numbers they present either, but rather look at the graphs! And when you do, always always keep in mind that it isn't a measurement done directly from sensor data, but that it has gone the route through downsizing and printing. If you know how those processes function, you also know how that will impact the test outcome.

If you care about using the full 30 MP images and how the noise pattern from a camera looks like at 100%, DXOmark is definitely NOT the place to look. It's a good estimation of noise performance at print-sized photos in a magazine - or for photos downsized to fit screens / websites. If you want very large size prints or like to export 100% crops of photos, the noise characteristics can't be extrapolated from the DXOmark data - you'll have to find other tests than theirs.

A short summary on how to make use of DXOmark when you're going to buy cameras:

1) Realise that it is only the sensor & processing performance of raw files that get tested

2) Make sure to read and understand the methodology

3) Look at the graphs and compare the cameras that interest you!

4) Due to the downsizing methodology - check elsewhere for sensor and processing performance at different ISOs to get a more complete picture.

5) Now you know a bit about the sensor performance for still photography, which shouldn't really be the main reason for camera choice. Handling, lens choice, video performance etc will have to be tried & tested and read and learnt about elsewhere.

[Don Kotlos]

1 hour ago, dahlfors said:

They do the noise measurements from prints

They also offer "screen" results for the pixel peepers.

[dahlfors]

18 minutes ago, Don Kotlos said:

They also offer "screen" results for the pixel peepers.

Indeed! The issue with the screen results is that I haven't seen any description of their screen setup (but maybe it exists somewhere in their writing / on their site?).

The results of that one will be very dependent on how they do it. The print process is described well enough to deduct the information you need. What I find interesting if you look at screen result is that dynamic range is shown to be less than on the print version, which I assume means that their screen setup limits the information you can pick out of the test in one way or the other.

[pss]

the overall ratings of cameras are based on how well they perform in different areas, so one camera might not be the best in any criteria but might still be the "best" all around...the A7RII is a pretty good example....

as for the helium ratings, i believe they are based on raw capture off the sensor, the problem is that all red files are always compressed....the weapon helium probably starts at 5:1 and  it goes up from there (depending on cropping, fps,...) this makes a big difference....

in all fairness, anyone can go to red.com, download raw files (in different sizes and compression rates) as well as their raw converter...all this is free and allows for great pixel peeping....

i used to own a red epic, i did pull stills fro print from that sensor and this helium is miles ahead of the MX and dragon sensors, especially when it comes to noise/high ISO performance.....but stills pulled from the available helium test files are nowhere near stills from A7RII....detail, noise, even DR and color.....

but all this is still pretty amazing and definitely a sign of things to come....

[leeys]

I'll echo what many have said: DXO is a fantastic resource, but you need to ignore the silly score they give and look at the test charts in the areas that are important to you. Also make sure the chart you're reading is showing the ACTUAL resolution, not the downsampled one.

[Andrew Reid]

The problem with these charts and scores is that they don't take into account subjectivity by their very nature, they are only about technical dynamics and figures.

The Leica M9 full frame CCD sensor is a case in point. Different tonal response, colour, noise pattern, more filmic to my eye to CMOS - subjectively so. Loses badly to a lot of crop sensor crap on the DXOMark charts. So you need to be careful, there are a lot of people taking DXOMark as gospel and the RED Helium sensor, which I am sure is incredible, will also have a look that either subjectively you like to dislike, that only a real world test can show... yet because it got a high score on DXOMark the internet went wild and a consensus grew that this could be one of the best sensors ever created.

Personally, a sensor cannot be judged independently of the lenses you use on it and the format of the sensor - i.e. Super 16mm, 35mm, 65mm.

And as we see from the Leica M9, how the dynamic range is subjectively handled is more important than the number of stops on a test chart, which may not even all be usable any way.