Bad colors with Canon 5D Mark II (esp with dark photos)
It seems that Darktable renders the colors from 40D much better than it does with 5D Mark II, it is especially bad in case of photos shot in the dark (daylight photos more or less ok) - the contrast is much lower than of the camera JPEG and there the red component is much less saturated.
Can you help me with identifying the problem?
I guess it may be the problem with color matrices in src/common/colormatrices.c?
#2 Updated by Pascal de Bruijn over 8 years ago
I just checked the images and I don't immediately see a problem.
The enhanced color matrix renders reds redder than the standard color matrix, as expected.
But to compare we really need the same scene shot with both a 40D and a 5DmkII. And even then it's not that unusual to see some difference from camera to camera.
#3 Updated by angryziber - over 8 years ago
Actually, I have discovered the root cause: I have been using the Landscape Picture Style, which boosts some colors in the embedded jpeg, making it very different from what darktable produces.
Still, I have spent some time comparing the results of Canon's DPP with darktable and I see some discrepancies in rendering of yellows and blues: even beige becomes too yellow, while blues go more to the cyan side at some brightnesses. This is something that DPP doesn't do, and rawstudio was able to render yellows and blues more correctly as well with it's 5D mark II dcp profile.
Pascal, maybe you have an idea on how to extract color matrices from rawstudio dcp files to correct the values in src/common/colormatrices.c? How do you calculate those values while you calibrate cameras?
And another thing is the base curve, which seems to be way off in case of 5d (but not in case of 40d). I have tried to tune the basecurve and here is the closest I could get to what DPP/Camera itself produces:
it's too difficult to extract the values from op_params blob field in the db, so this is an image for comparison. Again, I tuned it manually to make pictures look more like DPP renders them. With the built-in 'canon eos like' base curve photos look darker and with lower contrast. Am I missing something?
#4 Updated by Pascal de Bruijn over 8 years ago
The Picture Style doesn't only affect the color rendition, but the basecurve as well.
Darktable's basecurve are aimed most reproducing the camera's default style usually called Standard or something (without any dynamic optimization like ALO, etc).
Our color rendition (enhanced color matrix) is aimed at rendered fairly accurate color (when viewed on calibrated displays). (This is basically more or less what's you get if you'd calibrate your own camera, we just did that for you).
I'm currently investigated how we can more accurately reproduce the color rendition of the camera's default style. But those won't replace the enhanced color matrix, but might become available in the future side by side. As not everybody actually wants to reproduce camera output.
So for non-standard picture styles, you'd need to modify the basecurve like you did, we have little interest in them for darktable itself.
With regards to DCP profiles, those aren't originated at rawstudio, they come from Adobe as far as I know. And I'm reluctant to make Darktable dependent on Adobe's stuff. As there is no open source software to produce DCP profiles. So this is of very little interest to me personally at the moment.
#6 Updated by Pascal de Bruijn over 8 years ago
I did quickly check the rawstudio stuff, but from what I can see, they actually don't ship the adobe dcp profiles, but dcp profiles created by them selves. But they seem to implement the dcraw matrix, which we ship as well as the "standard color matrix".
#7 Updated by angryziber - over 8 years ago
Here is another example, shot with the Standard picture style:
It still has the problem of low contrast and being too yellowish in darktable.
My base curve looks much closer to the original in-camera jpeg in terms of lightness and contrast. Darktable's curve is too dark. And rawstudio does a similar job to this concrete image, also dark and yellowish.
Can the yellow cast come from imprecise calibration using the color chart?
Btw, my screen is calibrated.
#8 Updated by angryziber - over 8 years ago
Here is the same scene shot with 40D:
- the default 'eos like' base curve is much more correct for this image, imho the image actually looks better than the in-camera jpeg, which unfortunately is not the case for 5D mkII.
It seems to me that having the same base curve for all Canon cameras doesn't really work nicely...
#9 Updated by Pascal de Bruijn over 8 years ago
I do see there is a difference, however, all similar cases until now could be explained by dynamic camera processing (which is impossible to reliably replicate).
If I'm not mistaken the 5DmkII has ALO, do you have that enabled by any chance? Try turning that off, then take some new shots, and compare again.
#10 Updated by angryziber - over 8 years ago
Great guess :-)
I had my ALO set to Standard (Canon's default), however making 3 test shots with ALO set to Disabled, Standard and Strong produced visually indistinguishable results for this particular scene.
In other words, shot with disabled ALO is still as dark in Darktable, like the previous shots I have posted - can't notice any difference. And also this yellow cast :-(
Pascal, can you give me an advice on how to get rid of the yellow cast in shadows at least, can it be done by experimenting with values in colormatrices.c? Is there any description on what do these values mean?
#11 Updated by Pascal de Bruijn over 8 years ago
Pretty much all reported basecurve defects have been due to dynamic processing in the camera.
These numbers in colormatrices.c are XYZ matrices, which are in essence color profiles (generated from user submitted data). These are represented by enhanced color matrix in the input profile plugin. These values are not to be experimented manually with. So no changes will be accepted (even if you stumble onto a would-be ""fix""). In essence the enhanced color matrix is what color would most likely look like if you would calibrate your own camera, we just did the work for you.
As I said before, I'm working on a solution to match camera color rendition more closely. However it's not done yet.
#12 Updated by angryziber - over 8 years ago
I can't find any more settings that I could disable in-camera. And, given the fact that this is a rather high-end camera, I guess it should have the possibility to disable all kinds of overprocessing. Anyway, have you seen any 5DmkII image that would be matched by the default eos-like curve in darktable? So far, all of my images, no matter what the lightning conditions were, are too dark in darktable with the default curve.
Another thing, is there a chance to recalibrate 5DmkII to get rid of the yellow cast? I guess I need a well-known color chart for that?
BTW, there's one more interesting read here: http://dcptool.sourceforge.net/Hue%20Twists.html
And, can you elaborate a bit more on what exactly are you working on? Maybe I can help in some way?
Sorry for too many questions at once :-)
#15 Updated by angryziber - over 8 years ago
Another guess on why the brightness is too low is that darktable uses the wrong default exposure, so instead of applying my custom basecurve I see that exposure compensation of +0.7-0.8 does the same trick!
Is there a map of camera-model/default-exposure somewhere in darktable?
#16 Updated by Pascal de Bruijn over 8 years ago
Huh, default exposure? We just process the RAW data as-is, without any exposure compensation. Do you have any other weird camera settings I should know about?
You can make your own exposure plugin preset with custom exposure compensation, where you can have it auto-applied to whatever make/model you want.
BTW, I still have to take a look at the samples.