Project

General

Profile

Bug #11786

Heap buffer overflow in amaze demosaic code

Added by Andreas Schneider 26 days ago. Updated 26 days ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Darkroom
Target version:
Start date:
10/25/2017
Due date:
% Done:

0%

Affected Version:
git master branch
System:
openSUSE
bitness:
64-bit
hardware architecture:
amd64/x86

Description

Hello,

there is a buffer overflow in the amaze demosaic code. It is detected if you build darktable with AddressSanetizer.

See frame #7

#0  0x00007ffff5eaa0d0 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
        set = 
            {__val = {0, 140737336338546, 140734962386352, 0, 140737354095496, 140734962386336, 140734962386320, 5322390845, 140737354094640, 140737335516350, 4294967295, 206158430216, 140737318989696, 140737354098416, 4409925841731009853, 5641132720752050493}}
        pid = <optimized out>
        tid = <optimized out>
#1  0x00007ffff5eab6b1 in __GI_abort () at abort.c:79
        save_stage = 1
        act = 
          {__sigaction_handler = {sa_handler = 0xffffffff, sa_sigaction = 0xffffffff}, sa_mask = {__val = {4294967295, 140734962387063, 140734962390240, 140734962390232, 140735894407436, 0, 140737336447953, 0, 206158430216, 140737343479808, 1, 140737343479968, 0, 4294967291, 58533, 0}}, sa_flags = -1593947892, sa_restorer = 0x7ffff7dd6838}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00007ffff6f1ff8b in  () at /usr/lib64/libasan.so.4
#3  0x00007ffff6f27728 in  () at /usr/lib64/libasan.so.4
#4  0x00007ffff6f0a5fd in  () at /usr/lib64/libasan.so.4
#5  0x00007ffff6f0b5f5 in __asan_report_load_n () at /usr/lib64/libasan.so.4
#6  0x00007fffb6744e17 in _mm_loadu_ps (__P=0x7fffa0fe490c) at /usr/lib64/gcc/x86_64-suse-linux/7/include/xmmintrin.h:934
        indx1 = <optimized out>
        cc = <optimized out>
        row = 899
        rr = 147
        right = 1345
        rrmax = <optimized out>
        nystartcol = <optimized out>
        selmask = <optimized out>
        cc1 = 81
        offset = <optimized out>
        sgn3v = <optimized out>
        nyendrow = <optimized out>
        rrmin = <optimized out>
        sgnv = <optimized out>
        bottom = 912
        nystartrow = <optimized out>
        left = 1264
        rr1 = 160
        ccmax = 65
        top = 752
        nyendcol = <optimized out>
        ccmin = 0
        doNyquist = <optimized out>
        buffer = <optimized out>
        hcdalt = <optimized out>
        rbm = <optimized out>
        rgbgreen = <optimized out>
        dirwts0 = <optimized out>
        vcd = <optimized out>
        rbint = <optimized out>
        pmwt = <optimized out>
        cldf = 2
        dirwts1 = <optimized out>
        hvwt = <optimized out>
        Dgrb = <optimized out>
        rbp = <optimized out>
        nyquist2 = <optimized out>
        delhvsqsum = <optimized out>
        hcd = <optimized out>
        cddiffsq = <optimized out>
        dginth = <optimized out>
        Dgrbsq1m = <optimized out>
        cfa = <optimized out>
        nyqutest = <optimized out>
        delm = <optimized out>
        Dgrb2 = <optimized out>
        dgintv = <optimized out>
        Dgrbsq1p = <optimized out>
        delp = <optimized out>
        data = <optimized out>
        vcdalt = <optimized out>
        nyquist = <optimized out>
        winx = <optimized out>
        winy = <optimized out>
        winw = <optimized out>
        winh = <optimized out>
        width = <optimized out>
        height = <optimized out>
        clip_pt = <optimized out>
        clip_pt8 = <optimized out>
        ts = <optimized out>
        tsh = <optimized out>
        ex = <optimized out>
        ey = <optimized out>
        v1 = <optimized out>
        v2 = <optimized out>
        v3 = <optimized out>
        p1 = <optimized out>
        p2 = <optimized out>
        p3 = <optimized out>
        m1 = <optimized out>
        m2 = <optimized out>
        m3 = <optimized out>
        eps = <optimized out>
        epssq = <optimized out>
        arthresh = <optimized out>
        nyqthresh = <optimized out>
#7  0x00007fffb6744e17 in amaze_demosaic_RT(dt_iop_module_t*, dt_dev_pixelpipe_iop_t*, float const*, float*, dt_iop_roi_t const*, dt_iop_roi_t const*, int) (self=<optimized out>, piece=<optimized out>, in=<optimized out>, out=<optimized out>, roi_in=<optimized out>, roi_out=<optimized out>, filters=<optimized out>)
    at /home/asn/workspace/projects/darktable/src/iop/amaze_demosaic_RT.cc:522
        indx1 = <optimized out>
        cc = <optimized out>
        row = 899
        rr = 147
        right = 1345
        rrmax = <optimized out>
        nystartcol = <optimized out>
        selmask = <optimized out>
        cc1 = 81
        offset = <optimized out>
        sgn3v = <optimized out>
        nyendrow = <optimized out>
        rrmin = <optimized out>
        sgnv = <optimized out>
        bottom = 912
        nystartrow = <optimized out>
        left = 1264
        rr1 = 160
        ccmax = 65
        top = 752
        nyendcol = <optimized out>
        ccmin = 0
        doNyquist = <optimized out>
        buffer = <optimized out>
        hcdalt = <optimized out>
        rbm = <optimized out>
        rgbgreen = <optimized out>
        dirwts0 = <optimized out>
        vcd = <optimized out>
        rbint = <optimized out>
        pmwt = <optimized out>
        cldf = 2
        dirwts1 = <optimized out>
        hvwt = <optimized out>
        Dgrb = <optimized out>
        rbp = <optimized out>
        nyquist2 = <optimized out>
        delhvsqsum = <optimized out>
        hcd = <optimized out>
        cddiffsq = <optimized out>
        dginth = <optimized out>
        Dgrbsq1m = <optimized out>
        cfa = <optimized out>
        nyqutest = <optimized out>
        delm = <optimized out>
        Dgrb2 = <optimized out>
        dgintv = <optimized out>
        Dgrbsq1p = <optimized out>
        delp = <optimized out>
        data = <optimized out>
        vcdalt = <optimized out>
        nyquist = <optimized out>
        winx = <optimized out>
        winy = <optimized out>
        winw = <optimized out>
        winh = <optimized out>
        width = <optimized out>
        height = <optimized out>
        clip_pt = <optimized out>
        clip_pt8 = <optimized out>
        ts = <optimized out>
        tsh = <optimized out>
        ex = <optimized out>
        ey = <optimized out>
        v1 = <optimized out>
        v2 = <optimized out>
        v3 = <optimized out>
        p1 = <optimized out>
        p2 = <optimized out>
        p3 = <optimized out>
        m1 = <optimized out>
        m2 = <optimized out>
        m3 = <optimized out>
        eps = <optimized out>
        epssq = <optimized out>
        arthresh = <optimized out>
        nyqthresh = <optimized out>
#8  0x00007fffef2efe5e in  () at /usr/lib64/libgomp.so.1
#9  0x00007ffff6235558 in start_thread (arg=0x7fff6971c700) at pthread_create.c:465
        pd = 0x7fff6971c700
        now = <optimized out>
        unwind_buf = 
              {cancel_jmp_buf = {{jmp_buf = {140734962452224, -5532860128589514449, 140736840172766, 140736840172767, 140734962452224, 0, 5533103042173335855, 5532875833888209199}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
        pagesize_m1 = <optimized out>
        sp = <optimized out>
        freesize = <optimized out>
        __PRETTY_FUNCTION__ = "start_thread" 
#10 0x00007ffff5f6c45f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

AddressSanetizer reports:

==6885==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f05d2c8c90c at pc 0x7f05ed4afe17 bp 0x7f0607b28a30 sp 0x7f0607b28a28
READ of size 16 at 0x7f05d2c8c90c thread T14 (worker res 1)
==6885==AddressSanitizer: while reporting a bug found another one. Ignoring.
    #0 0x7f05ed4afe16 in amaze_demosaic_RT._omp_fn.0 /usr/lib64/gcc/x86_64-suse-linux/7/include/xmmintrin.h:934
    #1 0x7f061d861e9e in GOMP_parallel (/usr/lib64/libgomp.so.1+0xee9e)
    #2 0x7f05ed4b050c in amaze_demosaic_RT /home/asn/workspace/projects/darktable/src/iop/amaze_demosaic_RT.cc:403
    #3 0x7f05ed49bad7 in process /home/asn/workspace/projects/darktable/src/iop/demosaic.c:1757
    #4 0x7f0624c6b620 in default_process /home/asn/workspace/projects/darktable/src/develop/imageop.c:186
    #5 0x7f0624c9d1dc in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:1429
    #6 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #7 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #8 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #9 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #10 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #11 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #12 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #13 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #14 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #15 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #16 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #17 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #18 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #19 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #20 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #21 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #22 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #23 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #24 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #25 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #26 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #27 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #28 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #29 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #30 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #31 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #32 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #33 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #34 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #35 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #36 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #37 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #38 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #39 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #40 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #41 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #42 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #43 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #44 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #45 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #46 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #47 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #48 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #49 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #50 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #51 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #52 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #53 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #54 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #55 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #56 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #57 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #58 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #59 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #60 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #61 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #62 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #63 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #64 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #65 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #66 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #67 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #68 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #69 0x7f0624ca6e18 in dt_dev_pixelpipe_process_rec_and_backcopy /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:2188
    #70 0x7f0624ca6e18 in dt_dev_pixelpipe_process /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:2274
    #71 0x7f0624c651ce in dt_dev_process_preview_job /home/asn/workspace/projects/darktable/src/develop/develop.c:271
    #72 0x7f0624c50fdb in dt_dev_process_preview_job_run /home/asn/workspace/projects/darktable/src/control/jobs/develop_jobs.c:25
    #73 0x7f0624c3f534 in dt_control_run_job_res /home/asn/workspace/projects/darktable/src/control/jobs.c:219
    #74 0x7f0624c3f534 in dt_control_work_res /home/asn/workspace/projects/darktable/src/control/jobs.c:523
    #75 0x7f06247b0557 in start_thread (/lib64/libpthread.so.0+0x7557)
    #76 0x7f06244e745e in clone (/lib64/libc.so.6+0xf845e)

0x7f05d2c8c910 is located 0 bytes to the right of 4784400-byte region [0x7f05d27fc800,0x7f05d2c8c910)
allocated by thread T14 (worker res 1) here:
    #0 0x7f062547c088 in __interceptor_posix_memalign (/usr/lib64/libasan.so.4+0xdd088)
    #1 0x7f0624b1bcc1 in dt_alloc_align /home/asn/workspace/projects/darktable/src/common/darktable.c:1118
    #2 0x7f0624c961bb in dt_dev_pixelpipe_cache_get_weighted /home/asn/workspace/projects/darktable/src/develop/pixelpipe_cache.c:173
    #3 0x7f0624c962dc in dt_dev_pixelpipe_cache_get /home/asn/workspace/projects/darktable/src/develop/pixelpipe_cache.c:132
    #4 0x7f0624c98899 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:727
    #5 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #6 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #7 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #8 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #9 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #10 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #11 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #12 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #13 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #14 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #15 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #16 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #17 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #18 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #19 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #20 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #21 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #22 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #23 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #24 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #25 0x7f0624c97702 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:702
    #26 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #27 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #28 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583
    #29 0x7f0624c973a9 in dt_dev_pixelpipe_process_rec /home/asn/workspace/projects/darktable/src/develop/pixelpipe_hb.c:583

Thread T14 (worker res 1) created by T0 here:
    #0 0x7f06253d8cc0 in pthread_create (/usr/lib64/libasan.so.4+0x39cc0)
    #1 0x7f0624b341cf in dt_pthread_create /home/asn/workspace/projects/darktable/src/common/dtpthread.c:69
    #2 0x7f0624c424f2 in dt_control_jobs_init /home/asn/workspace/projects/darktable/src/control/jobs.c:638
    #3 0x7f0624c35ec9 in dt_control_init /home/asn/workspace/projects/darktable/src/control/control.c:70
    #4 0x7f0624b207d7 in dt_init /home/asn/workspace/projects/darktable/src/common/darktable.c:830
    #5 0x4009ef in main /home/asn/workspace/projects/darktable/src/main.c:68
    #6 0x7f062440ff49 in __libc_start_main (/lib64/libc.so.6+0x20f49)

SUMMARY: AddressSanitizer: heap-buffer-overflow /usr/lib64/gcc/x86_64-suse-linux/7/include/xmmintrin.h:934 in amaze_demosaic_RT._omp_fn.0
Shadow bytes around the buggy address:
  0x0fe13a5898d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe13a5898e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe13a5898f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe13a589900: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe13a589910: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0fe13a589920: 00[00]fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fe13a589930: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fe13a589940: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fe13a589950: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fe13a589960: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fe13a589970: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==6885==ABORTING

History

#1 Updated by Roman Lebedev 26 days ago

some irc log for context:

[22:00:24] <LebedevRI> qogniw / DrSlony may be interested
[22:00:37] <qogniw> ?
[22:01:08] <LebedevRI> qogniw: there appears to be some overflow in amaze_demosaic_RT.cc#L522
[22:01:23] <qogniw> ah, ok, looking
[22:01:27] <LebedevRI> https://cpaste.org/?48eed6e416b4e284#MjHntS2mrd1aYGlBf1sHSw6rXU5wRRLIwg0iNjJmwHo=
[22:01:40] <gladiac> build with -O0 running
[22:03:26] <LebedevRI> that file looks so funny :) i bet if you fuzz it, you will find some more interesting stuff
[22:06:57] <qogniw> LebedevRI: in rt the input for amaze has a 4 pixel border (for all borders), which maybe the reason it doesn't overflow in rt
[22:07:19] <qogniw> still looking
[22:10:45] <LebedevRI> sounds like a missing prerequisite check
[22:11:00] <gladiac> https://hastebin.com/vagebemuqa.go
[22:12:09] <gladiac> damn
[22:12:39] <gladiac> that was too late
[22:12:45] <gladiac> that's the wrong trace
[22:13:07] <gladiac> that console output with debug is a bit much and slows everything even more down
[22:13:59] <qogniw> LebedevRI: gladiac: As the rt amaze code in rt and dt is not the same, I will won't look further at the code except you can reproduce the bug also in rt
[22:22:59] <LebedevRI> different? you mean our copy is not up-to-date?
[22:24:51] <qogniw> LebedevRI: no, but it's still not the same code
[22:29:08] <LebedevRI> you lost me there, there were no functional changes since last sync
[22:34:20] <qogniw> LebedevRI: no functional changes, true. But look for example at https://github.com/darktable-org/darktable/blob/master/src/iop/amaze_demosaic_RT.cc#L522 and https://github.com/Beep6581/RawTherapee/blob/dev/rtengine/amaze_demosaic_RT.cc#L229
[22:35:37] <qogniw> our rawdata array may allow overflowing (because it doesn't overflow as it has a 4 pix border) while I don't know anything about your `in` buffer
[22:38:14] <LebedevRI> so in other words if we pass width=2 height=2 to amaze demosaic, it just assumes that you lied, and uses width=6 height=6, correct?
[22:38:29] <LebedevRI> good to know, thank you
[22:39:02] <qogniw> no
[22:39:25] <qogniw> it's just too long ago that I modified the code to tell exactly about
[22:39:37] <qogniw> atm
[22:43:23] <qogniw> LebedevRI: the main difference between rt and dt concerning amaze is (imho) that rt always demosaics the full raw file, while dt demosaics a roi. Am I right?
[22:45:10] <LebedevRI> i don't know what are the differences between rt and dt concerning amaze, but i can confirm that  dt demosaics a roi. +no division by 65535
[23:00:26] <gladiac> row=4015, width=5935, cc=156, left=5360, tidx=23834541
[23:01:39] <gladiac> tidx is the index it accesses in the pointer/array
[23:04:59] <gladiac> there is no size passed, so you can check if you access valid memory
[23:06:37] <LebedevRI> gladiac: it sounded like that is all correct and we are simply misusing the code because we were not aware of some precondition

Also available in: Atom PDF