• httpSess coredump just appeared

    From Nigel Reed@VERT to GitLab issue in main/sbbs on Thu Jan 19 22:56:38 2023
    open https://gitlab.synchro.net/main/sbbs/-/issues/495

    <code>
    $ gdb /sbbs/exec/sbbs '/tmp/core.sbbs!httpSess.3560723'

    [Thread debugging using libthread_db enabled]
    Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
    Core was generated by `/sbbs/exec/sbbs d'.
    Program terminated with signal SIGSEGV, Segmentation fault.
    #0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
    120 ../sysdeps/x86_64/multiarch/../strlen.S: No such file or directory. [Current thread is 1 (Thread 0x7f7ccadfa700 (LWP 3676796))]


    (gdb) bt
    #0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
    #1 0x00007f7d8f1fdd15 in __vfprintf_internal (s=s@entry=0x7f7ccadf3e80, format=format@entry=0x7f7d8fb9a5f1 "%ld\t%s\t%s\t%s\t%s\t%u\t%lu",
    ap=ap@entry=0x7f7ccadf3ff0, mode_flags=mode_flags@entry=0) at vfprintf-internal.c:1688
    #2 0x00007f7d8f210bca in __vasprintf_internal (result_ptr=0x7f7ccadf3fe0, format=0x7f7d8fb9a5f1 "%ld\t%s\t%s\t%s\t%s\t%u\t%lu", args=0x7f7ccadf3ff0,
    mode_flags=0) at vasprintf.c:57
    #3 0x00007f7d8fb6c993 in strListAppendFormat (list=0x7f7ccadf4110, format=0x7f7d8fb9a5f1 "%ld\t%s\t%s\t%s\t%s\t%u\t%lu") at str_list.c:321
    #4 0x00007f7d8f6a66f5 in mqtt_client_on (mqtt=0x7f7d8f43b440 <mqtt>, on=0, sock=141, client=0x0, update=0) at mqtt.c:628
    #5 0x00007f7d8f40a356 in client_off (sock=141) at websrvr.c:772
    #6 0x00007f7d8f4203be in http_session_thread (arg=0x0) at websrvr.c:6776
    #7 0x00007f7d8f37f609 in start_thread (arg=<optimized out>) at pthread_create.c:477
    #8 0x00007f7d8f2a4133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
    (gdb)
    </code>

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net
  • From Rob Swindell@VERT to GitLab note in main/sbbs on Fri Jan 20 07:29:23 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/495#note_3082

    Interesting.. this looks like the same crash that took down vert.synchro.net (on Windows) a couple of weeks ago. Good to know it's not a Windows-only issue!

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net
  • From Nigel Reed@VERT to GitLab note in main/sbbs on Fri Jan 20 08:23:37 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/495#note_3083

    Of course, I just realized I recompiled sbbs on 17th Jan and the crash was on 16th so the core and binary files are not going to line up I expect so not sure if that dump is completely useful or not.

    I have another core from Jan 13th and then one from core.sbbs!termNode.3347010 on 15th. Let me know if you want a bt from those as new issues or I can add them here.

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net
  • From Rob Swindell@VERT to GitLab note in main/sbbs on Fri Jan 20 20:49:36 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/495#note_3085

    BT's from those other core dumps could be still useful.

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net
  • From Rob Swindell@VERT to GitLab note in main/sbbs on Sat Jan 21 11:05:23 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/495#note_3091

    So here's the BT from a crash on Windows today (using code from a week ago, so mqtt.c line numbers don't match current SHA), but it's clearly the same crash in strlen() by way of vasprintf():

    ```
    sbbs.dll!common_strnlen_c<unsigned char>(const unsigned char * const string, const unsigned int maximum_count) Line 36 C++
    sbbs.dll!common_strnlen_simd<0,unsigned char>(const unsigned char * const string, const unsigned int maximum_count) Line 94 C++
    sbbs.dll!common_strnlen<unsigned char>(const unsigned char * const string, const unsigned int maximum_count) Line 153 C++
    sbbs.dll!strnlen(const char * string, unsigned int maximum_count) Line 165 C++
    sbbs.dll!__crt_stdio_output::output_processor<char,__crt_stdio_output::string_output_adapter<char>,__crt_stdio_output::standard_base<char,__crt_stdio_output::string_output_adapter<char>>>::type_case_s_compute_narrow_string_length(const int maximum_length, char __formal) Line 2268 C++
    sbbs.dll!__crt_stdio_output::output_processor<char,__crt_stdio_output::string_output_adapter<char>,__crt_stdio_output::standard_base<char,__crt_stdio_output::string_output_adapter<char>>>::type_case_s() Line 2255 C++
    sbbs.dll!__crt_stdio_output::output_processor<char,__crt_stdio_output::string_output_adapter<char>,__crt_stdio_output::standard_base<char,__crt_stdio_output::string_output_adapter<char>>>::state_case_type() Line 1999 C++
    sbbs.dll!__crt_stdio_output::output_processor<char,__crt_stdio_output::string_output_adapter<char>,__crt_stdio_output::standard_base<char,__crt_stdio_output::string_output_adapter<char>>>::process() Line 1644 C++
    sbbs.dll!common_vsprintf<__crt_stdio_output::standard_base,char>(const unsigned __int64 options, char * const buffer, const unsigned int buffer_count, const char * const format, __crt_locale_pointers * const locale, char * const arglist) Line 163 C++
    sbbs.dll!__stdio_common_vsprintf(unsigned __int64 options, char * buffer, unsigned int buffer_count, const char * format, __crt_locale_pointers * locale, char * arglist) Line 235 C++
    [External Code]
    sbbs.dll!vasprintf(char * * strptr, const char * format, char * va) Line 60 C
    sbbs.dll!strListAppendFormat(char * * * list, const char * format, ...) Line 321 C
    sbbs.dll!mqtt_client_on(mqtt * mqtt, int on, int sock, client_t * client, int update) Line 614 C
    websrvr.dll!client_off(unsigned int sock) Line 772 C
    websrvr.dll!http_session_thread(void * arg) Line 6775 C
    websrvr.dll!invoke_thread_procedure(void(*)(void *) procedure, void * const context) Line 82 C++
    websrvr.dll!thread_start<void (__cdecl*)(void *)>(void * const parameter) Line 115 C++
    [External Code]

    mqtt->client_list = {first=0x0760c020 {data=0x06cb0710 next=0x077ca988 {data=0x06cd4f48 next=0x076666c0 {data=0x06cd49f0 ...} ...} ...} ...}

    client_list.mutex = {DebugInfo=0x00782620 {Type=0 CreatorBackTraceIndex=0 CriticalSection=websrvr.dll!0x01a945dc {DebugInfo=...} ...} ...}

    client_list.count = 9
    client_list.sem = 0x00000000


    list = 0x131e8af0 {0x07bf2cc0 "2152\tHTTP\tGuest\t98.7.221.143\t<no name>\t56584\t1674288933"}
    ```
    I don't see anything obviously wrong and this problem only happens like once in every millions calls to mqtt_client_on(), so I'm probably just going to punt and use an alternative approach to appending a new formatted string to the (temporary) client_list.

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net
  • From Rob Swindell@VERT to GitLab note in main/sbbs on Mon Jan 23 12:11:51 2023
    https://gitlab.synchro.net/main/sbbs/-/issues/495#note_3132

    This crash is still occurring with the latest changes and I think I know why: the client_list maintained in the mqtt object has pointers to the username for each connected client and in the web server, that points to char buffer in an ephemeral http_session_t instance which is likely being freed already at this point in the code in same race-conditions. Changing the client_t definition to use a char array for the username rather than a pointer should fix this.

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net
  • From Rob Swindell@VERT to GitLab issue in main/sbbs on Mon Jan 23 17:05:16 2023
    close https://gitlab.synchro.net/main/sbbs/-/issues/495

    ---
    þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net