open https://gitlab.synchro.net/main/sbbs/-/issues/541

Some characters in messages (such as the apostrophe used apparently by Apple Mail or other software on iPads) don't display properly in DDMsgReader (and show up as odd-looking characters). This may be related to MIME encoding? msglist.js seems to display them properly, so maybe that can be used as an example.

Some notes on dumping a message, which may help in debugging:

<DigitalMan> 'smbutil v /sbbs/data/mail' will dump all the headers
<DigitalMan> 'smbutil r /sbbs/data/mail' will read the message text <DigitalMan> if you know the message number, you can skip all the other messages easily, or just pipe output to a file and edit to suit
<nelgin> I did, thank you. uploaded bademail.txt and badheader.txt and I'm using SyncTerm, of course :)
<nelgin> The sender was using MuffinTerm on an iphone fwiw.
<DigitalMan> the terminals (on the sending or receiving side) shouldn't really matter
<DigitalMan> the sender was using Apple Mail (2.3445.9.7) according to those headers
<git_nelgin> GitLab issue from Nigel Reed
<git_nelgin> open https://gitlab.synchro.net/main/sbbs/-/issues/540 <git_nelgin> add ssh passphrase authentication
<nelgin> Ah right, he was using muffinterm to access the bbs but yeah, that was via gmail. My bad.
<DigitalMan> Can you try viewing that same message using 'smbutil -r r /sbbs/data/mail' ? Does it look the same or add a like of MIME noise to the output?
<DigitalMan> -r = display raw message body text (not MIME-decoded)

One of DM's recent commits:
git_rswindell> https://gitlab.synchro.net/main/sbbs/-/commit/1ca7f708fe87ea825cb0e5af
<git_rswindell> Modified src/smblib/smbtxt.c
<git_rswindell> Fix decoding of tab-indented 'charset' of multi-part MIME text parts
<git_rswindell> Apple Mail apparently uses tabs to indent the charset of nested MIME parts:

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/issues/541#note_3414

Does DDMsgReader translate UTF-8 encoded messages to CP437 or ASCII when appropriate for the viewing terminal?

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/issues/541#note_3420

I did add some code quite a while ago that used utf8_cp437() (from utf8_cp437.js) for that purpose, but it seems that wasn't enough. I hadn't gotten back to that until today.

I had a look at what msglist.js is doing, and it has the function msg_pmode(). It looks like if the message header's is_utf8 property is true, then it sets the P_UTF8 mode bit, and it looks like that's required to pass to console.print() in order for it to look valid on a terminal.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/issues/541#note_3421

Replace "a terminal" with "a non-UTF-8" terminal and yeah, that'd be true. Some terminals support UTF-8 just fine (and in those cases, don't support CP437).

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

close https://gitlab.synchro.net/main/sbbs/-/issues/541

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

reopen https://gitlab.synchro.net/main/sbbs/-/issues/541

Some characters in messages (such as the apostrophe used apparently by Apple Mail or other software on iPads) don't display properly in DDMsgReader (and show up as odd-looking characters). This may be related to MIME encoding? msglist.js seems to display them properly, so maybe that can be used as an example.

Some notes on dumping a message, which may help in debugging:
```
<DigitalMan> 'smbutil v /sbbs/data/mail' will dump all the headers
<DigitalMan> 'smbutil r /sbbs/data/mail' will read the message text <DigitalMan> if you know the message number, you can skip all the other messages easily, or just pipe output to a file and edit to suit
<nelgin> I did, thank you. uploaded bademail.txt and badheader.txt and I'm using SyncTerm, of course :)
<nelgin> The sender was using MuffinTerm on an iphone fwiw.
<DigitalMan> the terminals (on the sending or receiving side) shouldn't really matter
<DigitalMan> the sender was using Apple Mail (2.3445.9.7) according to those headers
<nelgin> Ah right, he was using muffinterm to access the bbs but yeah, that was via gmail. My bad.
<DigitalMan> Can you try viewing that same message using 'smbutil -r r /sbbs/data/mail' ? Does it look the same or add a like of MIME noise to the output?
<DigitalMan> -r = display raw message body text (not MIME-decoded)
```

One of DM's recent commits:
```
<git_rswindell> https://gitlab.synchro.net/main/sbbs/-/commit/1ca7f708fe87ea825cb0e5af
<git_rswindell> Modified src/smblib/smbtxt.c
<git_rswindell> Fix decoding of tab-indented 'charset' of multi-part MIME text parts
<git_rswindell> Apple Mail apparently uses tabs to indent the charset of nested MIME parts:
```

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/issues/541#note_3437

Issue not fixed by latest commits.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/issues/541#note_3552

Have you found anything new regarding this issue? I'm wondering if this can be closed.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net

https://gitlab.synchro.net/main/sbbs/-/issues/541#note_3553

Closing, as this appears to be fixed and there's no new information for this issue.

---
þ Synchronet þ Vertrauen þ Home of Synchronet þ [vert/cvs/bbs].synchro.net