fusion wrote to Gamgee <=-
On 31 Mar 2021, Gamgee said the following...
What he's describing is what happens with automated BBS ads and/or echo rules postings (just two examples). Same body, different MSGID, and
they won't get distributed into the echos because the hub will drop it
as a dupe. Those aren't malicious/spam messages, but they get dropped.
this, of course, nobody agrees on. i for one have removed just
about every area with a non stop barage of rules postings.. which
everyone ignores except the one stickler who wants to refer to
them. usually the otherwise completely dead area will spring back
to life the second you post though, because the one dude posting
the rules non stop doesn't agree with you and he's the only other
person there..
tbh i hate automated posts in general. every ham radio area on
every net i've been on is completely ruined with the same non
stop messages you get in your local club's mailing lists. and
when's the last time you met a ham that isn't on their local
club's mailing list?
Looks like your <SHIFT> key is broken, BTW. Do you actually prefer to not use proper capitalization?
It sounds like you're suggesting that *both* the text and the
MSGID should be compared and equal in order to be considered a duplicate - to which I would ask: why bother? if the MSGID is a duplicate, it's a duplicate. No need to compare the message body
in that case. But 2 messages with different MSGIDs but the same
body text is either a network screw-up or someone purposely or accidentally posting the same content repeatedly.
What he's describing is what happens with automated BBS ads and/or echo rules postings (just two examples). Same body, different MSGID, and
they won't get distributed into the echos because the hub will drop it
as a dupe. Those aren't malicious/spam messages, but they get dropped.
There's really
no way for software to tell the difference. But you can import
and forward those messages if you prefer to, you have that
option.
They realistically can't be forwarded, because the next system in the
link (probably a hub) will drop it as a dupe.
I *think* he's saying that if the MSGID's are different, the message
should not be considered a dupe, even if the body is identical to a
previous message. I agree with that thinking.
Re: sbbs as fidonet hub
By: Digital Man to Al on Tue Mar 30 2021 09:27 pm
It sounds like you're suggesting that *both* the text and the MSGID should be compared and equal in order to be considered a duplicate -
Yes, I think that would do what I am hoping to do.
to which I would ask: why bother?
These messages (rules posts or BBS ads) do have identical message bodies but they have been newly posted, these are new posts with a different MSGID. We still want to be vigilant about dupes, in fact today we must be vigilant about dupes.
Stepping away from the BBS side of this and looking at it from a mail flow point of view.. when linked to someones node, linked nodes have a reasonable expectation that they will receive all new messages in a given area that have been posted, and also that messages that enter the flow from their node will also be sent along to connected nodes.
Hence my position that the dupes should maybe only be
filtered out based on MSGID.
If the MSGID is duplicate, there's no point in comparing the message body text. You already know it's a dupe.
Re: sbbs as fidonet hub
By: Digital Man to Al on Wed Mar 31 2021 16:22:13
If the MSGID is duplicate, there's no point in comparing the message body text. You already know it's a dupe.
not always true...
Digital Man wrote to Gamgee <=-
There's really
no way for software to tell the difference. But you can import
and forward those messages if you prefer to, you have that
option.
They realistically can't be forwarded, because the next system in the
link (probably a hub) will drop it as a dupe.
I *think* he's saying that if the MSGID's are different, the message
should not be considered a dupe, even if the body is identical to a
previous message. I agree with that thinking.
Disable duplicate message checking and you will have that
behavior.
Digital Man wrote to Gamgee <=-
Hence my position that the dupes should maybe only be
filtered out based on MSGID.
And you have that option.
fusion wrote to Gamgee <=-
Looks like your <SHIFT> key is broken, BTW. Do you actually prefer to
not use proper capitalization?
ya
Yes, I think that would do what I am hoping to do.
If the MSGID is duplicate, there's no point in comparing the message body text. You already know it's a dupe.
I'm not clear what your definitino of "vigilant about dupes" means. If you only care about duplicate MSGIDs, then disable the duplicate message checking and that is what you will get - but *I* don't consider that configuration to be vigilant.
I hope that my uplink will deduplicate messages using effective measures. If he doesn't, then I will. Duplicates should be detected and removed as upstream as possible, in my opinion.
But if you want different behavior on your system, you have that option.
I feel like we're talking past each other and I'm not really sure how else to state it: if you want to allow multiple messages to be imported into your message bases and passed to downlinks with the identical message body text of a previously imported message, then disable the duplicate message checking in SCFG. This will have no impact on duplicate MSGID checking. There is no configuration setting to disable dulpicate MSGID checking.
I suppose it comes down to just what is a dupe. Is it a dupe if the msg body CRC is identical to a past post? That's a pretty good indication but I am suggesting we campare the MSGID of the two messages before we call it a dupe.
Currently SBBSecho is dropping new messages from the flow of traffic because it has incorrectly trapped them as dupes when they are in fact new posts.
not always true... just ask daryl stout of thunderbolt bbs about the
times i reported to him about his messages being detected as dupes
If the MSGID is duplicate, there's no point in comparing the message
body text. You already know it's a dupe.
not always true... just ask daryl stout of thunderbolt bbs about the times i reported to him about his messages being detected as dupes because the sofware he was using at the time was creating basically random MSGIDs and some were duplicates of others only a few months old... in some cases, it may have been attributed to reinstallations and/or the loss of a/the msgid file if there was such in use... in any case, i know that he can tell about
the problem... one of those packages was virtual advanced IIRC... there was
another one or two that he tried before setteling on synchronet...
I don't know how often it happens, because most probably go
undetected. I'm not reading my dupe area on a regular basis,
let alone scan it for false dupes... ;)
in any case, dupe checking in FTN is not done my /just/ detecting duplicate MSGIDs and rejecting the others... the header, including the time stamp, as well as the message body should be taken into
account...
it should also be said that CRC16/CRC32 on the message bodies is also
not sifficient... even with filtering out white space and the various EoLs... this because, and most programmers know this, there's a
limited supply of CRC values in the tables and it is all too easy to
find "hash clashes"... CRC16 has only 65536 values... CRC32 has only 4294967296 values... the "Birthday Problem" also comes into play...
these days, MD5 and SHA1 are also out due to defects in them...
SHA256 would be the first really useful algorithm or SHA512...
but the real key is to filter out the stuff that can change and hash
only that which won't...
anyway, i'm done with this topic in this area... the discussion really belongs elsewhere for those that are truely interested in implementing proper duplicate detection in FTNs...
in any case, dupe checking in FTN is not done my /just/ detecting duplicate MSGIDs and rejecting the others... the header, including the time stamp, as well as the message body should be taken into account...
it should also be
said that CRC16/CRC32 on the message bodies is also not sifficient... even with filtering out white space and the various EoLs... this because, and most programmers know this, there's a limited supply of CRC values in the tables and it is all too easy to find "hash clashes"... CRC16 has only 65536 values... CRC32 has only 4294967296 values... the "Birthday Problem" also comes into play...
these days, MD5 and SHA1 are also out due to defects in them...
SHA256 would
be the first really useful algorithm or SHA512... but the real key is to filter out the stuff that can change and hash only that which won't...
anyway, i'm done with this topic in this area... the discussion really belongs elsewhere for those that are truely interested in implementing proper duplicate detection in FTNs...
in any case, dupe checking in FTN is not done my /just/ detecting
duplicate MSGIDs and rejecting the others... the header,
including the time stamp, as well as the message body should be
taken into account...
Dupe checking in FTN is done via many methods - its implementation dependant. SBBSecho does not take the time stamp of the message
into account when considering if a message is a duplicate.
anyway, i'm done with this topic in this area... the discussion
really belongs elsewhere for those that are truely interested in
implementing proper duplicate detection in FTNs...
Okay, then I guess you won't see this reply. <shrug>
I don't want to defeat duplicate checking. If I do disable
duplicate message checking then I will have message checking
based solely on MSGID?
I don't want to defeat duplicate checking. If I do disable
duplicate message checking then I will have message checking
based solely on MSGID?
Have you considered appendingprepending the date-time to your autopost content?
Deepend wrote to Ragnarok <=-
Any detailed information on using SBBS as a hub for a FTN network? I
have been wondering the same thing but Either I'm not looking hard
enough or the FTN tools are still too new to have good documentation
for certain uses but Any information on the procedures to setup a SBBS
as a Hub for a FTN network would be appreciated.
Deepend wrote to Ragnarok <=-
De> Any detailed information on using SBBS as a hub for a FTN network? I
De> have been wondering the same thing but Either I'm not looking hard
De> enough or the FTN tools are still too new to have good documentation
De> for certain uses but Any information on the procedures to setup a SBBS
De> as a Hub for a FTN network would be appreciated.
I don't know what information's out there, some of the tools are a bit new.
It's possible, I'm Fidonet RC10 and a hub for Micronet. I'm using Synchronet as a mailer/areafix manager and hosting file echoes. I'm using Makenl to create the nodelist segments and it all mostly works OK.
Before this setup, I used Argus as a mailer, Allfix for file processing, and sbbsecho for message area requests and mail tossing.
The only thing missing at this point, and I need to look into it more, is passthrough requests for file areas. I don't carry a lot of them, and would like to have the ability to pass on file echoes without hosting them.
The wiki entries on binkit and tickit are pretty good. Nodelist support in Echocfg gets a little tricky if you're a hub for more than one network.
The #synchronet IRC channel on DOVEnet is a good place to ask for help when you're setting things up. I'm usually on tilde.club lately, I'm currently going through a #trivia phase... :)
... How did you find this place?
--- MultiMail/DOS v0.52
¨ Synchronet ¨ realitycheckBBS -- http://realitycheckBBS.org
Any detailed information on using SBBS as a hub for a FTN network? I have been
wondering the same thing but Either I'm not looking hard enough or the FTN tools are still too new to have good documentation for certain uses but Any information on the procedures to setup a SBBS as a Hub for a FTN network would
be appreciated.
Hey underminer. Not sure what my exact plan is yet.. but want to set something
up to test and learn so I actually can do something when its needed. Pretty sure I've actually sent you netmail or something within the last month.. or atleast I thought I did.. maybe it didn't go through for some reason.
Deepend wrote to poindexter FORTRAN <=-
If you could possibly share some more detail on how you have things
setup it'd be very helpful. More detail the better if you don't mind
:)
| Sysop: | Chris Crash |
|---|---|
| Location: | Huntington Beach, CA. |
| Users: | 578 |
| Nodes: | 8 (0 / 8) |
| Uptime: | 29:35:17 |
| Calls: | 10,736 |
| Calls today: | 1 |
| Files: | 5 |
| Messages: | 443,197 |
| Posted today: | 1 |