> This experience fits a pattern I keep running into with European business-facing APIs and services
honestly, it's a pattern I have been running into with many start ups, fintech, banks and some other places _no matter where_.
It also often makes sense. For many large orgs (e.g. Banks) this APIs are often a side business, sometimes one they don't want but have to have for compliance (or market pressure). But for strip it's their live blood.
And many startups (like actual start ups, not 200 man companies running by investor money) often simply don't have the resources to prioritize a "very nice to use API".
Lastly API design which is both nice to use and stable for existing integrations is surprisingly hard to get right (if you don't have some senior engine prioritizing it very highly and forcing it being kept prioritized; Or a surprisingly "clean"/"clear" use case.).
Message-Id being basically required for _automated_ mails (very similar mail send to a lot of people) requiring Message-Id is a de-facto industry standard. Sure some providers don't care and some might make it just more likely that your mail ends up in spam. But this could have happened with pretty much any mail provider widely used by companies.
much more funny (/s) is if you start out in a startup and still use the default template for password reset/on-baording links of a widely used system (e.g. Keycloak) and it turns out multiple larger but "cheap" phishing campaigns did the same and now MS/Google and other suspect you are running a phishing operation
or when you use a local data center and can't send mails to MS/Outlook anymore because it turns out someone did some legal questionable things on them and MS wanted the personal information to be handed over _without court order or any ongoing legal proceeding against this people_ and they didn't hand it over (partially because they legally aren't allowed to...) and MS decided to retaliate by permanently blacklist the IPv4 range of that data center(s) which just happen to locally compete with Azure while self hosted mails competes with Outlooks...
it differs by country (EU gives countries some similar overlapping laws, but they still are distinct countries with a lot of different practices and different laws in many many ways.)
and it differs by severity
E.g. deutsche bank has become notorious to appear in most large scale bank scandals _in the US_ as a "German Representative". They also repeatedly get into trouble for relations to money laundering operations and other issues. At the same time they are the Bank which has to give you a (expensive but not completely absurd priced) bank account no matter your credit score, criminal history etc. and is the bank you mostly likely have a escrow account parking money if the person who gets it "if you mess up" is the state/local government (like e.g. in some long term visa context).
In general banks move very slow, including with software protocol updates/changes. And that isn't just the case in the EU AFIK. Actually some of the most crazy cases of "old software" I have seen in "the west" where as far as I remember in the UK and the issue was that it was deeply fundamentally impossible for them handle a lot of non English names correctly, like in 2017~2020 or so. Through at least they still could use the bank, just with a crippled/en-ified name. Funnily the same but worse (as in not working at all) I have heard about Japan banks, where they had(have?) really big issues if you have a middle name.).
message-id is (or at least was ~5-10 years ago) only required for automated mails, i.e. if you send a mail which looks mostly the same to a lot of users
so if you (ab-)use protocols for mail clients to send automated mails they might very well not add Message-Id. In general many mail providers have both a way to send a mail where "they do some clever parts" (like adding Message-Id) and interfaces where they mostly just send what you give them. It's possible that they migrated from one solution which did automatically add Message-Id to one using interfaces where it doesn't happen without realizing that there is a mismatch in this solutions do implicitly add.
Message-Id being required for automated mails is a de-facto industry standard
while the consequences differ between mail provider, it missing will also make it much more likely for mail to be reject or put into the spam folder
It's also well known. Pretty much viva engineers fucked up doing proper research.
Now to be fair:
- it sucks that you can't just implement the RFC(s)
- the standards suck, docent of different RFCs overlapping and replacing each other and referencing often older versions of other RFCs, with docents of ways to do the same things of which only some can be used reliable in practice and a common gaps in the standards about edge cases or about the "higher level semantics" of constructs.
- so overall mail seems very simple at first but if you want to automated send mails reliable internationally it's a total pain and Message-Id is just the head of the iceberg.
something the RFC standard recommend but doesn't require
but it being required is a de-facto industry standard for sending automated mails
and is clearly documented by support sides of large mail providers (like Google)
the mail standards only defines what parts you can put together, but widely fail to define how this parts can be interpreted, what are sensible combinations, etc.
and they don't cover spam/suspicious mail detection at all
so you can't just go by RFC, you need to read up on what all larger mail providers have as additional requirements (which mostly are the same, and Message-Id being the most common dominator) and then hope that another provider you didn't read up one doesn't have some other surprising rule (which doesn't tent to be the case if you don't do anything surprising, but it sucks anyway).
yeah, but that RFC isn't the only relevant document
Mail RFCs do not cover at all spam detection and malicious mail rejection, but it's a thing every large mail provider has and you really have to care about when producing automated mails all looking similar. And large mail providers like google tend to document what "base line" of additional requirements they have for accepting (automated mail). Having a Message-Id is in there, and in pretty much any larger mail providers documentation about that topic. Tbh. I have worked with mail before a bunch of years ago and the need for Message-Id was back then really no hidden gotcha but pretty well known.
and the design space mail provides is larger then any client could reasonable support (like it's really a huge mess covering docent of standards which allow all kind of nonsense and hypothetical use cases practical unsupported), so you anyway have to look at "what everyone does" and only then make sure it's also RFC compatible, instead of starting with the RFC. That was a painful lessen to learn.
In addition there are some de-facto standards not pinned down in any RFC, like e.g.:
- Message-Id being required for any automated mails by many mail providers (through how bad the consequences are if you don't have it diverges largely).
- You can't punycode encode the local part of an email address (it would be a different email), and there is no standard way (as far as I remember) to convert non us-ascii local parts to us-ascii. This is based on the fact that iff your mail server allows you to have non us-ascii local parts it should also support "internationalized mail" (SMTPUTF8 and co.). But it's a semi industry standard to give the user with an unicode local part also the mail with the punycode encoding of the local part so it often "just works" and dev are frequently surprised when it fails to work...
- You can have quoted text in local part, like whitespaces. But most of the industry decided to not give users such mail addresses so you can see it as soft deprecated and using it is asking for trouble.
- Attachements. The MIME encoding allows a lot of different ways to put mails together and doesn't force a specific semantic interpretation by mail clients. As such you if you naively use it you might run into surprises how/if your attachments or embedding(s) are displayed. Through today embedded resources often are either not done or uses data URLs. Again which ways work well and which don't is somewhat an industry standard and not in any RFC.
- A lot of different ways to encode Unicode to us-ascii. If you produce any mails you probably should by default use the latest revision (where encoding is often just not needed as things are utf8), but might need a fallback with it often being fully unclear if . But if you are a client you probably have to support older versions. And in some parts of the world/business segments usage of very very old mail servers is a thing which is a major pain if you run into it.
so quoting that something isn't strictly required by mail RFCs is kinda pointless as even many things explicitly allowed won't work well in practice
As a rule of thump: If you can afford it test you system will all widely used mail providers as if you where an external customers. And redo the tests yearly.
oh and as a bonus, if you mails looks too similar to known phishing mails it will also just disappear. That seems irrelevant, but e.g. if you use Keycloak with the default mail templates for password reset and co. there is a high chance of your mails ending up in spam or not even being delivered as scammers have used Keycloak for their means, too. And that isn't just a case for Keycloak but any "decently widely used open source software producing mails and having default templates". So you pretty much always need to change the default templates (you will do so anyway for branding, but skipping it in the earliest stages of a startup where branding might still be in flux isn't that uncommon either).
It is de-facto required and has been for many years.
Should in most RFCs also mean "do it as long as you don't have a very good technical reason not to do it". Like it's most times a "weak must". And in that case the only reason it isn't must is for backward compatibility with older mail system not used for sending automated mails.
And it is documented if you read any larger mail providers docs about "what to do that my automated mails don't get misclassified as spam". And spam rejection is a whole additional non-standardized layer on top of RFCs anyone working with mail should be aware of. In any decades old non centralized communication system without ever green standards having other "industry standard/de-factor" but not "standardized" requirements is pretty normal btw.
1. SHOULD means, do it if you can/you have to have a really good reason if you don't do it. The only reason it's SHOULD and not MUST is backward compatibility. Mostly in context of "personal send mails", i.e. not automated mails. (For automated mail sending the expectations of you running somewhat up to date software is much higher).
2. You can't really implement mail stuff just based on RFCs:
- There docent overlapping RFCs which can sometimes influence each other and many of them obsolete older versions why others still relevant RFCs reference this older versions. This makes it hard to even know what actually is required/recommendation.
- Then you have a lot of "irrelevant" parts, which where standardized but are hardly supported/if at all. You probably should somewhat support them as recipient but should never produce them as sender today (mostly stuff related to pro-"everything is utf8" days). Like in general the ideas of "how mail should probably work" in old RFCs and "how it is done IRL today" are in some aspects _very_ far away.
- Lastly RFCs are not sufficient by themself. They don't cover large parts of the system for "spam detection/suspicious mail rejection". So it's a must have to go to the support pages of all large mail providers and read through what they expect of mails. And "automated mails need a message id" is a pretty common requirement. In addition you have to e.g. make sure the domain you use isn't black listed (e.g. due to behavior of a previous user), and that your servers IP addresses aren't black listed (they never should be black listed long term, but happens anyway, and e.g. MS has based on very questionable excuses "conveniently" black listed smaller local data center competition while also being one of the most widely used providers for commercial mail in that area).
> but then the obvious question is, for areas where it almost never gets above freezing, why doesn't the snow get infinitely thick?
this is how glaciers are created
snow getting stuck up, not melting, compressing by weight into much much smaller ice and then more stacking up. And during the last ice age this repeating for a very long time (because snow is mostly air, so the amount of ice you get from it is very little).
The reasons why this isn't too big of an issue on the north/south pool, Antarctica etc. is because this places are also very dry/don't have a lot of snow fall.
To have snowfall you need water in the air. Which mostly comes from heat evaporating water. This doesn't happen in non stop freezing cold places.
So the wind needs to carry the wet air over.
But there is a gradient between hot wet air places and very cold places. So a lot of water rains or snows off before reaching the places where snow doesn't melt.
A large part of the south pool is technically a desert as it has hardly any _new_ snow fall. Just a lot of years old snow getting moved around by wind.
honestly, it's a pattern I have been running into with many start ups, fintech, banks and some other places _no matter where_.
It also often makes sense. For many large orgs (e.g. Banks) this APIs are often a side business, sometimes one they don't want but have to have for compliance (or market pressure). But for strip it's their live blood.
And many startups (like actual start ups, not 200 man companies running by investor money) often simply don't have the resources to prioritize a "very nice to use API".
Lastly API design which is both nice to use and stable for existing integrations is surprisingly hard to get right (if you don't have some senior engine prioritizing it very highly and forcing it being kept prioritized; Or a surprisingly "clean"/"clear" use case.).
reply