SP and HT when being tolerant

[mnot@…]

3.1.1 Request Line and 3.1.2 Status line -- RFC2616 specifically allowed multiple SP and HT in these constructs --

Clients SHOULD be tolerant in parsing the Status-Line and servers tolerant when parsing the Request-Line. In particular, they SHOULD accept any amount of SP or HT characters between fields, even though only a single SP is required. (19.3 Tolerant Applications)

Yet this text seems to have been lost. Given that HT isn't even in the BNF, it should still be somewhere.

[fielding@…]

I thought we removed this requirement on purpose. Specifically, no known client sends extra spaces or HTABs. We could add them to BWS and spread that around a bit more.

[fielding@…]

In other words, we could change

request-line = method SP request-target SP HTTP-version CRLF

to

request-line = CRLF* BWS method SP BWS request-target SP BWS HTTP-version BWS CRLF

if we want the old 2616 robustness rules in the grammar. Personally, I'd rather not.

[fielding@…]

From [2028]:

Expand on message parsing robustness regarding whitespace. Clarify that whitespace-delimited parsing by recipients is allowed.

Add requirement on recipients of whitespace between start-line and first header field to be consistent with implementations and safe from the header spoofing issues.

Reduce ABNF conformance by recipients to a SHOULD.

Addresses #411, #412, and #414.