Opened 7 years ago

Closed 6 years ago

#484 closed editorial (incorporated)

MUSTs and other feedback

Reported by: mnot@… Owned by: draft-ietf-httpbis-p1-messaging@…
Priority: normal Milestone: 24
Component: p1-messaging Severity: In WG Last Call
Keywords: Cc:

Description

[ see further discussion / proposals on list ]

A sender MUST NOT generate protocol elements that do not match the grammar defined by the ABNF rules for those protocol elements that are applicable to the sender's role.

The "for those protocol elements..." part should be dropped IMO. A sender MUST NOT generate invalid protocol elements even if they are not applicable to the sender's role. Note that we are talking about _generation_ and not forwarding here.

If a received protocol element is processed, the recipient MUST be able to parse any value that would match the ABNF rules

"processed" seems too broad because simply buffering a header may be called "processing". "Interpreted" may be better. Or did I miss the definition of "process" that clarifies this?

If a received protocol element is processed, the recipient MUST be able to parse any value that would match the ABNF rules for that protocol element, excluding only those rules not applicable to the recipient's role.

The "excluding only those rules not applicable..." part seems to contradict the "processed" verb. Why would a recipient want to process something inapplicable? Perhaps this is related to the "process" versus "interpreted" issue mentioned above.

the recipient MUST be able to parse any value that would match the ABNF rules for that protocol element, excluding only those rules not applicable to the recipient's role.

Please rephrase to avoid double negation in "excluding not applicable". For example: "the recipient MUST be able to parse any value matching the corresponding ABNF protocol element rules applicable to the recipient's role"

Senders MUST exclude the userinfo subcomponent (and its "@" delimiter) when an "http" URI is transmitted within a message as a request target or header field value.

The above MUST should not apply to proxies, right? Policing forwarded URLs will break applications and policing forwarded extension header fields is not possible at all and would violate the "MUST forward" rule. How about "senders MUST NOT generate"?

A server MUST be prepared to receive URIs of unbounded length

This MUST may be demoted to "ought" because "be prepared" is too vague (but see below for a related missing MUST).

A server MUST be prepared to receive URIs of unbounded length and respond with the 414

Please insert a second MUST after "and": "and MUST respond".

Multiple header fields with the same field name can be combined into one "field-name: field-value" pair

Should this be a MAY as in "The recipient MAY combine multiple ...". As worded now, it is not clear whether a proxy is allowed to combine headers when forwarding them. Note that this affects extension and other headers that a proxy may not understand (but may still want to combine if allowed to do so).

A server MUST be prepared to receive request header fields of unbounded length and respond

Consider removing the above MUST but please add MUST after "and": A server ought to be prepared to receive ... and MUST respond ... See above for discussion of a similar MUST that applies to URIs of unbounded length.

A client MUST be prepared to receive response header fields of unbounded length.

Same here, except no new MUST is needed.

If chunked is applied to a payload body, the sender MUST NOT apply chunked more than once

The precondition is bogus: If chunked is NOT [yet?] applied to a payload body, the sender still MUST NOT apply chunked more than once!

the sender MUST NOT apply chunked more than once

This needs to be rephrased to make it clear that proxies are not responsible for dechunking multiple chunked encodings to make the forwarded message comply with this MUST. For example, we could say: "the sender MUST NOT generate messages with multiple chunked encodings".

Please note that both the proposed "multiple chunked encodings" and the existing "more than once" wordings imply that foo,chunked,bar,chunked combination is also not allowed.

A server MUST send an empty trailer with the chunked transfer coding unless at least one of the following is true:

This should be relaxed to "A server MUST generate ..." because a proxy, in general, does not know whether bullet #2 ("the trailer fields consist entirely of optional metadata...") is true. Even though chunking is a hop-by-hop mechanism, proxies ought to forward Trailers whenever possible, right?

a client MUST send only the absolute path and query components of the target URI as the request-target

To allow for transition to the absolute-form for all requests in some future version of HTTP, HTTP/1.1 servers MUST accept the absolute-form in requests

Should the first "MUST send" be relaxed to "MUST generate" so that the proxies do not block the apparently anticipated "transition to the absolute-form for all requests" by stripping URIs as they forward them?

In order to avoid request loops, a proxy that forwards requests to other proxies MUST be able to recognize and exclude all of its own server names

Several intermingled issues here:

1) The "other proxies" prerequisite is a red herring IMO. Any proxy should avoid request loops. If a proxy that is not configured to forward request to other proxies sends an "origin server" request to itself, such a request may still create a loop. I think the following would be better: "In order to avoid request loops, a proxy MUST ..."

2) If we want the proxy to recognize and exclude, let's demand that instead of just demanding that the proxy is able to do that (but possibly does not do it): "a proxy ... MUST recognize and exclude ...".

3) This MUST may benefit from some polishing to clarify what "exclude" means. I think it means "reject" in this context.

4) The "recognize" part can be dropped because recognition is implied by the "exclude" requirement.

At the end, we may arrive at something like this:

"In order to avoid request loops, a proxy MUST reject requests for itself, including requests where the server address is formed using a proxy domain name, its aliases, local variations, or literal IP addresses."

A client that does not support persistent connections MUST send the "close" connection option in every request message.

Including a CONNECT request message?

A client that pipelines requests MUST be prepared to retry those requests

MUST be prepared to retry but does not have to retry? Or MUST retry?

A client that pipelines requests MUST be prepared to retry those requests if the connection closes before it receives all of the corresponding responses.

Please clarify that the client MUST retry unanswered requests and not all "those requests" it pipelined.

MUST NOT pipeline on a retry connection until it knows the connection is persistent.

Is it really possible to know that a connection _is_ persistent? And what does that really mean for a connection to be persistent "now"? I think all it means is that the connection seems to be "open" -- that the sender has not received an error or connection close notification [yet].

It is possible to know that the connection was persistent (i.e., handled multiple messages), but since the server may close an idle (from the server point of view) connection at any time, I do not think it is possible to know that the next message will reach the server, unless the client and the server are coordinating out of band somehow.

What are we trying to achieve with this requirement? Avoid multiple and possibly endless retries? Minimize the chances that a retry will have to be retried? Perhaps the MUST can be relaxed or reworded to reflect the true intent?

And here is a list of MUST-level requirements that are missing an explicit actor on which the requirement is placed. Most of these should be easy to rephrase to place the requirement on the intended actor (e.g., "A proxy MUST" instead of "header field MUST":

An unrecognized header field received by a proxy MUST be forwarded downstream

The host MUST NOT be empty; if an "http" URI is received with an empty host, then it MUST be rejected as invalid.

the TCP connection MUST be secured,

These special characters MUST be in a quoted string

the message framing is invalid and MUST be treated as an error

a response message received by a user agent, it MUST be treated as an error

The trailer MUST NOT contain fields

the Host field-value MUST be identical

the Host header field MUST be sent with an empty field-value.

The "Via" header field MUST be sent by a proxy

the connection MUST be closed after the current request/response is complete

all messages on a connection MUST have a self-defined message length

the first action after changing the protocol MUST be a response

Please be careful with "send" and "generate" when fixing the above actorless rules so that the proxies do not accidentally become responsible for policing traffic where unnecessary.

Change History (4)

comment:1 Changed 6 years ago by fielding@…

From [2332]:

Explain in more detail the conformance requirements related to specific roles, recipients, and workarounds; addresses #484

comment:2 Changed 6 years ago by fielding@…

A sender MUST NOT generate protocol elements that do not match the grammar defined by the ABNF rules for those protocol elements that are applicable to the sender's role.

The "for those protocol elements..." part should be dropped IMO. A sender MUST NOT generate invalid protocol elements even if they are not applicable to the sender's role. Note that we are talking about _generation_ and not forwarding here.

That isn't why it is being described. There are ABNF rules that define various alternative syntax to be generated based on the role of the sender (and, in some cases, based on the role of the recipient). It is hard to capture in a small number of words.

If a received protocol element is processed, the recipient MUST be able to parse any value that would match the ABNF rules

"processed" seems too broad because simply buffering a header may be called "processing". "Interpreted" may be better. Or did I miss the definition of "process" that clarifies this?

Actually, no, simply buffering a header is not called processing. "Movement of data or material towards a known goal or end result, by passing it through a series of stages or a sequence of actions." However, I can rephrase it.

If a received protocol element is processed, the recipient MUST be able to parse any value that would match the ABNF rules for that protocol element, excluding only those rules not applicable to the recipient's role.

The "excluding only those rules not applicable..." part seems to contradict the "processed" verb. Why would a recipient want to process something inapplicable? Perhaps this is related to the "process" versus "interpreted" issue mentioned above.

A client does not need to parse syntax that is only sent to servers. An origin server does not need to parse syntax that is only sent by a server. ...

the recipient MUST be

able to parse any value that would match the ABNF rules for that protocol element, excluding only those rules not applicable to the recipient's role.

Please rephrase to avoid double negation in "excluding not applicable". For example: "the recipient MUST be able to parse any value matching the corresponding ABNF protocol element rules applicable to the recipient's role"

Okay.

One more attempt at describing the overall conformance requirements has been committed in [2332] to address this part of #484 (and related concerns).

comment:3 Changed 6 years ago by fielding@…

From [2334]:

Many changes to properly target MUST requirements by role; addresses #484

comment:4 Changed 6 years ago by fielding@…

  • Milestone changed from unassigned to 24
  • Resolution set to incorporated
  • Status changed from new to closed

The following was included in [2334]:

Senders MUST exclude the userinfo subcomponent (and its "@" delimiter) when an "http" URI is transmitted within a message as a request target or header field value.

The above MUST should not apply to proxies, right? Policing forwarded URLs will break applications and policing forwarded extension header fields is not possible at all and would violate the "MUST forward" rule. How about "senders MUST NOT generate"?

Right, fixed.

A server MUST be prepared to receive URIs of unbounded length

This MUST may be demoted to "ought" because "be prepared" is too vague (but see below for a related missing MUST).

Fixed.

A server MUST be prepared to receive URIs of unbounded length and respond with the 414

Please insert a second MUST after "and": "and MUST respond".

Fixed.

Multiple header fields with the same field name can be combined into one "field-name: field-value" pair

Should this be a MAY as in "The recipient MAY combine multiple ...". As worded now, it is not clear whether a proxy is allowed to combine headers when forwarding them. Note that this affects extension and other headers that a proxy may not understand (but may still want to combine if allowed to do so).

Fixed.

A server MUST be prepared to receive request header fields of unbounded length and respond

Consider removing the above MUST but please add MUST after "and": A server ought to be prepared to receive ... and MUST respond ... See above for discussion of a similar MUST that applies to URIs of unbounded length.

Fixed.

A client MUST be prepared to receive response header fields of unbounded length.

Same here, except no new MUST is needed.

Fixed.

If chunked is applied to a payload body, the sender MUST NOT apply chunked more than once

The precondition is bogus: If chunked is NOT [yet?] applied to a payload body, the sender still MUST NOT apply chunked more than once!

Fixed.

the sender MUST NOT apply chunked more than once

This needs to be rephrased to make it clear that proxies are not responsible for dechunking multiple chunked encodings to make the forwarded message comply with this MUST. For example, we could say: "the sender MUST NOT generate messages with multiple chunked encodings".

Please note that both the proposed "multiple chunked encodings" and the existing "more than once" wordings imply that foo,chunked,bar,chunked combination is also not allowed.

Both are intentional. A proxy is required to fix framing problems in received messages (or reject them without forwarding). Such messages might be crafted to bypass security filters.

A server MUST send an empty trailer with the chunked transfer coding unless at least one of the following is true:

This should be relaxed to "A server MUST generate ..." because a proxy, in general, does not know whether bullet #2 ("the trailer fields consist entirely of optional metadata...") is true. Even though chunking is a hop-by-hop mechanism, proxies ought to forward Trailers whenever possible, right?

Fixed.

a client MUST send only the absolute path and query components of the target URI as the request-target

To allow for transition to the absolute-form for all requests in some future version of HTTP, HTTP/1.1 servers MUST accept the absolute-form in requests

Should the first "MUST send" be relaxed to "MUST generate" so that the proxies do not block the apparently anticipated "transition to the absolute-form for all requests" by stripping URIs as they forward them?

No, since then such proxies won't interoperate with older servers. I wish we could just delete the second requirement, since it has been proven to be unreliable in practice and wouldn't be reliable for any HTTP/1.x (it isn't needed for support of HTTP/2).

In order to avoid request loops, a proxy that forwards requests to=20 other proxies MUST be able to recognize and exclude all of its own server names

Several intermingled issues here:

1) The "other proxies" prerequisite is a red herring IMO. Any proxy should avoid request loops. If a proxy that is not configured to forward request to other proxies sends an "origin server" request to itself, such a request may still create a loop.

Well, if it sends an origin server request to itself, then as the recipient it takes on the role of an origin server or gateway, not a proxy, and the requirement (as stated) no longer applies ...

I think the following would be better: "In order to avoid request loops, a proxy MUST ..."

Since this section is about forwarding, I have changed it to

An intermediary MUST NOT forward a message to itself unless it is protected from an infinite request loop. In general, an intermediary ought to recognize its own server names, including any aliases, local variations, or literal IP addresses, and respond to such requests directly.

2) If we want the proxy to recognize and exclude, let's demand that instead of just demanding that the proxy is able to do that (but possibly does not do it): "a proxy ... MUST recognize and exclude ...".

Fixed.

3) This MUST may benefit from some polishing to clarify what "exclude" means. I think it means "reject" in this context.

Not necessarily.

4) The "recognize" part can be dropped because recognition is implied by the "exclude" requirement.

At the end, we may arrive at something like this:

"In order to avoid request loops, a proxy MUST reject requests for itself, including requests where the server address is formed using a proxy domain name, its aliases, local variations, or literal IP addresses."

It can handle the requests itself. I have rephrased it.

A client that does not support persistent connections MUST send the "close" connection option in every request message.

Including a CONNECT request message?

Yes (they are orthogonal).

A client that pipelines requests MUST be prepared to retry those requests

MUST be prepared to retry but does not have to retry? Or MUST retry?

Changed to "SHOULD retry unanswered requests".

A client that pipelines requests MUST be prepared to retry those requests if the connection closes before it receives all of the corresponding responses.

Please clarify that the client MUST retry unanswered requests and not all "those requests" it pipelined.

Fixed.

MUST NOT pipeline on a retry connection until it knows the connection is persistent.

Is it really possible to know that a connection _is_ persistent?

Fixed, and rephrased to note that it is due to the TCP reset problem.

...

And here is a list of MUST-level requirements that are missing an explicit actor on which the requirement is placed. Most of these should be easy to rephrase to place the requirement on the intended actor (e.g., "A proxy MUST" instead of "header field MUST":

An unrecognized header field received by a proxy MUST be forwarded downstream

Fixed (and then removed as it was a redundant copy).

The host MUST NOT be empty; if an "http" URI is received with an empty host, then it MUST be rejected as invalid.

Fixed.

the TCP connection MUST be secured,

Fixed.

These special characters MUST be in a quoted string

Removed as it was redundant to ABNF.

the message framing is invalid and MUST be treated as an error

Fixed.

a response message received by a user agent, it MUST be treated as an error

Fixed.

The trailer MUST NOT contain fields

Fixed.

the Host field-value MUST be identical

Fixed.

the Host header field MUST be sent with an empty field-value.

Fixed.

The "Via" header field MUST be sent by a proxy

Fixed already for another ticket.

the connection MUST be closed after the current request/response is complete

Removed as redundant.

all messages on a connection MUST have a self-defined message length

Changed to "need to".

the first action after changing the protocol MUST be a response

Rephrased.

Note: See TracTickets for help on using tickets.