Welcome to the tutorial.
+ +---- + +The status line reports the protocol version, a numeric status code +(`200`), and a human-readable reason phrase (`OK`). The headers describe +the payload: its type (`text/html`) and size in bytes (`137`). After a +blank line, the body carries the actual content. + +This request-response pattern is the heartbeat of the Web. Every link +you click, every image that loads, every API call your app makes follows +exactly this structure. + +== Methods + +The _method_ in the request line tells the server what action to +perform. HTTP defines several methods, but three dominate everyday use: + +[cols="1,4"] +|=== +|Method |Purpose + +|**GET** +|Retrieve a resource. This is what browsers use when you navigate to a +page or load an image. A GET should never change anything on the server; +it is purely a read operation. + +|**POST** +|Send data to the server for processing. Form submissions, file uploads, +and API calls that create new records typically use POST. The data +travels in the request body. + +|**HEAD** +|Identical to GET, but the server returns only the headers--no body. +This is useful for checking whether a resource exists or has changed +without downloading the entire thing. +|=== + +Other methods exist (`PUT`, `DELETE`, `PATCH`, `OPTIONS`), and they play +important roles in RESTful APIs. But GET and POST account for the vast +majority of traffic on the Web. + +The distinction between GET and POST matters. GET requests can be +bookmarked, cached, and repeated safely. POST requests carry side +effects--submitting an order twice is not the same as submitting it +once. This is why browsers warn you before resubmitting a form. + +== Status Codes + +The three-digit status code in the response is how the server +communicates outcome. Codes are grouped by their first digit: + +[cols="1,3,3"] +|=== +|Range |Category |Examples + +|**1xx** +|Informational +|`100 Continue` + +|**2xx** +|Success +|`200 OK`, `201 Created`, `204 No Content` + +|**3xx** +|Redirection +|`301 Moved Permanently`, `302 Found`, `304 Not Modified` + +|**4xx** +|Client error +|`400 Bad Request`, `403 Forbidden`, `404 Not Found` + +|**5xx** +|Server error +|`500 Internal Server Error`, `503 Service Unavailable` +|=== + +A `200` means everything worked. A `404` means the resource does not +exist--probably the most recognizable error code in the world. A `301` +tells the client the resource has moved and provides the new URL in a +`Location` header. A `500` means something went wrong on the server's +side. + +The reason phrase (`OK`, `Not Found`) is for humans reading raw +messages. Software uses the numeric code exclusively. You could replace +`200 OK` with `200 All Good` and nothing would break. + +== Headers + +Headers are the metadata layer of HTTP. They appear in both requests +and responses as `Name: value` pairs, one per line, between the start +line and the body. + +Some headers you will encounter constantly: + +**Request headers:** + +* `Host` -- the domain name of the server (required in HTTP/1.1) +* `User-Agent` -- identifies the client software +* `Accept` -- media types the client is willing to receive +* `Accept-Language` -- preferred human languages +* `If-Modified-Since` -- makes the request conditional, asking the + server to send the resource only if it changed after a given date + +**Response headers:** + +* `Content-Type` -- the media type of the body +* `Content-Length` -- size of the body in bytes +* `Date` -- when the response was generated +* `Server` -- identifies the server software +* `Last-Modified` -- when the resource was last changed +* `Location` -- the URL to redirect to (used with 3xx status codes) +* `Cache-Control` -- directives for caching behavior + +Headers make HTTP extensible. A client and server can agree on new +headers without changing the protocol itself. This openness is one of +the reasons HTTP has survived and thrived for over three decades. + +== Content Types + +When a server sends a response, how does the client know whether the +body is HTML, a JPEG image, or a JSON document? The answer is the +`Content-Type` header, which carries a _media type_ (historically +called a MIME type). + +A media type has two parts separated by a slash: a top-level type and +a subtype. + +[cols="2,3"] +|=== +|Media Type |Meaning + +|`text/html` +|An HTML document + +|`text/plain` +|Plain text, no formatting + +|`image/jpeg` +|A JPEG photograph + +|`image/png` +|A PNG image + +|`application/json` +|A JSON data structure + +|`application/octet-stream` +|Arbitrary binary data (the catch-all) + +|`video/mp4` +|An MP4 video file +|=== + +The top-level type gives the client a general idea even if it does not +recognize the specific subtype. If the browser receives `image/webp` +and has no WebP decoder, it at least knows the content is an image and +can decide what to do--perhaps show a placeholder or offer to +download the file. + +Media types are the reason HTTP can carry anything, not just hypertext. +A single protocol serves web pages, streams video, delivers fonts, and +transfers API payloads, all because every response says exactly what it +contains. + +== Statelessness + +HTTP is _stateless_: the server does not remember anything about +previous requests. Each request is independent--the server processes +it in isolation and forgets about it the moment the response is sent. + +This sounds like a limitation, but it is actually a powerful design +choice. Statelessness means any server in a cluster can handle any +request. There is no session to maintain, no affinity to preserve. +Scaling becomes straightforward: add more servers and distribute the +load. + +Of course, real applications need state. A shopping cart must persist +between page views. A login must be remembered. HTTP solves this +through _cookies_--small pieces of data that the server sends in a +`Set-Cookie` header and the client returns in a `Cookie` header on +subsequent requests. Cookies bolt stateful sessions onto a stateless +protocol without changing the protocol itself. + +== Connections and TCP + +HTTP does not define how bytes travel across the network. It delegates +that responsibility to TCP (Transmission Control Protocol), which +guarantees reliable, in-order delivery. Before any HTTP message can be +exchanged, the client and server establish a TCP connection through a +three-way handshake: + +. The client sends a SYN packet. +. The server responds with SYN-ACK. +. The client replies with ACK. + +Only then can data flow. This handshake adds one full round-trip of +latency before the first byte of the request is even sent. Between +New York and London, that round-trip alone takes roughly 56 +milliseconds over fiber. + +Early HTTP (1.0) opened a new TCP connection for every single request. +A page with ten images meant eleven connections: one for the HTML and +one for each image. The overhead was enormous. + +HTTP/1.1 changed this with _persistent connections_. By default, the +connection stays open after a response, ready for the next request. +This eliminates repeated handshakes and lets TCP ramp up to full +throughput. A `Connection: close` header signals that one party is +done and the connection should be torn down. + +HTTP/1.1 also introduced _pipelining_: a client can send several +requests in a row without waiting for responses. In practice, +pipelining proved fragile and was rarely used, but the idea of +eliminating round-trip delays carried forward into HTTP/2's +multiplexing. + +== A Brief History + +HTTP has evolved in discrete steps, each addressing the shortcomings +of the previous version: + +**HTTP/0.9 (1991)** -- The original one-line protocol. Only GET was +supported. No headers, no status codes, no content types. The server +sent back raw HTML and closed the connection. + +**HTTP/1.0 (1996)** -- Added version numbers, headers, status codes, +and content-type negotiation. For the first time, HTTP could carry +images, video, and other media--not just hypertext. Each request still +required its own TCP connection. + +**HTTP/1.1 (1997)** -- The workhorse of the modern Web. Persistent +connections, chunked transfer encoding, the `Host` header for virtual +hosting, and cache-control directives. HTTP/1.1 corrected architectural +flaws and introduced the performance optimizations that made the Web +commercially viable. + +**HTTP/2 (2015)** -- A binary framing layer over the same semantics. +Multiplexed streams eliminate head-of-line blocking at the application +level. Header compression (`HPACK`) reduces overhead. Server push +allows preemptive delivery of resources. + +**HTTP/3 (2022)** -- Replaces TCP with QUIC, a UDP-based transport +with built-in encryption and independent stream multiplexing. This +eliminates head-of-line blocking at the transport level and reduces +connection setup to a single round-trip. + +Despite these changes, the fundamental model has not moved: a client +sends a request, a server sends a response, and headers describe +everything. Code written against HTTP/1.1 semantics still works in +an HTTP/3 world. + +== Putting It All Together + +Here is a complete HTTP/1.1 exchange, the kind that happens invisibly +thousands of times as you browse: + +[source] +---- +GET /about.html HTTP/1.1 <1> +Host: www.example.com <2> +User-Agent: Mozilla/5.0 <3> +Accept: text/html <4> + +---- + +[source] +---- +HTTP/1.1 200 OK <1> +Date: Sat, 07 Feb 2026 12:00:00 GMT <2> +Server: Apache/2.4.54 <3> +Content-Type: text/html <4> +Content-Length: 82 <5> + + <6> +About us.
+ +---- + +The request identifies the resource, the protocol version, and what the +client is prepared to accept. The response confirms success, describes +the payload, and delivers the content. Two small text messages, and a +page appears on screen. + +Every layer of the Web--browsers, servers, proxies, caches, CDNs, +APIs--is built on this exchange. Understanding it gives you a +foundation for everything that follows. diff --git a/doc/modules/ROOT/pages/2.http-tutorial/2b.urls-and-resources.adoc b/doc/modules/ROOT/pages/2.http-tutorial/2b.urls-and-resources.adoc new file mode 100644 index 00000000..73704e54 --- /dev/null +++ b/doc/modules/ROOT/pages/2.http-tutorial/2b.urls-and-resources.adoc @@ -0,0 +1,319 @@ +// +// Copyright (c) 2025 Vinnie Falco (vinnie.falco@gmail.com) +// +// Distributed under the Boost Software License, Version 1.0. (See accompanying +// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) +// +// Official repository: https://github.com/cppalliance/http +// + += URLs and Resources + +Every building in a city has an address. Without addresses, you could +describe a building by its appearance or its neighborhood, but you could +never tell a taxi driver exactly where to go. Before URLs existed, finding +something on the Internet was like that -- you needed to know which +application to open, which server to connect to, which protocol to speak, +and which directory to look in. URLs collapsed all of that into a single +string that anyone could share, bookmark, or click. + +This section explains what resources are, how URLs name them, and how +the pieces of a URL work together to tell an HTTP client exactly what +to fetch and from where. + +== Resources + +A resource is anything that can be served over the web. The term is +deliberately broad. A resource might be a static file sitting on disk -- an +HTML page, a JPEG photograph, a PDF manual. It might also be a program +that generates content on demand: a search engine returning results for +your query, a stock ticker streaming live prices, or an API endpoint +returning JSON. + +What makes something a resource is not its format or its origin, but the +fact that it can be identified by a name and retrieved by a client. HTTP +does not care whether the bytes come from a file, a database, a camera +feed, or a script. As far as the protocol is concerned, a resource is +whatever the server sends back. + +=== Media Types + +When a server sends a resource, the client needs to know what kind of data +it is receiving. A stream of bytes could be an image, a web page, or a +compressed archive -- the bits alone do not say. HTTP solves this with +*media types* (also called MIME types), a labeling system borrowed from +email. + +A media type is a two-part string: a primary type and a subtype, separated +by a slash: + +[cols="1a,2a"] +|=== +|Media Type|Meaning + +|`text/html` +|An HTML document + +|`text/plain` +|Plain text with no formatting + +|`image/jpeg` +|A JPEG photograph + +|`image/png` +|A PNG image + +|`application/json` +|JSON-formatted data + +|`application/octet-stream` +|Arbitrary binary data (the catch-all) + +|=== + +The server communicates the media type in the `Content-Type` header: + +[source] +---- +HTTP/1.1 200 OK +Content-Type: image/png +Content-Length: 4096 + +<...4096 bytes of image data...> +---- + +The client reads this header and decides how to handle the body. A browser +renders `text/html` as a web page, displays `image/jpeg` as a picture, and +might offer to download `application/octet-stream` as a file. Media types +are the reason the web can serve every kind of content through a single +protocol. + +== URLs + +A Uniform Resource Locator (URL) is the address of a resource on the +Internet. It tells a client three things at once: _how_ to access the +resource (the protocol), _where_ the resource lives (the server), and +_which_ resource to retrieve (the path). + +[source] +---- +http://www.example.com/seasonal/index-fall.html +---- + +This single string replaces what used to be a paragraph of instructions: +"Open your FTP client, connect to this server, log in with these +credentials, navigate to this directory, switch to binary mode, and +download this file." A URL encodes all of that context into a compact, +shareable format. + +URLs are a subset of a broader concept called Uniform Resource Identifiers +(URIs). The HTTP specification uses the term URI, but in practice nearly +every URI you encounter is a URL. The distinction matters mainly in +specifications; for day-to-day work with HTTP, the two terms are +interchangeable. + +=== Anatomy of a URL + +A URL can contain up to nine components. Most URLs use only a few of them, +but the full general form is: + +[source] +---- +scheme://user:password@host:port/path?query#fragment +---- + +The three most important parts are the *scheme*, the *host*, and the +*path*. Here is how they break down for a typical HTTP URL: + +[source] +---- + http://www.example.com:8080/tools/hammers?color=blue&sort=price#reviews + \__/ \______________/\__/\____________/ \____________________/\_____/ +scheme host port path query fragment +---- + +[cols="1a,3a"] +|=== +|Component|Description + +|*scheme* +|The protocol to use. For web traffic this is `http` or `https`. The scheme +ends at the first `:` character and is case-insensitive. + +|*host* +|The server's address -- either a domain name like `www.example.com` or an +IP address like `192.168.1.1`. This is where the client will open a +connection. + +|*port* +|The TCP port on the server. If omitted, the default for the scheme is used +(80 for `http`, 443 for `https`). + +|*path* +|The specific resource on the server, structured like a filesystem path. +Each segment is separated by `/`. + +|*query* +|Additional parameters passed to the server, introduced by `?`. Query +strings are typically formatted as `name=value` pairs separated by `&`. + +|*fragment* +|A reference to a specific section _within_ the resource, introduced by +`#`. Fragments are used only by the client -- they are never sent to the +server. + +|=== + +=== Schemes + +The scheme is the first thing a client reads. It determines which protocol +to use for retrieving the resource. Although HTTP and HTTPS dominate the +web, URLs support many schemes: + +[cols="1a,3a"] +|=== +|Scheme|Example + +|`http` +|`\http://www.example.com/index.html` -- standard web traffic, port 80 + +|`https` +|`\https://www.example.com/secure` -- HTTP over TLS, port 443 + +|`ftp` +|`ftp://ftp.example.com/pub/readme.txt` -- file transfer + +|`mailto` +|`mailto:user@example.com` -- email address + +|`file` +|`file:///home/user/notes.txt` -- local filesystem + +|=== + +For HTTP programming, you will work almost exclusively with `http` and +`https`. The scheme tells your code whether to open a plain TCP connection +or negotiate a TLS handshake before sending the first request. + +=== The Request-Target + +When a client sends an HTTP request, the URL does not appear in the +message exactly as you see it in a browser's address bar. The scheme and +host are stripped away, and only the *request-target* is placed on the +request line. For most requests, the request-target is the path plus any +query string: + +[source] +---- +GET /tools/hammers?color=blue HTTP/1.1 +Host: www.example.com +---- + +The host is conveyed separately in the `Host` header. This split exists +because a single server can host many domain names (virtual hosting), and +the request-target alone would not identify which site the client wants. + +For requests sent through a proxy, the full URL (called the *absolute +form*) may appear on the request line instead: + +[source] +---- +GET http://www.example.com/tools/hammers HTTP/1.1 +---- + +Understanding the request-target matters because when you build or parse +HTTP messages, you are working with this extracted piece of the URL, not +the full address. + +== Percent-Encoding + +URLs were designed to be transmitted safely across every protocol on the +Internet, so they are restricted to a small set of characters: letters, +digits, and a handful of punctuation marks. Any character outside this safe +set must be *percent-encoded* -- replaced with a `%` sign followed by two +hexadecimal digits representing the character's byte value. + +[cols="1a,1a,1a"] +|=== +|Character|ASCII Code|Encoded Form + +|space +|32 (0x20) +|`%20` + +|`#` +|35 (0x23) +|`%23` + +|`%` +|37 (0x25) +|`%25` + +|`/` +|47 (0x2F) +|`%2F` + +|`?` +|63 (0x3F) +|`%3F` + +|=== + +For example, a search query containing spaces and special characters: + +[source] +---- +GET /search?q=hello%20world%21 HTTP/1.1 +Host: www.example.com +---- + +Here `%20` represents a space and `%21` represents an exclamation mark. + +Several characters have reserved meanings inside a URL -- `/` separates +path segments, `?` introduces the query, `#` marks a fragment, and `:` +separates the scheme. If you need these characters to appear as literal +data (for instance, a filename that contains a question mark), you must +percent-encode them. Conversely, encoding characters that are already safe +is technically allowed but can cause interoperability problems, so it is +best avoided. + +Applications should encode unsafe characters before transmitting a URL and +decode them when processing one. Getting this wrong is a common source of +bugs: double-encoding a URL that is already encoded, or failing to encode +a user-supplied value before inserting it into a path or query string. + +== Relative URLs + +Not every URL needs to spell out the scheme and host. A *relative URL* +is a shorthand that omits the parts which can be inferred from context. +If you are already viewing a page at +`\http://www.example.com/tools/index.html`, a link to `./hammers.html` is +understood to mean `\http://www.example.com/tools/hammers.html`. + +The URL from which missing parts are inherited is called the *base URL*. +It is usually the URL of the document that contains the link: + +[source] +---- +Base URL: http://www.example.com/tools/index.html +Relative URL: ./hammers.html +Resolved URL: http://www.example.com/tools/hammers.html +---- + +Relative URLs make content portable. A set of HTML pages that link to +each other with relative paths can be moved to a different server or a +different directory without breaking any links, because the references +adjust automatically to the new base. + +In HTTP messages, the request-target is already relative to the server, +so the concept shows up naturally. When your code constructs a request, +it uses the path portion of a URL -- which is itself a relative reference +resolved against the connection's host. + +== Next Steps + +You now know what resources are, how URLs name them, and how the pieces +of a URL map onto an HTTP request. The next section breaks open the +messages themselves: + +* xref:2.http-tutorial/2c.message-anatomy.adoc[Message Anatomy] -- start lines, headers, and bodies diff --git a/doc/modules/ROOT/pages/2.http-tutorial/2c.message-anatomy.adoc b/doc/modules/ROOT/pages/2.http-tutorial/2c.message-anatomy.adoc new file mode 100644 index 00000000..cb757951 --- /dev/null +++ b/doc/modules/ROOT/pages/2.http-tutorial/2c.message-anatomy.adoc @@ -0,0 +1,376 @@ +// +// Copyright (c) 2025 Vinnie Falco (vinnie.falco@gmail.com) +// +// Distributed under the Boost Software License, Version 1.0. (See accompanying +// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) +// +// Official repository: https://github.com/cppalliance/http +// + += Message Anatomy + +If HTTP is the language that moves the Web, then HTTP messages are the +sentences. Every image that loads, every form that submits, every API +call that returns JSON--each one is carried inside a small, precisely +formatted block of text that both sides of the conversation agree to +understand. + +The beauty of HTTP messages is that they are _plain text_. You can +read them with your own eyes, type them by hand, and debug them with +nothing more than a terminal. This transparency is not an accident; it +is a deliberate design choice that made the early Web possible for +anyone with a text editor and a network socket. Binary protocols may +be more compact, but HTTP's readability is the reason it was adopted +so fast, and the reason you can learn its structure in a single +sitting. + +== The Three Parts + +Every HTTP message--request or response--consists of exactly three +parts, always in the same order: + +[source] +---- +start-line CRLF +header-field CRLF +header-field CRLF +... +CRLF +[ message-body ] +---- + +. The **start line** says what the message is about. For a request, + it identifies the action and the target. For a response, it reports + what happened. +. The **header fields** carry metadata: the type and size of the body, + caching instructions, authentication credentials, and anything else + the sender wants the receiver to know. +. The **body** (optional) carries the actual payload--an HTML page, + a JSON document, an image, or nothing at all. + +The start line and headers are ASCII text, one item per line, each +terminated by a carriage return followed by a line feed (written +`CRLF`, ASCII 13 then ASCII 10). After the last header, a blank +line--a bare `CRLF` with nothing before it--marks the boundary +between metadata and data. Everything after that blank line is the +body. + +This structure never changes. Every HTTP/1.x message you will ever +encounter follows it. Once you can identify these three parts, you can +read any HTTP exchange. + +== Request Messages + +A request message begins with a _request line_ that contains three +fields separated by spaces: + +[source] +---- +method SP request-target SP HTTP-version CRLF +---- + +Here is a concrete example: + +[source] +---- +GET /docs/tutorial.html HTTP/1.1 +Host: www.example.com +Accept: text/html +User-Agent: curl/8.4.0 + +---- + +The request line `GET /docs/tutorial.html HTTP/1.1` says three things +at once: + +* **GET** is the method--the action the client wants the server to + perform. Methods are covered in detail in a later section. +* **/docs/tutorial.html** is the request target--the resource being + addressed, usually the path component of the URL. +* **HTTP/1.1** is the protocol version, telling the server which + dialect of HTTP the client speaks. + +After the request line come the headers (`Host`, `Accept`, +`User-Agent`), and after the blank line comes the body. This +particular request has no body because GET asks for data rather than +sending it. + +A POST request, by contrast, typically carries a body: + +[source] +---- +POST /api/login HTTP/1.1 +Host: www.example.com +Content-Type: application/json +Content-Length: 42 + +{"username":"alice","password":"s3cret!"} +---- + +Here `Content-Type` tells the server the body is JSON, and +`Content-Length` tells it the body is exactly 42 bytes long. The +body begins immediately after the blank line. + +== Response Messages + +A response message begins with a _status line_ instead of a request +line, but the rest of the structure is identical: + +[source] +---- +HTTP-version SP status-code SP reason-phrase CRLF +---- + +For example: + +[source] +---- +HTTP/1.1 200 OK +Date: Sat, 07 Feb 2026 12:00:00 GMT +Content-Type: text/html +Content-Length: 137 + + +Welcome to the tutorial.
+ +---- + +The status line `HTTP/1.1 200 OK` reports: + +* **HTTP/1.1** -- the protocol version the server is using. +* **200** -- a numeric status code indicating success. Codes are + grouped by their first digit: 2xx means success, 3xx means + redirection, 4xx means client error, 5xx means server error. + Status codes are covered in their own section. +* **OK** -- a human-readable reason phrase. Software ignores this; + you could replace `OK` with `All Good` and nothing would break. + The phrase exists solely to help humans scanning raw traffic. + +After the status line come headers, the blank line, and the body +carrying the HTML document. + +== On the Wire + +Understanding the physical format matters when debugging. Here is +what the request from the earlier example looks like as raw bytes +flowing across the network, with invisible characters made visible: + +[source] +---- +G E T / d o c s / t u t o r i a l . h t m l H T T P / 1 . 1 CR LF +H o s t : w w w . e x a m p l e . c o m CR LF +A c c e p t : t e x t / h t m l CR LF +CR LF +---- + +Every line, including the start line, ends with `CR LF` (bytes `0x0D +0x0A`). The blank line after the headers is just `CR LF` by itself--no +characters before it. That bare `CRLF` is what separates the header +section from the body. It is so important that the HTTP specification +requires it even when the message has no body and no headers at all. + +A few practical details worth knowing: + +* **Whitespace in headers.** A header field is a name, a colon, optional + whitespace, and a value: `Content-Type: text/html`. Leading and + trailing whitespace around the value is stripped by the parser. +* **Case insensitivity.** Header field names are case-insensitive. + `Content-Type`, `content-type`, and `CONTENT-TYPE` all mean the + same thing. +* **One header per line.** Older specifications allowed "folding" a + long header across multiple lines by starting continuation lines + with whitespace, but HTTP/1.1 (RFC 9112) has deprecated this + practice. Modern implementations reject folded headers. +* **Robustness.** The specification says `CRLF`, but many real-world + implementations also accept a bare `LF`. Robust parsers tolerate + this, although strict parsers reject it. + +== Header Fields + +Header fields are the metadata layer of HTTP. They appear between the +start line and the body as `Name: value` pairs, one per line. + +[source] +---- +Content-Type: text/html; charset=utf-8 +Content-Length: 4821 +Cache-Control: max-age=3600 +---- + +Headers serve many roles. Some describe the body (`Content-Type`, +`Content-Length`). Some control caching (`Cache-Control`, `ETag`). +Some carry authentication tokens (`Authorization`). Some influence +connection behavior (`Connection`, `Transfer-Encoding`). The protocol +is extensible--any sender can introduce new header names, and +receivers that do not recognize them simply pass them through. + +Headers are classified into broad categories: + +* **Request headers** supply information about the request or the + client: `Host`, `User-Agent`, `Accept`, `Authorization`. +* **Response headers** supply information about the response or the + server: `Server`, `Retry-After`, `WWW-Authenticate`. +* **Representation headers** describe the body: `Content-Type`, + `Content-Length`, `Content-Encoding`, `Content-Language`. +* **General headers** apply to the message as a whole rather than to + the body: `Date`, `Connection`, `Transfer-Encoding`, `Via`. + +A deeper exploration of individual headers appears in a later section. +What matters here is the _structure_: every header is a name-value +pair on its own line, headers can appear in any order, the same name +can appear more than once, and the entire header block ends with a +blank line. + +== The Message Body + +The body is the payload. It is everything that comes after the blank +line separating headers from data. It can contain HTML, JSON, XML, +images, video, compressed archives, or nothing at all. + +Not every message has a body. Responses to HEAD requests never +include one. Responses with status codes 1xx, 204, and 304 never +include one. GET requests rarely carry a body, though the protocol +does not forbid it. + +When a body _is_ present, the receiver needs to know how large it is. +HTTP provides three mechanisms: + +**Content-Length.** The simplest case. The sender states the exact +number of bytes: + +[source] +---- +Content-Length: 4821 +---- + +The receiver reads exactly 4821 bytes after the blank line and knows +the body is complete. + +**Chunked transfer encoding.** When the sender does not know the +total size in advance--for example, when generating content +dynamically--it can send the body in chunks. Each chunk is preceded +by its size in hexadecimal, and the stream ends with a zero-length +chunk: + +[source] +---- +Transfer-Encoding: chunked + +1a +This is the first chunk. +10 +Second chunk!!! +0 + +---- + +The receiver reads each chunk size, consumes that many bytes, and +repeats until it sees `0`. This mechanism lets servers begin +transmitting before the entire response is generated. + +**Connection close.** A server may simply close the connection when +the body is finished. This only works for responses (a client cannot +close and still expect a reply) and only when neither `Content-Length` +nor chunked encoding is in use. It is the least desirable method +because it prevents connection reuse. + +== Requests vs. Responses + +Requests and responses share the same three-part structure. The only +structural difference is the start line: + +[cols="1,2,2"] +|=== +|Part |Request |Response + +|**Start line** +|`GET /index.html HTTP/1.1` +|`HTTP/1.1 200 OK` + +|**Headers** +|`Host: example.com` + +`Accept: text/html` +|`Content-Type: text/html` + +`Content-Length: 512` + +|**Body** +|_(often empty for GET)_ +|`...` +|=== + +This symmetry is intentional. Because both directions use the same +header syntax and the same body framing rules, a single parser can +handle either direction with only the start-line logic swapped out. + +== Message Flow + +HTTP messages travel in one direction: _downstream_. Every sender is +upstream of the receiver, regardless of whether the message is a +request or a response. When a client sends a request through two +proxies to an origin server, the request flows downstream from client +to server. When the server sends the response back through those same +proxies, the response also flows downstream--this time from server to +client. + +The terms _inbound_ and _outbound_ describe the two legs of the +journey. Messages travel inbound toward the origin server and outbound +back to the client: + +[source] +---- +Client ──► Proxy A ──► Proxy B ──► Server (inbound) +Client ◄── Proxy A ◄── Proxy B ◄── Server (outbound) +---- + +This distinction matters when proxies modify or inspect messages in +transit. A proxy that adds a `Via` header, for instance, annotates the +message as it passes downstream in either direction. + +== A Complete Exchange + +Putting it all together, here is an annotated HTTP/1.1 exchange--the +kind that happens thousands of times while you browse a single page: + +[source] +---- +GET /about.html HTTP/1.1 <1> +Host: www.example.com <2> +User-Agent: Mozilla/5.0 <3> +Accept: text/html <4> + <5> +---- + +<1> Request line: method, target, version +<2> Required in HTTP/1.1--identifies which site on the server +<3> Identifies the client software +<4> Tells the server what the client prefers to receive +<5> Blank line--end of headers, no body follows + +[source] +---- +HTTP/1.1 200 OK <1> +Date: Sat, 07 Feb 2026 12:00:00 GMT <2> +Server: Apache/2.4.54 <3> +Content-Type: text/html <4> +Content-Length: 82 <5> + <6> + <7> +About us.
+ +---- + +<1> Status line: version, code, reason +<2> When the response was generated +<3> Server software +<4> The body is HTML +<5> The body is exactly 82 bytes +<6> Blank line--end of headers, body follows +<7> The body itself + +Two small text messages, a handful of headers each, and a page appears +on screen. Every layer of the Web--browsers, servers, proxies, caches, +CDNs, APIs--is built on this exchange. The format has not changed in +any fundamental way since 1997. Code that can parse these three parts +correctly can participate in the largest distributed system ever built. diff --git a/doc/modules/ROOT/pages/2.http-tutorial/2d.methods.adoc b/doc/modules/ROOT/pages/2.http-tutorial/2d.methods.adoc new file mode 100644 index 00000000..717ce148 --- /dev/null +++ b/doc/modules/ROOT/pages/2.http-tutorial/2d.methods.adoc @@ -0,0 +1,427 @@ +// +// Copyright (c) 2025 Vinnie Falco (vinnie.falco@gmail.com) +// +// Distributed under the Boost Software License, Version 1.0. (See accompanying +// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) +// +// Official repository: https://github.com/cppalliance/http +// + += Methods + +The method is the verb at the beginning of every HTTP request. It tells +the server what the client wants done. When your browser loads a page, +it sends `GET`. When you submit a login form, it sends `POST`. When a +deployment script uploads a new version of a file, it sends `PUT`. The +rest of the message--headers, body, URL--supplies the nouns and +adjectives, but the method is the action word that drives the entire +conversation. + +HTTP defines a small set of standard methods, each with precise +semantics that the rest of the Web's infrastructure relies on. Caches +decide what to store based on the method. Proxies decide what to retry +based on the method. Browsers decide whether to warn you about +resubmitting a form based on the method. Understanding what each method +means--and two critical properties called _safety_ and +_idempotency_--is the key to understanding why HTTP works the way it +does. + +== GET + +GET is the workhorse of the Web. It asks the server to return a +representation of the resource identified by the request target. Every +link you click, every image that loads, every stylesheet and script +your browser fetches--all of them use GET. + +[source] +---- +GET /docs/tutorial.html HTTP/1.1 +Host: www.example.com +Accept: text/html +---- + +The server responds with the resource: + +[source] +---- +HTTP/1.1 200 OK +Content-Type: text/html +Content-Length: 4821 + + +You found the new page.
+ +---- + +<1> The client requests the original URL +<2> The server says the resource has permanently moved +<3> The `Location` header provides the new URL +<4> The client automatically follows the redirect +<5> The resource is delivered from its new home + +Browsers perform this redirect chain automatically and invisibly. +The user sees only the final page and the new URL in the address +bar. Programs and libraries do the same, though most limit the +number of redirects they will follow (typically 20) to avoid +infinite loops. + +== Codes You Will See Every Day + +Out of the dozens of defined status codes, a handful appear in the +vast majority of real-world traffic: + +[cols="1,3"] +|=== +|Code |When you will see it + +|`200` +|Almost every successful page load, API call, and resource fetch. + +|`201` +|After creating a resource via POST in a REST API. + +|`204` +|After a successful DELETE, or a PUT that needs no response body. + +|`301` +|When a website changes its URL structure or migrates to a new +domain. + +|`304` +|When the browser checks its cache and the resource has not changed. + +|`400` +|When an API request has missing or invalid parameters. + +|`401` +|When you forget to include your authentication token. + +|`403` +|When you are authenticated but lack permission. + +|`404` +|When the URL does not match any resource on the server. + +|`500` +|When the server has an unhandled bug. + +|`502` +|When the reverse proxy cannot reach the application server. + +|`503` +|When the server is overloaded or down for maintenance. +|=== + +Memorizing these twelve codes covers the overwhelming majority of +HTTP traffic. The rest are important in specific contexts--range +requests, content negotiation, WebDAV--but these are the ones that +appear in logs, error pages, and debugging sessions day after day. + +== Why the First Digit Matters + +The five-class design is not just a convenience for humans. The HTTP +specification (RFC 9110) requires that software treat an unrecognized +status code as equivalent to the x00 code of its class. If a client +receives a response with status code `299`, and it does not know what +`299` means, it must treat it as `200`. If it receives `599`, it must +treat it as `500`. + +This rule makes the protocol future-proof. New status codes can be +defined without breaking existing clients. As long as the first digit +is correct, old software will do something reasonable--it just will +not do anything _specific_ to the new code. This is one of the small +design decisions that allowed HTTP to evolve for over three decades +without fragmenting the Web. diff --git a/doc/modules/ROOT/pages/2.http-tutorial/2f.headers.adoc b/doc/modules/ROOT/pages/2.http-tutorial/2f.headers.adoc new file mode 100644 index 00000000..97d9ade2 --- /dev/null +++ b/doc/modules/ROOT/pages/2.http-tutorial/2f.headers.adoc @@ -0,0 +1,635 @@ +// +// Copyright (c) 2025 Vinnie Falco (vinnie.falco@gmail.com) +// +// Distributed under the Boost Software License, Version 1.0. (See accompanying +// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) +// +// Official repository: https://github.com/cppalliance/http +// + += Headers + +An HTTP message without headers is like a package with no shipping label. +The bytes inside might be a web page, a photograph, or a stock quote, +but neither the sender nor the receiver would know what to do with them. +Headers are the metadata layer that gives HTTP its power: they describe +the payload, declare the sender's preferences, control caching, carry +authentication tokens, and negotiate the terms of every exchange. Two +programs that have never communicated before can cooperate perfectly, as +long as they read each other's headers. + +You already know from the previous section that headers sit between the +start line and the body as `Name: value` pairs, one per line. This +section goes deeper. You will learn _which_ headers matter, _why_ they +exist, and how they work together to orchestrate everything from a +simple page load to a complex API transaction. + +== Header Syntax + +Every header field follows the same format: + +[source] +---- +Field-Name: field-value +---- + +A name, a colon, optional whitespace, and a value. The name is +case-insensitive -- `Content-Type`, `content-type`, and `CONTENT-TYPE` +all mean the same thing. The value, however, _is_ case-sensitive for +most headers unless the specification for that header says otherwise. + +[source] +---- +Content-Type: text/html; charset=utf-8 +Cache-Control: max-age=3600, must-revalidate +Accept: text/html, application/json;q=0.9, */*;q=0.8 +---- + +Some values are simple tokens. Others carry parameters separated by +semicolons, or lists of items separated by commas. The `Accept` header +above says: "I prefer HTML, I can handle JSON almost as well, and in a +pinch I will take anything." Those `q` values are quality weights -- +a number from 0 to 1 that ranks the client's preference. + +A message can contain any number of headers, and the same header name +can appear more than once. When it does, the effect is the same as if +the values were combined into a single comma-separated list. These two +forms are equivalent: + +[source] +---- +Accept-Encoding: gzip +Accept-Encoding: deflate +---- + +[source] +---- +Accept-Encoding: gzip, deflate +---- + +Headers can appear in any order, with one exception: the `Host` header +should appear first in an HTTP/1.1 request, because servers that support +virtual hosting need it immediately. + +== Categories of Headers + +HTTP defines dozens of standard headers. Remembering them all is +unnecessary, but understanding how they are organized makes the list +manageable. Headers fall into four broad categories based on their role +in the message. + +=== Request Headers + +Request headers carry information _from the client to the server_. +They describe who is making the request, what the client can accept, +and any conditions attached to the request. + +[cols="1a,3a"] +|=== +|Header|Purpose + +|`Host` +|The domain name (and optional port) of the server. Required in +HTTP/1.1 because a single machine often hosts many sites. + +|`User-Agent` +|Identifies the client software -- browser name, version, and platform. +Servers use this to tailor responses or log traffic patterns. + +|`Accept` +|Media types the client is willing to receive, ranked by preference. + +|`Accept-Language` +|Human languages the client prefers, such as `en`, `fr`, or `de`. + +|`Accept-Encoding` +|Compression algorithms the client supports, such as `gzip` or `br` +(Brotli). + +|`Referer` +|The URL of the page that led the client to make this request. Useful +for analytics and back-link tracking. + +|`Authorization` +|Credentials for authenticating the client -- a bearer token, a Basic +username/password pair, or other scheme. + +|`Cookie` +|Previously stored cookies being returned to the server. + +|=== + +Here is a realistic request with several of these headers in action: + +[source] +---- +GET /api/products?category=tools HTTP/1.1 +Host: www.example.com +User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) +Accept: application/json +Accept-Language: en-US,en;q=0.9 +Accept-Encoding: gzip, br +Authorization: Bearer eyJhbGciOiJIUzI1NiJ9... +Cookie: session_id=abc123 +---- + +Every header here helps the server make a better decision. `Accept` +says the client wants JSON, not HTML. `Accept-Encoding` says the server +can compress the response. `Authorization` proves the client is logged +in. Without these headers, the server would have to guess -- or refuse +the request. + +=== Response Headers + +Response headers carry information _from the server to the client_. +They describe the server itself, provide instructions about the +response, and sometimes set up future interactions. + +[cols="1a,3a"] +|=== +|Header|Purpose + +|`Server` +|Identifies the server software, similar to how `User-Agent` identifies +the client. + +|`Date` +|The date and time when the response was generated. + +|`Location` +|The URL to redirect to. Used with 3xx status codes to send the client +somewhere else. + +|`Retry-After` +|Tells the client how long to wait before retrying, typically used with +`503 Service Unavailable`. + +|`Set-Cookie` +|Sends a cookie to the client for storage. The client will return it in +subsequent `Cookie` headers. + +|`WWW-Authenticate` +|Challenges the client to provide credentials. Sent with +`401 Unauthorized` to initiate authentication. + +|=== + +=== Representation Headers + +Representation headers describe the body of the message -- what it is, +how big it is, and how it has been encoded. They appear in both requests +and responses, because both directions can carry a body. + +[cols="1a,3a"] +|=== +|Header|Purpose + +|`Content-Type` +|The media type of the body: `text/html`, `application/json`, +`image/png`, and so on. This is the single most important header for +interpreting the payload. + +|`Content-Length` +|The size of the body in bytes. Lets the receiver know exactly how much +data to expect. + +|`Content-Encoding` +|Any compression applied to the body, such as `gzip` or `br`. The +receiver must decompress before processing. + +|`Content-Language` +|The natural language of the body, such as `en` or `fr`. + +|`Transfer-Encoding` +|How the body is framed for transport. The value `chunked` means the +body arrives in pieces, each prefixed by its size. + +|=== + +When a server sends a compressed JSON response, the representation +headers make the whole arrangement explicit: + +[source] +---- +HTTP/1.1 200 OK +Content-Type: application/json; charset=utf-8 +Content-Encoding: gzip +Content-Length: 1843 +Transfer-Encoding: chunked +---- + +The client sees `Content-Type` and knows it is JSON. It sees +`Content-Encoding: gzip` and knows it must decompress. It sees +`Transfer-Encoding: chunked` and knows the body will arrive in +pieces rather than all at once. Every step is unambiguous. + +=== General Headers + +Some headers belong to the message as a whole rather than to the +request, the response, or the body. These are general headers -- +they can appear in either direction and affect how the message is +transported or processed. + +[cols="1a,3a"] +|=== +|Header|Purpose + +|`Connection` +|Controls whether the TCP connection stays open after the response. +`keep-alive` is the HTTP/1.1 default; `close` signals that the +connection should be torn down. + +|`Via` +|Records the intermediate proxies or gateways a message has passed +through. Each hop appends its own entry. + +|`Cache-Control` +|Directives for how caches along the path should handle the message. +This header is so important it gets its own section below. + +|`Date` +|The timestamp of the message, used for caching calculations and log +correlation. + +|=== + +== Conditional Headers + +One of the most practical things headers do is make requests +_conditional_. A conditional request says: "Only give me this +resource if something has changed." This avoids transferring data the +client already has, saving bandwidth and time. + +The mechanism relies on validators -- values that identify a specific +version of a resource. HTTP supports two kinds: + +**Last-Modified dates.** The server includes a `Last-Modified` header +in the response, telling the client when the resource was last changed: + +[source] +---- +HTTP/1.1 200 OK +Last-Modified: Wed, 15 Jan 2026 08:30:00 GMT +Content-Type: text/html +Content-Length: 5120 + +... +---- + +When the client requests the resource again, it includes the date in an +`If-Modified-Since` header: + +[source] +---- +GET /index.html HTTP/1.1 +Host: www.example.com +If-Modified-Since: Wed, 15 Jan 2026 08:30:00 GMT +---- + +If the resource has not changed, the server responds with +`304 Not Modified` and no body -- the client uses its cached copy. +If it _has_ changed, the server sends the full `200 OK` response +with the updated resource. + +**Entity tags.** An `ETag` is an opaque identifier -- often a hash -- +that the server assigns to a specific version of a resource: + +[source] +---- +HTTP/1.1 200 OK +ETag: "a3f2b7c" +Content-Type: application/json +Content-Length: 892 + +{"products": [...]} +---- + +The client echoes it back in an `If-None-Match` header: + +[source] +---- +GET /api/products HTTP/1.1 +Host: www.example.com +If-None-Match: "a3f2b7c" +---- + +If the ETag still matches, the server returns `304 Not Modified`. +ETags are more precise than dates because they change whenever the +content changes, regardless of the clock. They also handle the edge +case where a resource is modified, then reverted -- the date changes +but the content is the same. An ETag catches that. + +[cols="1a,2a"] +|=== +|Request Header|Meaning + +|`If-Modified-Since` +|Send the resource only if it changed after this date. + +|`If-None-Match` +|Send the resource only if its ETag differs from this value. + +|`If-Match` +|Proceed only if the resource's current ETag matches. Used to prevent +overwriting someone else's changes during an update. + +|`If-Unmodified-Since` +|Proceed only if the resource has not changed since this date. + +|=== + +Conditional requests are the foundation of efficient caching. Without +them, every page view would require a full download even if nothing +changed. + +== Cache-Control + +Caching is one of the most consequential behaviors that headers govern. +The `Cache-Control` header tells every cache along the path -- the +browser, any proxy, any CDN -- how to store, reuse, and validate +responses. Getting caching right can make a site feel instant; getting +it wrong can serve stale data or defeat the cache entirely. + +`Cache-Control` carries one or more comma-separated directives: + +[source] +---- +Cache-Control: public, max-age=86400, must-revalidate +---- + +This says: any cache may store the response (`public`), it is fresh +for 86,400 seconds -- one day (`max-age`), and after that the cache +must check with the server before reusing it (`must-revalidate`). + +The most important directives: + +[cols="1a,3a"] +|=== +|Directive|Meaning + +|`public` +|Any cache (browser or shared proxy) may store the response. Even +responses that would normally be private become cacheable. + +|`private` +|Only the user's browser may cache this response. Shared caches like +CDNs must not store it. Useful for responses tailored to one user. + +|`no-cache` +|A cache may store the response, but must revalidate with the server +before every reuse. This guarantees freshness without giving up caching +entirely. + +|`no-store` +|The response must not be stored by any cache at all. Used for +sensitive data like banking pages or personal health records. + +|`max-age=The requested resource is only available in Japanese.
+ +---- + +In practice, many servers choose to send the closest available variant +anyway, reasoning that _something_ is better than an error page. The +specification permits this--`406` is a tool, not a mandate. + +== The Vary Header + +Content negotiation creates a complication for caches. A cache stores a +response keyed by its URL. But if the same URL can produce different +responses depending on `Accept-Language`, a cache that blindly serves +the first response it stored will send French pages to English speakers. + +The `Vary` header solves this. The server includes it in the response to +tell caches which request headers influenced the choice of variant: + +[source] +---- +HTTP/1.1 200 OK +Content-Type: text/html +Content-Language: fr +Vary: Accept-Language +---- + +This tells any cache: "I chose this response based on the +`Accept-Language` header. If a future request has a different +`Accept-Language` value, do not serve this cached copy--ask the origin +server again." + +A `Vary` header can list multiple fields: + +[source] +---- +Vary: Accept-Language, Accept-Encoding +---- + +Caches that implement `Vary` correctly store multiple variants of the +same URL and match incoming requests against the stored request headers. +Getting `Vary` right is essential for any system that sits between +clients and origin servers--proxies, CDNs, and reverse caches all +depend on it. + +== Content Encoding + +Content negotiation decides _what_ to send. Content encoding decides +_how to compress it_ before it travels across the network. + +When a server has a large HTML page, sending it uncompressed wastes +bandwidth and time. If the client supports compression, the server can +encode the body with an algorithm like gzip, Brotli, or deflate, and +the client decompresses it on arrival. The original media type does not +change--a gzip-compressed HTML page is still `text/html`. Only the +transport representation changes. + +=== The Content-Encoding Process + +The flow is straightforward: + +. The client sends `Accept-Encoding` listing the algorithms it supports. +. The server picks one (or none) and compresses the body. +. The server adds a `Content-Encoding` header naming the algorithm. +. The client reads `Content-Encoding`, decompresses, and processes the + original content. + +[source] +---- +GET /report.html HTTP/1.1 +Host: www.example.com +Accept-Encoding: gzip, br +---- + +[source] +---- +HTTP/1.1 200 OK +Content-Type: text/html +Content-Encoding: gzip +Content-Length: 3907 + +<...3907 bytes of gzip-compressed HTML...> +---- + +The `Content-Length` reflects the compressed size, not the original. +The `Content-Type` still says `text/html` because that is what the +body _is_ once decompressed. + +=== Common Content-Encoding Algorithms + +[cols="1a,3a"] +|=== +|Token|Description + +|`gzip` +|The most widely supported algorithm. Based on the DEFLATE algorithm +wrapped in the gzip file format. Virtually every HTTP client and server +understands it. + +|`deflate` +|Raw DEFLATE compression in the zlib format. Less common than gzip in +practice due to historical ambiguity in implementations. + +|`br` +|Brotli, a newer algorithm developed by Google. Achieves better +compression ratios than gzip, especially for text. Supported by all +modern browsers, typically only over HTTPS. + +|`identity` +|No encoding applied. This token exists so clients can explicitly +express a preference for uncompressed content using quality values. + +|=== + +A client that wants to explicitly reject uncompressed responses can +send: + +[source] +---- +Accept-Encoding: gzip;q=1.0, identity;q=0.0 +---- + +If the server cannot compress the response, it should send a +`406 Not Acceptable` rather than ignore the prohibition--though, again, +real-world servers vary in how strictly they follow this. + +== Transfer Encoding + +Content encoding compresses the _payload_. Transfer encoding changes +how the _message_ is delivered. The distinction matters: content +encoding is about the resource, transfer encoding is about the +transport. + +The primary transfer encoding in HTTP/1.1 is _chunked encoding_. It +exists to solve a specific problem: how do you send a response when +you do not know its total size in advance? + +=== The Problem + +Normally, a server declares the body size in `Content-Length` so the +client knows when the body ends. But if the server is generating +content dynamically--streaming search results, compressing on the fly, +assembling a page from multiple database queries--it may not know the +total size until it is finished. Without `Content-Length`, and on a +persistent connection, the client has no way to tell where one response +ends and the next begins. + +=== Chunked Encoding + +Chunked transfer encoding breaks the body into a series of chunks, each +preceded by its size in hexadecimal. A zero-length chunk signals the +end of the body: + +[source] +---- +HTTP/1.1 200 OK +Content-Type: text/plain +Transfer-Encoding: chunked + +1a +We hold these truths to be +1c +self-evident, that all men +0 + +---- + +Each chunk begins with a line containing the chunk size (in hex), +followed by a CRLF, then that many bytes of data, then another CRLF. +The final chunk has a size of `0`, and after it the body is complete. + +This mechanism lets the server begin transmitting before the entire +response is generated, reducing latency for the client. It also +preserves persistent connections--the client reads chunks until it +sees the terminating zero-length chunk, then it knows the next bytes +on the connection belong to a new response. + +=== Combining Content and Transfer Encoding + +Content encoding and transfer encoding can be applied together. A +server might gzip-compress a dynamically generated HTML page and then +send the compressed data in chunks: + +[source] +---- +HTTP/1.1 200 OK +Content-Type: text/html +Content-Encoding: gzip +Transfer-Encoding: chunked + +a3f +<...chunk of gzip-compressed data...> +7b2 +<...another chunk...> +0 + +---- + +The client first reassembles the chunks, then decompresses the gzip +payload, and finally processes the HTML. The two encodings serve +different purposes and are reversed in opposite order: transfer +encoding is unwrapped first, content encoding second. + +== Character Sets + +Text-based media types carry an optional `charset` parameter on the +`Content-Type` header that tells the client how to decode bytes into +characters: + +[source] +---- +Content-Type: text/html; charset=utf-8 +---- + +Without this parameter, the client must guess the encoding--and guessing +is a reliable source of garbled text. UTF-8 has become the dominant +encoding on the web, handling virtually every script in use today. Older +encodings like `iso-8859-1` (Latin-1) still appear, particularly on +legacy systems. + +Clients can declare character-set preferences in the `Accept-Charset` +header, but modern practice has largely moved past this. Most clients +support UTF-8 and most servers send it. The header remains in the +specification for completeness, but you will rarely need to set it +explicitly. + +== A Complete Negotiated Exchange + +Here is an exchange that exercises several negotiation mechanisms at +once. The client is a browser in France requesting a documentation page: + +[source] +---- +GET /guide/getting-started HTTP/1.1 +Host: docs.example.com +Accept: text/html;q=1.0, application/json;q=0.5 +Accept-Language: fr;q=1.0, en;q=0.7 +Accept-Encoding: gzip, br +---- + +The server has a French HTML variant and decides to compress it with +Brotli: + +[source] +---- +HTTP/1.1 200 OK +Content-Type: text/html; charset=utf-8 +Content-Language: fr +Content-Encoding: br +Content-Length: 8421 +Vary: Accept-Language, Accept-Encoding + + + +... +---- + +The response headers tell the full story: the body is HTML in UTF-8 +(`Content-Type`), written in French (`Content-Language`), compressed +with Brotli (`Content-Encoding`), and 8421 bytes in compressed form +(`Content-Length`). The `Vary` header warns caches that both language +and encoding influenced the choice, so future requests with different +values for those headers need a fresh lookup. + +The client decompresses the Brotli payload and renders the French HTML +page. The entire negotiation--language selection, format preference, +compression--happened in a single round-trip, guided entirely by headers. diff --git a/doc/modules/ROOT/pages/2.http-tutorial/2h.connection-management.adoc b/doc/modules/ROOT/pages/2.http-tutorial/2h.connection-management.adoc new file mode 100644 index 00000000..ce3e7f97 --- /dev/null +++ b/doc/modules/ROOT/pages/2.http-tutorial/2h.connection-management.adoc @@ -0,0 +1,439 @@ +// +// Copyright (c) 2025 Vinnie Falco (vinnie.falco@gmail.com) +// +// Distributed under the Boost Software License, Version 1.0. (See accompanying +// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) +// +// Official repository: https://github.com/cppalliance/http +// + += Connection Management + +Opening a TCP connection is like dialing a phone number before you can +speak. There is a pause--a handshake--where both sides agree that a +line is open. In the early days of the Web, every single HTTP request +paid that cost: dial, say one thing, hang up, dial again, say the next +thing. A page with ten images meant eleven phone calls. The Web +worked, but the wasted time was enormous. Connection management is the +story of how HTTP learned to keep the line open, send more per call, +and eventually carry entire conversations over a single wire. + +== The Cost of a New Connection + +Before any HTTP message can travel, the client and server must +establish a TCP connection through a three-way handshake: + +. The client sends a *SYN* packet. +. The server responds with *SYN+ACK*. +. The client replies with *ACK*. + +Only after this exchange can data flow. The handshake adds one full +round-trip of latency before the first byte of the request is even +sent. Between New York and London, that round-trip takes roughly 56 +milliseconds over fiber. For a small resource--a 304 Not Modified +response, a tiny icon--the handshake can consume more than half the +total time. + +[source] +---- +Client Server + |--- SYN ----------------------->| + |<------------- SYN+ACK --------| + |--- ACK ----------------------->| + |--- GET /index.html HTTP/1.1 -->| + |<------------ HTTP/1.1 200 OK --| +---- + +TCP also starts slowly on purpose. A new connection uses _slow start_, +throttling the amount of data in flight until the network proves it +can handle more. Each successful acknowledgment doubles the sender's +congestion window. A fresh connection cannot send data at full speed; +it needs time to ramp up. A connection that has already exchanged a +modest amount of data is significantly faster than a new one. + +These two costs--handshake delay and slow start--make opening a new +connection surprisingly expensive. Every optimization in this section +exists to avoid paying them more than necessary. + +== HTTP/1.0: One Request, One Connection + +The original HTTP/1.0 protocol treated every request as an isolated +event. The client opened a TCP connection, sent one request, received +one response, and the connection was torn down. Loading a web page +with an HTML document and three images required four separate TCP +connections: + +[source] +---- +[connect] GET /page.html → 200 OK [close] +[connect] GET /logo.png → 200 OK [close] +[connect] GET /photo.jpg → 200 OK [close] +[connect] GET /style.css → 200 OK [close] +---- + +Each connection paid the handshake cost. Each started with a fresh +slow-start window. As web pages grew richer--dozens of images, +stylesheets, and scripts--the accumulated overhead became the dominant +source of latency. Users stared at blank screens while their browsers +quietly opened and closed connections. + +== Persistent Connections + +The fix was obvious: keep the connection open. Rather than hanging up +after each response, the client and server could reuse the same TCP +connection for multiple requests. + +HTTP/1.0 introduced this informally through a `Connection: +Keep-Alive` header. If the client included this header, and the server +echoed it back, the connection stayed open after the response: + +[source] +---- +GET /page.html HTTP/1.0 +Host: www.example.com +Connection: Keep-Alive + +---- + +[source] +---- +HTTP/1.0 200 OK +Content-Type: text/html +Content-Length: 3104 +Connection: Keep-Alive + +... +---- + +Both sides had to agree. If the server did not return `Connection: +Keep-Alive`, the client assumed the connection would close. Every +request that wanted persistence had to ask for it explicitly. + +HTTP/1.1 reversed the default. Persistent connections became automatic. +An HTTP/1.1 connection stays open after every response unless one side +explicitly signals otherwise with `Connection: close`: + +[source] +---- +GET /style.css HTTP/1.1 +Host: www.example.com +Connection: close + +---- + +This single change eliminated enormous overhead. With connection reuse, +the handshake cost is paid once, TCP slow start ramps up once, and all +subsequent requests on that connection benefit from a warmed-up pipe. +For a page requiring N resources from the same server, persistent +connections save (N-1) round trips--often seconds of real-world +latency. + +Either side can close a persistent connection at any time, even without +sending `Connection: close` first. Servers close idle connections to +free resources. Clients close connections they no longer need. The +protocol requires that both sides tolerate unexpected closes and be +prepared to retry requests. + +== The Connection Header + +The `Connection` header controls per-hop connection behavior. It is +a hop-by-hop header, meaning it applies only to the immediate link +between two participants and must not be forwarded by proxies. + +The header carries three kinds of values: + +* **`close`** -- signals that the connection should be shut down after + the current request/response. +* **`Keep-Alive`** -- in HTTP/1.0, explicitly requests persistence. + Unnecessary in HTTP/1.1, where persistence is the default. +* **Header field names** -- lists other hop-by-hop headers that must + be removed before forwarding. This "protects" headers from + accidental propagation through proxy chains. + +[source] +---- +Connection: close +---- + +[source] +---- +Connection: Keep-Alive +Keep-Alive: timeout=30, max=100 +---- + +The `Keep-Alive` header (when present alongside `Connection: +Keep-Alive`) can include hints about how long the sender expects to +hold the connection open and how many more requests it anticipates. +These are advisory, not guarantees. A server that says `timeout=30` +may still close the connection after five seconds if it needs the +resources. + +A critical rule for intermediaries: proxies must parse the +`Connection` header, remove it and every header it names, and then +forward the message. A proxy that blindly relays `Connection: +Keep-Alive` to an origin server creates a well-known failure called +the "dumb proxy" problem--the server believes the proxy wants +persistence, the proxy does not understand persistence, and the +connection hangs. + +== Parallel Connections + +While persistent connections eliminated repeated handshakes, they did +not solve another problem: serialization. On a single persistent +connection, each request had to wait for the previous response to +finish. If the server took 500 milliseconds to generate one resource, +everything behind it queued up. + +Browsers worked around this by opening multiple TCP connections in +parallel. Instead of one pipe, they opened several--typically six per +host in modern browsers: + +[source] +---- +Connection 1: GET /page.html → 200 OK → GET /app.js → 200 OK +Connection 2: GET /style.css → 200 OK → GET /font.woff → 200 OK +Connection 3: GET /logo.png → 200 OK +Connection 4: GET /hero.jpg → 200 OK +Connection 5: GET /icon.svg → 200 OK +Connection 6: GET /data.json → 200 OK +---- + +Parallel connections overlap the delays. While one connection waits +for a response, others are already transferring data. Users see +images loading simultaneously across the page, which feels faster even +when the wall-clock time is similar. + +The downsides are real: + +* Each connection pays its own handshake and slow-start costs. +* Six connections consume six times the memory and CPU on both client + and server. +* Under limited bandwidth, parallel streams compete for the same + pipe, and each one moves proportionally slower. +* A hundred users each opening six connections means 600 connections + for the server to manage. + +Parallel connections are a pragmatic workaround, not an elegant +solution. They exist because HTTP/1.x lacks multiplexing. + +== Pipelining + +HTTP/1.1 introduced _pipelining_ as an attempt at better concurrency +within a single connection. With pipelining, a client can send several +requests in a row without waiting for responses: + +[source] +---- +Client Server + |--- GET /a.html ----------------->| + |--- GET /b.css ------------------>| + |--- GET /c.js ------------------->| + |<------------- 200 OK (a.html) ---| + |<------------- 200 OK (b.css) ----| + |<------------- 200 OK (c.js) -----| +---- + +By dispatching requests early, the client eliminates the dead time +between sending a request and receiving the previous response. The +server can even begin processing requests in parallel internally. + +But pipelining has a fatal constraint: **responses must arrive in the +same order as the requests.** HTTP/1.1 messages carry no sequence +numbers, so neither side can match responses to requests if they +arrive out of order. This requirement creates _head-of-line blocking_. + +=== Head-of-Line Blocking + +Suppose the client pipelines three requests and the server can +generate the second and third responses quickly, but the first takes +a long time. The fast responses must wait, fully buffered, until the +slow one finishes: + +[source] +---- +Client Server + |--- GET /slow ------------------->| + |--- GET /fast1 ------------------->| fast1 ready, but must wait + |--- GET /fast2 ------------------->| fast2 ready, but must wait + | | ...processing /slow... + |<------------ 200 OK (/slow) -----| + |<------------ 200 OK (/fast1) ----| + |<------------ 200 OK (/fast2) ----| +---- + +A single slow response blocks everything behind it. The server wastes +memory buffering completed responses. If the connection fails +mid-pipeline, the client must re-request everything it has not +received--possibly triggering duplicate processing for non-idempotent +requests. + +Additional problems made pipelining fragile in practice: + +* Many proxies and intermediaries did not support it correctly. +* Servers had to buffer potentially large responses out of order. +* Detecting whether an intermediary supports pipelining was unreliable. +* Only idempotent requests (GET, HEAD) were safe to pipeline. + +Due to these issues, browser support for pipelining remained limited. +Most browsers shipped with it disabled by default. The idea was +sound--eliminating round-trip delays is always valuable--but the +execution within HTTP/1.1's constraints was impractical. The real +solution required a protocol-level change. + +== Closing Connections + +Connection management is not just about opening and keeping +connections alive. Knowing _when_ and _how_ to close them correctly +is equally important. + +=== Signaled Close + +Either party can signal its intent to close by including `Connection: +close` in a request or response. After the client sends this header, +it must not send additional requests on that connection. After the +server sends it, the client knows the connection will end once the +response is fully received. + +=== Idle Timeouts + +Servers close persistent connections that sit idle too long. A +connection consuming resources but carrying no traffic is a liability. +Typical idle timeouts range from 5 to 120 seconds, depending on the +server's load and configuration. Clients must be prepared for the +connection to vanish at any time and should reopen a new one when +needed. + +=== Graceful Close + +TCP connections are bidirectional--each side has an independent +input and output channel. A _full close_ shuts down both channels +at once. A _half close_ shuts down only one, leaving the other +open. + +The HTTP specification recommends that applications perform a _graceful +close_ by first closing their output channel (signaling "I have +nothing more to send") and then waiting for the peer to close its +output channel. This avoids a dangerous race condition: if you close +the input channel while the peer is still sending, the operating +system may issue a TCP RST (reset), which erases any data the peer +had buffered but you had not yet read. + +This matters most with pipelined connections. Imagine you pipelined +ten requests and have received responses for the first eight, sitting +unread in your buffer. Now request eleven arrives at a server that +has already decided to close. The server's RST wipes your buffer, and +you lose the eight perfectly good responses you already had. + +The graceful close protocol: + +. Close your output channel (half close). +. Continue reading from the input channel. +. When the peer also closes, or a timeout expires, close fully. + +=== Retries and Idempotency + +When a connection closes unexpectedly, the client must decide whether +to retry the request. For idempotent methods--GET, HEAD, PUT, +DELETE--retrying is safe because repeating the operation produces +the same result. For non-idempotent methods like POST, retrying risks +duplication. This is why browsers warn before resubmitting a form: +the connection may have closed after the server processed the request +but before the response arrived. + +== How HTTP/2 Changed the Picture + +HTTP/2 addressed HTTP/1.1's connection limitations at the protocol +level. Rather than opening multiple TCP connections or attempting +fragile pipelining, HTTP/2 introduced _multiplexing_ over a single +connection. + +An HTTP/2 connection breaks messages into small binary _frames_, +each tagged with a stream identifier. Multiple streams flow +concurrently over the same TCP connection, and their frames can +interleave freely: + +[source] +---- +Single TCP connection: + [stream 1: request ] + [stream 3: request ] + [stream 1: response frame 1] + [stream 3: response frame 1] + [stream 1: response frame 2] + [stream 3: response frame 2] +---- + +Because each frame identifies its stream, responses can arrive in any +order and be reassembled correctly. Head-of-line blocking at the HTTP +layer is eliminated. A slow response on stream 1 no longer blocks a +fast response on stream 3. + +The consequences for connection management are dramatic: + +* **One connection per origin.** HTTP/2 uses a single TCP connection + between client and server, eliminating the overhead of multiple + parallel connections. +* **No domain sharding.** With multiplexing, the workaround of + splitting resources across subdomains becomes counterproductive--it + prevents the protocol from prioritizing and compressing effectively. +* **Stream prioritization.** The client can indicate which streams + matter most, allowing the server to allocate bandwidth intelligently. +* **Flow control.** Both per-stream and per-connection flow control + prevent any one stream from monopolizing the pipe. + +However, HTTP/2 still runs over TCP, which imposes its own ordering +constraints. If a TCP packet is lost, the entire connection stalls +until that packet is retransmitted--even streams whose data was not +in the lost packet. This is _transport-layer_ head-of-line blocking, +and it is the problem that HTTP/3, built on QUIC over UDP, was +designed to solve. But that is a story for a later section. + +== Practical Summary + +The evolution of HTTP connection management follows a clear arc toward +doing more with fewer connections: + +[cols="1,2,3"] +|=== +|Era |Strategy |Connection behavior + +|**HTTP/1.0** +|One request per connection +|Open, request, respond, close. Expensive and wasteful. + +|**HTTP/1.0+** +|Keep-Alive +|Opt-in persistence via `Connection: Keep-Alive`. A significant +improvement, but both sides had to agree explicitly. + +|**HTTP/1.1** +|Persistent by default +|Connections stay open unless `Connection: close` is sent. Pipelining +attempted but largely failed due to head-of-line blocking. + +|**Browsers** +|Parallel connections +|Up to six TCP connections per host to work around HTTP/1.x +serialization. Effective but resource-heavy. + +|**HTTP/2** +|Multiplexed streams +|One connection per origin. Binary framing eliminates HTTP-layer +head-of-line blocking. Stream priorities and flow control replace +the need for multiple connections. +|=== + +The lesson running through all of this is that connections are +expensive, and the protocol's history is a series of increasingly +elegant solutions to that single economic fact. Keep connections open. +Reuse them. And when one connection is not enough, multiplex rather +than multiply. + +== Next Steps + +You now understand how HTTP connections are opened, reused, and closed, +and why each generation of the protocol refined the strategy. The next +section covers the mechanism that prevents the Web from doing the same +work twice: + +* xref:2.http-tutorial/2i.caching.adoc[Caching] -- how clients and servers avoid redundant transfers diff --git a/doc/modules/ROOT/pages/2.http-tutorial/2i.caching.adoc b/doc/modules/ROOT/pages/2.http-tutorial/2i.caching.adoc new file mode 100644 index 00000000..839b7d61 --- /dev/null +++ b/doc/modules/ROOT/pages/2.http-tutorial/2i.caching.adoc @@ -0,0 +1,497 @@ +// +// Copyright (c) 2025 Vinnie Falco (vinnie.falco@gmail.com) +// +// Distributed under the Boost Software License, Version 1.0. (See accompanying +// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) +// +// Official repository: https://github.com/cppalliance/http +// + += Caching + +The fastest HTTP request is the one that never reaches the server. Every +time a browser fetches a stylesheet it already downloaded five seconds ago, +or a CDN re-asks an origin for a logo that has not changed in a year, the +network does work that produces no new information. The bytes travel the +same wires, consume the same bandwidth, and impose the same latency--all +to deliver an answer both sides already know. + +HTTP caching exists to eliminate this waste. A _cache_ stores a copy of a +response and reuses it for subsequent matching requests, avoiding the +round-trip to the origin server entirely when the stored copy is still +valid. The result is faster page loads, lower bandwidth costs, and reduced +server load. A single busy origin serving millions of users would collapse +under the weight of redundant traffic if caches did not absorb the vast +majority of it. + +The mechanism is deceptively simple in concept--save the response, serve it +again later--but the details matter. How long is a stored response usable? +How does a cache know when the original has changed? Who is allowed to +store what? HTTP answers these questions through a set of headers and rules +that give servers precise control over how their responses are cached, and +give caches the tools to serve content efficiently without ever delivering +stale data by accident. + +== Freshness: When a Stored Response is Good Enough + +A cached response does not stay valid forever. The server that generated +it knows how volatile its content is, and HTTP provides two ways for the +server to express this: a relative lifetime and an absolute expiration +date. + +=== Cache-Control: max-age + +The modern and preferred approach is the `Cache-Control: max-age` +directive. The value is the number of seconds the response may be +considered _fresh_ from the moment it was generated: + +[source] +---- +HTTP/1.1 200 OK +Content-Type: text/html +Cache-Control: max-age=3600 + + +... +---- + +This response tells any cache that stores it: "You may serve this copy +for the next 3600 seconds (one hour) without contacting me." During that +window the cache satisfies requests instantly--a _cache hit_. After the +window closes the stored copy becomes _stale_, and the cache must check +with the server before using it again. + +=== The Expires Header + +Before `Cache-Control` existed, HTTP/1.0 used the `Expires` header to +specify an absolute date and time after which the response should no +longer be considered fresh: + +[source] +---- +HTTP/1.1 200 OK +Content-Type: text/html +Expires: Thu, 01 Jan 2026 00:00:00 GMT + + +... +---- + +Absolute dates depend on the server's clock being accurate, which proved +unreliable in practice. If both `Expires` and `Cache-Control: max-age` +are present, `max-age` takes priority. New implementations should use +`max-age`. + +=== The Age Header + +When a shared cache (such as a CDN) stores a response and later serves it, +the `Age` header tells the next recipient how many seconds the response +has been sitting in that cache: + +[source] +---- +HTTP/1.1 200 OK +Content-Type: text/html +Cache-Control: max-age=3600 +Age: 1800 + + +... +---- + +A client receiving this response knows that 1800 of the original 3600 +seconds of freshness have already elapsed, leaving 1800 seconds of +remaining freshness. Without the `Age` header, downstream caches would +have no way to account for time spent in upstream caches. + +=== Heuristic Caching + +If a response carries no `Cache-Control` or `Expires` header at all, +caches do not simply refuse to store it. HTTP allows them to apply a +_heuristic_: if the `Last-Modified` header is present, a common rule of +thumb is to treat the response as fresh for roughly 10% of the time since +it was last modified. A page last modified a year ago might be cached for +about five weeks; a page modified yesterday, for about two hours. + +Heuristic caching is a sensible default, but it is unpredictable. Servers +that care about caching behavior should always include an explicit +`Cache-Control` header. + +== Validation: Checking Without Re-Downloading + +When a cached copy goes stale, the cache does not have to throw it away +and fetch the entire response from scratch. Instead it can ask the server: +"Has this resource changed since I last fetched it?" If the answer is no, +the server sends back a tiny `304 Not Modified` response with no body, +and the cache marks its existing copy as fresh again. This is called +_revalidation_, and it can save enormous amounts of bandwidth. + +HTTP supports two revalidation mechanisms, each based on a different kind +of identifier that the server attaches to the original response. + +=== Last-Modified and If-Modified-Since + +The simplest approach uses timestamps. The server includes a +`Last-Modified` header in the response: + +[source] +---- +HTTP/1.1 200 OK +Content-Type: text/html +Last-Modified: Mon, 15 Jan 2026 10:00:00 GMT +Cache-Control: max-age=3600 + + +... +---- + +When the cached copy becomes stale and a client requests the same +resource, the cache sends a _conditional request_ with an +`If-Modified-Since` header carrying the stored timestamp: + +[source] +---- +GET /index.html HTTP/1.1 +Host: www.example.com +If-Modified-Since: Mon, 15 Jan 2026 10:00:00 GMT +---- + +If the resource has not changed, the server responds: + +[source] +---- +HTTP/1.1 304 Not Modified +Cache-Control: max-age=3600 +---- + +No body is transferred. The cache refreshes the freshness lifetime of its +stored copy and serves it to the client. If the resource _has_ changed, +the server responds with a full `200 OK` and the new content. + +=== ETags and If-None-Match + +Timestamps have limitations. A file might be rewritten with identical +content, changing its modification date without changing its meaning. Or +changes might happen faster than the one-second granularity of HTTP dates. +_Entity tags_ (ETags) solve both problems. An ETag is an opaque identifier +--often a hash or version string--that the server generates for a specific +version of a resource: + +[source] +---- +HTTP/1.1 200 OK +Content-Type: text/html +ETag: "a1b2c3d4" +Cache-Control: max-age=3600 + + +... +---- + +When revalidating, the cache sends the stored ETag in an `If-None-Match` +header: + +[source] +---- +GET /index.html HTTP/1.1 +Host: www.example.com +If-None-Match: "a1b2c3d4" +---- + +If the server's current ETag for the resource matches, nothing has changed +and the server returns `304 Not Modified`. If the ETag differs, the server +returns the new content with a `200 OK` and a new ETag. + +When both `If-Modified-Since` and `If-None-Match` are present in the same +request, the ETag comparison takes precedence. Servers are encouraged to +send both `ETag` and `Last-Modified` in responses, because each serves +different consumers: ETags provide precise cache validation, while +`Last-Modified` is useful for crawlers, content-management systems, and +HTTP/1.0 caches that do not understand ETags. + +=== Weak Validators + +Sometimes a cosmetic change--a whitespace fix, a comment edit--should +not force every cache in the world to re-download the resource. HTTP +supports _weak validators_ for this purpose. A weak ETag is prefixed +with `W/`: + +[source] +---- +ETag: W/"v2.6" +---- + +A weak ETag signals that the resource is _semantically equivalent_ even +if the bytes are not identical. Caches can still use weak ETags for +revalidation, but certain operations that require exact byte-level +matching (such as range requests) demand strong validators. + +== Cache-Control Directives + +The `Cache-Control` header is the primary tool for controlling caching +behavior. Directives can appear in both responses and requests, each +serving a different purpose. The most important response directives are +summarized below. + +=== Controlling Who May Cache + +[cols="2a,3a"] +|=== +|Directive|Meaning + +|`public` +|Any cache--browser, proxy, CDN--may store the response. This is the +default for most responses, but stating it explicitly can override +restrictions that would otherwise apply (for example, to responses that +required authentication). + +|`private` +|Only the end user's browser may store the response. Shared caches such +as proxies and CDNs must not. Use this for personalized content--a user's +account page, a shopping cart, anything tied to a session. + +|=== + +=== Controlling Storage and Reuse + +[cols="2a,3a"] +|=== +|Directive|Meaning + +|`no-store` +|The response must not be stored by any cache at all. Use this for +sensitive data that should never be written to disk--bank statements, +medical records, authentication tokens. + +|`no-cache` +|The response _may_ be stored, but must not be served to a client without +first revalidating with the origin server. Despite its misleading name, +`no-cache` does not prevent caching--it prevents _unvalidated_ reuse. + +|`max-age=