AS ISO 28500:2018 pdf download.Information and documentation WARC file format.
4 File and record model
A WARC format file is the simple concatenation of one or more WARC records. The first record usually describes the records to follow. In general, record content is either the direct result of a retrieval attempt (web pages, inline images, URL redirection information, DNS hostname lookup results, standalone files, etc.) or is synthesized material (e.g. metadata, transformed content) that provides additional information about archived content.
A WARC record shall consist of a record header followed by a record content block and two new lines. The WARC record header shall consist of one first line declaring the record to be in the WARC format with a given version number, then a variable number of line-oriented named fields terminated by a blank line. The WARC record header format shall follow the general rules of HTTP/1.1 [RFC2616j and (RFC5322] headers with one major exception: it shall also allow UTF-8 characters, as specified in [RFC3629].
The top-level view of a WARC file can be expressed in an ABNF grammar, reusing the augmented constructs defined in section 2.1 of HTTP/1.1 [RFC2616]. (In particular, note that to avoid the risk of confusion, where any WARC rule has the same name as an [RFC2616j rule, the definition here has been made the same, except in the case of the CHAR rule, which in WARC includes multibyte UTF8 characters.)
The record version shall appear first in every record and hence shall also begin the WARC file itself.
The WARC record relies heavily on named fields. Each named field consists of a name followed by a colon (“:“) and the field value. Field names are not case-sensitive. The field value may be preceded by any amount of linear white space (LWS), though a single space is preferred. Header fields can be extended over multiple lines by preceding each extra line with at least one space or tab character.
Named fields may appear in any order and field values may contain any UTF-8 character. Both definedfields and extension-fields follow the generic named-field format. Extension-fields may be used in extensions of the core format.
The record block shall contain octet content, interpreted based on the record type and other header values. All records shall include a Content-Length field to specify the length of the block.
Some record types (and possibly future record types) also define a payload, such as a meaningful subset of the block or content from a predecessor record. Some headers pertain to the payload of a record rather than the block directly.
For example, in a ‘response’ record with a content block consisting of HTTP headers and a data object, the payload would be the data object. All ‘response’, ‘resource’, ‘request’, ‘revisit’, ‘conversion’ and ‘continuation’ records may have a payload. All ‘warcinfo’ and ‘metadata’ records shall not have a payload.
Content matching the warc-file rule shall have the MIME content-type “application/warc”, as specified in Clause 8.
Content matching only the warc-fields rule is useful as a simple descriptive format, and has MIME content-type “application/warc-fields”, as specified in Clause 8.
New named fields and new records types may be defined in extensions of the core format. However, it is strongly recommended to discuss any addition to verify that a suitable field or type does not already exist to avoid collision. Discussion should notably be held within the IIPC. See Reference [11] for more information.
A ‘conversion’ record shall contain an alternative version of another record’s content that was created as the result of an archival process. Typically, this is used to hold content transformations that maintain viability of content after widely available rendering tools for the originally stored format disappear. As needed, the original content may be migrated (transformed) to a more viable format in order to keep the information usable with current tools while minimizing loss of information (intellectual content, look and feel, etc.). Any number of ‘conversion’ records may be created that reference a specific source record, which may itself contain transformed content. Each transformation should result in a freestanding, complete record, with no dependency on survival of the original record.
Wherever practical, a ‘conversion’ record should contain a WARC-Refers-To field so as to identify the prior material converted.AS ISO 28500 pdf download.