Skip to content

#1998 WARC writer: WARC-Protocol header to follow WARC field proposals - #2034

Open
sebastian-nagel wants to merge 1 commit into
apache:mainfrom
sebastian-nagel:sc-1998-warc-protocol-header
Open

#1998 WARC writer: WARC-Protocol header to follow WARC field proposals#2034
sebastian-nagel wants to merge 1 commit into
apache:mainfrom
sebastian-nagel:sc-1998-warc-protocol-header

Conversation

@sebastian-nagel

Copy link
Copy Markdown
Contributor

This PR addresses #1998.

OkHttp protocol:

  • add protocol response header key _cipher_suites_ to hold the SSL/TLS Cipher suite separate from _protocol_versions_.

WARC writer:

  • add WARC header WARC-Cipher-Suite
  • split multiple values in WARC-Protocol header and repeat header

Example WARC header with this change applied:

WARC/1.0
WARC-Record-ID: <urn:uuid:64d3f1a3-2293-4581-b676-a4d715b65ebc>
Content-Length: 30510
WARC-Date: 2026-07-29T17:20:01Z
WARC-Type: response
WARC-IP-Address: 2620:cb:2000::1
WARC-Target-URI: https://commoncrawl.org/blog/common-crawl-foundation-at-lrec-2026
Content-Type: application/http; msgtype=response
WARC-Protocol: h2
WARC-Protocol: TLS_1_3
WARC-Cipher-Suite: TLS_AES_128_GCM_SHA256
WARC-Payload-Digest: sha1:72ZYLF3J57X4GOO6UNSPQHE366QPTWVL
WARC-Block-Digest: sha1:M34FZULWWPMGUNTPJZLUGUPQ7TE5G5JS

HTTP/1.1 200 
date: Wed, 29 Jul 2026 17:20:01 GMT
...

(fixes apache#1998)

OkHttp protocol: add protocol response header key `_cipher_suites_`
to hold the SSL/TLS Cipher suite separate from `_protocol_versions_`.

WARC writer:
1. add WARC header `WARC-Cipher-Suite`
2. split multiple values in `WARC-Protocol` header and
   repeat header
.header(ProtocolResponse.RESPONSE_IP_KEY, ipAddress)
.header(ProtocolResponse.REQUEST_TIME_KEY, Long.toString(startFetchTime))
.header(ProtocolResponse.PROTOCOL_VERSIONS_KEY, protocols.toString())
.header(ProtocolResponse.CIPHER_SUITES_KEY, cipherSuite)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens if cipherSuite is null?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARC writer checks for null anyway, because another protocol implementation could have been used which does not collect the necessary information. If it's not available, it does not add the WARC-Cipher-Suite header.

But I can change the code, so that no header is added, if no Cipher suite was used (connection over http://).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants