Added codecs: sha0, keccak, md6

Author: dhondta

README.md

@@ -377,8 +377,9 @@ This category also contains `ascii85`, `adobe`, `[x]btoa`, `zeromq` with the `ba
 - [X] `blake`: includes BLAKE2b and BLAKE2s (Python 3 only ; relies on `hashlib`)
 - [X] `crypt`: Unix's crypt hash for passwords (Python 3 and Unix only ; relies on `crypt`)
 - [X] `fnv`: aka Fowler-Noll-Vo ; includes FNV-1 and FNV-1a in 32, 64, 128, 256, 512 and 1024 bits
-- [X] `md`: aka Message Digest ; includes MD4 and MD5 (relies on `hashlib`)
-- [X] `sha`: aka Secure Hash Algorithms ; includes SHA1, 224, 256, 384, 512 (Python2/3) but also SHA3-224, -256, -384 and -512 (Python 3 only ; relies on `hashlib`)
+- [X] `keccak`: original Keccak (pre-SHA-3 padding, as used e.g. by Ethereum) in 224, 256, 384 and 512 bits
+- [X] `md`: aka Message Digest ; includes MD2, MD4 and MD5 (relies on `hashlib`), and MD6 (digest sizes multiple of 8, up to 512 bits)
+- [X] `sha`: aka Secure Hash Algorithms ; includes SHA0, SHA1, 224, 256, 384, 512 (Python2/3) but also SHA3-224, -256, -384 and -512 (Python 3 only ; relies on `hashlib`)
 - [X] `shake`: aka SHAKE hashing (Python 3 only ; relies on `hashlib`)
 
 > :warning: Hash functions are of course definitely **NOT** encoding functions ; they are implemented for convenience with the `.encode(...)` API from `codecs` and useful for chaning codecs.

docs/pages/enc/hashing.md

@@ -4,7 +4,7 @@
 
 ### BLAKE
 
-These one-way transformation functions all rely on the native [`hashlib`](https://docs.python.org/3/library/hashlib.html) library.
+These one-way transformation functions rely on the native [`hashlib`](https://docs.python.org/3/library/hashlib.html) library, except SHA-0 (withdrawn shortly after its publication and replaced by SHA-1), which is implemented in pure Python.
 
 **Codec** | **Conversions** | **Aliases** | **Comment**
 :---: | :---: | --- | ---
@@ -94,13 +94,31 @@ These one-way transformation functions are non-cryptographic hashes, used e.g. i
 
 -----
 
+### Keccak
+
+This one-way transformation function is the original Keccak, as submitted to the SHA-3 competition: it only differs from SHA-3 by the padding of the data (e.g. Ethereum uses Keccak-256).
+
+**Codec** | **Conversions** | **Aliases** | **Comment**
+:---: | :---: | --- | ---
+`keccak` | data --> Keccak(data, size) | `keccak224`, `keccak-256`, `keccak_512`, ... | parametrized ; *size* belongs to {224, 256, 384, 512} (default: 256)
+
+```python
+>>> codext.encode("This is a test string !", "keccak")
+'cee1238ea4e44163d88c21979e42fedb5451d60c1f97b0221f757c1e77b37883'
+>>> codext.encode("This is a test string !", "keccak-512")
+'d5cbccbd447103a06c2fd3fb3b566373395ce9c573015d83d5b75f518c92ca013fc73130bf0e941d29d2283176f1816f658064ff2ee09b470e314e1d61bda761'
+```
+
+-----
+
 ### Message Digest
 
 **Codec** | **Conversions** | **Aliases** | **Comment**
 :---: | :---: | --- | ---
 `md2` | data --> MD2(data) |  | 
 `md4` | data --> MD4(data) |  | relies on [`hashlib`](https://docs.python.org/3/library/hashlib.html)
 `md5` | data --> MD5(data) |  | relies on [`hashlib`](https://docs.python.org/3/library/hashlib.html)
+`md6` | data --> MD6(data, size) | `md6-128`, `md6_512`, ... | parametrized ; *size* is a multiple of 8 in [8, 512] (default: 256) ; unkeyed, with the default tree height (64)
 
 ```python
 >>> codext.encode("This is a test string !", "md2")
@@ -109,16 +127,21 @@ These one-way transformation functions are non-cryptographic hashes, used e.g. i
 'ee4170b214eaac5be6a13d64a31b60b3'
 >>> codext.encode("This is a test string !", "md5")
 '5ba93d5b8e8efd9135f0030c978dd64e'
+>>> codext.encode("This is a test string !", "md6")
+'dc29b9f3d9cd16cf27b71f915b386bdd8fc3daa5eb7476a3408442d52bf8d869'
+>>> codext.encode("This is a test string !", "md6-128")
+'091d9f116b997823999ac64cfe3bfbad'
 ```
 
 -----
 
 ### Secure Hash Algorithm
 
-These one-way transformation functions all rely on the native [`hashlib`](https://docs.python.org/3/library/hashlib.html) library.
+These one-way transformation functions rely on the native [`hashlib`](https://docs.python.org/3/library/hashlib.html) library, except SHA-0 (withdrawn shortly after its publication and replaced by SHA-1), which is implemented in pure Python.
 
 **Codec** | **Conversions** | **Aliases** | **Comment**
 :---: | :---: | --- | ---
+`sha0` | data --> SHA0(data) |  | 
 `sha1` | data --> SHA1(data) |  | 
 `sha224` | data --> SHA224(data) |  | 
 `sha256` | data --> SHA256(data) |  | 
@@ -130,6 +153,8 @@ These one-way transformation functions all rely on the native [`hashlib`](https:
 `sha512` | data --> SHA512(data) |  | 
 
 ```python
+>>> codext.encode("This is a test string !", "sha0")
+'2cffee460320a8c226b9a62819f9409367d97e1b'
 >>> codext.encode("This is a test string !", "shake_256_64")
 '01a14a746d7c1d28927fe6078fdb9dcc8fabc45da58b3d1af13175b6278a6e824241927c47b4c5ced2ff629833574c9d985410d97c5c3d54d0f15b548cf2713d'
 >>> codext.encode("This is a test string !", "sha224")

src/codext/hashing/__init__.py

@@ -2,6 +2,7 @@
 from .blake import *
 from .crypt import *
 from .fnv import *
+from .keccak import *
 from .md import *
 from .mmh3 import *
 from .sha import *

src/codext/hashing/keccak.py

@@ -0,0 +1,87 @@
+# -*- coding: UTF-8 -*-
+"""Keccak Codecs - string hashing with Keccak.
+
+Keccak was designed by Guido Bertoni, Joan Daemen, Michaël Peeters and Gilles Van Assche, and won the SHA-3 competition
+ in 2012. It is a sponge construction over the Keccak-f[1600] permutation, whose capacity is twice the digest size. The
+ SHA-3 standard (FIPS 202, provided by hashlib) differs from the original submission only in the padding of the data
+ (domain separation bits "01" added before the "10*1" padding). The original Keccak, implemented here, is still used
+ e.g. by Ethereum (Keccak-256).
+
+The digest size (224, 256, 384 or 512 bits) is given with the codec name (e.g. "keccak-512") ; it defaults to 256. These
+ codecs return the hexadecimal digest (one-way), for use with other codecs in encoding chains.
+
+These codecs:
+- transform strings from str to str
+- transform strings from bytes to bytes
+- transform file content from str to bytes (write)
+
+Reference: https://keccak.team/files/Keccak-reference-3.0.pdf
+"""
+from ..__common__ import *
+
+
+M64 = (1 << 64) - 1
+
+
+def __rc(t):
+    # output bit of the LFSR x^8 + x^6 + x^5 + x^4 + 1 (see the reference, section 1.2)
+    r = 1
+    for _ in range(t % 255):
+        r <<= 1
+        if r & 0x100:
+            r ^= 0x171
+    return r & 1
+
+
+def __rot():
+    r, x, y = [0] * 25, 1, 0
+    for t in range(24):
+        r[x + 5 * y], x, y = (t + 1) * (t + 2) // 2 % 64, y, (2 * x + 3 * y) % 5
+    return r
+
+
+# round constants and rotation offsets (see the reference, sections 1.2 and 2.3)
+RC = [sum(__rc(j + 7 * i) << (2 ** j - 1) for j in range(7)) for i in range(24)]
+ROT = __rot()
+
+
+def __keccak_f(a):
+    rol = lambda v, n: (v << n | v >> (64 - n)) & M64 if n else v
+    for rc in RC:
+        # theta
+        c = [a[x] ^ a[x+5] ^ a[x+10] ^ a[x+15] ^ a[x+20] for x in range(5)]
+        d = [c[(x - 1) % 5] ^ rol(c[(x + 1) % 5], 1) for x in range(5)]
+        a = [v ^ d[i % 5] for i, v in enumerate(a)]
+        # rho and pi
+        r = [0] * 25
+        for i, v in enumerate(a):
+            x, y = i % 5, i // 5
+            r[y + 5 * ((2 * x + 3 * y) % 5)] = rol(v, ROT[i])
+        # chi and iota
+        a = [r[i] ^ (~r[(i + 1) % 5 + i - i % 5] & r[(i + 2) % 5 + i - i % 5]) for i in range(25)]
+        a[0] ^= rc
+    return a
+
+
+def keccak(n=""):
+    # a bad digest size is only reported when coding, as the lookup of the bare codec name must succeed (e.g. when
+    #  listing the encodings)
+    n = n or 256
+    def _hash(data, errors="strict"):
+        if n not in (224, 256, 384, 512):
+            raise LookupError("Bad parameter for encoding 'keccak': digest size must be 224, 256, 384 or 512")
+        # rate in bytes, then pad10*1 (without the SHA-3 domain separation bits)
+        d, rate = b(data), 200 - n // 4
+        d += b"\x01" + b"\x00" * (-(len(d) + 1) % rate)
+        d = d[:-1] + bytes([d[-1] | 0x80])
+        a = [0] * 25
+        for i in range(0, len(d), rate):
+            for j in range(rate // 8):
+                a[j] ^= int.from_bytes(d[i+8*j:i+8*j+8], "little")
+            a = __keccak_f(a)
+        # the digest is always shorter than the rate, hence a single squeeze
+        return b"".join(v.to_bytes(8, "little") for v in a)[:n // 8].hex(), len(data)
+    return _hash
+
+
+add("keccak", keccak, pattern=r"^keccak(?:[-_]?(\d+))?$", guess=None)

src/codext/hashing/md.py

@@ -3,8 +3,11 @@
 
 The Message Digest algorithms were designed by Ronald Rivest: MD2 (1989, implemented here as it is not provided by
  hashlib anymore), MD4 (1990, only when available in hashlib) and MD5 (1991), all producing 128-bit digests ; they are
- now considered broken for cryptographic use. These codecs return the hexadecimal digest (one-way), for use with other
- codecs in encoding chains.
+ now considered broken for cryptographic use. MD6 (2008, submitted to the SHA-3 competition) is unrelated to them: it
+ compresses blocks of 512 bytes into chaining values of 128 bytes along a 4-ary tree, which the last compression
+ truncates to the digest size (from 8 to 512 bits, given with the codec name, e.g. "md6-512" ; it defaults to 256). It
+ is implemented here in its default mode, unkeyed and with a tree height of 64. These codecs return the hexadecimal
+ digest (one-way), for use with other codecs in encoding chains.
 
 These codecs:
 - transform strings from str to str
@@ -14,7 +17,10 @@
 Reference: https://datatracker.ietf.org/doc/html/rfc1319
            https://datatracker.ietf.org/doc/html/rfc1320
            https://datatracker.ietf.org/doc/html/rfc1321
+           https://people.csail.mit.edu/rivest/pubs/RABCx08.pdf
 """
+from math import isqrt
+
 from ..__common__ import *
 
 
@@ -59,8 +65,59 @@ def md2(data):
     return "".join("{:02x}".format(x) for x in digest[:16])
 
 
+# MD6 constants (see the specification, section 9): Q (15 words of the fractional part of sqrt(6)), the tap positions,
+#  the shift amounts and the round constants (initial value and mask)
+M64 = (1 << 64) - 1
+MD6_Q = [isqrt(6 << 1920) >> (64 * (14 - i)) & M64 for i in range(15)]
+MD6_T = (17, 18, 21, 31, 67, 89)
+MD6_R = (10, 5, 13, 10, 11, 12, 2, 7, 14, 15, 7, 13, 11, 7, 6, 12)
+MD6_L = (11, 24, 9, 16, 15, 9, 27, 15, 6, 2, 29, 8, 15, 5, 31, 9)
+MD6_S0, MD6_SM = 0x0123456789abcdef, 0x7311c2812425cfa0
+
+
+def __md6_compress(n, rounds):
+    a, s, (t0, t1, t2, t3, t4, t5) = n[:], MD6_S0, MD6_T
+    for i in range(89, 89 + 16 * rounds):
+        j = (i - 89) % 16
+        if j == 0 and i > 89:
+            s = (s << 1 | s >> 63) & M64 ^ s & MD6_SM
+        x = s ^ a[i-t5] ^ a[i-t0] ^ (a[i-t1] & a[i-t2]) ^ (a[i-t3] & a[i-t4])
+        x ^= x >> MD6_R[j]
+        a.append((x ^ x << MD6_L[j]) & M64)
+    return a[-16:]
+
+
+def md6(n=""):
+    # a bad digest size is only reported when coding, as the lookup of the bare codec name must succeed (e.g. when
+    #  listing the encodings)
+    n = n or 256
+    def _hash(data, errors="strict"):
+        if n not in range(8, 513, 8):
+            raise LookupError("Bad parameter for encoding 'md6': digest size must be a multiple of 8 from 8 to 512")
+        # default mode (see the specification, section 2.4): no key, tree of height L=64 and r=40+d/4 rounds ; with
+        #  L=64, the sequential mode at level L+1 is never reached
+        m, rounds, lvl, key = b(data), 40 + n // 4, 0, [0] * 8
+        while True:
+            lvl += 1
+            nb, out = max(1, -(-len(m) // 512)), b""
+            for i in range(nb):
+                blk = m[512*i:512*(i+1)]
+                # control word: rounds, L, final compression flag (z), padding bits (p), key length and digest size
+                p = 8 * (512 - len(blk))
+                v = rounds << 48 | 64 << 40 | (nb == 1) << 36 | p << 20 | n
+                blk += b"\x00" * (p // 8)
+                w = [int.from_bytes(blk[k:k+8], "big") for k in range(0, 512, 8)]
+                out += b"".join(x.to_bytes(8, "big") for x in __md6_compress(MD6_Q + key + [lvl << 56 | i, v] + w,
+                                                                               rounds))
+            m = out
+            if nb == 1:
+                return m[-n // 8:].hex(), len(data)
+    return _hash
+
+
 add("md2", lambda s, error="strict": (md2(s), len(s)), guess=None)
 add("md5", lambda s, error="strict": (hashlib.new("md5", b(s)).hexdigest(), len(s)), guess=None)
 if "md4" in hashlib.algorithms_available:
     add("md4", lambda s, error="strict": (hashlib.new("md4", b(s)).hexdigest(), len(s)), guess=None)
 
+add("md6", md6, pattern=r"^md6(?:[-_]?(\d+))?$", guess=None)

src/codext/hashing/sha.py

@@ -1,22 +1,49 @@
 # -*- coding: UTF-8 -*-
 """SHA Hashing Codecs - string hashing with Secure Hash Algorithms.
 
-The Secure Hash Algorithms are published by the NIST: SHA-1 (1995, 160 bits, now broken), the SHA-2 family designed by
- the NSA (SHA-224, SHA-256, SHA-384 and SHA-512) and the SHA-3 family (SHA3-224 to SHA3-512, based on Keccak,
- standardised in 2015). These codecs return the hexadecimal digest (one-way), for use with other codecs in encoding
- chains.
+The Secure Hash Algorithms are published by the NIST: SHA-0 (1993, 160 bits, withdrawn soon after its publication and
+ implemented here as it is not provided by hashlib), SHA-1 (1995, which only adds a one-bit rotation in the message
+ schedule of SHA-0, now broken), the SHA-2 family designed by the NSA (SHA-224, SHA-256, SHA-384 and SHA-512) and the
+ SHA-3 family (SHA3-224 to SHA3-512, based on Keccak, standardised in 2015). These codecs return the hexadecimal digest
+ (one-way), for use with other codecs in encoding chains.
 
 These codecs:
 - transform strings from str to str
 - transform strings from bytes to bytes
 - transform file content from str to bytes (write)
 
-Reference: https://csrc.nist.gov/pubs/fips/180-4/upd1/final
+Reference: https://en.wikipedia.org/wiki/SHA-1#SHA-0
+           https://csrc.nist.gov/pubs/fips/180-4/upd1/final
            https://csrc.nist.gov/pubs/fips/202/final
 """
 from ..__common__ import *
 
 
+def sha0(data):
+    # same as SHA-1 (see FIPS 180-4, section 6.1.2), without the 1-bit left rotation in the message schedule
+    d, m = b(data), 0xffffffff
+    d += b"\x80" + b"\x00" * (-(len(d) + 9) % 64) + (8 * len(d)).to_bytes(8, "big")
+    h = [0x67452301, 0xefcdab89, 0x98badcfe, 0x10325476, 0xc3d2e1f0]
+    for i in range(0, len(d), 64):
+        w = [int.from_bytes(d[i+j:i+j+4], "big") for j in range(0, 64, 4)]
+        for t in range(16, 80):
+            w.append(w[t-3] ^ w[t-8] ^ w[t-14] ^ w[t-16])
+        a, b_, c, d_, e = h
+        for t in range(80):
+            if t < 20:
+                f, k = (b_ & c) | (~b_ & d_), 0x5a827999
+            elif t < 40:
+                f, k = b_ ^ c ^ d_, 0x6ed9eba1
+            elif t < 60:
+                f, k = (b_ & c) | (b_ & d_) | (c & d_), 0x8f1bbcdc
+            else:
+                f, k = b_ ^ c ^ d_, 0xca62c1d6
+            a, b_, c, d_, e = ((a << 5 | a >> 27) + f + e + k + w[t]) & m, a, (b_ << 30 | b_ >> 2) & m, c, d_
+        h = [(x + y) & m for x, y in zip(h, [a, b_, c, d_, e])]
+    return "".join(f"{x:08x}" for x in h)
+
+
+add("sha0", lambda s, error="strict": (sha0(s), len(s)), guess=None)
 add("sha1", lambda s, error="strict": (hashlib.sha1(b(s)).hexdigest(), len(s)), guess=None)
 add("sha224", lambda s, error="strict": (hashlib.sha224(b(s)).hexdigest(), len(s)), guess=None)
 add("sha256", lambda s, error="strict": (hashlib.sha256(b(s)).hexdigest(), len(s)), guess=None)

tests/test_manual.py

@@ -208,6 +208,39 @@ def test_codec_hash_functions(self):
         self.assertRaises(LookupError, codecs.encode, STR, "fnv1-16")
         self.assertRaises(LookupError, codecs.encode, STR, "fnv1a-2048")
         self.assertRaises(NotImplementedError, codecs.decode, STR, "fnv1a")
+        # SHA-0, Keccak (original padding) and MD6, with test vectors of the specifications and reference
+        #  implementations
+        for enc, vectors in {
+            'sha0': ["f96cea198ad1dd5617ac084a3d92c6107708c0ef", "0164b8a914cd2a5e74c4f7ff082c4d97f1edf880"],
+            'keccak|keccak256|keccak-256|keccak_256': [
+                "c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470",
+                "4e03657aea45a94fc7d47ba826c8d667c0d1e6e33a64a036ec44f58fa12d6c45"],
+            'keccak224': ["f71837502ba8e10837bdd8d365adb85591895602fc552b48b7390abd"],
+            'keccak-384': ["2c23146a63a29acf99e73b88f8c24eaa7dc60aa771780ccc006afbfa8fe2479b"
+                           "2dd2b21362337441ac12b515911957ff"],
+            'keccak_512': ["0eab42de4c3ceb9235fc91acffe746b29c29a8c366b7c60e4e67c466f36a4304"
+                           "c00fa9caf9d87976ba469bcbe06713b435f091ef2769fb160cdab33d3670680e"],
+            'md6|md6256|md6-256|md6_256': ["bca38b24a804aa37d821d31af00f5598230122c5bbfc4c4ad5ed40e4258f04ca",
+                                           "230637d4e6845cf0d092b558e87625f03881dd53a7439da34cf3b94ed0d8b2c5"],
+            'md6-224': ["d2091aa2ad17f38c51ade2697f24cafc3894c617c77ffe10fdc7abcb",
+                        "510c30e4202a5cdd8a4f2ae9beebb6f5988128897937615d52e6d228"],
+            'md6-512': ["6b7f33821a2c060ecdd81aefddea2fd3c4720270e18654f4cb08ece49ccb469f"
+                        "8beeee7c831206bd577f9f2630d9177979203a9489e47e04df4e6deaa0f8e0c0",
+                        "00918245271e377a7ffb202b90f3bda5477d8feab12d8a3a8994ebc55fe6e74c"
+                        "a8341520032eeea3fdef892f2882378f636212af4b2683ccf80bf025b7d9b457"],
+        }.items():
+            for e in enc.split("|"):
+                for s, r in zip(["", "abc"], vectors):
+                    self.assertEqual(codecs.encode(s, e), r)
+                    self.assertEqual(codecs.encode(b(s), e), b(r))
+        # MD6 over several levels of its tree (8192 bytes, i.e. 16 blocks, then 4, then 1)
+        self.assertEqual(codecs.encode(bytes(i & 0xff for i in range(8192)), "md6"),
+                         b"2a40338156df221b18b20e4003f51f61284cacd01935e9e87414e6ae40a6bd25")
+        self.assertEqual(len(codecs.encode(STR, "md6-8")), 2)
+        for e in ["keccak-128", "keccak_1024", "md6-0", "md6-12", "md6-520"]:
+            self.assertRaises(LookupError, codecs.encode, STR, e)
+        for e in ["sha0", "keccak", "md6"]:
+            self.assertRaises(NotImplementedError, codecs.decode, STR, e)
         if UNIX:
             try:
                 import crypt