Checksum Utils

22 min read

Six non-cryptographic checksums you can use to verify save-file integrity, detect corrupted downloads, and generate stable content fingerprints for asset deduplication.

Summary

The Scylla checksum utilities give you six small classes, all behind the same IChecksum interface: feed bytes in, read a number out, optionally reset and reuse. The classes cover the standard non-cryptographic checksum and hash algorithms that every game project ends up needing – ChecksumAdler32, ChecksumCRC16, ChecksumCRC32, ChecksumFletcher32, ChecksumXXHash32, and ChecksumXXHash64. There is no factory and no facade; the classes are the surface. The interface is the contract.

Each algorithm picks a different point on the speed-vs-strength-vs-output-width curve:

  • CRC32 (Cyclic Redundancy Check) produces the IEEE 802.3 result that GZip, ZIP, and PNG embed in their headers, so it is the right pick when interop with external tools matters.
  • Adler32 is the zlib companion checksum – smaller header, slightly weaker error detection, faster.
  • CRC16 is a compact 16-bit value useful for serial-style protocols.
  • Fletcher32 is the simpler two-accumulator design specified by ATM AAL5 and SCTP.
  • xxHash32 and xxHash64 are the modern non-cryptographic hashes that outpace CRC32 by a wide margin and produce well-distributed values suitable for hash-table keys; they also accept a seed so two consumers can derive distinct hashes from the same input.

All six algorithms answer the same question: “did the bytes I’m holding match the bytes someone else produced?” without paying for cryptographic strength. Authenticated encryption (see Crypto) covers the case where an attacker might rewrite the data and the checksum to match. These utilities cover everything else: detecting accidental corruption in save files, validating that two cache entries are identical, generating stable dictionary keys, and producing fingerprints over arbitrary byte streams. The uniform IChecksum interface means swapping algorithms is a one-line change.

Use the checksum utilities for:

  • Verifying that a save file or asset on disk has not been silently corrupted by a failed write, bit-flip, or partial copy.
  • Generating fast cache keys, content-addressable identifiers, or dictionary hash codes over byte buffers.
  • Detecting changed files between two builds – compute the checksum of each file; mismatches are the candidates for re-upload or re-extract.
  • Producing a quick “are these two buffers identical?” comparison without comparing every byte (a checksum mismatch is conclusive; a match is conclusive at the strength of the chosen algorithm).
  • Interoperating with external formats that already embed a known checksum: GZip and ZIP both store CRC32; PNG stores Adler32; many serial protocols carry CRC16.

Features

  • Uniform IChecksum interface. Three methods (Update(buffer), Update(buffer, offset, count), Reset) and one property (Value) cover every algorithm. Code typed against the interface switches algorithms in a single line.
  • Six shipped algorithms. ChecksumAdler32, ChecksumCRC16, ChecksumCRC32, ChecksumFletcher32, ChecksumXXHash32, ChecksumXXHash64. Each occupies a different point on the speed-vs-strength-vs-width curve; pick by purpose, not by name recognition.
  • Standard CRC32. IEEE 802.3 polynomial with reflected 0xEDB88320, initial 0xFFFFFFFF, final XOR 0xFFFFFFFF. The value matches what gzip -l, unzip -l, pngcheck, and Ethernet hardware produce; mismatches with third-party tools usually indicate a different polynomial variant.
  • CCITT CRC16 for protocol tags. Polynomial 0x1021, initial 0xFFFF, no final XOR. Compatible with HDLC, X.25, XMODEM, Bluetooth, and SD-card protocols when a 16-bit checksum budget is the constraint.
  • Adler32 for zlib compatibility. Two-accumulator design with prime modulus 65521, matching RFC 1950. Faster than CRC32 with weaker error detection on very short inputs.
  • Fletcher32 for ATM and SCTP. Two-accumulator design with modulus 65535 specified by ATM AAL5 and SCTP. Correctly handles odd-length inputs by deferring the trailing byte across Update calls.
  • xxHash32 and xxHash64 for fast modern hashing. The Yann Collet xxHash family with strong avalanche and good distribution, faster than CRC32 per byte, with seedable variants for hash-table hardening and multi-fingerprint derivation.
  • Streaming over chunked input. Update is incremental and produces the same value for update(a); update(b) as for update(a + b). You can hash a multi-gigabyte file in fixed-size chunks without loading it into memory.
  • Slice updates. Update(buffer, offset, count) checksums a sub-range of a larger array in place – no sub-array allocation, no manual copy step.
  • Instance reuse via Reset. Hash many files (or many independent inputs) from a single IChecksum instance by calling Reset between computations. Avoids the small per-class state allocation in tight loops.
  • Non-destructive Value reads. Reading Value does not mutate the running state; subsequent Update calls continue to accumulate into the same checksum, so the next Value read reflects the combined input.

Algorithms

A checksum (and the related non-cryptographic hash) is a function that compresses an arbitrary number of input bytes into a fixed-size summary number. Two identical inputs always produce the same output; two different inputs usually produce different outputs, but the output is small enough that two specific inputs sometimes coincide – a collision. Algorithms differ in how big the output is, how well they spread similar inputs across the output range, how strongly they detect particular kinds of corruption, and how quickly they run.

A few terms appear repeatedly in the algorithm descriptions below:

  • Output width is the number of bits in the result. A 16-bit checksum has 65,536 possible values; a 32-bit checksum has roughly 4.3 billion; a 64-bit hash has roughly 1.8 x 10^19. Wider outputs make accidental collisions less likely but cost more to store and transmit.
  • Polynomial is the mathematical divisor each CRC algorithm uses. Different polynomials produce different CRC values for the same input, so a CRC32 produced with the IEEE 802.3 polynomial (the one Scylla uses, the one GZip/ZIP use) will not match a CRC32 produced with the Castagnoli polynomial. Cross-tool comparisons only work when both ends use the same polynomial.
  • Seed is an optional starting value for the xxHash algorithms. Different seeds produce different hashes from the same input. Hash tables use seeded hashing to defend against collision-based denial-of-service attacks; content-addressable systems use it to derive multiple independent fingerprints from one buffer.
  • Avalanche describes how well a hash function spreads small changes in the input across the output. Good avalanche means flipping one input bit changes about half of the output bits. xxHash has strong avalanche; CRC has weaker avalanche but stronger detection of certain error patterns.
ClassOutput widthSeedable?SpeedBest for
ChecksumAdler3232 bitsNoVery fastThe zlib/PNG companion checksum (RFC 1950). Two-accumulator design with prime modulus 65521. Faster than CRC32; weaker error detection on short inputs. Use when interop with zlib output matters.
ChecksumCRC1616 bitsNoFastCompact 16-bit CRC using the CCITT polynomial 0x1021. Initial state 0xFFFF, no final XOR. Used by HDLC, X.25, XMODEM, Bluetooth, SD-card protocols. Use when a 16-bit budget is sufficient.
ChecksumCRC3232 bitsNoFastThe standard IEEE 802.3 CRC32 with reflected polynomial 0xEDB88320, initial state 0xFFFFFFFF, final XOR 0xFFFFFFFF. Matches GZip, ZIP, PNG, and Ethernet. The right pick for file integrity.
ChecksumFletcher3232 bitsNoVery fastTwo-accumulator design with modulus 65535, similar to Adler32 but with different error-detection properties. Specified by ATM AAL5 and SCTP. Choose for interop with those protocols; otherwise pick Adler32 or CRC32.
ChecksumXXHash3232 bitsYesFastestModern non-cryptographic hash by Yann Collet. Faster than CRC32 with strong avalanche and good distribution. Use for hash-table keys, fast cache fingerprints, and content-addressable storage on 32-bit platforms.
ChecksumXXHash6464 bitsYesFastestThe 64-bit variant of xxHash. On 64-bit platforms typically faster per byte than xxHash32. Use when collision probability matters – dedup, very large keysets, long-running content stores.

All six classes share the same lifecycle: construct, call Update one or more times with byte buffers (or slices via Update(buffer, offset, count)), read the result from Value, and optionally call Reset to reuse the instance for a new computation. The Value property returns a long regardless of the algorithm’s native width: 16-bit and 32-bit results are widened to the lower bits; the 64-bit xxHash result is reinterpreted as a signed long, so a (ulong)checksum.Value cast is required when the canonical unsigned value matters.

Recipes

These recipes each solve one practical game-development problem and are self-contained – you don’t need to read them in order. The first recipe introduces the IChecksum lifecycle (construct, update, read, reset) that every later recipe builds on, so start there if you haven’t used the API before.

Verify save-file integrity with CRC32

Problem. You’ve shipped a save system and you want a safety net: compute a checksum on write, embed it in the save file, and on load refuse to continue if the values don’t match. Silent corruption from a failed write, a power-loss mid-save, or a bit-flip on storage shouldn’t produce a game that loads into a broken state. CRC32 is the right pick here because it’s the same algorithm GZip, ZIP, and PNG use in their headers, so it’s the most universally understood reference value if you ever need to inspect a save externally.

Solution. The IChecksum lifecycle is: construct, call Update one or more times (multiple calls produce the same result as one call with the concatenated input), read Value, then Reset and reuse for the next computation.

/* Construct an algorithm. Typing the variable as IChecksum lets you swap
 * algorithms with a single-line change anywhere in the surrounding code. */
IChecksum checksum = new ChecksumCRC32();

/* Update one or more times with byte buffers. Here the save header and
 * payload are fed separately; the result is identical to a single call
 * with both concatenated. */
checksum.Update(Encoding.UTF8.GetBytes("SaveHeader:v2:"));
checksum.Update(saveBlob);

/* Read the running checksum. The long return type is the same across all
 * algorithms; cast to (uint) for the natural 32-bit form. */
uint computed = (uint)checksum.Value;
Log.Info($"Save checksum: 0x{computed:X8}", LogCategory.Core);

/* On load: compare the embedded checksum against a fresh computation. */
byte[] fileBytes = File.ReadAllBytes("save.dat");
uint embedded    = LoadEmbeddedChecksum(fileBytes);

checksum.Reset();                /* clear state for a new computation */
checksum.Update(fileBytes, offset: 0, count: fileBytes.Length - 4); /* exclude the trailing 4-byte checksum field */

uint actual = (uint)checksum.Value;

if (actual != embedded)
{
    Log.Warning($"Save file CRC32 mismatch: expected 0x{embedded:X8}, got 0x{actual:X8}", LogCategory.Core);
    /* Treat as corruption: prompt for the backup save, refuse to load,
     * or fall back to a known-good snapshot. Never silently continue. */
}

/* Reuse the same instance for a second, independent computation. Reset
 * clears internal state to the algorithm's initial value and avoids
 * allocating a fresh object for each file. */
checksum.Reset();
checksum.Update(configBlob);
uint configHash = (uint)checksum.Value;

Tag a network packet with a CRC16 checksum

Problem. Your game sends compact binary packets over a custom protocol and you want a two-byte integrity tag that the receiver can verify before processing the payload. A CRC (Cyclic Redundancy Check) using the CCITT polynomial is what Bluetooth, XMODEM, SD-card readers, and a handful of other standards chose for this exact budget, and your receiver might be one of those devices or a tool that speaks the same protocol.

Solution. ChecksumCRC16 uses polynomial 0x1021, initial state 0xFFFF, no final XOR – matching HDLC, X.25, XMODEM, Bluetooth, and SD-card protocols. Append the tag big-endian after the payload and strip it before re-computing on the receiving end.

/* Append a CRC16 tag to a packet before sending. */
byte[] payload = BuildPacket();

var crc16 = new ChecksumCRC16();
crc16.Update(payload);
ushort tag = (ushort)crc16.Value;

/* Two extra bytes for the tag, sent big-endian. */
byte[] framed = new byte[payload.Length + 2];
Buffer.BlockCopy(payload, 0, framed, 0, payload.Length);
framed[payload.Length    ] = (byte)(tag >> 8);
framed[payload.Length + 1] = (byte)(tag & 0xFF);

Send(framed);

/* Receiver: re-compute over the payload, compare against the trailing tag. */
ushort received = (ushort)((framed[payload.Length] << 8) | framed[payload.Length + 1]);

crc16.Reset();
crc16.Update(framed, 0, framed.Length - 2); /* exclude the trailing tag */

if ((ushort)crc16.Value != received)
{
    Log.Warning("Packet CRC16 mismatch; dropping.", LogCategory.Core);
}

Compute an Adler32 value for zlib-compatible output

Problem. You’re producing output in a zlib-compatible format – or validating data from one – and you need the Adler32 value that zlib embeds in its stream header and that PNG carries in its zTXt and iTXt chunks. You want the exact same number the reference implementation would produce, which means using RFC 1950’s two-accumulator design with prime modulus 65521.

Solution. ChecksumAdler32 matches RFC 1950 and is faster than CRC32 on most CPUs. Its error detection is slightly weaker on very short inputs, but for zlib interop that is a fixed-format concern, not a design choice you’re making fresh.

/* Compute the Adler32 of a payload to embed in a zlib-format header,
 * or compare against the trailing 4 bytes of a zlib stream. */
var adler = new ChecksumAdler32();
adler.Update(plaintextBeforeCompression);

uint adlerValue = (uint)adler.Value;
Log.Info($"Adler32: 0x{adlerValue:X8}", LogCategory.Core);

/* If you also need to compress the data, pair this with ScyllaCompression
 * (see the Compression page) so the checksum and the payload stay in sync. */

Hash an ATM/SCTP frame with Fletcher32

Problem. You’re implementing a protocol that specifically requires Fletcher32 – ATM AAL5, SCTP, or a network stack that adopted it for its simplicity and competitive error-detection. Fletcher32 sits between Adler32 and CRC32 in strength; picking it here is usually a compatibility requirement, not a personal preference.

Solution. ChecksumFletcher32 uses the two-accumulator design with modulus 65535. It correctly handles odd-length inputs by deferring the trailing byte, so splits across chunk boundaries still produce the right value.

/* Fletcher32 over a multi-chunk message. The class handles odd-length inputs
 * by deferring the trailing byte until Value is read; updates that split a
 * 16-bit word across chunks still produce the same result as a single update. */
var fletcher = new ChecksumFletcher32();
fletcher.Update(header);  /* header is 13 bytes (odd) */
fletcher.Update(body);    /* body completes the byte-pair correctly */

uint fletcherValue = (uint)fletcher.Value;

Build a seeded hash-table key with xxHash32

Problem. You’re using byte-buffer content as dictionary keys – perhaps an equipment fingerprint cache, a texture-dedup table, or a content-addressable asset store – and you need a fast hash with good distribution. CRC32 has weak avalanche (small input changes shift only a few output bits), which leads to clustering in hash tables. xxHash32 is faster and has much better avalanche, and it takes a seed so you can harden the table against hash-flooding attacks by randomizing the seed per process startup.

Solution. ChecksumXXHash32 is the Yann Collet xxHash32 implementation. Default seed is zero; pass a runtime-generated seed when the keys might be attacker-controlled.

/* Default-seeded xxHash32. Two consumers with the same seed always compute
 * the same value for the same input, so this is safe for stable content keys. */
var xh32 = new ChecksumXXHash32();
xh32.Update(Encoding.UTF8.GetBytes("equipment-id-1024"));
uint contentKey = (uint)xh32.Value;

/* Custom seed: different seeds produce different hashes from the same input.
 * Draw the seed from the crypto source at startup so an attacker cannot
 * pre-compute collision sets against your hash table. */
var cryptoRng  = RandomUtil.CreateCrypto();
uint runtimeSeed = cryptoRng.NextUInt32();
cryptoRng.Dispose();

var seeded = new ChecksumXXHash32(seed: runtimeSeed);
seeded.Update(Encoding.UTF8.GetBytes("equipment-id-1024"));
uint hardenedKey = (uint)seeded.Value;

/* Reuse the seeded instance for subsequent lookups in the same session. */
seeded.Reset();
seeded.Update(Encoding.UTF8.GetBytes("equipment-id-2048"));
uint secondKey = (uint)seeded.Value;

Fingerprint assets for deduplication with xxHash64

Problem. You’re building a content pipeline or a runtime asset store where you want to deduplicate identical files – textures submitted under two names, audio clips that ended up as duplicates after a refactor, downloadable content bundles where re-downloads of unchanged data should be skipped. You need a fingerprint that’s stable, fast to compute, and wide enough that accidental collisions are vanishingly rare even across millions of assets.

Solution. ChecksumXXHash64 gives you a 64-bit fingerprint. On 64-bit platforms it’s typically faster per byte than xxHash32 because each round processes 8 bytes through 64-bit accumulators. The output space is roughly 1.8 x 10^19, which makes accidental collisions negligible for any realistic asset count.

/* Fingerprint a large asset for deduplication. Two assets with the same
 * content always produce the same fingerprint regardless of file name or path. */
byte[] assetBytes = File.ReadAllBytes(assetPath);

var xh64 = new ChecksumXXHash64(seed: 0);
xh64.Update(assetBytes);

/* Cast to ulong to recover the canonical unsigned form; the interface returns
 * long, which may be negative for large 64-bit values. */
ulong fingerprint = (ulong)xh64.Value;
string contentID  = fingerprint.ToString("X16");
Log.Info($"Asset fingerprint: {contentID}", LogCategory.Core);

/* Use the fingerprint as the key in a dedup store. If two paths map to the
 * same fingerprint, they're byte-for-byte identical and only one copy needs
 * to be stored or shipped. */

Hash a large file in chunks without loading it into memory

Problem. Your game ships large asset bundles or produces multi-gigabyte replay files, and you want to verify their integrity on download or on load without pulling the whole file into memory first. Every IChecksum implementation supports incremental updates, so you can feed the file in whatever chunk size fits your memory budget and get the exact same result as a single all-at-once update.

Solution. Feed the stream in fixed-size chunks. Update(buffer, offset, count) checksums exactly the bytes you specify – which matters on the final read, where stream.Read may return fewer bytes than the buffer holds.

/* Fingerprint a multi-gigabyte file in 64 KB chunks. The xxHash64 instance
 * maintains the running state internally; the loop only holds one chunk at
 * a time in memory. */
const int CHUNK = 64 * 1024;
var buffer = new byte[CHUNK];

var hash = new ChecksumXXHash64();

using (var stream = File.OpenRead("Builds/large-bundle.bin"))
{
    int read;
    while ((read = stream.Read(buffer, 0, buffer.Length)) > 0)
    {
        /* Pass the exact byte count actually read. Passing buffer.Length
         * on a short final read would incorporate stale bytes from the
         * previous iteration and corrupt the fingerprint. */
        hash.Update(buffer, offset: 0, count: read);
    }
}

ulong fingerprint = (ulong)hash.Value;
Log.Info($"Bundle fingerprint: {fingerprint:X16}", LogCategory.Core);

/* The same chunked loop works for every IChecksum implementation. Swap the
 * class to switch algorithms without touching the surrounding loop. */
IChecksum fileIntegrityCheck = new ChecksumCRC32();
foreach (var chunk in chunkedSource)
{
    fileIntegrityCheck.Update(chunk);
}
uint crc = (uint)fileIntegrityCheck.Value;

API Sketch

The surface is one interface plus six sealed classes. Every class implements the interface and adds at most one extra public member: a seed-accepting constructor on the two xxHash classes. There is no facade, no factory, and no static configuration; pick the class that fits the requirement and instantiate it directly.

public interface IChecksum
{
    /* Current running checksum value. Reading is non-destructive; subsequent
     * Update calls continue from the same internal state. */
    long Value { get; }

    /* Incorporate all bytes from `buffer`. A null or empty buffer is a no-op. */
    void Update(byte[] buffer);

    /* Incorporate `count` bytes starting at `buffer[offset]`. A null buffer
     * or zero count is a no-op. Caller ensures offset + count <= buffer.Length. */
    void Update(byte[] buffer, int offset, int count);

    /* Reset the internal state to the algorithm's documented initial value,
     * discarding any previously accumulated data. */
    void Reset();
}

public sealed class ChecksumAdler32    : IChecksum { public ChecksumAdler32();    }
public sealed class ChecksumCRC16      : IChecksum { public ChecksumCRC16();      }
public sealed class ChecksumCRC32      : IChecksum { public ChecksumCRC32();      }
public sealed class ChecksumFletcher32 : IChecksum { public ChecksumFletcher32(); }

public sealed class ChecksumXXHash32   : IChecksum
{
    public ChecksumXXHash32(uint seed = 0);
}

public sealed class ChecksumXXHash64   : IChecksum
{
    public ChecksumXXHash64(ulong seed = 0);
}

See the API reference for full signatures, remarks, and exception contracts on every member.

Algorithm reference

The table below summarises the algorithm-specific details that are useful when reading or producing a value to match an external reference. The CRC variants are sensitive to all four parameters: a value computed with a different polynomial, a different initial state, or a different final XOR will not match, even though it is still called “CRC32”.

ClassOutput widthInitial stateFinal XORPolynomial / modulusSeed typeNotes
ChecksumAdler3232 bitss1 = 1, s2 = 0nonemod 65521 (prime)noneOutput is (s2 << 16) | s1. Compatible with zlib (RFC 1950) and PNG.
ChecksumCRC1616 bits0xFFFFnonepoly 0x1021 (CCITT)noneNon-reflected, big-endian bit order. Used by HDLC, X.25, XMODEM.
ChecksumCRC3232 bits0xFFFFFFFF0xFFFFFFFF (applied in Value)poly 0xEDB88320 (IEEE 802.3, reflected)noneMatches GZip, ZIP, PNG, and Ethernet. Standard, widely-interoperable CRC32.
ChecksumFletcher3232 bitsc0 = 0, c1 = 0nonemod 65535noneOperates on 16-bit words. Odd-length inputs pad the trailing byte with zero (deferred until Value is read).
ChecksumXXHash3232 bitsfrom seed (default 0)nonefive 32-bit primesuintxxHash specification by Yann Collet. Strong avalanche; designed for hash-table use.
ChecksumXXHash6464 bitsfrom seed (default 0)nonefive 64-bit primesulong64-bit variant of xxHash. Reinterpreted as signed long by the interface; cast to ulong to recover the canonical unsigned form.

Best Practices

  • Pick by purpose, not by name recognition. CRC32 for file integrity and interop with GZip/ZIP/PNG. CRC16 for compact protocol tags. Adler32 for zlib compatibility. Fletcher32 only for ATM/SCTP interop. xxHash32 for hash-table keys on 32-bit-leaning workloads. xxHash64 for fingerprints, dedup, and 64-bit content addressing.
  • Reuse instances across computations. Constructing a fresh ChecksumXxx for every input wastes the small per-class state allocation. Call Reset() between independent computations and feed the next buffer.
  • Type the variable as IChecksum when the algorithm should be swappable. Code that hashes via IChecksum checksum = ... can switch from CRC32 to xxHash64 with a one-line change and no other edits.
  • Cast Value to the correct unsigned type at the call site. (uint), (ushort), or (ulong) matches the natural width of each algorithm. The long return type is a side effect of the interface; the algorithms themselves work in unsigned space.
  • Use the offset / count overload for slices. Allocating a sub-array to checksum a fragment of a larger buffer is wasted work; the overload checksums the slice in place.
  • Stream very large inputs in fixed-size chunks. Avoid File.ReadAllBytes followed by a single Update; that loads the whole file into memory. A 64 KB or 256 KB chunked loop produces the same checksum without the memory cost.
  • Use a runtime-generated seed for xxHash hash-table use. A constant seed makes the hash table vulnerable to precomputed collision attacks if an attacker can control the keys. Draw the seed from RandomUtil.CreateCrypto or another per-process source at startup.
  • For 64-bit fingerprints, prefer xxHash64 over deriving from two xxHash32 values. The 64-bit variant has its own primes and merge step; concatenating two xxHash32 results does not produce the same distribution.
  • Treat the checksum as a comparison key, not a unique identifier. Two different inputs can in principle produce the same checksum (a collision). Use the checksum to filter likely matches, then fall back to a byte-by-byte compare when uniqueness must be guaranteed.
  • For real tamper detection, reach for Crypto. A checksum can be recomputed by anyone who can rewrite the data. HMAC and AEAD ciphertext tags cannot, because they depend on a key that the attacker does not hold.
  • Match the polynomial when interoperating. A CRC32 written by one tool only matches a CRC32 read by another if both ends use the same polynomial. ChecksumCRC32 matches the IEEE 802.3 polynomial; a value produced with the Castagnoli polynomial will not match.
  • Pair checksum verification with serialization round-trips. When you’re protecting Serialization output (JSON save data, config files), compute the checksum over the raw serialized bytes before writing and after reading. A mismatch on load means the bytes changed on disk, not that your deserializer has a bug.
  • Combine with compression for transport payloads. If you’re sending compressed data (see Compression), compute the checksum over the compressed bytes so the receiver can verify the compressed payload before decompressing – a corrupted compressed stream can produce garbled output that is hard to diagnose otherwise.

Pitfalls

Updated on July 9, 2026