Lore: Epic Games Open-Sources a VCS Built for Massive Binaries

Game repositories are not web repositories. In an engine the size of Unreal, most of the weight is binary assets, textures, meshes, world partitions, hundreds of MB to several GB each, churned by thousands of artists and developers. Git handles that badly: LFS is a bolt-on dragging whole 2 GB files, dedup is capped by pack deltas, partial clone plus sparse checkout is experimental, and offline means fail.

Epic sat on this problem longest: it wrote its own VCS for UEFN, then open-sourced it as Lore, MIT, close to 4.9k stars, pre-1.0. Formats can still change, so treat this as a portrait of the architecture, not an API promise.

Why a game studio writes its own VCS

The motivation is three workload properties. Content-agnostic: repos mix source, config, build artifacts and binaries. Large on every axis: millions of files, terabyte-scale files, millions of revisions, hundreds of branches, thousands of users, a hundred repos on one backend. Centrally coordinated: one source of truth for access control, durability and conflicts, while developers still work offline.

Prior art: Git owns content-addressed stores and commit graphs, Perforce centralized file-level locking, Sapling text-shaped monorepos. Git caps binary dedup with pack deltas and leaves multi-tenancy to outside infrastructure. Perforce hashes with MD5, delta-encodes text, stores binary whole-file, proprietary protocol. Lore fills the intersection: content-addressed integrity, a centralized server of record, sparse and lazy fetch at every granularity, fragment-level dedup, multi-tenancy by design, everything open under MIT.

Architecture: storage split from version control

Lore is two systems stacked: a content-addressed, partition-based storage subsystem with strict access boundaries, plus a version control subsystem building revisions, branches, merges and staging on its primitives. Version control is just one consumer; storage also works standalone as a deduplicating blob store.

The two layers

graph TD
    Tool["CLI / IDE / App"] --> API["Rust core, C API + bindings (Go, Python, JS, C#)"]
    API --> VC["Version control subsystem"]
    VC --> SS["Storage subsystem"]
    subgraph Store["Inside the storage subsystem"]
        Imm["Immutable store: fragments keyed by (hash, context)"]
        Mut["Mutable store: branch pointers, name-to-ID"]
    end
    SS --> Imm
    SS --> Mut
    Imm --> Back["Local packfiles / S3 / ReplicatedStore"]
    Mut --> MBack["Local file / DynamoDB"]

The immutable store holds every byte ever written, named by its 32-byte BLAKE3 hash: faster than SHA-256 on long inputs, Merkle-structured for multicore hashing, and collision resistance of 2^128 fragments before anything gets weird. The mutable store is a small key-value table for branch pointers, name-to-ID mappings and catalogs; mutation lives only here, behind four ops: load, store, cas, list.

A fragment address is (hash, context), 32 plus 16 bytes. Hash names the content, context names the entity, via a stable file ID surviving moves, copies and obliteration. The 16-byte partition is the access boundary: content identity, entity identity, permission.

A 320-byte revision and a prebuilt Merkle tree

A revision is a frozen whole-tree snapshot: a 320-byte fragment of hashes, magic and format version, a revision number along the first-parent chain, parent hash (one for normal revisions, two for merges), hashes of the Merkle tree, metadata and link list, plus a repository ID for cross-repo merges. 320 bytes is deliberate: loading a revision is one round trip.

The tree is a chain of node blocks. Nodes are 96 bytes; 512 per block plus a 128-byte header makes 49,280 bytes, exactly one fragment. A node is one file or directory: flags, mode, a 32-bit index to parent/child/sibling, a name reference into the block name table with a 64-bit lowercase-name hash, size, content address. Fixed size is what makes it cheap: blocks mmap straight from disk, no parse or copy.

Blocks share structural dedup for free: revisions differing in a few files share every untouched block, so a new revision costs only changed blocks. Every revision hash includes its parent hash, so tampering breaks the chain. A branch is a mutable pointer, a name-to-ID entry plus a latest pointer, no new revision: free branching.

Chunking: FastCDC, fixed-size, and the price of canonicality

Large files are hashed as chunks, each chunk an independent fragment. Lore ships two chunking strategies; the calling application picks one through the storage API.

Two strategies, two trade-off sets

graph TD
    F["File 8 GB (uasset)"] --> C1["Chunk 1 (~64 KiB, FastCDC)"]
    F --> C2["Chunk 2 (~64 KiB)"]
    F --> CN["Chunk N (<= 256 KiB)"]
    C1 --> H1["BLAKE3 hash"]
    C2 --> H2["BLAKE3 hash"]
    CN --> HN["BLAKE3 hash"]
    H1 --> L["Fragment list: (hash, offset), keeps order"]
    H2 --> L
    HN --> L
    L --> LR["List over 256 KiB? Split into a tree of fragment lists"]

FastCDC is content-defined chunking: a rolling hash scans the file and boundaries land on a magic pattern, 64 KiB average, 32 KiB floor, 256 KiB ceiling. Boundaries follow content, so inserting data shifts only what follows; untouched chunks keep their hashes, one byte changed inside an 8 GB file re-uploads a few chunks.

But boundaries inherit history: re-chunking from scratch mints new boundaries even in untouched regions, cascading rewrites that kill dedup. Temporal coherence, reusing old boundaries where bytes are unchanged, saves dedup but forfeits canonicality: with CDC, different addresses need not mean different content. Fixed-size chunking is canonical, one content one address, but one inserted byte shifts every later boundary. Lore lets the app choose; 256 KiB is also the protocol’s fragment threshold.

Fragment lists for multi-terabyte files would balloon to hundreds of MB, so oversized lists split into fragments flagged as lists: a file becomes a tree of fragment lists, each level addressed and deduped independently. References carry chunk hash plus byte offset, ordered for O(log n) binary search. Range reads fetch only overlapping fragments: cost scales with range, not file size.

Compression stays out of addressing: Zstd per fragment, address is the hash of the uncompressed payload, so codec changes never alter addresses. Git instead hashes objects with a blob <size>\0 header; a file’s SHA-1 is not its raw bytes. Lore hashes raw bytes, reproducible with b3sum.

Sparse by default, binary-first by nature

Lore assumes the whole tree never materializes anywhere. Clones are sparse by default: .lore/view declares the subset (inbound filter), .loreignore excludes paths (outbound). FilterMode: committed-state ops consult only the view, working-tree ops both, so new ignore rules never touch committed files.

Fetch is lazy: loading a revision walks only the viewed slice, pulls its backing fragments, leaving the rest on the remote and edge cache. The local cache is an LRU with a user-set budget; a 4 MiB range read from a multi-GB file fetches nothing more.

Binary-first means storage and transport never touch content beyond moving bytes: no CRLF translation, no encoding inference, no clean/smudge. Text diff and three-way merge live in the version control layer. Unmergeable content, engine world state, serialized scenes, gets file-level locking, acquire, release, query: a push touching a lock held elsewhere is rejected server-side before it becomes visible.

Centralized but offline-capable: two-phase push

Centralized because access control, durability and conflict resolution need one decision point, but not always-online: staging, commit, branch, switch and diff all run on the local mutable store and fragment cache.

Push has two phases. First, the client lists the fragments the new revision and its ancestors reference that the remote lacks, queries which exist, and uploads the missing ones in parallel, out of order, resumable. Then, once everything is durable, a conditional put moves latest from H_old to H_new: a compare-and-swap, the single serialization point where concurrent pushes to one branch queue.

Two-phase push, a single CAS

sequenceDiagram
    participant C as Lore client
    participant S as Lore server
    C->>S: Query which fragments are missing
    S-->>C: N fragments missing
    loop Parallel, resumable upload
        C->>S: Put each fragment (Zstd, out of order)
    end
    C->>S: MutableCas: latest = H_new if still H_old
    alt CAS wins
        S-->>C: OK, branch moved to H_new
    else CAS lost, branch already moved
        S-->>C: Conflict
        C->>C: Sync + merge locally (merge revision, two parents)
        C->>S: Retry push
    end

Losing the CAS means someone pushed first: sync, merge locally into a two-parent revision, retry. Optionally the server fast-forwards, building the merge revision itself, parent_self the remote latest, parent_other the incoming, in one round trip; genuine conflicts return to the client. Fragments-first keeps push atomic: dying between phases leaves unreferenced fragments; readers still see the old latest.

Multi-tenant: partition is the boundary, hash is no pass

For backend people the highlight is multi-tenancy: unrelated repos share one backend, tenants do not trust each other. The 16-byte partition is the access boundary, derived server-side from the session. A session on partition A cannot read partition B, even byte-identical content.

Content addressing gives identical bytes identical hashes, so knowing a hash must not mean readable bytes: a knows-the-hash attack would leak content from hashes seen in logs, artifacts or shared dependencies. The protocol defends: Put always carries the bytes; the server never registers a fragment from hash knowledge alone, even when those bytes already exist. Copy is the only cross-partition shortcut, rights on both sides checked, no bytes transferred; dedup sits under the access model.

Side channels get the same discipline: existence queries run only in the session’s partition, answering FoundInContext, Found, NotFound or Unknown, near constant time, so timing leaks nothing. Errors follow a strict precedence. Auth is JWT over QUIC (ALPN lore-storage/0.4) and gRPC; each QUIC connection carries up to 8 streams, two for control, pipelined with out-of-order replies.

Replaceable backends, horizontal scale

Storage is defined by two traits, ImmutableStore and MutableStore. Immutable: local packfiles (append-only, mmappable index), S3 (object per fragment keyed by hash), ReplicatedStore (peers warm read misses, writes propagate immediately). Mutable: a local file with filesystem locking, or DynamoDB conditional writes. Typical layout: local cache, network replica, S3 for durability, one interface; a new backend is two trait implementations.

Scale comes from a stateless read path: address A in partition P maps to bytes, session-free, embarrassingly parallel, and every server returns the same bytes because content addresses content; writes never contend, since bytes decide the address. Contention lives only in the mutable store: sizing it is the single most consequential capacity decision. Hot/warm/cold tiering has edges serve 90% or more of fragment traffic without touching upstream; overload returns SlowDown and clients back off exponentially, slow but never wrong. Hash-based sharding is an open problem: tail bytes of a hash pick the shard, front bytes handle disk fan-out.

Obliteration, legal takedowns or an accidentally committed secret, deletes payload but keeps addresses, crash-safe, PayloadObliterating then PayloadObliterated, scoped to a file ID. Non-goals: no P2P decentralization, no defense against malicious servers.

Verdict: does Lore kill Git?

No, and it never aimed to: Git stays unbeatable for distributed, code-heavy work. Lore targets Perforce’s seat, the centralized binary-heavy world: MD5 out, BLAKE3 in; delta encoding out, content addressing in; always-online out, offline-capable in; closed out, MIT in. Against plain Git it wins where Git refuses to go: multi-GB files first-class, sparse clones by default, multi-tenancy in the storage layer.

The real strengths: a clean subsystem split, one serialization point easy to reason about, raw-byte hashing any tool can reproduce with b3sum, chunking, fragment lists and sparse reads as one circuit.

What is unfinished, plainly: pre-1.0; rebase, cherry-pick and squash exist only at the data-model level; roadmap items remain, fork copy-on-read, branch-aware locking, multiple stages, hash sharding, a desktop client. UEFN runs Lore internally, but the open build cannot talk to it: the compression is proprietary, and Epic is moving to open compression.

For backend engineers the value is the lessons: immutable versus mutable stores, addressing separated from compression, one CAS as the serialization point, raw-byte hashing. If your asset repo outgrew a few tens of GB on Git plus LFS, or the Perforce license hurts, try Lore small; the Go SDK lore-go exists. Big production: wait for 1.0 and maturing tooling.

Sources

• Lore system design doc: https://epicgames.github.io/lore/explanation/system-design/
• GitHub repository: https://github.com/EpicGames/lore
• Homepage: https://lore.org/