BinarySync
A high-performance, cloud-native file versioning system that uses content-defined chunking (CDC) to transfer and store only the modified portions of large binary assets — minimizing bandwidth and storage for multi-gigabyte files.
Overview
BinarySync is a versioning and sync engine built for large binary assets. Instead of treating a file as one opaque blob, it slices it into content-addressed chunks, so a version history can share storage and transfers can move only what actually changed.
Problem
Large binary files break conventional versioning. A 4 GB dataset that gets a 100 MB edit forces a full re-upload with typical tools, and fixed-size chunking means even an insertion in the middle of a file invalidates every subsequent chunk. The result is wasted bandwidth, duplicated storage, and slow syncs for exactly the assets where data changes matter most.
Architecture
Solution
BinarySync computes chunk boundaries from the content itself using a rolling Gear hash, so a local edit only alters the chunks around it. Chunks are addressed by their SHA-256 hash, making deduplication automatic across versions. A manifest tracks which chunks each version references, and a delta-sync protocol asks the server only for the chunks a client is missing.
Key Features
- Content-defined chunking using a rolling Gear hash
- Delta synchronization with chunk-level uploads
- SHA-256 content-addressed deduplication
- Cloud-native object storage using AWS S3
- Java 21 Virtual Threads for parallel transfers
- Constant-memory chunk processing for large binary assets
Technical Decisions
- Gear hash over cryptographic hashing for boundary detection — it is fast enough to run over every byte while staying resistant to edits shifting the stream.
- Content addressing with SHA-256 so identical chunks are stored once, regardless of which version references them.
- Virtual Threads for transfer parallelism — blocking I/O without the complexity of a thread-per-chunk pool.
- A partitioned S3 key scheme so chunk lookup stays flat and predictable at scale.
- H2 for chunk metadata to keep a self-contained, testable control plane.
Lessons Learned
- Fixed-size chunking is deceptively simple until you insert one byte in the middle of a multi-gigabyte file. Content-defined chunking is the correct mental model from day one.
- The client-server protocol design matters as much as the storage layer — declaring local chunks up front turns a transfer problem into a set-difference problem.
- Constant memory is a feature. The system stays usable on developer laptops even for files that exceed RAM.
Future Roadmap
- cdc-java: publish the core chunking library as a reusable artifact
- SDK for other languages and a documented client API
- Dedicated documentation site
- A reproducible benchmarks page with real transfer numbers