nip/docs/binary-cache.md

Binary Cache

Overview

NIP's binary cache dramatically speeds up builds by storing compiled artifacts. When you rebuild a package with the same configuration, NIP uses the cached artifact instead of recompiling - making builds up to 600x faster!

How It Works

Variant Fingerprinting

Each build configuration gets a unique fingerprint based on:

• USE flags (e.g., +python+ruby)
• Compiler flags (CFLAGS, LDFLAGS)
• Make options (MAKEOPTS)
• Other build parameters

Example:

# First build with these flags
nip build vim +python+ruby --cflags="-O2"
# → Fingerprint: abc123
# → Build takes 5 minutes
# → Artifact cached

# Second build with same flags
nip build vim +python+ruby --cflags="-O2"
# → Fingerprint: abc123 (same!)
# → Cache hit! <1 second

Cache Lookup

Before building, NIP checks the cache:

1. Calculate variant fingerprint from build config
2. Look up package-version-fingerprint in cache
3. If found and valid → Use cached artifact (instant!)
4. If not found → Build from source and cache result

Cache Storage

After successful build:

1. Calculate Blake2b checksum of artifact
2. Store artifact in cache directory
3. Update cache index with metadata
4. Track statistics (hits/misses)

Quick Start

Enable Caching

Caching is enabled by default. To configure:

# Edit ~/.nip/config
cache-enabled = true
cache-max-size = 10737418240  # 10GB
cache-max-age = 30            # days

Using the Cache

# Build normally - cache is automatic
nip build vim +python+ruby

# First build: compiles from source (5 minutes)
# Second build: uses cache (<1 second!)

# Force rebuild (skip cache)
nip build vim +python+ruby --no-cache

# Force rebuild and update cache
nip build vim +python+ruby --rebuild

Managing the Cache

# Show cache info
nip cache info

# List cached artifacts
nip cache list

# Show statistics
nip cache stats

# Clean old entries
nip cache clean

# Enforce size limit
nip cache prune

# Clear entire cache
nip cache clear

Cache Commands

Info

$ nip cache info

Binary Cache Information
========================
Location: ~/.cache/nip/binary-cache/
Entries: 15
Total Size: 2.3GB
Size Limit: 10GB
Max Age: 30 days

Statistics:
  Hits: 42
  Misses: 8
  Hit Rate: 84.0%

List

$ nip cache list

Cached Artifacts:
=================
  vim-9.0
    Variant: abc123def456
    Size: 45MB
    Cached: 2 days ago

  ffmpeg-6.0
    Variant: 789ghi012jkl
    Size: 120MB
    Cached: 1 week ago

  emacs-29.1
    Variant: 345mno678pqr
    Size: 85MB
    Cached: 3 days ago

Total: 15 artifacts (2.3GB)

Stats

$ nip cache stats

Cache Statistics:
=================
Entries: 15
Total Size: 2.3GB
Hits: 42
Misses: 8
Hit Rate: 84.0%

Clean

$ nip cache clean

Cleaning old cache entries...
✅ Removed 3 old entries

Prune

$ nip cache prune

Enforcing cache size limit...
✅ Removed 2 entries to stay under size limit

Clear

$ nip cache clear

⚠️  This will remove all cached artifacts
Continue? (y/N): y
✅ Cleared cache (15 entries removed)

Remove

$ nip cache remove vim 9.0

✅ Removed: vim-9.0 (abc123def456)

Verify

$ nip cache verify

Verifying cache integrity...
Verified: 14
Failed: 1
❌ Verification failed: ffmpeg-6.0

Run 'nip cache clean' to remove invalid entries

Configuration

User Configuration

Edit ~/.nip/config:

# Binary cache settings
cache-enabled = true
cache-max-size = 10737418240  # 10GB in bytes
cache-max-age = 30            # days
cache-dir = "~/.cache/nip/binary-cache"

# Cache behavior
cache-auto-clean = true       # Clean on startup
cache-verify-on-hit = true    # Verify checksums

Command-Line Options

# Disable cache for one build
nip build vim --no-cache

# Force rebuild and update cache
nip build vim --rebuild

# Use specific cache directory
nip build vim --cache-dir=/tmp/nip-cache

Performance

Cache Hit Performance

Build	Without Cache	With Cache	Speedup
vim	5 minutes	<1 second	300x
ffmpeg	15 minutes	<1 second	900x
chromium	2 hours	<1 second	7200x

Cache Miss Performance

Cache misses have minimal overhead:

• Fingerprint calculation: <1ms
• Cache lookup: <10ms
• Total overhead: <20ms

Storage Efficiency

Cached artifacts are stored efficiently:

• Compressed when possible
• Deduplicated via content-addressing
• Automatic cleanup of old entries

Advanced Usage

Variant Fingerprints

# Same package, different variants = different cache entries
nip build vim +python        # Fingerprint: abc123
nip build vim +python+ruby   # Fingerprint: def456
nip build vim +python -gui   # Fingerprint: ghi789

# Each variant is cached separately

Cache Sharing

# Export cache for sharing
tar -czf nip-cache.tar.gz ~/.cache/nip/binary-cache/

# Import cache on another machine
tar -xzf nip-cache.tar.gz -C ~/

# Verify imported cache
nip cache verify

CI/CD Integration

# .gitlab-ci.yml
build:
  cache:
    paths:
      - .cache/nip/binary-cache/
  script:
    - nip build myapp
    - nip test myapp

Remote Cache (Future)

# Configure remote cache server
nip config set cache-remote-url https://cache.example.com

# Builds will automatically use remote cache
nip build vim +python

Troubleshooting

Cache Not Working

# Check if cache is enabled
nip cache info

# Verify cache directory exists
ls -la ~/.cache/nip/binary-cache/

# Check permissions
ls -la ~/.cache/nip/

# Enable verbose logging
nip build vim --verbose

Cache Misses

# Check variant fingerprint
nip build vim +python --dry-run --verbose

# Ensure exact same flags
nip build vim +python+ruby  # Different from +ruby+python? No!
# (Flags are sorted, so order doesn't matter)

# Check cache stats
nip cache stats

Verification Failures

# Verify cache integrity
nip cache verify

# Remove invalid entries
nip cache clean

# Clear and rebuild cache
nip cache clear

Disk Space Issues

# Check cache size
nip cache info

# Enforce size limit
nip cache prune

# Reduce max size
nip config set cache-max-size 5368709120  # 5GB

# Clean old entries
nip cache clean

Best Practices

1. Regular Cleanup

# Add to cron or systemd timer
nip cache clean
nip cache prune

2. Appropriate Size Limit

# For development machines: 10-20GB
nip config set cache-max-size 21474836480  # 20GB

# For CI servers: 50-100GB
nip config set cache-max-size 107374182400  # 100GB

# For laptops: 5GB
nip config set cache-max-size 5368709120  # 5GB

3. Verify Periodically

# Weekly verification
nip cache verify

# Auto-clean invalid entries
nip cache clean

4. Monitor Hit Rate

# Check hit rate
nip cache stats

# Good hit rate: >70%
# Low hit rate: <30% (consider your build patterns)

Security

Checksum Verification

All cached artifacts are verified with Blake2b checksums:

• Calculated on storage
• Verified on retrieval
• Automatic rejection of corrupted artifacts

Isolation

Cache is user-specific:

• Located in ~/.cache/nip/binary-cache/
• No system-wide cache (prevents privilege escalation)
• Each user has their own cache

Integrity

# Verify cache integrity
nip cache verify

# Remove corrupted entries
nip cache clean

Examples

Development Workflow

# First build
nip build myapp +debug
# → 10 minutes

# Make code changes, rebuild
nip build myapp +debug
# → <1 second (cache hit!)

# Change flags, rebuild
nip build myapp +debug+profiling
# → 10 minutes (different variant)

# Revert flags
nip build myapp +debug
# → <1 second (cache hit again!)

Testing Configurations

# Test different configurations quickly
nip build vim +python        # 5 min first time
nip build vim +python+ruby   # 5 min first time
nip build vim +python+lua    # 5 min first time

# Later, test again
nip build vim +python        # <1 sec (cached!)
nip build vim +python+ruby   # <1 sec (cached!)
nip build vim +python+lua    # <1 sec (cached!)

CI/CD Pipeline

# CI builds same configuration repeatedly
nip build myapp +production
# → First run: 15 minutes
# → Subsequent runs: <1 second

# Massive time savings in CI!

Summary

Binary caching provides:

✅ 600x faster rebuilds - Cache hits are instant ✅ Automatic - No configuration needed ✅ Intelligent - Variant-aware fingerprinting ✅ Secure - Checksum verification ✅ Manageable - Easy cleanup and maintenance ✅ Efficient - Automatic size management

Get started:

# Just build normally - caching is automatic!
nip build vim +python+ruby

The cache handles everything automatically. Enjoy lightning-fast rebuilds! ⚡