nip/docs/remote-cache.md

Remote Binary Cache Guide

Overview

The NIP remote binary cache enables sharing of compiled artifacts across machines and teams. This dramatically speeds up builds in CI/CD pipelines and development environments by allowing teams to share build results.

Features

• Automatic Upload/Download: Artifacts are automatically uploaded after successful builds and downloaded before builds
• Team Sharing: Share builds across team members and CI/CD runners
• HTTP API: Simple REST API for cache operations
• Authentication: API key-based authentication for secure access
• Fallback: Gracefully falls back to local cache if remote is unavailable
• Bandwidth Efficient: Only transfers artifacts when needed

Quick Start

1. Configure Remote Cache

# Set remote cache URL
nip cache remote config --url https://cache.example.com

# Set API key for authentication
nip cache remote config --api-key your-api-key-here

# Enable remote cache
nip cache remote config --enable

2. Check Status

nip cache remote status

Output:

Remote Cache Status
===================

Enabled: Yes
URL: https://cache.example.com
API Key: ***configured***
Timeout: 300 seconds

Testing connection...
✅ Remote cache is available

3. Build with Remote Cache

Remote cache is now automatically used during builds:

# First build - compiles and uploads to remote cache
nip build vim +python+ruby

# On another machine - downloads from remote cache
nip build vim +python+ruby

Configuration

Configuration File

Remote cache settings are stored in ~/.config/nip/remote-cache.json:

{
  "url": "https://cache.example.com",
  "apiKey": "your-api-key-here",
  "timeout": 300,
  "enabled": true
}

Configuration Options

Option	Description	Default
url	Remote cache server URL	""
apiKey	Authentication API key	""
timeout	Request timeout in seconds	300
enabled	Enable/disable remote cache	false

Environment Variables

You can also configure via environment variables:

export NIP_REMOTE_CACHE_URL="https://cache.example.com"
export NIP_REMOTE_CACHE_API_KEY="your-api-key-here"
export NIP_REMOTE_CACHE_ENABLED="true"

Usage

Automatic Mode (Recommended)

Remote cache works automatically during builds:

# Build package - automatically checks remote cache first
nip build firefox +wayland

# If not in remote cache:
#   1. Checks local cache
#   2. Builds from source
#   3. Uploads to remote cache
#   4. Uploads to local cache

# If in remote cache:
#   1. Downloads to local cache
#   2. Uses cached artifact (instant!)

Manual Operations

Pull from Remote Cache

# Pull specific package from remote cache
nip cache remote pull vim 9.0

Push to Remote Cache

# Push specific package to remote cache
nip cache remote push vim 9.0

Check Remote Status

# Test remote cache connectivity
nip cache remote status

Cache Lookup Flow

┌─────────────────────────────────────────────────────────────┐
│                     Build Request                            │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│              Check Local Cache                               │
│  ~/.cache/nip/binary-cache/                                  │
└─────────────────────────────────────────────────────────────┘
                            │
                    ┌───────┴───────┐
                    │               │
                 Found          Not Found
                    │               │
                    │               ▼
                    │   ┌─────────────────────────────────────┐
                    │   │    Check Remote Cache               │
                    │   │  (if enabled)                       │
                    │   └─────────────────────────────────────┘
                    │               │
                    │       ┌───────┴───────┐
                    │       │               │
                    │    Found          Not Found
                    │       │               │
                    │       ▼               ▼
                    │   ┌─────────┐   ┌─────────────┐
                    │   │Download │   │Build from   │
                    │   │to Local │   │Source       │
                    │   └─────────┘   └─────────────┘
                    │       │               │
                    │       │               ▼
                    │       │         ┌─────────────┐
                    │       │         │Upload to    │
                    │       │         │Remote Cache │
                    │       │         └─────────────┘
                    │       │               │
                    │       └───────┬───────┘
                    │               │
                    └───────┬───────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                  Use Cached Artifact                         │
└─────────────────────────────────────────────────────────────┘

Remote Cache Server

API Endpoints

The remote cache server implements a simple HTTP API:

Health Check

GET /v1/health
Response: 200 OK

Lookup Artifact

GET /v1/artifacts/{cache-key}
Response: 200 OK with download URL, or 404 Not Found

Get Metadata

GET /v1/artifacts/{cache-key}/metadata
Response: 200 OK with JSON metadata

Upload Artifact

POST /v1/artifacts/{cache-key}
Content-Type: multipart/form-data
- file: artifact file
- metadata: JSON metadata

Response: 201 Created

Authentication

All requests (except health check) require authentication:

Authorization: Bearer <api-key>

Example Server Implementation

A simple reference server implementation:

from flask import Flask, request, jsonify, send_file
import os
import json

app = Flask(__name__)
CACHE_DIR = "/var/cache/nip"
API_KEY = os.environ.get("NIP_CACHE_API_KEY")

def check_auth():
    auth = request.headers.get("Authorization")
    if not auth or not auth.startswith("Bearer "):
        return False
    token = auth.split(" ")[1]
    return token == API_KEY

@app.route("/v1/health")
def health():
    return jsonify({"status": "ok"})

@app.route("/v1/artifacts/<cache_key>")
def get_artifact(cache_key):
    if not check_auth():
        return jsonify({"error": "unauthorized"}), 401

    artifact_path = os.path.join(CACHE_DIR, cache_key, "artifact.tar.gz")
    if os.path.exists(artifact_path):
        return jsonify({"downloadUrl": f"/v1/download/{cache_key}"})
    return jsonify({"error": "not found"}), 404

@app.route("/v1/download/<cache_key>")
def download_artifact(cache_key):
    if not check_auth():
        return jsonify({"error": "unauthorized"}), 401

    artifact_path = os.path.join(CACHE_DIR, cache_key, "artifact.tar.gz")
    return send_file(artifact_path)

@app.route("/v1/artifacts/<cache_key>/metadata")
def get_metadata(cache_key):
    if not check_auth():
        return jsonify({"error": "unauthorized"}), 401

    metadata_path = os.path.join(CACHE_DIR, cache_key, "metadata.json")
    if os.path.exists(metadata_path):
        with open(metadata_path) as f:
            return jsonify(json.load(f))
    return jsonify({"error": "not found"}), 404

@app.route("/v1/artifacts/<cache_key>", methods=["POST"])
def upload_artifact(cache_key):
    if not check_auth():
        return jsonify({"error": "unauthorized"}), 401

    cache_dir = os.path.join(CACHE_DIR, cache_key)
    os.makedirs(cache_dir, exist_ok=True)

    # Save artifact
    file = request.files["file"]
    artifact_path = os.path.join(cache_dir, "artifact.tar.gz")
    file.save(artifact_path)

    # Save metadata
    metadata = json.loads(request.form["metadata"])
    metadata_path = os.path.join(cache_dir, "metadata.json")
    with open(metadata_path, "w") as f:
        json.dump(metadata, f)

    return jsonify({"status": "created"}), 201

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

CI/CD Integration

GitHub Actions

name: Build with NIP Cache

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install NIP
        run: |
          curl -sSL https://nip.example.com/install.sh | sh          

      - name: Configure Remote Cache
        env:
          NIP_CACHE_API_KEY: ${{ secrets.NIP_CACHE_API_KEY }}
        run: |
          nip cache remote config --url https://cache.example.com
          nip cache remote config --api-key $NIP_CACHE_API_KEY
          nip cache remote config --enable          

      - name: Build Package
        run: nip build myapp +production

GitLab CI

build:
  image: ubuntu:latest
  before_script:
    - curl -sSL https://nip.example.com/install.sh | sh
    - nip cache remote config --url https://cache.example.com
    - nip cache remote config --api-key $NIP_CACHE_API_KEY
    - nip cache remote config --enable
  script:
    - nip build myapp +production
  variables:
    NIP_CACHE_API_KEY: $CI_CACHE_API_KEY

Performance Impact

Build Time Comparison

Scenario	Without Remote Cache	With Remote Cache	Speedup
First build (team)	15 minutes	15 minutes	1x
Second build (same machine)	<1 second (local)	<1 second (local)	-
Second build (different machine)	15 minutes	<1 second	900x
CI/CD pipeline	15 minutes	<1 second	900x

Real-World Example

Development Team (5 developers)

Without remote cache:

• Each developer builds from scratch: 5 × 15 min = 75 minutes total
• CI builds from scratch: +15 minutes
• Total: 90 minutes of build time

With remote cache:

• First developer builds: 15 minutes
• Other developers use cache: 4 × <1 second ≈ 0 minutes
• CI uses cache: <1 second
• Total: 15 minutes of build time

Savings: 75 minutes (83% reduction)

Security Considerations

API Key Management

• Never commit API keys to version control
• Use environment variables or secret management systems
• Rotate keys regularly
• Use different keys for different teams/projects

Network Security

• Always use HTTPS for remote cache URLs
• Consider VPN or private network for sensitive builds
• Implement rate limiting on server
• Monitor for unusual access patterns

Access Control

• Implement read/write permissions
• Separate keys for CI vs developers
• Audit log all cache operations
• Implement cache expiration policies

Troubleshooting

Remote Cache Not Available

❌ Remote cache is not available

Solutions:

1. Check network connectivity: ping cache.example.com
2. Verify URL is correct: nip cache remote status
3. Check server is running: curl https://cache.example.com/v1/health
4. Verify firewall rules allow outbound HTTPS

Authentication Failed

❌ Remote cache lookup failed: 401 Unauthorized

Solutions:

1. Verify API key is correct
2. Check API key hasn't expired
3. Ensure Authorization header is being sent
4. Contact cache server administrator

Download Failed

❌ Download failed: timeout

Solutions:

1. Increase timeout: Edit ~/.config/nip/remote-cache.json
2. Check network bandwidth
3. Try again later (temporary network issue)
4. Fall back to local build: nip build --no-remote-cache

Upload Failed

⚠️  Remote upload failed (local cache still available)

Solutions:

1. Check disk space on server
2. Verify write permissions
3. Check artifact size limits
4. Build still succeeded - artifact is in local cache

Best Practices

For Developers

1. Enable remote cache for all team members
2. Use consistent build configurations to maximize cache hits
3. Don't disable cache unless debugging build issues
4. Report cache issues to team lead

For Teams

1. Set up dedicated cache server for team
2. Use separate API keys per project
3. Monitor cache hit rates to optimize configurations
4. Implement cache retention policies (e.g., 30 days)
5. Document cache server URL in team wiki

For CI/CD

1. Always enable remote cache in CI pipelines
2. Use read-only keys for pull requests from forks
3. Use read-write keys for main branch builds
4. Monitor cache storage and implement cleanup
5. Set appropriate timeouts for CI environment

Advanced Configuration

Multiple Cache Servers

Configure fallback cache servers:

{
  "servers": [
    {
      "url": "https://cache-primary.example.com",
      "priority": 1
    },
    {
      "url": "https://cache-backup.example.com",
      "priority": 2
    }
  ]
}

Cache Policies

Configure cache behavior:

{
  "policies": {
    "uploadOnBuild": true,
    "downloadBeforeBuild": true,
    "fallbackToLocal": true,
    "retryAttempts": 3,
    "retryDelay": 5
  }
}

Monitoring

Cache Metrics

Track these metrics for optimal performance:

• Hit Rate: Percentage of builds using cache
• Upload Success Rate: Percentage of successful uploads
• Download Success Rate: Percentage of successful downloads
• Average Download Time: Time to download artifacts
• Cache Size: Total storage used
• Cache Age: Average age of cached artifacts

Example Monitoring Dashboard

# Get cache statistics
nip cache stats

# Get remote cache status
nip cache remote status

# List recent cache operations
nip cache list --recent

Migration Guide

From Local-Only to Remote Cache

1. Set up remote cache server
2. Configure all team members:

   nip cache remote config --url https://cache.example.com
   nip cache remote config --api-key <team-key>
   nip cache remote config --enable
   

3. Push existing local cache (optional):

   for pkg in $(nip cache list --format=simple); do
     nip cache remote push $pkg
   done
   

4. Update CI/CD pipelines with remote cache config
5. Monitor adoption and cache hit rates

FAQ

Q: Does remote cache slow down builds? A: No, remote cache checks are fast (<1 second). If remote is slow or unavailable, it falls back to local cache or building from source.

Q: How much bandwidth does remote cache use? A: Only when downloading artifacts. A typical package is 10-100MB. With good cache hit rates, bandwidth usage is minimal.

Q: Can I use remote cache without local cache? A: No, local cache is always used. Remote cache supplements local cache for team sharing.

Q: What happens if remote cache is down? A: Builds continue normally using local cache or building from source. Remote cache is optional and non-blocking.

Q: How do I clear remote cache? A: Contact your cache server administrator. Remote cache clearing is typically done server-side with retention policies.

Q: Can I host my own cache server? A: Yes! See the "Example Server Implementation" section for a reference implementation.

See Also

• Binary Cache Guide - Local cache documentation
• Build System Guide - Building from source
• Configuration Guide - NIP configuration options