# Remote Binary Cache Guide

## Overview

The NIP remote binary cache enables sharing of compiled artifacts across machines and teams. This dramatically speeds up builds in CI/CD pipelines and development environments by allowing teams to share build results.

## Features

- **Automatic Upload/Download**: Artifacts are automatically uploaded after successful builds and downloaded before builds
- **Team Sharing**: Share builds across team members and CI/CD runners
- **HTTP API**: Simple REST API for cache operations
- **Authentication**: API key-based authentication for secure access
- **Fallback**: Gracefully falls back to local cache if remote is unavailable
- **Bandwidth Efficient**: Only transfers artifacts when needed

## Quick Start

### 1. Configure Remote Cache

```bash
# Set remote cache URL
nip cache remote config --url https://cache.example.com

# Set API key for authentication
nip cache remote config --api-key your-api-key-here

# Enable remote cache
nip cache remote config --enable
```

### 2. Check Status

```bash
nip cache remote status
```

Output:
```
Remote Cache Status
===================

Enabled: Yes
URL: https://cache.example.com
API Key: ***configured***
Timeout: 300 seconds

Testing connection...
✅ Remote cache is available
```

### 3. Build with Remote Cache

Remote cache is now automatically used during builds:

```bash
# First build - compiles and uploads to remote cache
nip build vim +python+ruby

# On another machine - downloads from remote cache
nip build vim +python+ruby
```

## Configuration

### Configuration File

Remote cache settings are stored in `~/.config/nip/remote-cache.json`:

```json
{
  "url": "https://cache.example.com",
  "apiKey": "your-api-key-here",
  "timeout": 300,
  "enabled": true
}
```

### Configuration Options

| Option | Description | Default |
|--------|-------------|---------|
| `url` | Remote cache server URL | "" |
| `apiKey` | Authentication API key | "" |
| `timeout` | Request timeout in seconds | 300 |
| `enabled` | Enable/disable remote cache | false |

### Environment Variables

You can also configure via environment variables:

```bash
export NIP_REMOTE_CACHE_URL="https://cache.example.com"
export NIP_REMOTE_CACHE_API_KEY="your-api-key-here"
export NIP_REMOTE_CACHE_ENABLED="true"
```

## Usage

### Automatic Mode (Recommended)

Remote cache works automatically during builds:

```bash
# Build package - automatically checks remote cache first
nip build firefox +wayland

# If not in remote cache:
#   1. Checks local cache
#   2. Builds from source
#   3. Uploads to remote cache
#   4. Uploads to local cache

# If in remote cache:
#   1. Downloads to local cache
#   2. Uses cached artifact (instant!)
```

### Manual Operations

#### Pull from Remote Cache

```bash
# Pull specific package from remote cache
nip cache remote pull vim 9.0
```

#### Push to Remote Cache

```bash
# Push specific package to remote cache
nip cache remote push vim 9.0
```

#### Check Remote Status

```bash
# Test remote cache connectivity
nip cache remote status
```

## Cache Lookup Flow

```
┌─────────────────────────────────────────────────────────────┐
│                     Build Request                            │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│              Check Local Cache                               │
│  ~/.cache/nip/binary-cache/                                  │
└─────────────────────────────────────────────────────────────┘
                            │
                    ┌───────┴───────┐
                    │               │
                 Found          Not Found
                    │               │
                    │               ▼
                    │   ┌─────────────────────────────────────┐
                    │   │    Check Remote Cache               │
                    │   │  (if enabled)                       │
                    │   └─────────────────────────────────────┘
                    │               │
                    │       ┌───────┴───────┐
                    │       │               │
                    │    Found          Not Found
                    │       │               │
                    │       ▼               ▼
                    │   ┌─────────┐   ┌─────────────┐
                    │   │Download │   │Build from   │
                    │   │to Local │   │Source       │
                    │   └─────────┘   └─────────────┘
                    │       │               │
                    │       │               ▼
                    │       │         ┌─────────────┐
                    │       │         │Upload to    │
                    │       │         │Remote Cache │
                    │       │         └─────────────┘
                    │       │               │
                    │       └───────┬───────┘
                    │               │
                    └───────┬───────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                  Use Cached Artifact                         │
└─────────────────────────────────────────────────────────────┘
```

## Remote Cache Server

### API Endpoints

The remote cache server implements a simple HTTP API:

#### Health Check
```
GET /v1/health
Response: 200 OK
```

#### Lookup Artifact
```
GET /v1/artifacts/{cache-key}
Response: 200 OK with download URL, or 404 Not Found
```

#### Get Metadata
```
GET /v1/artifacts/{cache-key}/metadata
Response: 200 OK with JSON metadata
```

#### Upload Artifact
```
POST /v1/artifacts/{cache-key}
Content-Type: multipart/form-data
- file: artifact file
- metadata: JSON metadata

Response: 201 Created
```

### Authentication

All requests (except health check) require authentication:

```
Authorization: Bearer <api-key>
```

### Example Server Implementation

A simple reference server implementation:

```python
from flask import Flask, request, jsonify, send_file
import os
import json

app = Flask(__name__)
CACHE_DIR = "/var/cache/nip"
API_KEY = os.environ.get("NIP_CACHE_API_KEY")

def check_auth():
    auth = request.headers.get("Authorization")
    if not auth or not auth.startswith("Bearer "):
        return False
    token = auth.split(" ")[1]
    return token == API_KEY

@app.route("/v1/health")
def health():
    return jsonify({"status": "ok"})

@app.route("/v1/artifacts/<cache_key>")
def get_artifact(cache_key):
    if not check_auth():
        return jsonify({"error": "unauthorized"}), 401

    artifact_path = os.path.join(CACHE_DIR, cache_key, "artifact.tar.gz")
    if os.path.exists(artifact_path):
        return jsonify({"downloadUrl": f"/v1/download/{cache_key}"})
    return jsonify({"error": "not found"}), 404

@app.route("/v1/download/<cache_key>")
def download_artifact(cache_key):
    if not check_auth():
        return jsonify({"error": "unauthorized"}), 401

    artifact_path = os.path.join(CACHE_DIR, cache_key, "artifact.tar.gz")
    return send_file(artifact_path)

@app.route("/v1/artifacts/<cache_key>/metadata")
def get_metadata(cache_key):
    if not check_auth():
        return jsonify({"error": "unauthorized"}), 401

    metadata_path = os.path.join(CACHE_DIR, cache_key, "metadata.json")
    if os.path.exists(metadata_path):
        with open(metadata_path) as f:
            return jsonify(json.load(f))
    return jsonify({"error": "not found"}), 404

@app.route("/v1/artifacts/<cache_key>", methods=["POST"])
def upload_artifact(cache_key):
    if not check_auth():
        return jsonify({"error": "unauthorized"}), 401

    cache_dir = os.path.join(CACHE_DIR, cache_key)
    os.makedirs(cache_dir, exist_ok=True)

    # Save artifact
    file = request.files["file"]
    artifact_path = os.path.join(cache_dir, "artifact.tar.gz")
    file.save(artifact_path)

    # Save metadata
    metadata = json.loads(request.form["metadata"])
    metadata_path = os.path.join(cache_dir, "metadata.json")
    with open(metadata_path, "w") as f:
        json.dump(metadata, f)

    return jsonify({"status": "created"}), 201

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)
```

## CI/CD Integration

### GitHub Actions

```yaml
name: Build with NIP Cache

on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install NIP
        run: |
          curl -sSL https://nip.example.com/install.sh | sh

      - name: Configure Remote Cache
        env:
          NIP_CACHE_API_KEY: ${{ secrets.NIP_CACHE_API_KEY }}
        run: |
          nip cache remote config --url https://cache.example.com
          nip cache remote config --api-key $NIP_CACHE_API_KEY
          nip cache remote config --enable

      - name: Build Package
        run: nip build myapp +production
```

### GitLab CI

```yaml
build:
  image: ubuntu:latest
  before_script:
    - curl -sSL https://nip.example.com/install.sh | sh
    - nip cache remote config --url https://cache.example.com
    - nip cache remote config --api-key $NIP_CACHE_API_KEY
    - nip cache remote config --enable
  script:
    - nip build myapp +production
  variables:
    NIP_CACHE_API_KEY: $CI_CACHE_API_KEY
```

## Performance Impact

### Build Time Comparison

| Scenario | Without Remote Cache | With Remote Cache | Speedup |
|----------|---------------------|-------------------|---------|
| First build (team) | 15 minutes | 15 minutes | 1x |
| Second build (same machine) | <1 second (local) | <1 second (local) | - |
| Second build (different machine) | 15 minutes | <1 second | 900x |
| CI/CD pipeline | 15 minutes | <1 second | 900x |

### Real-World Example

**Development Team (5 developers)**

Without remote cache:
- Each developer builds from scratch: 5 × 15 min = 75 minutes total
- CI builds from scratch: +15 minutes
- **Total: 90 minutes of build time**

With remote cache:
- First developer builds: 15 minutes
- Other developers use cache: 4 × <1 second ≈ 0 minutes
- CI uses cache: <1 second
- **Total: 15 minutes of build time**

**Savings: 75 minutes (83% reduction)**

## Security Considerations

### API Key Management

- **Never commit API keys to version control**
- Use environment variables or secret management systems
- Rotate keys regularly
- Use different keys for different teams/projects

### Network Security

- **Always use HTTPS** for remote cache URLs
- Consider VPN or private network for sensitive builds
- Implement rate limiting on server
- Monitor for unusual access patterns

### Access Control

- Implement read/write permissions
- Separate keys for CI vs developers
- Audit log all cache operations
- Implement cache expiration policies

## Troubleshooting

### Remote Cache Not Available

```bash
❌ Remote cache is not available
```

**Solutions:**
1. Check network connectivity: `ping cache.example.com`
2. Verify URL is correct: `nip cache remote status`
3. Check server is running: `curl https://cache.example.com/v1/health`
4. Verify firewall rules allow outbound HTTPS

### Authentication Failed

```bash
❌ Remote cache lookup failed: 401 Unauthorized
```

**Solutions:**
1. Verify API key is correct
2. Check API key hasn't expired
3. Ensure Authorization header is being sent
4. Contact cache server administrator

### Download Failed

```bash
❌ Download failed: timeout
```

**Solutions:**
1. Increase timeout: Edit `~/.config/nip/remote-cache.json`
2. Check network bandwidth
3. Try again later (temporary network issue)
4. Fall back to local build: `nip build --no-remote-cache`

### Upload Failed

```bash
⚠️  Remote upload failed (local cache still available)
```

**Solutions:**
1. Check disk space on server
2. Verify write permissions
3. Check artifact size limits
4. Build still succeeded - artifact is in local cache

## Best Practices

### For Developers

1. **Enable remote cache** for all team members
2. **Use consistent build configurations** to maximize cache hits
3. **Don't disable cache** unless debugging build issues
4. **Report cache issues** to team lead

### For Teams

1. **Set up dedicated cache server** for team
2. **Use separate API keys** per project
3. **Monitor cache hit rates** to optimize configurations
4. **Implement cache retention policies** (e.g., 30 days)
5. **Document cache server URL** in team wiki

### For CI/CD

1. **Always enable remote cache** in CI pipelines
2. **Use read-only keys** for pull requests from forks
3. **Use read-write keys** for main branch builds
4. **Monitor cache storage** and implement cleanup
5. **Set appropriate timeouts** for CI environment

## Advanced Configuration

### Multiple Cache Servers

Configure fallback cache servers:

```json
{
  "servers": [
    {
      "url": "https://cache-primary.example.com",
      "priority": 1
    },
    {
      "url": "https://cache-backup.example.com",
      "priority": 2
    }
  ]
}
```

### Cache Policies

Configure cache behavior:

```json
{
  "policies": {
    "uploadOnBuild": true,
    "downloadBeforeBuild": true,
    "fallbackToLocal": true,
    "retryAttempts": 3,
    "retryDelay": 5
  }
}
```

## Monitoring

### Cache Metrics

Track these metrics for optimal performance:

- **Hit Rate**: Percentage of builds using cache
- **Upload Success Rate**: Percentage of successful uploads
- **Download Success Rate**: Percentage of successful downloads
- **Average Download Time**: Time to download artifacts
- **Cache Size**: Total storage used
- **Cache Age**: Average age of cached artifacts

### Example Monitoring Dashboard

```bash
# Get cache statistics
nip cache stats

# Get remote cache status
nip cache remote status

# List recent cache operations
nip cache list --recent
```

## Migration Guide

### From Local-Only to Remote Cache

1. **Set up remote cache server**
2. **Configure all team members**:
   ```bash
   nip cache remote config --url https://cache.example.com
   nip cache remote config --api-key <team-key>
   nip cache remote config --enable
   ```
3. **Push existing local cache** (optional):
   ```bash
   for pkg in $(nip cache list --format=simple); do
     nip cache remote push $pkg
   done
   ```
4. **Update CI/CD pipelines** with remote cache config
5. **Monitor adoption** and cache hit rates

## FAQ

**Q: Does remote cache slow down builds?**
A: No, remote cache checks are fast (<1 second). If remote is slow or unavailable, it falls back to local cache or building from source.

**Q: How much bandwidth does remote cache use?**
A: Only when downloading artifacts. A typical package is 10-100MB. With good cache hit rates, bandwidth usage is minimal.

**Q: Can I use remote cache without local cache?**
A: No, local cache is always used. Remote cache supplements local cache for team sharing.

**Q: What happens if remote cache is down?**
A: Builds continue normally using local cache or building from source. Remote cache is optional and non-blocking.

**Q: How do I clear remote cache?**
A: Contact your cache server administrator. Remote cache clearing is typically done server-side with retention policies.

**Q: Can I host my own cache server?**
A: Yes! See the "Example Server Implementation" section for a reference implementation.

## See Also

- [Binary Cache Guide](binary-cache.md) - Local cache documentation
- [Build System Guide](source-build-guide.md) - Building from source
- [Configuration Guide](configuration.md) - NIP configuration options