591 lines
16 KiB
Markdown
591 lines
16 KiB
Markdown
# Remote Binary Cache Guide
|
||
|
||
## Overview
|
||
|
||
The NIP remote binary cache enables sharing of compiled artifacts across machines and teams. This dramatically speeds up builds in CI/CD pipelines and development environments by allowing teams to share build results.
|
||
|
||
## Features
|
||
|
||
- **Automatic Upload/Download**: Artifacts are automatically uploaded after successful builds and downloaded before builds
|
||
- **Team Sharing**: Share builds across team members and CI/CD runners
|
||
- **HTTP API**: Simple REST API for cache operations
|
||
- **Authentication**: API key-based authentication for secure access
|
||
- **Fallback**: Gracefully falls back to local cache if remote is unavailable
|
||
- **Bandwidth Efficient**: Only transfers artifacts when needed
|
||
|
||
## Quick Start
|
||
|
||
### 1. Configure Remote Cache
|
||
|
||
```bash
|
||
# Set remote cache URL
|
||
nip cache remote config --url https://cache.example.com
|
||
|
||
# Set API key for authentication
|
||
nip cache remote config --api-key your-api-key-here
|
||
|
||
# Enable remote cache
|
||
nip cache remote config --enable
|
||
```
|
||
|
||
### 2. Check Status
|
||
|
||
```bash
|
||
nip cache remote status
|
||
```
|
||
|
||
Output:
|
||
```
|
||
Remote Cache Status
|
||
===================
|
||
|
||
Enabled: Yes
|
||
URL: https://cache.example.com
|
||
API Key: ***configured***
|
||
Timeout: 300 seconds
|
||
|
||
Testing connection...
|
||
✅ Remote cache is available
|
||
```
|
||
|
||
### 3. Build with Remote Cache
|
||
|
||
Remote cache is now automatically used during builds:
|
||
|
||
```bash
|
||
# First build - compiles and uploads to remote cache
|
||
nip build vim +python+ruby
|
||
|
||
# On another machine - downloads from remote cache
|
||
nip build vim +python+ruby
|
||
```
|
||
|
||
## Configuration
|
||
|
||
### Configuration File
|
||
|
||
Remote cache settings are stored in `~/.config/nip/remote-cache.json`:
|
||
|
||
```json
|
||
{
|
||
"url": "https://cache.example.com",
|
||
"apiKey": "your-api-key-here",
|
||
"timeout": 300,
|
||
"enabled": true
|
||
}
|
||
```
|
||
|
||
### Configuration Options
|
||
|
||
| Option | Description | Default |
|
||
|--------|-------------|---------|
|
||
| `url` | Remote cache server URL | "" |
|
||
| `apiKey` | Authentication API key | "" |
|
||
| `timeout` | Request timeout in seconds | 300 |
|
||
| `enabled` | Enable/disable remote cache | false |
|
||
|
||
### Environment Variables
|
||
|
||
You can also configure via environment variables:
|
||
|
||
```bash
|
||
export NIP_REMOTE_CACHE_URL="https://cache.example.com"
|
||
export NIP_REMOTE_CACHE_API_KEY="your-api-key-here"
|
||
export NIP_REMOTE_CACHE_ENABLED="true"
|
||
```
|
||
|
||
## Usage
|
||
|
||
### Automatic Mode (Recommended)
|
||
|
||
Remote cache works automatically during builds:
|
||
|
||
```bash
|
||
# Build package - automatically checks remote cache first
|
||
nip build firefox +wayland
|
||
|
||
# If not in remote cache:
|
||
# 1. Checks local cache
|
||
# 2. Builds from source
|
||
# 3. Uploads to remote cache
|
||
# 4. Uploads to local cache
|
||
|
||
# If in remote cache:
|
||
# 1. Downloads to local cache
|
||
# 2. Uses cached artifact (instant!)
|
||
```
|
||
|
||
### Manual Operations
|
||
|
||
#### Pull from Remote Cache
|
||
|
||
```bash
|
||
# Pull specific package from remote cache
|
||
nip cache remote pull vim 9.0
|
||
```
|
||
|
||
#### Push to Remote Cache
|
||
|
||
```bash
|
||
# Push specific package to remote cache
|
||
nip cache remote push vim 9.0
|
||
```
|
||
|
||
#### Check Remote Status
|
||
|
||
```bash
|
||
# Test remote cache connectivity
|
||
nip cache remote status
|
||
```
|
||
|
||
## Cache Lookup Flow
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Build Request │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Check Local Cache │
|
||
│ ~/.cache/nip/binary-cache/ │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
│
|
||
┌───────┴───────┐
|
||
│ │
|
||
Found Not Found
|
||
│ │
|
||
│ ▼
|
||
│ ┌─────────────────────────────────────┐
|
||
│ │ Check Remote Cache │
|
||
│ │ (if enabled) │
|
||
│ └─────────────────────────────────────┘
|
||
│ │
|
||
│ ┌───────┴───────┐
|
||
│ │ │
|
||
│ Found Not Found
|
||
│ │ │
|
||
│ ▼ ▼
|
||
│ ┌─────────┐ ┌─────────────┐
|
||
│ │Download │ │Build from │
|
||
│ │to Local │ │Source │
|
||
│ └─────────┘ └─────────────┘
|
||
│ │ │
|
||
│ │ ▼
|
||
│ │ ┌─────────────┐
|
||
│ │ │Upload to │
|
||
│ │ │Remote Cache │
|
||
│ │ └─────────────┘
|
||
│ │ │
|
||
│ └───────┬───────┘
|
||
│ │
|
||
└───────┬───────┘
|
||
│
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────┐
|
||
│ Use Cached Artifact │
|
||
└─────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Remote Cache Server
|
||
|
||
### API Endpoints
|
||
|
||
The remote cache server implements a simple HTTP API:
|
||
|
||
#### Health Check
|
||
```
|
||
GET /v1/health
|
||
Response: 200 OK
|
||
```
|
||
|
||
#### Lookup Artifact
|
||
```
|
||
GET /v1/artifacts/{cache-key}
|
||
Response: 200 OK with download URL, or 404 Not Found
|
||
```
|
||
|
||
#### Get Metadata
|
||
```
|
||
GET /v1/artifacts/{cache-key}/metadata
|
||
Response: 200 OK with JSON metadata
|
||
```
|
||
|
||
#### Upload Artifact
|
||
```
|
||
POST /v1/artifacts/{cache-key}
|
||
Content-Type: multipart/form-data
|
||
- file: artifact file
|
||
- metadata: JSON metadata
|
||
|
||
Response: 201 Created
|
||
```
|
||
|
||
### Authentication
|
||
|
||
All requests (except health check) require authentication:
|
||
|
||
```
|
||
Authorization: Bearer <api-key>
|
||
```
|
||
|
||
### Example Server Implementation
|
||
|
||
A simple reference server implementation:
|
||
|
||
```python
|
||
from flask import Flask, request, jsonify, send_file
|
||
import os
|
||
import json
|
||
|
||
app = Flask(__name__)
|
||
CACHE_DIR = "/var/cache/nip"
|
||
API_KEY = os.environ.get("NIP_CACHE_API_KEY")
|
||
|
||
def check_auth():
|
||
auth = request.headers.get("Authorization")
|
||
if not auth or not auth.startswith("Bearer "):
|
||
return False
|
||
token = auth.split(" ")[1]
|
||
return token == API_KEY
|
||
|
||
@app.route("/v1/health")
|
||
def health():
|
||
return jsonify({"status": "ok"})
|
||
|
||
@app.route("/v1/artifacts/<cache_key>")
|
||
def get_artifact(cache_key):
|
||
if not check_auth():
|
||
return jsonify({"error": "unauthorized"}), 401
|
||
|
||
artifact_path = os.path.join(CACHE_DIR, cache_key, "artifact.tar.gz")
|
||
if os.path.exists(artifact_path):
|
||
return jsonify({"downloadUrl": f"/v1/download/{cache_key}"})
|
||
return jsonify({"error": "not found"}), 404
|
||
|
||
@app.route("/v1/download/<cache_key>")
|
||
def download_artifact(cache_key):
|
||
if not check_auth():
|
||
return jsonify({"error": "unauthorized"}), 401
|
||
|
||
artifact_path = os.path.join(CACHE_DIR, cache_key, "artifact.tar.gz")
|
||
return send_file(artifact_path)
|
||
|
||
@app.route("/v1/artifacts/<cache_key>/metadata")
|
||
def get_metadata(cache_key):
|
||
if not check_auth():
|
||
return jsonify({"error": "unauthorized"}), 401
|
||
|
||
metadata_path = os.path.join(CACHE_DIR, cache_key, "metadata.json")
|
||
if os.path.exists(metadata_path):
|
||
with open(metadata_path) as f:
|
||
return jsonify(json.load(f))
|
||
return jsonify({"error": "not found"}), 404
|
||
|
||
@app.route("/v1/artifacts/<cache_key>", methods=["POST"])
|
||
def upload_artifact(cache_key):
|
||
if not check_auth():
|
||
return jsonify({"error": "unauthorized"}), 401
|
||
|
||
cache_dir = os.path.join(CACHE_DIR, cache_key)
|
||
os.makedirs(cache_dir, exist_ok=True)
|
||
|
||
# Save artifact
|
||
file = request.files["file"]
|
||
artifact_path = os.path.join(cache_dir, "artifact.tar.gz")
|
||
file.save(artifact_path)
|
||
|
||
# Save metadata
|
||
metadata = json.loads(request.form["metadata"])
|
||
metadata_path = os.path.join(cache_dir, "metadata.json")
|
||
with open(metadata_path, "w") as f:
|
||
json.dump(metadata, f)
|
||
|
||
return jsonify({"status": "created"}), 201
|
||
|
||
if __name__ == "__main__":
|
||
app.run(host="0.0.0.0", port=8080)
|
||
```
|
||
|
||
## CI/CD Integration
|
||
|
||
### GitHub Actions
|
||
|
||
```yaml
|
||
name: Build with NIP Cache
|
||
|
||
on: [push, pull_request]
|
||
|
||
jobs:
|
||
build:
|
||
runs-on: ubuntu-latest
|
||
steps:
|
||
- uses: actions/checkout@v3
|
||
|
||
- name: Install NIP
|
||
run: |
|
||
curl -sSL https://nip.example.com/install.sh | sh
|
||
|
||
- name: Configure Remote Cache
|
||
env:
|
||
NIP_CACHE_API_KEY: ${{ secrets.NIP_CACHE_API_KEY }}
|
||
run: |
|
||
nip cache remote config --url https://cache.example.com
|
||
nip cache remote config --api-key $NIP_CACHE_API_KEY
|
||
nip cache remote config --enable
|
||
|
||
- name: Build Package
|
||
run: nip build myapp +production
|
||
```
|
||
|
||
### GitLab CI
|
||
|
||
```yaml
|
||
build:
|
||
image: ubuntu:latest
|
||
before_script:
|
||
- curl -sSL https://nip.example.com/install.sh | sh
|
||
- nip cache remote config --url https://cache.example.com
|
||
- nip cache remote config --api-key $NIP_CACHE_API_KEY
|
||
- nip cache remote config --enable
|
||
script:
|
||
- nip build myapp +production
|
||
variables:
|
||
NIP_CACHE_API_KEY: $CI_CACHE_API_KEY
|
||
```
|
||
|
||
## Performance Impact
|
||
|
||
### Build Time Comparison
|
||
|
||
| Scenario | Without Remote Cache | With Remote Cache | Speedup |
|
||
|----------|---------------------|-------------------|---------|
|
||
| First build (team) | 15 minutes | 15 minutes | 1x |
|
||
| Second build (same machine) | <1 second (local) | <1 second (local) | - |
|
||
| Second build (different machine) | 15 minutes | <1 second | 900x |
|
||
| CI/CD pipeline | 15 minutes | <1 second | 900x |
|
||
|
||
### Real-World Example
|
||
|
||
**Development Team (5 developers)**
|
||
|
||
Without remote cache:
|
||
- Each developer builds from scratch: 5 × 15 min = 75 minutes total
|
||
- CI builds from scratch: +15 minutes
|
||
- **Total: 90 minutes of build time**
|
||
|
||
With remote cache:
|
||
- First developer builds: 15 minutes
|
||
- Other developers use cache: 4 × <1 second ≈ 0 minutes
|
||
- CI uses cache: <1 second
|
||
- **Total: 15 minutes of build time**
|
||
|
||
**Savings: 75 minutes (83% reduction)**
|
||
|
||
## Security Considerations
|
||
|
||
### API Key Management
|
||
|
||
- **Never commit API keys to version control**
|
||
- Use environment variables or secret management systems
|
||
- Rotate keys regularly
|
||
- Use different keys for different teams/projects
|
||
|
||
### Network Security
|
||
|
||
- **Always use HTTPS** for remote cache URLs
|
||
- Consider VPN or private network for sensitive builds
|
||
- Implement rate limiting on server
|
||
- Monitor for unusual access patterns
|
||
|
||
### Access Control
|
||
|
||
- Implement read/write permissions
|
||
- Separate keys for CI vs developers
|
||
- Audit log all cache operations
|
||
- Implement cache expiration policies
|
||
|
||
## Troubleshooting
|
||
|
||
### Remote Cache Not Available
|
||
|
||
```bash
|
||
❌ Remote cache is not available
|
||
```
|
||
|
||
**Solutions:**
|
||
1. Check network connectivity: `ping cache.example.com`
|
||
2. Verify URL is correct: `nip cache remote status`
|
||
3. Check server is running: `curl https://cache.example.com/v1/health`
|
||
4. Verify firewall rules allow outbound HTTPS
|
||
|
||
### Authentication Failed
|
||
|
||
```bash
|
||
❌ Remote cache lookup failed: 401 Unauthorized
|
||
```
|
||
|
||
**Solutions:**
|
||
1. Verify API key is correct
|
||
2. Check API key hasn't expired
|
||
3. Ensure Authorization header is being sent
|
||
4. Contact cache server administrator
|
||
|
||
### Download Failed
|
||
|
||
```bash
|
||
❌ Download failed: timeout
|
||
```
|
||
|
||
**Solutions:**
|
||
1. Increase timeout: Edit `~/.config/nip/remote-cache.json`
|
||
2. Check network bandwidth
|
||
3. Try again later (temporary network issue)
|
||
4. Fall back to local build: `nip build --no-remote-cache`
|
||
|
||
### Upload Failed
|
||
|
||
```bash
|
||
⚠️ Remote upload failed (local cache still available)
|
||
```
|
||
|
||
**Solutions:**
|
||
1. Check disk space on server
|
||
2. Verify write permissions
|
||
3. Check artifact size limits
|
||
4. Build still succeeded - artifact is in local cache
|
||
|
||
## Best Practices
|
||
|
||
### For Developers
|
||
|
||
1. **Enable remote cache** for all team members
|
||
2. **Use consistent build configurations** to maximize cache hits
|
||
3. **Don't disable cache** unless debugging build issues
|
||
4. **Report cache issues** to team lead
|
||
|
||
### For Teams
|
||
|
||
1. **Set up dedicated cache server** for team
|
||
2. **Use separate API keys** per project
|
||
3. **Monitor cache hit rates** to optimize configurations
|
||
4. **Implement cache retention policies** (e.g., 30 days)
|
||
5. **Document cache server URL** in team wiki
|
||
|
||
### For CI/CD
|
||
|
||
1. **Always enable remote cache** in CI pipelines
|
||
2. **Use read-only keys** for pull requests from forks
|
||
3. **Use read-write keys** for main branch builds
|
||
4. **Monitor cache storage** and implement cleanup
|
||
5. **Set appropriate timeouts** for CI environment
|
||
|
||
## Advanced Configuration
|
||
|
||
### Multiple Cache Servers
|
||
|
||
Configure fallback cache servers:
|
||
|
||
```json
|
||
{
|
||
"servers": [
|
||
{
|
||
"url": "https://cache-primary.example.com",
|
||
"priority": 1
|
||
},
|
||
{
|
||
"url": "https://cache-backup.example.com",
|
||
"priority": 2
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
### Cache Policies
|
||
|
||
Configure cache behavior:
|
||
|
||
```json
|
||
{
|
||
"policies": {
|
||
"uploadOnBuild": true,
|
||
"downloadBeforeBuild": true,
|
||
"fallbackToLocal": true,
|
||
"retryAttempts": 3,
|
||
"retryDelay": 5
|
||
}
|
||
}
|
||
```
|
||
|
||
## Monitoring
|
||
|
||
### Cache Metrics
|
||
|
||
Track these metrics for optimal performance:
|
||
|
||
- **Hit Rate**: Percentage of builds using cache
|
||
- **Upload Success Rate**: Percentage of successful uploads
|
||
- **Download Success Rate**: Percentage of successful downloads
|
||
- **Average Download Time**: Time to download artifacts
|
||
- **Cache Size**: Total storage used
|
||
- **Cache Age**: Average age of cached artifacts
|
||
|
||
### Example Monitoring Dashboard
|
||
|
||
```bash
|
||
# Get cache statistics
|
||
nip cache stats
|
||
|
||
# Get remote cache status
|
||
nip cache remote status
|
||
|
||
# List recent cache operations
|
||
nip cache list --recent
|
||
```
|
||
|
||
## Migration Guide
|
||
|
||
### From Local-Only to Remote Cache
|
||
|
||
1. **Set up remote cache server**
|
||
2. **Configure all team members**:
|
||
```bash
|
||
nip cache remote config --url https://cache.example.com
|
||
nip cache remote config --api-key <team-key>
|
||
nip cache remote config --enable
|
||
```
|
||
3. **Push existing local cache** (optional):
|
||
```bash
|
||
for pkg in $(nip cache list --format=simple); do
|
||
nip cache remote push $pkg
|
||
done
|
||
```
|
||
4. **Update CI/CD pipelines** with remote cache config
|
||
5. **Monitor adoption** and cache hit rates
|
||
|
||
## FAQ
|
||
|
||
**Q: Does remote cache slow down builds?**
|
||
A: No, remote cache checks are fast (<1 second). If remote is slow or unavailable, it falls back to local cache or building from source.
|
||
|
||
**Q: How much bandwidth does remote cache use?**
|
||
A: Only when downloading artifacts. A typical package is 10-100MB. With good cache hit rates, bandwidth usage is minimal.
|
||
|
||
**Q: Can I use remote cache without local cache?**
|
||
A: No, local cache is always used. Remote cache supplements local cache for team sharing.
|
||
|
||
**Q: What happens if remote cache is down?**
|
||
A: Builds continue normally using local cache or building from source. Remote cache is optional and non-blocking.
|
||
|
||
**Q: How do I clear remote cache?**
|
||
A: Contact your cache server administrator. Remote cache clearing is typically done server-side with retention policies.
|
||
|
||
**Q: Can I host my own cache server?**
|
||
A: Yes! See the "Example Server Implementation" section for a reference implementation.
|
||
|
||
## See Also
|
||
|
||
- [Binary Cache Guide](binary-cache.md) - Local cache documentation
|
||
- [Build System Guide](source-build-guide.md) - Building from source
|
||
- [Configuration Guide](configuration.md) - NIP configuration options
|