nip/docs/bootstrap-detection-flow.md

# Bootstrap Detection Flow

## Overview

The NIP bootstrap system automatically detects the best available build tool installation method based on the current system state. This document describes the detection logic, fallback mechanisms, and decision flow.

## Detection Hierarchy

The system follows this priority order when determining how to install build tools:

1. **Native Package Manager** (Highest Priority)
2. **Existing Bootstrap Installation** (Already installed)
3. **Recipe-Based Bootstrap** (Automatic download and install)
4. **Container-Based Build** (Fallback for complex scenarios)
5. **Manual Installation** (Last resort - user guidance)

## Flow Diagram

```
User runs: nip build <package>
           |
           v
    [Check Build Tools]
           |
           +---> Tools Available? --YES--> [Proceed with Build]
           |
           NO
           |
           v
    [Automatic Detection]
           |
           +---> Check Native Package Manager
           |     (pacman, apt, dnf, etc.)
           |     |
           |     +---> Available? --YES--> [Install via Package Manager]
           |                                        |
           |                                        v
           NO                              [Verify Installation]
           |                                        |
           v                                        v
    [Check Existing Bootstrap]            [Proceed with Build]
           |
           +---> Bootstrap Installed? --YES--> [Verify and Use]
           |                                          |
           NO                                         v
           |                                 [Proceed with Build]
           v
    [Recipe-Based Bootstrap]
           |
           +---> Select Recipe by Platform
           |     (Nix, PKGSRC, Gentoo)
           |     |
           |     v
           |  [Download Binary]
           |     |
           |     v
           |  [Verify Checksum]
           |     |
           |     v
           |  [Install to ~/.nip/bootstrap/]
           |     |
           |     v
           |  [Create Wrappers]
           |     |
           |     v
           |  [Verify Installation]
           |     |
           |     +---> Success? --YES--> [Proceed with Build]
           |
           NO
           |
           v
    [Container Fallback]
           |
           +---> Podman Available? --YES--> [Build in Container]
           |                                        |
           |                                        v
           NO                              [Extract Artifacts]
           |                                        |
           v                                        v
    [Manual Installation Guide]          [Proceed with Build]
           |
           v
    [Show Instructions]
           |
           v
    [Exit with Error]
```

## Detection Logic Details

### 1. Native Package Manager Detection

**Purpose**: Use system package manager when available for official support and updates.

**Detection Method**:
```nim
proc detectNativePackageManager(): Option[string] =
  # Check for common package managers
  if commandExists("pacman"): return some("pacman")
  if commandExists("apt-get"): return some("apt")
  if commandExists("dnf"): return some("dnf")
  if commandExists("zypper"): return some("zypper")
  return none(string)
```

**Installation Commands**:
- **Arch Linux**: `pacman -S --needed base-devel`
- **Debian/Ubuntu**: `apt-get install build-essential`
- **Fedora**: `dnf groupinstall "Development Tools"`
- **openSUSE**: `zypper install -t pattern devel_basis`

**Advantages**:
- Official packages with security updates
- System integration and dependency management
- No additional maintenance required

**When Used**:
- System has a recognized package manager
- User has appropriate permissions (or can use sudo)
- Package manager has build tools available

### 2. Existing Bootstrap Check

**Purpose**: Avoid redundant installations if bootstrap tools are already present.

**Detection Method**:
```nim
proc checkExistingBootstrap(): BootstrapStatus =
  let bootstrapDir = getHomeDir() / ".nip" / "bootstrap"

  # Check for Nix
  if fileExists(bootstrapDir / "nix" / "bin" / "nix-build"):
    if verifyNixInstallation(bootstrapDir / "nix"):
      return BootstrapStatus(kind: bsNix, path: bootstrapDir / "nix")

  # Check for PKGSRC
  if fileExists(bootstrapDir / "pkgsrc" / "bin" / "bmake"):
    if verifyPkgsrcInstallation(bootstrapDir / "pkgsrc"):
      return BootstrapStatus(kind: bsPkgsrc, path: bootstrapDir / "pkgsrc")

  # Check for Gentoo
  if fileExists(bootstrapDir / "gentoo" / "bin" / "emerge"):
    if verifyGentooInstallation(bootstrapDir / "gentoo"):
      return BootstrapStatus(kind: bsGentoo, path: bootstrapDir / "gentoo")

  return BootstrapStatus(kind: bsNone)
```

**Verification Steps**:
1. Check binary exists and is executable
2. Test basic functionality (version check)
3. Verify required dependencies are present
4. Confirm configuration is valid

**When Used**:
- Previous bootstrap installation detected
- Installation passes verification checks
- No user override requested

### 3. Recipe-Based Bootstrap

**Purpose**: Automatically download and install minimal build tools without manual intervention.

**Platform Selection Logic**:
```nim
proc selectBootstrapRecipe(): string =
  # Platform-specific preferences
  when defined(linux):
    # Check distribution
    let distro = detectLinuxDistro()
    case distro
    of "arch", "manjaro":
      return "nix"  # Lightweight, no conflicts with pacman
    of "debian", "ubuntu":
      return "nix"  # Clean isolation from apt
    of "gentoo":
      return "gentoo"  # Native toolchain
    of "nixos":
      return "nix"  # Already Nix-based
    else:
      return "nix"  # Default to Nix for most Linux

  when defined(macos):
    return "nix"  # Best macOS support

  when defined(bsd):
    return "pkgsrc"  # Native BSD tool

  # Fallback
  return "nix"
```

**Installation Flow**:
1. **Recipe Selection**: Choose appropriate recipe for platform
2. **Download**: Fetch minimal binary from trusted source
3. **Verification**: Validate Blake2b-512 checksum
4. **Extraction**: Unpack to `~/.nip/bootstrap/<tool>/`
5. **Configuration**: Run setup scripts
6. **Wrapper Creation**: Create convenience wrappers in `~/.nip/bin/`
7. **Verification**: Test installation functionality
8. **PATH Update**: Add to user's PATH if needed

**Advantages**:
- Fully automatic, no user intervention
- Isolated installation (no system pollution)
- Cryptographically verified binaries
- Consistent across platforms

**When Used**:
- No native package manager available
- User prefers isolated installation
- System package manager lacks build tools
- Cross-platform consistency needed

### 4. Container Fallback

**Purpose**: Provide isolated build environment when native tools unavailable or problematic.

**Container Runtime Detection**:
```nim
proc detectContainerRuntime(): Option[ContainerRuntime] =
  # Prefer Podman (rootless, daemonless)
  if commandExists("podman"):
    return some(ContainerRuntime.Podman)

  # Fall back to Docker
  if commandExists("docker"):
    return some(ContainerRuntime.Docker)

  # Try containerd with nerdctl
  if commandExists("nerdctl"):
    return some(ContainerRuntime.Containerd)

  return none(ContainerRuntime)
```

**Build Process**:
1. **Image Selection**: Choose appropriate base image (Gentoo, Alpine, etc.)
2. **Container Creation**: Spin up isolated build environment
3. **Source Mounting**: Mount package source into container
4. **Build Execution**: Run build inside container
5. **Artifact Extraction**: Copy built packages out
6. **Cleanup**: Remove container and temporary files

**Advantages**:
- Complete isolation from host system
- Reproducible builds
- No host system modification
- Works when native tools fail

**When Used**:
- Recipe-based bootstrap fails
- Complex build requirements
- User explicitly requests container build
- System incompatibilities detected

### 5. Manual Installation Guide

**Purpose**: Provide clear instructions when automatic methods fail.

**Guidance Provided**:
```
Build tools are required but could not be installed automatically.

Please install one of the following:

Option 1: Nix Package Manager (Recommended)
  curl -L https://nixos.org/nix/install | sh

Option 2: PKGSRC Bootstrap
  Download: https://pkgsrc.org/bootstrap/
  Follow platform-specific instructions

Option 3: System Package Manager
  Arch Linux: sudo pacman -S base-devel
  Debian/Ubuntu: sudo apt-get install build-essential
  Fedora: sudo dnf groupinstall "Development Tools"

After installation, run: nip build <package>
```

**When Used**:
- All automatic methods failed
- No container runtime available
- User needs to understand requirements
- Platform not supported by recipes

## User Control and Overrides

### Command-Line Options

Users can override automatic detection:

```bash
# Force specific bootstrap method
nip build --bootstrap=nix <package>
nip build --bootstrap=pkgsrc <package>
nip build --bootstrap=container <package>

# Skip bootstrap and use system tools
nip build --no-bootstrap <package>

# Force re-bootstrap even if installed
nip build --force-bootstrap <package>
```

### Configuration File

Set preferences in `~/.nip/config.kdl`:

```kdl
bootstrap {
  # Preferred method: "auto", "nix", "pkgsrc", "gentoo", "container", "system"
  preferred "auto"

  # Allow automatic installation
  auto-install true

  # Container runtime preference
  container-runtime "podman"

  # Installation directory
  install-dir "~/.nip/bootstrap"
}
```

## Error Handling and Recovery

### Detection Failures

**Scenario**: Cannot detect any suitable method

**Response**:
1. Log detailed detection results
2. Show manual installation guide
3. Suggest container-based build as alternative
4. Exit with clear error message

### Installation Failures

**Scenario**: Bootstrap installation fails

**Response**:
1. Automatic rollback of partial installation
2. Clean up downloaded files
3. Try next method in fallback chain
4. Log failure details for debugging

### Verification Failures

**Scenario**: Installed tools fail verification

**Response**:
1. Mark installation as invalid
2. Attempt reinstallation
3. Try alternative bootstrap method
4. Report specific verification failure

## Performance Considerations

### Caching

- **Detection Results**: Cache for 5 minutes to avoid repeated checks
- **Downloaded Binaries**: Keep in `~/.nip/cache/` for reinstallation
- **Verification Status**: Cache successful verifications

### Parallel Operations

- Download and verify checksums concurrently
- Run multiple verification checks in parallel
- Background recipe updates don't block builds

### Minimal Overhead

- Detection adds < 100ms to build startup
- Cached results eliminate repeated detection
- Lazy loading of bootstrap modules

## Security Considerations

### Checksum Verification

All downloaded binaries verified with Blake2b-512:
- Checksums stored in recipes
- Verification before extraction
- Automatic rejection of mismatches

### Signature Verification

Future enhancement for GPG signature checking:
- Public keys embedded in recipes
- Optional signature verification
- Warn on unsigned binaries

### Isolation

- Bootstrap tools installed in user directory
- No system-wide modifications
- Container builds provide additional isolation

## Platform-Specific Behavior

### Arch Linux

1. Check for `base-devel` via pacman
2. If unavailable, install Nix bootstrap
3. Use Nix for source builds
4. Graft results into pacman database

### Gentoo

1. Check for native Portage tools
2. If unavailable, use container with Gentoo stage3
3. Build inside container with proper USE flags
4. Extract and install artifacts

### NixOS

1. Use native Nix (already available)
2. No bootstrap needed
3. Integrate with system Nix store

### macOS

1. Check for Xcode Command Line Tools
2. If unavailable, install Nix bootstrap
3. Use Nix for consistent builds

### BSD

1. Prefer PKGSRC (native BSD tool)
2. Fall back to Nix if PKGSRC unavailable
3. Respect BSD conventions

## Future Enhancements

### Planned Improvements

- **Smart Caching**: Predict needed tools and pre-download
- **Parallel Bootstrap**: Install multiple tools simultaneously
- **Health Monitoring**: Periodic verification of installed tools
- **Auto-Updates**: Keep bootstrap tools current
- **Telemetry**: Anonymous usage data to improve detection

### Under Consideration

- **Binary Mirrors**: Multiple download sources for reliability
- **Incremental Updates**: Delta updates for bootstrap tools
- **Shared Installations**: System-wide bootstrap for multi-user systems
- **Cloud Builds**: Offload builds to remote servers

## Troubleshooting

### Common Issues

**Issue**: Detection fails on supported platform

**Solution**:
```bash
# Force specific method
nip build --bootstrap=nix <package>

# Check detection manually
nip bootstrap info

# Enable debug logging
nip --debug build <package>
```

**Issue**: Bootstrap installation hangs

**Solution**:
```bash
# Cancel and retry with different method
Ctrl+C
nip build --bootstrap=container <package>

# Check network connectivity
nip bootstrap test-connection
```

**Issue**: Verification fails after installation

**Solution**:
```bash
# Remove and reinstall
nip bootstrap remove nix
nip bootstrap install nix

# Try alternative
nip bootstrap install pkgsrc
```

## Summary

The automatic detection flow provides a seamless experience:

1. **Intelligent**: Chooses best method for platform and situation
2. **Automatic**: No user intervention required in common cases
3. **Flexible**: Multiple fallback options ensure success
4. **Transparent**: Clear feedback about what's happening
5. **Controllable**: User can override any automatic decision

This design ensures that users can start building packages immediately, while advanced users retain full control over the bootstrap process.