nip/docs/bootstrap-detection-flow.md

Bootstrap Detection Flow

Overview

The NIP bootstrap system automatically detects the best available build tool installation method based on the current system state. This document describes the detection logic, fallback mechanisms, and decision flow.

Detection Hierarchy

The system follows this priority order when determining how to install build tools:

1. Native Package Manager (Highest Priority)
2. Existing Bootstrap Installation (Already installed)
3. Recipe-Based Bootstrap (Automatic download and install)
4. Container-Based Build (Fallback for complex scenarios)
5. Manual Installation (Last resort - user guidance)

Flow Diagram

User runs: nip build <package>
           |
           v
    [Check Build Tools]
           |
           +---> Tools Available? --YES--> [Proceed with Build]
           |
           NO
           |
           v
    [Automatic Detection]
           |
           +---> Check Native Package Manager
           |     (pacman, apt, dnf, etc.)
           |     |
           |     +---> Available? --YES--> [Install via Package Manager]
           |                                        |
           |                                        v
           NO                              [Verify Installation]
           |                                        |
           v                                        v
    [Check Existing Bootstrap]            [Proceed with Build]
           |
           +---> Bootstrap Installed? --YES--> [Verify and Use]
           |                                          |
           NO                                         v
           |                                 [Proceed with Build]
           v
    [Recipe-Based Bootstrap]
           |
           +---> Select Recipe by Platform
           |     (Nix, PKGSRC, Gentoo)
           |     |
           |     v
           |  [Download Binary]
           |     |
           |     v
           |  [Verify Checksum]
           |     |
           |     v
           |  [Install to ~/.nip/bootstrap/]
           |     |
           |     v
           |  [Create Wrappers]
           |     |
           |     v
           |  [Verify Installation]
           |     |
           |     +---> Success? --YES--> [Proceed with Build]
           |
           NO
           |
           v
    [Container Fallback]
           |
           +---> Podman Available? --YES--> [Build in Container]
           |                                        |
           |                                        v
           NO                              [Extract Artifacts]
           |                                        |
           v                                        v
    [Manual Installation Guide]          [Proceed with Build]
           |
           v
    [Show Instructions]
           |
           v
    [Exit with Error]

Detection Logic Details

1. Native Package Manager Detection

Purpose: Use system package manager when available for official support and updates.

Detection Method:

proc detectNativePackageManager(): Option[string] =
  # Check for common package managers
  if commandExists("pacman"): return some("pacman")
  if commandExists("apt-get"): return some("apt")
  if commandExists("dnf"): return some("dnf")
  if commandExists("zypper"): return some("zypper")
  return none(string)

Installation Commands:

• Arch Linux: pacman -S --needed base-devel
• Debian/Ubuntu: apt-get install build-essential
• Fedora: dnf groupinstall "Development Tools"
• openSUSE: zypper install -t pattern devel_basis

Advantages:

• Official packages with security updates
• System integration and dependency management
• No additional maintenance required

When Used:

• System has a recognized package manager
• User has appropriate permissions (or can use sudo)
• Package manager has build tools available

2. Existing Bootstrap Check

Purpose: Avoid redundant installations if bootstrap tools are already present.

Detection Method:

proc checkExistingBootstrap(): BootstrapStatus =
  let bootstrapDir = getHomeDir() / ".nip" / "bootstrap"

  # Check for Nix
  if fileExists(bootstrapDir / "nix" / "bin" / "nix-build"):
    if verifyNixInstallation(bootstrapDir / "nix"):
      return BootstrapStatus(kind: bsNix, path: bootstrapDir / "nix")

  # Check for PKGSRC
  if fileExists(bootstrapDir / "pkgsrc" / "bin" / "bmake"):
    if verifyPkgsrcInstallation(bootstrapDir / "pkgsrc"):
      return BootstrapStatus(kind: bsPkgsrc, path: bootstrapDir / "pkgsrc")

  # Check for Gentoo
  if fileExists(bootstrapDir / "gentoo" / "bin" / "emerge"):
    if verifyGentooInstallation(bootstrapDir / "gentoo"):
      return BootstrapStatus(kind: bsGentoo, path: bootstrapDir / "gentoo")

  return BootstrapStatus(kind: bsNone)

Verification Steps:

1. Check binary exists and is executable
2. Test basic functionality (version check)
3. Verify required dependencies are present
4. Confirm configuration is valid

When Used:

• Previous bootstrap installation detected
• Installation passes verification checks
• No user override requested

3. Recipe-Based Bootstrap

Purpose: Automatically download and install minimal build tools without manual intervention.

Platform Selection Logic:

proc selectBootstrapRecipe(): string =
  # Platform-specific preferences
  when defined(linux):
    # Check distribution
    let distro = detectLinuxDistro()
    case distro
    of "arch", "manjaro":
      return "nix"  # Lightweight, no conflicts with pacman
    of "debian", "ubuntu":
      return "nix"  # Clean isolation from apt
    of "gentoo":
      return "gentoo"  # Native toolchain
    of "nixos":
      return "nix"  # Already Nix-based
    else:
      return "nix"  # Default to Nix for most Linux

  when defined(macos):
    return "nix"  # Best macOS support

  when defined(bsd):
    return "pkgsrc"  # Native BSD tool

  # Fallback
  return "nix"

Installation Flow:

1. Recipe Selection: Choose appropriate recipe for platform
2. Download: Fetch minimal binary from trusted source
3. Verification: Validate Blake2b-512 checksum
4. Extraction: Unpack to ~/.nip/bootstrap/<tool>/
5. Configuration: Run setup scripts
6. Wrapper Creation: Create convenience wrappers in ~/.nip/bin/
7. Verification: Test installation functionality
8. PATH Update: Add to user's PATH if needed

Advantages:

• Fully automatic, no user intervention
• Isolated installation (no system pollution)
• Cryptographically verified binaries
• Consistent across platforms

When Used:

• No native package manager available
• User prefers isolated installation
• System package manager lacks build tools
• Cross-platform consistency needed

4. Container Fallback

Purpose: Provide isolated build environment when native tools unavailable or problematic.

Container Runtime Detection:

proc detectContainerRuntime(): Option[ContainerRuntime] =
  # Prefer Podman (rootless, daemonless)
  if commandExists("podman"):
    return some(ContainerRuntime.Podman)

  # Fall back to Docker
  if commandExists("docker"):
    return some(ContainerRuntime.Docker)

  # Try containerd with nerdctl
  if commandExists("nerdctl"):
    return some(ContainerRuntime.Containerd)

  return none(ContainerRuntime)

Build Process:

1. Image Selection: Choose appropriate base image (Gentoo, Alpine, etc.)
2. Container Creation: Spin up isolated build environment
3. Source Mounting: Mount package source into container
4. Build Execution: Run build inside container
5. Artifact Extraction: Copy built packages out
6. Cleanup: Remove container and temporary files

Advantages:

• Complete isolation from host system
• Reproducible builds
• No host system modification
• Works when native tools fail

When Used:

• Recipe-based bootstrap fails
• Complex build requirements
• User explicitly requests container build
• System incompatibilities detected

5. Manual Installation Guide

Purpose: Provide clear instructions when automatic methods fail.

Guidance Provided:

Build tools are required but could not be installed automatically.

Please install one of the following:

Option 1: Nix Package Manager (Recommended)
  curl -L https://nixos.org/nix/install | sh

Option 2: PKGSRC Bootstrap
  Download: https://pkgsrc.org/bootstrap/
  Follow platform-specific instructions

Option 3: System Package Manager
  Arch Linux: sudo pacman -S base-devel
  Debian/Ubuntu: sudo apt-get install build-essential
  Fedora: sudo dnf groupinstall "Development Tools"

After installation, run: nip build <package>

When Used:

• All automatic methods failed
• No container runtime available
• User needs to understand requirements
• Platform not supported by recipes

User Control and Overrides

Command-Line Options

Users can override automatic detection:

# Force specific bootstrap method
nip build --bootstrap=nix <package>
nip build --bootstrap=pkgsrc <package>
nip build --bootstrap=container <package>

# Skip bootstrap and use system tools
nip build --no-bootstrap <package>

# Force re-bootstrap even if installed
nip build --force-bootstrap <package>

Configuration File

Set preferences in ~/.nip/config.kdl:

bootstrap {
  # Preferred method: "auto", "nix", "pkgsrc", "gentoo", "container", "system"
  preferred "auto"

  # Allow automatic installation
  auto-install true

  # Container runtime preference
  container-runtime "podman"

  # Installation directory
  install-dir "~/.nip/bootstrap"
}

Error Handling and Recovery

Detection Failures

Scenario: Cannot detect any suitable method

Response:

1. Log detailed detection results
2. Show manual installation guide
3. Suggest container-based build as alternative
4. Exit with clear error message

Installation Failures

Scenario: Bootstrap installation fails

Response:

1. Automatic rollback of partial installation
2. Clean up downloaded files
3. Try next method in fallback chain
4. Log failure details for debugging

Verification Failures

Scenario: Installed tools fail verification

Response:

1. Mark installation as invalid
2. Attempt reinstallation
3. Try alternative bootstrap method
4. Report specific verification failure

Performance Considerations

Caching

• Detection Results: Cache for 5 minutes to avoid repeated checks
• Downloaded Binaries: Keep in ~/.nip/cache/ for reinstallation
• Verification Status: Cache successful verifications

Parallel Operations

• Download and verify checksums concurrently
• Run multiple verification checks in parallel
• Background recipe updates don't block builds

Minimal Overhead

• Detection adds < 100ms to build startup
• Cached results eliminate repeated detection
• Lazy loading of bootstrap modules

Security Considerations

Checksum Verification

All downloaded binaries verified with Blake2b-512:

• Checksums stored in recipes
• Verification before extraction
• Automatic rejection of mismatches

Signature Verification

Future enhancement for GPG signature checking:

• Public keys embedded in recipes
• Optional signature verification
• Warn on unsigned binaries

Isolation

• Bootstrap tools installed in user directory
• No system-wide modifications
• Container builds provide additional isolation

Platform-Specific Behavior

Arch Linux

1. Check for base-devel via pacman
2. If unavailable, install Nix bootstrap
3. Use Nix for source builds
4. Graft results into pacman database

Gentoo

1. Check for native Portage tools
2. If unavailable, use container with Gentoo stage3
3. Build inside container with proper USE flags
4. Extract and install artifacts

NixOS

1. Use native Nix (already available)
2. No bootstrap needed
3. Integrate with system Nix store

macOS

1. Check for Xcode Command Line Tools
2. If unavailable, install Nix bootstrap
3. Use Nix for consistent builds

BSD

1. Prefer PKGSRC (native BSD tool)
2. Fall back to Nix if PKGSRC unavailable
3. Respect BSD conventions

Future Enhancements

Planned Improvements

• Smart Caching: Predict needed tools and pre-download
• Parallel Bootstrap: Install multiple tools simultaneously
• Health Monitoring: Periodic verification of installed tools
• Auto-Updates: Keep bootstrap tools current
• Telemetry: Anonymous usage data to improve detection

Under Consideration

• Binary Mirrors: Multiple download sources for reliability
• Incremental Updates: Delta updates for bootstrap tools
• Shared Installations: System-wide bootstrap for multi-user systems
• Cloud Builds: Offload builds to remote servers

Troubleshooting

Common Issues

Issue: Detection fails on supported platform

Solution:

# Force specific method
nip build --bootstrap=nix <package>

# Check detection manually
nip bootstrap info

# Enable debug logging
nip --debug build <package>

Issue: Bootstrap installation hangs

Solution:

# Cancel and retry with different method
Ctrl+C
nip build --bootstrap=container <package>

# Check network connectivity
nip bootstrap test-connection

Issue: Verification fails after installation

Solution:

# Remove and reinstall
nip bootstrap remove nix
nip bootstrap install nix

# Try alternative
nip bootstrap install pkgsrc

Summary

The automatic detection flow provides a seamless experience:

1. Intelligent: Chooses best method for platform and situation
2. Automatic: No user intervention required in common cases
3. Flexible: Multiple fallback options ensure success
4. Transparent: Clear feedback about what's happening
5. Controllable: User can override any automatic decision

This design ensures that users can start building packages immediately, while advanced users retain full control over the bootstrap process.