nexfs/README.md

# NexFS - Native Zig Flash Filesystem for NexusOS

> **The sovereign flash filesystem for Libertaria nodes and embedded devices**

[![License: LSL-1.0](https://img.shields.io/badge/License-LSL--1.0-blue.svg)](https://opensource.org/licenses/LSL-1.0)
[![Zig](https://img.shields.io/badge/Zig-0.13+-orange.svg)](https://ziglang.org)
[![Status: Alpha](https://img.shields.io/badge/Status-Alpha-yellow.svg)](https://git.sovereign-society.org/nexus/nexfs)

---

## What is NexFS?

**NexFS** is a native Zig implementation of a flash-aware filesystem designed for **Libertaria nodes** and **embedded sovereign devices**. It provides reliable, wear-leveling-aware storage for resource-constrained environments where data integrity and flash longevity are critical.

### Key Design Goals

- **Flash-First Architecture**: Optimized for raw NAND/NOR flash with wear leveling awareness
- **Zero Dynamic Allocation**: All buffers provided by caller - no runtime memory allocation
- **Platform Agnostic**: Works with any flash HAL via callback interface
- **Data Integrity**: CRC32C checksums on all metadata structures
- **Sovereign by Design**: No external dependencies, no vendor lock-in, fully auditable

---

## Build Profiles & Architecture

**NexFS** implements a **modular Baukasten architecture** (SPEC-004) with three build profiles. Not one binary - three configurations compiled from modules:

### Three Build Profiles

| Profile | Modules | Footprint | Use Case |
|---------|---------|-----------|----------|
| **nexfs-core** | CAS + Block Valve + Hash | ~40KB | IoT, Satellite, Sensor |
| **nexfs-sovereign** | core + CDC + DAG | ~120KB | Phone, Laptop, Desktop |
| **nexfs-mesh** | sovereign + UTCP + Gossip | ~200KB | Home Box, Chapter Node |

### Module Compilation

NexFS uses Nim's compile-time conditionals for module selection:

```nim
# nexfs/config.nim
const
  nexfs_cas* = defined(nexfs_cas)    # CAS Layer (always)
  nexfs_cdc* = defined(nexfs_cdc)    # Content-Defined Chunking
  nexfs_mesh* = defined(nexfs_mesh)  # UTCP Cluster + Gossip
  nexfs_dag* = defined(nexfs_dag)    # Merkle DAG Directories
  nexfs_dedup* = defined(nexfs_dedup) # Cross-Node Dedup
```

**Build Commands:**
```bash
# Core profile (IoT sensor)
nim c -d:nexfs_cas nexfs

# Sovereign profile (laptop)
nim c -d:nexfs_cas -d:nexfs_cdc -d:nexfs_dag nexfs

# Mesh profile (home box)
nim c -d:nexfs_cas -d:nexfs_cdc -d:nexfs_dag -d:nexfs_mesh nexfs
```

### Profile Comparison

**nexfs-core** is your LittleFS successor:
- ✅ Robust, power-safe
- ✅ Content-addressed storage
- ✅ Block-level wear leveling
- ❌ No network code
- ❌ No chunker
- ❌ No gossip
- **Target:** Root partition for everything that blinks and beeps

**nexfs-sovereign** adds local intelligence:
- ✅ CDC-Chunking for efficient snapshots
- ✅ Local deduplication
- ✅ Merkle DAGs for Time-Travel
- ❌ Still single-node
- **Target:** Phone, laptop, desktop

**nexfs-mesh** enables network layer:
- ✅ UTCP cluster communication
- ✅ Gossip-based peer discovery
- ✅ Cross-node deduplication
- ✅ Chapter-Mesh integration
- ✅ Kinetic Credits economy
- **Target:** Home box, Chapter node

---

## Specifications

NexFS is defined by three SPECs that build on each other:

| SPEC | Name | Scope | Size |
|------|------|-------|------|
| **SPEC-084** | NexFS Core Format | Superblock, Block Pointers, Hash-ID, Profile-Byte | Foundation |
| **SPEC-085** | NexFS Sovereign Extensions | CDC Chunking, Merkle DAG, Local Dedup, Time Travel | +80KB |
| **SPEC-704** | UTCP Storage Extensions | UTCP Storage Ops, Gossip, Chapter-Mesh, Kinetic Credits | +80KB |

**Dependency Chain:**
```
SPEC-084 (Core) → SPEC-085 (Sovereign) → SPEC-704 (Mesh)
     ↓                    ↓                      ↓
   40KB                +80KB                  +80KB
   IoT                 Laptop                 Home Box
```

Each SPEC is **independently implementable**. `core` compiles without the others. `sovereign` needs `core`. `mesh` needs both.

---

## Use Cases

### 1. Libertaria Mesh Nodes

**Primary Use Case**: Storage layer for Libertaria Capsule nodes

```
┌─────────────────────────────────────────┐
│        Libertaria Capsule Node          │
│                                         │
│  ┌─────────────────────────────────┐   │
│  │   L3 Gossip (QVL Trust Edges)   │   │
│  └─────────────────────────────────┘   │
│  ┌─────────────────────────────────┐   │
│  │   L2 Session (Noise Handshakes) │   │
│  └─────────────────────────────────┘   │
│  ┌─────────────────────────────────┐   │
│  │   L1 Identity (SoulKeys)        │   │
│  └─────────────────────────────────┘   │
│  ┌─────────────────────────────────┐   │
│  │   NexFS (Persistent Storage)    │◄──┘
│  └─────────────────────────────────┘
│  ┌─────────────────────────────────┐
│  │   Raw Flash (NAND/NOR/SPI)      │
│  └─────────────────────────────────┘
└─────────────────────────────────────────┘
```

**Why NexFS for Libertaria?**
- **Persistence**: SoulKeys, peer tables, trust graphs survive reboots
- **Integrity**: CRC32C ensures metadata hasn't been corrupted
- **Wear Leveling**: Tracks erase counts to maximize flash lifespan
- **Minimal Footprint**: Zero allocation design fits embedded constraints
- **Fast Boot**: No journal replay, direct mount from superblock

### 2. Embedded Sovereign Devices

**Secondary Use Case**: IoT devices, Raspberry Pi, ESP32, microcontrollers

**Examples:**
- **Solar Monitor Nodes**: Store sensor readings, config, firmware updates
- **Weather Network**: Log environmental data locally before sync
- **Pager Devices**: Message queue persistence
- **Home Automation**: Device state, automation rules, logs

**Why NexFS for Embedded?**
- **Raw Flash Support**: Works directly with SPI flash, no FTL layer needed
- **Power-Loss Resilience**: Dual superblock backup survives sudden power loss
- **Deterministic**: Fixed buffer sizes, predictable memory usage
- **No OS Dependencies**: Works bare-metal or with any RTOS

---

## Architecture

### On-Disk Layout

```
┌─────────────────────────────────────────────┐
│  Block 0: Primary Superblock (128 bytes)   │
├─────────────────────────────────────────────┤
│  Block 1: Backup Superblock (128 bytes)    │
├─────────────────────────────────────────────┤
│  Blocks 2-N: Block Allocation Map (BAM)    │
│  - Tracks allocation status                 │
│  - Records erase counts (wear leveling)    │
│  - Bad block marking                        │
├─────────────────────────────────────────────┤
│  Blocks N+1-N+4: Inode Table               │
│  - File/directory metadata                  │
│  - Inode IDs 1-128                          │
├─────────────────────────────────────────────┤
│  Blocks N+5+: Data Blocks                  │
│  - File/directory contents                  │
│  - Wear-leveled allocation                  │
└─────────────────────────────────────────────┘
```

### Key Components

**1. Superblock**
- Magic number: `0x4E455846` ("NEXF")
- Generation counter for crash recovery
- Mount count for health monitoring
- CRC32C checksum for integrity

**2. Block Allocation Map (BAM)**
- Per-block metadata: allocated, bad, reserved, needs_erase
- Erase count tracking for wear leveling
- Generation counter for block age

**3. Inode Table**
- File/directory metadata
- Supports: Regular, Directory, Symlink, Device nodes
- Max filename: 255 characters

**4. Flash Interface**
```zig
pub const FlashInterface = struct {
    read: *const fn (ctx: *anyopaque, addr: u64, buffer: []u8) NexFSError!usize,
    write: *const fn (ctx: *anyopaque, addr: u64, buffer: []const u8) NexFSError!void,
    erase: *const fn (ctx: *anyopaque, block_addr: BlockAddr) NexFSError!void,
    sync: *const fn (ctx: *anyopaque) NexFSError!void,
};
```

---

## Features

### ✅ Implemented (v0.1.0)

- **Format/Initialization**: `format()` creates fresh filesystem
- **Superblock Management**: Primary + backup with checksums
- **Block Allocation**: BAM-based allocation with wear tracking
- **Inode Operations**: Create, read, write, delete
- **Directory Operations**: mkdir, rmdir, readdir, lookup
- **File Operations**: open, read, write, close, seek
- **Path Resolution**: Full path support (`/path/to/file`)
- **Checksum Verification**: CRC32C on all metadata
- **Zero Allocation**: All buffers provided by caller

### 🚧 Planned (Future Versions)

- **Wear Leveling Algorithm**: Active block rotation based on erase counts
- **Bad Block Management**: Automatic bad block detection and marking
- **Defragmentation**: Reclaim fragmented data blocks
- **Snapshots**: Point-in-time filesystem snapshots
- **Compression**: Optional LZ4 compression for data blocks
- **Encryption**: Optional XChaCha20-Poly1305 encryption

---

## Chapter-Mesh Architecture (nexfs-mesh)

### The Strategic Moat

When a user installs a Sovereign device, NexFS creates a two-partition layout:

```
┌─ /dev/nvme0 ─────────────────────────────────┐
│                                              │
│  Partition 1: nexfs-sovereign (Root, Private)│
│  ├── /Cas (System, Immutable)                │
│  ├── /Nexus (Config, Encrypted)              │
│  └── /Data (User, Encrypted)                 │
│                                              │
│  Partition 2: nexfs-mesh (Chapter Pool)      │
│  ├── Encrypted at rest (Monolith)            │
│  ├── Auto-joins Chapter via UTCP Discovery   │
│  ├── Contributes: Storage + Bandwidth        │
│  └── Receives: Redundancy + Content Access   │
│                                              │
└────────────────────────────────────────────────┘
```

### Installation Flow

```bash
# During Sovereign installation
nexus forge --chapter "frankfurt-01"

# User prompt:
# "How much storage do you want to contribute to the Chapter Network?"
# [Slider: 10GB ... 500GB ... 2TB]
# Default: 10% of disk or 50GB, whichever is smaller

nexfs format /dev/nvme0p2 --mode sovereign-mesh \
  --chapter-id "frankfurt-01" \
  --quota 50G \
  --replication-factor 3
```

### What Happens in the Background

1. **UTCP Discovery** (SPEC-703 Pattern):
   - Box broadcasts `PEER_ANNOUNCE` on local network
   - Connects to known Chapter gateways

2. **Gossip Sync**:
   - Learns which other Sovereigns are online
   - Exchanges peer lists and chunk availability

3. **Lazy Replication**:
   - Popular chunks (Nip packages, community content) automatically cached
   - Replication factor enforced (default: 3 copies)

4. **Sealed Storage**:
   - User cannot see Chapter partition as normal directory
   - It's an opaque block pool
   - No user snooping on transit data possible

### The Economy (Kinetic Credits)

Integration with **Kinetic Economy Model** (SPEC-053):

```
You give: 100GB Mesh-Storage + Uptime
You get:  Kinetic Credits
You use:  Credits for Package-Downloads, Mesh-Access, Priority-Routing
```

**Engineered Reciprocity** - No altruism, no charity:
- More contribution = more credits
- No contribution = still can read (Commonwealth packages are free)
- Premium content and priority bandwidth cost credits

### The Torrent Killer

**Why NexFS Chapter-Mesh beats BitTorrent:**

| Feature | BitTorrent | NexFS Chapter-Mesh |
|---------|------------|-------------------|
| **FS Integration** | Separate client | Native FS (`nip install firefox`) |
| **Deduplication** | Full file transfer | CAS-Dedup (5% delta transfer) |
| **Encryption** | Optional | Encrypted at rest by default |
| **NAT Traversal** | Problematic | UTCP CellID routing (no NAT issues) |
| **Discovery** | Tracker/DHT | Gossip-based peer discovery |
| **Economy** | None | Kinetic Credits |

**Example:** Firefox 120.0.1 and 120.0.2 share 95% of chunks:
- BitTorrent: Transfer entire `.tar.gz` again
- NexFS: Transfer only 5% difference

### Superblock Extension

One byte in the superblock determines the profile:

```nim
type NexfsSuperblock* = object
  magic*: uint32          # "NXFS"
  version*: uint16
  hash_algo*: uint8       # 0x00=BLAKE3-256, 0x01=BLAKE3-128, 0x02=XXH3-128
  profile*: uint8         # 0x00=core, 0x01=sovereign, 0x02=mesh ← NEW
  chapter_id*: CellID     # 128-bit (only for mesh, otherwise 0)
  mesh_quota*: uint64     # Bytes dedicated to mesh (only for mesh)
  replication_target*: uint8 # Desired copies (default 3)
  # ... rest as usual
```

### Filesystem Hierarchy Extension

```
/Data/
├── Volume/
│   ├── Local/      # Standard
│   ├── Remote/     # Cloud-Mounts
│   ├── External/   # USB
│   └── Mesh/       # NEW: Chapter Pool
│       ├── .quota      # 50GB
│       ├── .chapter-id # frankfurt-01
│       └── .peers      # Known CellIDs
```

### UTCP Storage Protocol (SPEC-704)

**Wire Protocol Extensions:**

| Category | Operations | Size |
|----------|-----------|------|
| **Block Ops** | GET, PUT, DELETE, REPLICATE, VERIFY, REPAIR, STATS | 7 ops |
| **DAG Ops** | RESOLVE, ANCESTORS, DIFF | 3 ops |
| **Peer Ops** | ANNOUNCE, LEAVE, HEARTBEAT | 3 ops |

All operations use **16-byte extension header** on existing UTCP frames.

**Key Features:**
- Gossip-based peer discovery
- Credit-based flow control (SPEC-702 pattern reuse)
- Replication-factor enforcement
- Chapter-Mesh integration with Kinetic Credits

---

## Quick Start

### Installation

Add NexFS to your `build.zig.zon`:

```zig
.{
    .name = "your-project",
    .version = "0.1.0",
    .dependencies = .{
        .nexfs = .{
            .url = "https://git.sovereign-society.org/nexus/nexfs/archive/main.tar.gz",
            .hash = "...",
        },
    },
}
```

### Example: Basic Usage

```zig
const std = @import("std");
const nexfs = @import("nexfs");

// 1. Define your flash interface
const MyFlash = struct {
    flash_data: []u8,

    pub fn read(ctx: *anyopaque, addr: u64, buffer: []u8) nexfs.NexFSError!usize {
        const self = @ptrCast(*MyFlash, @alignCast(ctx));
        @memcpy(buffer, self.flash_data[addr..][0..buffer.len]);
        return buffer.len;
    }

    pub fn write(ctx: *anyopaque, addr: u64, buffer: []const u8) nexfs.NexFSError!void {
        const self = @ptrCast(*MyFlash, @alignCast(ctx));
        @memcpy(self.flash_data[addr..][0..buffer.len], buffer);
    }

    pub fn erase(ctx: *anyopaque, block_addr: nexfs.BlockAddr) nexfs.NexFSError!void {
        // Erase flash block (set to 0xFF for NAND)
    }

    pub fn sync(ctx: *anyopaque) nexfs.NexFSError!void {
        // Flush any caches
    }
};

pub fn main() !void {
    var flash = MyFlash{ .flash_data = try allocator.alloc(u8, 1024 * 1024) };

    // 2. Configure NexFS
    var read_buf: [4096]u8 = undefined;
    var write_buf: [4096]u8 = undefined;
    var workspace: [256]u8 = undefined;

    const config = nexfs.Config{
        .flash = .{
            .ctx = &flash,
            .read = MyFlash.read,
            .write = MyFlash.write,
            .erase = MyFlash.erase,
            .sync = MyFlash.sync,
        },
        .device_size = 1024 * 1024,
        .block_size = 4096,
        .block_count = 256,
        .page_size = 256,
        .checksum_algo = .CRC32C,
        .read_buffer = &read_buf,
        .write_buffer = &write_buf,
        .workspace = &workspace,
        .time_source = null,
        .verbose = true,
    };

    // 3. Format the filesystem
    try nexfs.format(&config.flash, &config, &write_buf);

    // 4. Create a file
    var fs = try nexfs.NexFS.init(allocator, config);
    const fd = try fs.create("/config.txt");
    try fs.write(fd, "hello nexfs");
    try fs.close(fd);

    // 5. Read it back
    var buf: [64]u8 = undefined;
    const fd2 = try fs.open("/config.txt");
    const len = try fs.read(fd2, &buf);
    try std.io.getStdOut().writeAll(buf[0..len]);
    try fs.close(fd2);
}
```

---

## Configuration Options

```zig
const Config = struct {
    flash: FlashInterface,      // Your flash HAL
    device_size: u64,            // Total flash size in bytes
    block_size: BlockSize,       // Flash block size (512, 1024, 2048, 4096)
    block_count: u32,            // Number of blocks
    page_size: PageSize,         // Flash page size for alignment
    checksum_algo: ChecksumAlgo, // None, CRC16, or CRC32C
    read_buffer: []u8,           // Buffer >= block_size
    write_buffer: []u8,          // Buffer >= block_size
    workspace: []u8,             // Buffer >= page_size
    time_source: ?TimeSource,    // Optional timestamp provider
    verbose: bool,               // Enable debug logging
};
```

### Recommended Configurations

**1. Raspberry Pi with SPI Flash (1MB)**
```zig
.block_size = 4096,
.page_size = 256,
.block_count = 256,
.checksum_algo = .CRC32C,
```

**2. ESP32 with Flash (4MB)**
```zig
.block_size = 4096,
.page_size = 256,
.block_count = 1024,
.checksum_algo = .CRC32C,
```

**3. Microcontroller with NOR Flash (512KB)**
```zig
.block_size = 2048,
.page_size = 256,
.block_count = 256,
.checksum_algo = .CRC16,  // Faster on limited CPUs
```

---

## Design Philosophy

### Sovereign Storage Principles

1. **No Secrets**: All code is open source and auditable (LSL-1.0)
2. **No Dependencies**: Zero external libraries, pure Zig
3. **No Vendor Lock-in**: Standard interfaces, portable anywhere
4. **No Hidden Allocation**: Explicit memory management
5. **No Trust Required**: Verify integrity with checksums

### Flash-Aware Design

**Why Raw Flash?**
- **Predictable Performance**: No FTL latency spikes
- **Full Control**: Wear leveling algorithm you control
- **Longer Lifespan**: Avoid consumer-grade FTL write amplification
- **Lower Power**: No background garbage collection

**Wear Leveling Strategy:**
- Track erase counts per block (BAM)
- Prefer blocks with lowest erase counts for writes
- Reserve high-erase-count blocks for cold data
- Target: Even wear distribution across flash lifetime

---

## Performance Characteristics

| Operation | Typical Latency | Notes |
|-----------|----------------|-------|
| Mount | < 10ms | Read superblock, validate checksum |
| Format | 100-500ms | Initialize all metadata blocks |
| File Create | 5-20ms | Allocate inode, write metadata |
| File Read (4KB) | 1-5ms | Single block read |
| File Write (4KB) | 10-30ms | Erase + write cycle |
| Directory Lookup | 1-5ms | Inode table scan |

**Memory Requirements:**
- **Minimum**: 2 × block_size + page_size (e.g., 8KB + 256B = ~8.5KB)
- **Recommended**: 2 × block_size + 2 × page_size (for async ops)
- **Allocator**: Not required (zero dynamic allocation)

---

## Roadmap

### Version 0.2.0 (Q2 2026) - Sovereign Profile
**Focus:** SPEC-085 Implementation
- [ ] FastCDC chunker implementation
- [ ] CAS store with inline dedup
- [ ] Merkle DAG directory structure
- [ ] TimeWarp snapshots (`nexfs snap create/rollback/diff`)
- [ ] Build profiles: core vs sovereign
- [ ] Active wear leveling algorithm
- [ ] Bad block management

### Version 0.3.0 (Q3 2026) - Mesh Profile
**Focus:** SPEC-704 Implementation
- [ ] UTCP storage protocol (7 Block Ops, 3 DAG Ops, 3 Peer Ops)
- [ ] Gossip-based peer discovery
- [ ] Cross-node deduplication
- [ ] Chapter-Mesh integration
- [ ] Kinetic Credits integration
- [ ] Credit-based flow control
- [ ] Compression support (LZ4)
- [ ] Defragmentation tool
- [ ] Filesystem check utility (fsck)

### Version 1.0.0 (Q4 2026) - Production Hardening
**Focus:** Production Readiness
- [ ] Encryption support (XChaCha20-Poly1305)
- [ ] CRDT for concurrent DAG edits
- [ ] Multi-Chapter federation
- [ ] Performance benchmarks
- [ ] Production-hardened
- [ ] Full Libertaria stack integration

### Future Considerations
- **CRDT-Specification** for concurrent DAG edits (Phase 50+)
- **Multi-Chapter-Federation** for cross-chapter chunk exchange (Phase 50+)

**Note:** Features grow into existing SPECs as Sections when needed, not as standalone documents.

---

## Testing

**Current Test Coverage:** 251/253 tests passing (99.2%)

```bash
# Run tests
zig build test

# Run with verbose output
zig build test -Dverbose
```

**Test Categories:**
- ✅ Superblock validation
- ✅ Checksum verification
- ✅ Block allocation/deallocation
- ✅ Inode operations
- ✅ Directory operations
- ✅ File operations
- ✅ Path resolution
- 🔄 Wear leveling (in progress)
- 🔄 Bad block handling (planned)

---

## Security Considerations

**Data Integrity:**
- CRC32C protects all metadata from silent corruption
- Dual superblock survives single-block corruption
- Bad block marking prevents data loss

**Power-Loss Resilience:**
- Primary + backup superblock
- Metadata writes are atomic (single block)
- No journal to replay

**Future Security Features:**
- Optional encryption at rest (v1.0)
- Authenticated encryption (AEAD)
- Key derivation from SoulKey (Libertaria integration)

---

## Contributing

**Development Status:** Alpha (v0.1.0)

**Contribution Areas:**
- Wear leveling algorithm improvements
- Bad block detection strategies
- Performance optimizations
- Test coverage improvements
- Documentation enhancements

**Code Style:**
- Follow Zig style guidelines
- SPDX license headers required
- BDD-style tests preferred
- Panopticum architecture compliance

---

## License

**License:** LSL-1.0 (Libertaria Source License 1.0)

**Summary:**
- ✅ Open source and auditable
- ✅ Free to use for sovereign applications
- ✅ Modifications must be contributed back
- ✅ No commercial restrictions for sovereign use cases

See [LICENSE](LICENSE) for full text.

---

## Community

**Repository:** https://git.sovereign-society.org/nexus/nexfs

**Organization:** [Nexus](https://git.sovereign-society.org/nexus)
- rumpk - Runtime package manager
- nip - Nexus package format
- nexus - Core utilities
- nipbox - Package repository
- **nexfs** - Flash filesystem

**Related Projects:**
- [Libertaria Stack](https://git.sovereign-society.org/libertaria/libertaria-stack) - P2P mesh networking
- [Janus Language](https://git.sovereign-society.org/janus/janus) - Systems programming language

---

## Acknowledgments

**Inspired By:**
- **LittleFS** - Flash-friendly embedded filesystem
- **JFFS2** - Journaling flash filesystem
- **YAFFS2** - Yet another flash filesystem

**Built With:**
- **Zig** - Systems programming language
- **Libertaria** - Sovereign P2P mesh network

---

**NexFS** - *Storage for Sovereign Systems*

*Part of the Nexus ecosystem for Libertaria nodes and embedded devices*