nushell-data-processing

# Nushell Data Processing

High-performance data processing using Polars DataFrames and LazyFrames in Nushell. Polars provides blazing-fast operations on large datasets with a familiar DataFrame API.

## Why Polars?

| Feature | Native Nushell | Polars |
|---------|---------------|--------|
| Performance | Good for small data | Optimized for millions of rows |
| Memory | Eager evaluation | Lazy evaluation, query optimization |
| Operations | Basic transforms | Full SQL-like operations |
| Parallelism | `par-each` | Built-in multi-threading |

**Rule of thumb:** Use Polars for datasets > 10,000 rows or complex aggregations.

## Getting Started

### Enable Polars Plugin

```nushell
# Add polars plugin
plugin add polars

# Verify installation
plugin list | where name == polars
```

### DataFrame vs LazyFrame

| Type | Description | Use Case |
|------|-------------|----------|
| DataFrame | Eager evaluation, immediate results | Small data, exploration |
| LazyFrame | Lazy evaluation, optimized execution | Large data, complex queries |

```nushell
# DataFrame - evaluates immediately
let df = [[a, b]; [1, 2], [3, 4]] | polars into-df

# LazyFrame - deferred execution
let lf = [[a, b]; [1, 2], [3, 4]] | polars into-lazy

# Execute lazy frame
$lf | polars collect
```

## Core Operations

### Creating DataFrames

```nushell
# From Nushell table
[[name, age, city]; ["Alice", 30, "NYC"], ["Bob", 25, "LA"]]
| polars into-df

# From file
polars open data.csv
polars open data.parquet
polars open data.json --json-lines

# From record
{a: [1, 2, 3], b: [4, 5, 6]} | polars into-df
```

### Selection and Filtering

```nushell
# Select columns
$df | polars select [name, age]

# Filter rows
$df | polars filter ((polars col age) > 25)

# Combined
$df
| polars filter ((polars col city) == "NYC")
| polars select [name, age]
```

### Expressions

Polars expressions are the building blocks:

```nushell
# Column reference
polars col age

# Literal value
polars lit 100

# Arithmetic
(polars c