> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-fix-nav-issues.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Key differences from pandas

> Important differences between DataStore and pandas

While DataStore is highly compatible with pandas, there are important differences to understand.

<h2 id="summary">
  Summary Table
</h2>

| Aspect           | pandas             | DataStore                                                                                                  |
| ---------------- | ------------------ | ---------------------------------------------------------------------------------------------------------- |
| **Execution**    | Eager (immediate)  | Lazy (deferred)                                                                                            |
| **Return types** | DataFrame/Series   | DataStore/ColumnExpr                                                                                       |
| **Row order**    | Preserved          | Preserved (automatic); not guaranteed in [performance mode](/products/chdb/configuration/performance-mode) |
| **inplace**      | Supported          | Not supported                                                                                              |
| **Index**        | Full support       | Simplified                                                                                                 |
| **Memory**       | All data in memory | Data at source                                                                                             |

***

<h2 id="lazy-execution">
  1. Lazy vs Eager Execution
</h2>

<h3 id="pandas-eager">
  pandas (Eager)
</h3>

Operations execute immediately:

```python theme={null}
import pandas as pd

df = pd.read_csv("data.csv")  # Loads entire file NOW
result = df[df['age'] > 25]   # Filters NOW
grouped = result.groupby('city')['salary'].mean()  # Aggregates NOW
```

<h3 id="datastore-lazy">
  DataStore (Lazy)
</h3>

Operations are deferred until results are needed:

```python theme={null}
from chdb import datastore as pd

ds = pd.read_csv("data.csv")  # Just records the source
result = ds[ds['age'] > 25]   # Just records the filter
grouped = result.groupby('city')['salary'].mean()  # Just records

# Execution happens here:
print(grouped)        # Executes when displaying
df = grouped.to_df()  # Or when converting to pandas
```

<h3 id="why-lazy">
  Why It Matters
</h3>

Lazy execution enables:

* **Query optimization**: Multiple operations compile to one SQL query
* **Column pruning**: Only needed columns are read
* **Filter pushdown**: Filters apply at the source
* **Memory efficiency**: Don't load data you don't need

***

<h2 id="return-types">
  2. Return Types
</h2>

<h3 id="pandas-return-types">
  pandas
</h3>

```python theme={null}
df['col']           # Returns pd.Series
df[['a', 'b']]      # Returns pd.DataFrame
df[df['x'] > 10]    # Returns pd.DataFrame
df.groupby('x')     # Returns DataFrameGroupBy
```

<h3 id="datastore-return-types">
  DataStore
</h3>

```python theme={null}
ds['col']           # Returns ColumnExpr (lazy)
ds[['a', 'b']]      # Returns DataStore (lazy)
ds[ds['x'] > 10]    # Returns DataStore (lazy)
ds.groupby('x')     # Returns LazyGroupBy
```

<h3 id="converting-to-pandas-types">
  Converting to pandas Types
</h3>

```python theme={null}
# Get pandas DataFrame
df = ds.to_df()
df = ds.to_pandas()

# Get pandas Series from column
series = ds['col'].to_pandas()

# Or trigger execution
print(ds)  # Automatically converts for display
```

***

<h2 id="triggers">
  3. Execution Triggers
</h2>

DataStore executes when you need actual values:

| Trigger              | Example            | Notes               |
| -------------------- | ------------------ | ------------------- |
| `print()` / `repr()` | `print(ds)`        | Display needs data  |
| `len()`              | `len(ds)`          | Need row count      |
| `.columns`           | `ds.columns`       | Need column names   |
| `.dtypes`            | `ds.dtypes`        | Need type info      |
| `.shape`             | `ds.shape`         | Need dimensions     |
| `.values`            | `ds.values`        | Need actual data    |
| `.index`             | `ds.index`         | Need index          |
| `to_df()`            | `ds.to_df()`       | Explicit conversion |
| Iteration            | `for row in ds`    | Need to iterate     |
| `equals()`           | `ds.equals(other)` | Need comparison     |

<h3 id="stay-lazy">
  Operations That Stay Lazy
</h3>

| Operation        | Returns     |
| ---------------- | ----------- |
| `filter()`       | DataStore   |
| `select()`       | DataStore   |
| `sort()`         | DataStore   |
| `groupby()`      | LazyGroupBy |
| `join()`         | DataStore   |
| `ds['col']`      | ColumnExpr  |
| `ds[['a', 'b']]` | DataStore   |
| `ds[condition]`  | DataStore   |

***

<h2 id="row-order">
  4. Row Order
</h2>

<h3 id="pandas-row-order">
  pandas
</h3>

Row order is always preserved:

```python theme={null}
df = pd.read_csv("data.csv")
print(df.head())  # Always same order as file
```

<h3 id="datastore-row-order">
  DataStore
</h3>

Row order is **automatically preserved** for most operations:

```python theme={null}
ds = pd.read_csv("data.csv")
print(ds.head())  # Matches file order

# Filter preserves order
ds_filtered = ds[ds['age'] > 25]  # Same order as pandas
```

DataStore automatically tracks original row positions internally (using `rowNumberInAllBlocks()`) to ensure order consistency with pandas.

<h3 id="order-preserved">
  When Order Is Preserved
</h3>

* File sources (CSV, Parquet, JSON, etc.)
* pandas DataFrame sources
* Filter operations
* Column selection
* After explicit `sort()` or `sort_values()`
* Operations that define order (`nlargest()`, `nsmallest()`, `head()`, `tail()`)

<h3 id="order-may-differ">
  When Order May Differ
</h3>

* After `groupby()` aggregations (use `sort_values()` to ensure consistent order)
* After `merge()` / `join()` with certain join types
* In **performance mode** (`config.use_performance_mode()`): row order is not guaranteed for any operation. See [Performance Mode](/products/chdb/configuration/performance-mode).

***

<h2 id="no-inplace">
  5. No inplace Parameter
</h2>

<h3 id="pandas-inplace">
  pandas
</h3>

```python theme={null}
df.drop(columns=['col'], inplace=True)  # Modifies df
df.fillna(0, inplace=True)              # Modifies df
df.rename(columns={'old': 'new'}, inplace=True)
```

<h3 id="datastore-inplace">
  DataStore
</h3>

`inplace=True` is not supported. Always assign the result:

```python theme={null}
ds = ds.drop(columns=['col'])           # Returns new DataStore
ds = ds.fillna(0)                       # Returns new DataStore
ds = ds.rename(columns={'old': 'new'})  # Returns new DataStore
```

<h3 id="why-no-inplace">
  Why No inplace?
</h3>

DataStore uses immutable operations to enable:

* Query building (lazy evaluation)
* Thread safety
* Easier debugging
* Cleaner code

***

<h2 id="index">
  6. Index Support
</h2>

<h3 id="pandas-index">
  pandas
</h3>

Full index support:

```python theme={null}
df = df.set_index('id')
df.loc['user123']           # Label-based access
df.loc['a':'z']             # Label-based slicing
df.reset_index()
df.index.name = 'user_id'
```

<h3 id="datastore-index">
  DataStore
</h3>

Simplified index support:

```python theme={null}
# Basic operations work
ds.loc[0:10]               # Integer position
ds.iloc[0:10]              # Same as loc for DataStore

# For pandas-style index operations, convert first
df = ds.to_df()
df = df.set_index('id')
df.loc['user123']
```

<h3 id="datastore-source-matters">
  DataStore Source Matters
</h3>

* **DataFrame source**: Preserves pandas index
* **File source**: Uses simple integer index

***

<h2 id="comparison">
  7. Comparison Behavior
</h2>

<h3 id="comparing-with-pandas">
  Comparing with pandas
</h3>

pandas doesn't recognize DataStore objects:

```python theme={null}
import pandas as pd
from chdb import datastore as ds

pdf = pd.DataFrame({'a': [1, 2, 3]})
dsf = ds.DataFrame({'a': [1, 2, 3]})

# This doesn't work as expected
pdf == dsf  # pandas doesn't know DataStore

# Solution: convert DataStore to pandas
pdf.equals(dsf.to_pandas())  # True
```

<h3 id="using-equals">
  Using equals()
</h3>

```python theme={null}
# DataStore.equals() also works
dsf.equals(pdf)  # Compares with pandas DataFrame
```

***

<h2 id="types">
  8. Type Inference
</h2>

<h3 id="pandas-types">
  pandas
</h3>

Uses numpy/pandas types:

```python theme={null}
df['col'].dtype  # int64, float64, object, datetime64, etc.
```

<h3 id="datastore-types">
  DataStore
</h3>

May use ClickHouse types:

```python theme={null}
ds['col'].dtype  # Int64, Float64, String, DateTime, etc.

# Types are converted when going to pandas
df = ds.to_df()
df['col'].dtype  # Now pandas type
```

<h3 id="explicit-casting">
  Explicit Casting
</h3>

```python theme={null}
# Force specific type
ds['col'] = ds['col'].astype('int64')
```

***

<h2 id="memory">
  9. Memory Model
</h2>

<h3 id="pandas-memory">
  pandas
</h3>

All data lives in memory:

```python theme={null}
df = pd.read_csv("huge.csv")  # 10GB in memory!
```

<h3 id="datastore-memory">
  DataStore
</h3>

Data stays at source until needed:

```python theme={null}
ds = pd.read_csv("huge.csv")  # Just metadata
ds = ds.filter(ds['year'] == 2024)  # Still just metadata

# Only filtered result is loaded
df = ds.to_df()  # Maybe only 1GB now
```

***

<h2 id="errors">
  10. Error Messages
</h2>

<h3 id="different-error-sources">
  Different Error Sources
</h3>

* **pandas errors**: From pandas library
* **DataStore errors**: From chDB or ClickHouse

```python theme={null}
# May see ClickHouse-style errors
# "Code: 62. DB::Exception: Syntax error..."
```

<h3 id="debugging-tips">
  Debugging Tips
</h3>

```python theme={null}
# View the SQL to debug
print(ds.to_sql())

# See execution plan
ds.explain()

# Enable debug logging
from chdb.datastore.config import config
config.enable_debug()
```

***

<h2 id="checklist">
  Migration Checklist
</h2>

When migrating from pandas:

* [ ] Change import statement
* [ ] Remove `inplace=True` parameters
* [ ] Add explicit `to_df()` where pandas DataFrame is required
* [ ] Add sorting if row order matters
* [ ] Use `to_pandas()` for comparison tests
* [ ] Test with representative data sizes

***

<h2 id="quick-ref">
  Quick Reference
</h2>

| pandas                  | DataStore                      |
| ----------------------- | ------------------------------ |
| `df[condition]`         | Same (returns DataStore)       |
| `df.groupby()`          | Same (returns LazyGroupBy)     |
| `df.drop(inplace=True)` | `ds = ds.drop()`               |
| `df.equals(other)`      | `ds.to_pandas().equals(other)` |
| `df.loc['label']`       | `ds.to_df().loc['label']`      |
| `print(df)`             | Same (triggers execution)      |
| `len(df)`               | Same (triggers execution)      |