batch-processing-optimizer

batch-processing-optimizer - Skill Dossier

batch-processing-optimizer

batch-processing-optimizer

Spark, pandas, polars, DuckDB optimization for batch data processing. Activate on: batch processing, Spark optimization, polars, DuckDB, pandas performance, data frame, shuffle, partition, memory optimization. NOT for: streaming pipelines (use streaming-pipeline-architect), warehouse queries (use data-warehouse-optimizer).

Backend & Infrastructure

#batch-processing#spark#polars#duckdb#performance

Allowed Tools

ReadWriteEditBash(npm:*npx:*python:*spark-submit:*)

Pairs With

data-warehouse-optimizer

Batch outputs often load into warehouses

airflow-dag-orchestrator

Airflow schedules batch processing jobs

lakehouse-architect

Batch jobs read/write lakehouse tables

batch-processing-optimizer preview

Share this skill

Twitter LinkedIn

⚡

Coming in Spring 2026 Beta

WinDAGs will match this skill automatically. Then ask:

"Use batch-processing-optimizer to help me build..."

Request Early Access