# PySpark Developer

> Weekday AI · Chennai, India · Full-time · Posted 2026-10-13

**Salary:** INR 2,000,000–3,000,000

**Workplace:** on_site

**Department:** Weekday's Client via platform

## Description

**This role is for one of Weekday’s clients  
Salary range: Rs 2000000 - Rs 3000000 (ie INR 20 - 30 LPA)**

  
Min Experience: 8+ years  
Location: Bengaluru, Chennai, Tamil Nadu, India  
JobType: full-time

We are looking for an experienced **PySpark Developer** with 8–13 years of experience in data engineering, Python programming, and large-scale data processing. The ideal candidate will have strong expertise in developing, optimizing, and maintaining scalable data pipelines using PySpark and Apache Spark. This role requires hands-on experience in processing structured and semi-structured data, implementing ETL/ELT workflows, and building reliable data solutions that support business intelligence, analytics, and data-driven decision-making.

The candidate will collaborate with data engineers, architects, analysts, and other technical stakeholders to design efficient data processing solutions, improve pipeline performance, and ensure data quality across the organization.

## Requirements

Key Responsibilities

### 1\. Data Pipeline Development

-   Design, develop, and maintain scalable, high-performance data pipelines using PySpark and Python.
-   Build robust ETL/ELT workflows to ingest, transform, validate, and load large volumes of data from multiple sources.
-   Develop batch and, where required, streaming data processing solutions to support business and analytical requirements.
-   Implement reusable data processing frameworks and ensure pipelines are reliable, maintainable, and scalable.
-   Integrate data from databases, APIs, cloud storage, and enterprise data platforms.

### 2\. PySpark Development and Optimization

-   Develop complex data transformations using PySpark DataFrames, Spark SQL, and distributed processing techniques.
-   Optimize Spark jobs by analyzing execution plans, partitioning strategies, caching, and resource utilization.
-   Troubleshoot performance bottlenecks, resolve processing failures, and improve job execution efficiency.
-   Work with large datasets while ensuring efficient memory management and distributed computing performance.
-   Apply best practices for handling joins, aggregations, data skew, and incremental data processing.

### 3\. Python Programming

-   Write clean, modular, reusable, and well-documented Python code for data engineering applications.
-   Develop utilities, validation scripts, and automation solutions to streamline data workflows.
-   Implement exception handling, logging, testing, and debugging practices to improve code quality.
-   Collaborate with engineering teams to maintain coding standards and version control practices.

### 4\. Data Quality and Reliability

-   Implement data validation, reconciliation, and monitoring mechanisms across pipelines.
-   Identify and resolve data inconsistencies, duplication, and transformation errors.
-   Maintain technical documentation, data lineage, and operational procedures.
-   Support production deployments, incident resolution, and continuous improvement initiatives.

### 5\. Collaboration and Technical Leadership

-   Work closely with business stakeholders and technical teams to translate requirements into scalable data solutions.
-   Participate in architecture discussions, code reviews, technical design sessions, and development planning.
-   Mentor junior developers and contribute to engineering best practices.

Required Skills and Qualifications

-   8–13 years of overall experience in software development, data engineering, or related technical roles.
-   Strong hands-on expertise in **PySpark, Apache Spark, and Python**.
-   Proven experience designing and maintaining scalable **data pipelines** and ETL/ELT processes.
-   Strong understanding of distributed computing, data partitioning, transformations, and performance optimization.
-   Experience working with SQL, relational databases, and large-scale structured or semi-structured datasets.
-   Familiarity with data warehousing concepts, data modeling, and data quality practices.
-   Experience with Git, debugging, testing, and production support.

Preferred Qualifications

-   Experience with cloud data platforms such as AWS, Microsoft Azure, or Google Cloud.
-   Exposure to platforms such as Databricks, Delta Lake, Snowflake, or cloud-based data lakes.
-   Familiarity with orchestration tools such as Apache Airflow and CI/CD practices.
-   Knowledge of Kafka or other streaming technologies is an advantage.

## Apply

[Apply at Weekday AI](https://apply.workable.com/weekday-1/j/5E322ACCCD/apply)

---
Powered by [Workable](https://www.workable.com)