# Senior Data Engineer (Databricks and Pyspark)

> Unison Group · Kuala Lumpur, Malaysia · Full-time · Posted 2026-11-05

**Salary:** unknown currency 15,000–25,000

**Workplace:** on_site

## Description

Position : Senior Data Engineer.  
Location : Malaysia and Singapore.  
Language : Mandarin Language must.  
Role : Full Time Perm role + one month Bonus.

## Requirements

We are looking for a **Senior Data Engineer** with strong expertise in **Databricks, PySpark, Delta Lake, and cloud-based data pipelines**. implement Lakehouse/Medallion architectures, and integrate data from multiple internal and external systems. This role requires strong technical leadership and hands-on architecture experience.

**Key Responsibilities**

-   Design, build, and optimize **data ingestion and transformation pipelines** using Databricks, PySpark, and Python.
-   Implement **Delta Lake and Medallion architecture** for scalable enterprise data platforms.
-   Develop ingestion frameworks for data from **SFTP, REST APIs, SharePoint/Graph API, AWS, and Azure sources**.
-   Automate workflows using **Databricks Workflows, ADF, Azure Functions, and CI/CD pipelines**.
-   Optimize Spark jobs for performance, reliability, and cost efficiency.
-   Implement **data validation, quality checks, and monitoring** with automated alerts and retries.
-   Design secure and governed datasets using **Unity Catalog** and cloud security best practices.
-   Collaborate with analysts, business users, and cross-functional teams to deliver curated datasets for reporting and analytics.
-   Provide technical leadership and guidance to junior team members.

**Required Skills**

-   5–8+ years of experience in **Data Engineering**.
-   Strong hands-on experience with **Databricks, PySpark, Delta Lake, SQL, Python**.
-   Experience with **Azure Data Lake, ADF, Azure Functions**, or AWS equivalents (S3, Lambda).
-   Experience integrating data from **APIs, SFTP servers, vendor data providers, and cloud storage**.
-   Knowledge of **ETL/ELT concepts, Lakehouse/Meddalion architecture**, and distributed processing.
-   Strong experience with **Git, Azure DevOps CI/CD**, and YAML pipelines.
-   Ability to optimize Spark workloads (partitioning, caching, Z-ordering, performance tuning).

**Good to Have**

-   Exposure to **Oil & Gas or trading analytics** (SPARTA, KPLER, IIR, OPEC).
-   Knowledge of **Power BI** or data visualization concepts.
-   Familiarity with **Terraform**, Scala, or PostgreSQL.
-   Experience with SharePoint development or .NET (optional)

## Apply

[Apply at Unison Group](https://apply.workable.com/unisongroup/j/49665455A4/apply)

---
Powered by [Workable](https://www.workable.com)