Python-Spark Scala

5–9 years of experience in Data Engineering / Big Data development. Strong hands-on experience with Python and Apache Spark (Scala & PySpark). Deep understanding of Spark architecture (RDD, DataFrames, Spark SQL). Experience in distributed computing and big data processing. Strong knowledge of SQL and data modeling concepts. Experience with data pipeline development (ETL/ELT). Familiarity with Linux/Unix environments. Experience with version control tools (Git).

Experience with cloud platforms (AWS, Azure, or GCP). Hands-on with Databricks / EMR / Spark clusters. Knowledge of streaming technologies (Kafka, Spark Streaming, Structured Streaming). Experience with workflow orchestration tools (Airflow, Oozie). Familiarity with Delta Lake / Lakehouse architecture. Exposure to NoSQL databases (MongoDB, Cassandra). Knowledge of CI/CD and DevOps practices.