Databricks Developer

About the Role

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark on Databricks
• Implement data processing logic in Java 8+, leveraging functional programming and OOP best practices
• Integrate with IRS data systems including IRMF, BMF, or IMF
• Optimize Spark jobs for performance, reliability, and cost-efficiency
• Collaborate with cross-functional teams to gather requirements and deliver data solutions
• Ensure compliance with data security, privacy, and governance standards
• Troubleshoot and debug production issues in distributed data environments

Requirements

• Active IRS MBI Clearance Required; IRS issued laptop strongly preferred
• Bachelor's degree in Computer Science, Information Systems, or a related field
• 8+ years of professional experience demonstrating expertise with IRS Data Systems
• Hands-on experience working with IRS IRMF, BMF, or IMF datasets
• Strong expertise in Java 8 or higher
• Experience with functional programming (Streams API, Lambdas)
• Familiarity with object-oriented design patterns and best practices
• Proficient in Spark Core, Spark SQL, and DataFrame/Dataset APIs
• Understanding of RDDs and when to use them
• Experience with Spark Streaming or Structured Streaming
• Familiarity with HDFS, Hive, or HBase
• Experience integrating with Kafka, S3, or Azure Data Lake
• Knowledge of Parquet, Avro, or ORC file formats
• Experience building ETL pipelines with Spark
• Experience with YARN, Kubernetes, or EMR for Spark deployment
• Familiarity with CI/CD tools like Jenkins or GitHub Actions
• Proficiency in Git, Maven, or Gradle
• Experience with unit testing frameworks like JUnit or TestNG
• Strong documentation skills and ability to troubleshoot production issues

Nice to Have

• Experience with Scala or Python in Spark environments
• Familiarity with Databricks or Google DataProc
• Knowledge of Delta Lake or Apache Iceberg
• Data modeling and performance design for big data systems