Senior Software Engineer - Distributed Data Systems
Databricks
Job Description
About the Role
At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to improve their business. Founded by engineers — and customer obsessed — we leap at every opportunity to solve technical challenges, from designing next-gen UI/UX for interfacing with data to scaling our services and infrastructure across millions of virtual machines. And we're only getting started. Modern data analysis employs sophisticated methods such as machine learning that go well beyond the roll-up and drill-down capabilities of traditional SQL query engines. As a software engineer on the Runtime team at Databricks, you will be building the next generation distributed data storage and processing systems that can outperform specialized SQL query engines in relational query performance, yet provide the expressiveness and programming abstractions to support diverse workloads ranging from ETL to data science.
Key Responsibilities
- Develop the de facto open source standard framework for big data, Apache Spark™.
- Provide reliable and high performance services and client libraries for storing and accessing large amounts of data on cloud storage backends such as AWS S3 and Azure Blob Store.
- Contribute to Delta Lake, a storage management system that combines the scale and cost-efficiency of data lakes with the performance and reliability of a data warehouse, including features like ACID transactions and time travel.
- Work on Delta Pipelines to simplify the management and operation of tens of thousands of data pipelines, providing higher level abstractions for expressing and deploying data pipelines.
- Build the next generation query optimizer and execution engine that is fast, tuning free, scalable, and robust.
Requirements
- BS (or higher) in Computer Science, related technical field or equivalent practical experience.
- Comfortable working towards a multi-year vision with incremental deliverables.
- Motivated by delivering customer value and impact.
- 5+ years of production level experience in Java, Scala, or C++.
- Strong foundation in algorithms and data structures and their real-world use cases.
- Experience with distributed systems, databases, and big data systems such as Apache Spark™, Hadoop.
Nice to Have
- Experience with cloud storage backends like AWS S3 or Azure Blob Store.
- Contributions to open source big data projects.
- Knowledge of data engineering and data science workflows.
Qualifications
- Educational background in Computer Science or related field, with a minimum of a bachelor's degree.
Benefits & Perks
- Competitive salary with a pay range of $166,000 - $225,000 USD depending on location and experience.
- Eligibility for annual performance bonus, equity, and comprehensive benefits.
- Work environment that supports diversity and inclusion.
Working at Databricks
Databricks is committed to fostering a diverse and inclusive culture where everyone can excel. We take great care to ensure that our hiring practices are inclusive and meet equal employment opportunity standards. We strive to provide comprehensive benefits and perks that meet the needs of all employees and promote a collaborative, innovative work environment.
Job Details
Job Skills
Key skills identified from this job posting
Sign upto access all insights for this job
About Databricks
Website
databricks.com
Location
San Francisco, CA
Industry
Other Computer Related Services
Similar Jobs
Other opportunities you might be interested in
Software Engineer
Actalent
Software Development Engineer
Adobe
Senior Software Engineer, AI/ML, Ads Bidding Optimization
Software Engineer - Developer (Associate or Experienced)
Boeing
Software Development Engineer
Adobe
Python, AI Developer
Infosys