Share this Job

Senior DevOps Engineer - Databricks Platform

Apply now »

Date: Aug 11, 2022

Location: Charlotte, NC, US, 28216

Company: Corning

Requisition Number: 56201

 

Corning is one of the world’s leading innovators in materials science. For more than 160 years, Corning has applied its unparalleled expertise in specialty glass, ceramics, and optical physics to develop products that have created new industries and transformed people’s lives.

Corning succeeds through sustained investment in R&D, a unique combination of material and process innovation, and close collaboration with customers to solve tough technology challenges.

The global Information Technology (IT) Function is leading efforts to align IT and Business Strategy, leverage IT investments, and optimize end to end business processes and associated information integration technologies.  Through these efforts, IT helps to improve the competitive position of Corning's businesses through IT enabled processes.  IT also delivers Information Technology applications, infrastructure, and project services in a cost efficient manner to Corning worldwide.

Overview

The Senior DevOps Engineer, Databricks Platform is responsible for the administration, maintenance, and provisioning of the Databricks-related elements of Corning’s AWS SaaS-based data lake and unified analytics service. Infrastructure as code (IaC) is a core aspect of this role, as you implement technical solutions to automatically provision compute cluster capacity, allocate compute and storage resources, monitor resource consumption, and assist with the migration of production interactive notebooks to more job-based compute capabilities. Assisting you in these efforts are “inner-sourced” implementation patterns your software engineering colleagues and the broader advanced analytics community contribute to as we continue to rapidly expand the use of machine and deep learning globally.

 

Responsibilities:

As a Senior Software Engineer, Databricks Platform, your primary responsibilities will be to deploy and automate, administer, and monitor the core platform services provided by our AWS-hosted Databricks unified analytics platform:

• Serve as the primary “Databricks account owner”, working with our platform operations staff to oversee billing, subscription levels, workspaces, host AWS accounts, audit logs, and high-level usage monitoring of the hundreds of users on the platform

• Work closely with the Privacera/Ranger DevOps Engineer on data access governance policy administration

• Work closely with our AWS cloud devops team on compute, storage and network resource management, ensuring that Corning has a comprehensive view of resource consumption across AWS and Databricks

• Collaborate closely with our IDP team to ensure proper operation of our SAML and SCIM identity management integrations

• Work to implement Databricks delegated account administration with the various divisions and functions across the company

• Serve as the primary Databricks workspace administrator and work to build out a team of workplace admins across the company

• In all admin use cases, leverage the Databricks platform APIs to build out the capability to work through IaC automation with Terraform

• Provide primary platform technical assist to our Qlik CDC Data Factory team and other teams building data engineering and ML engineering pipelines throughout Corning

• Work collaboratively and proactively with software engineers and site reliability engineers across the platform engineering organization and the user community to anticipate, troubleshoot and resolve Databricks performance, connectivity, and security issues as they present themselves

• Research, adopt and relate new technologies to solve issues that arise while adhering to company security and standards.

 

Education & Experience

• Undergraduate degree in Computer Science, IT, IS, or equivalent area of technical study

• At least 12-18 months’ experience managing a production Databricks deployment as an account owner and workspace admin

• 3 years’ builder experience on AWS, preferably with recent certification as solution architect, devops engineer

• 3-5 years of hands-on production programming experience with agile software development in a variety of languages, such as Python, .NET/C#, Go, Java, JavaScript/Node.JS

• 1-2 years of production Terraform IaC development and deployment experience

• 3 years of production site reliability engineering experience across both cloud and on-premise environments

 

Technical Qualifications:

A minimum of 2 years hands-on staging and production experience in each of the following areas

• Working with a team of 5 or more software engineers using Gitlab, Jenkins, or Azure DevOps.

• Infrastructure and configuration management tools incliuding Terraform, Ansible or Chef

• Container orchestration services, especially Docker & Kubernetes

• Systems administration scripting methods such as Python, Bash or PowerShell.

 

In-depth operational expertise in the following areas:

• Cloud architecture best practices around operational excellence, security, reliability, performance efficiency, and cost optimization (e.g. AWS Well Architected Framework)

• Best practices and IT operations for dynamic, always-up, always-available services

 

Other Qualifications:

• Demonstrated systems perspective when analyzing problems, thinking about overall operation, failure modes and how to address these problems proactively

• A strong sense for the importance of documentation, and the importance of not having to learn things twice

• Ability to work in an agile product team environment and balance a diverse set of stakeholder requests

• Excellent oral and written communication skills with an ability to break down complex technical systems to help business partners understand the value

• Ability to collaborate with other teams across the company, defining technology roadmaps, sharing experiences and lessons learned for continual improvement

• Excellent problem-solving and troubleshooting skills

• Process-oriented with great documentation skills

 

What sets us apart?

Corning’s unwavering commitment to Diversity. Diversity is integral to Corning’s belief in the fundamental dignity of the individual – one of Corning’s seven Values. We are committed to providing an environment where all employees can thrive. This begins with an understanding that our global workforce consists of a rich mixture of diverse people. This diversity will continue to be a source of our strength as well as a competitive advantage. If you have a passionate belief in the power of innovation to change the world; and if you are up to the challenge of working for a world-class organization that makes real, profitable advanced materials, then visit Corning’s website at www.corning.com

 

This position does not support immigration sponsorship.

 

We prohibit discrimination on the basis of race, color, gender, age, religion, national origin, sexual orientation, gender identity or expression, disability, veteran status or any other legally protected status.

 

We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.


Nearest Major Market: Charlotte