DevOps Engineer

Requisition Number: 64335

Corning is vital to progress – in the industries we help shape and in the world we share.

We invent life-changing technologies using materials science. Our scientific and manufacturing expertise, boundless curiosity, and commitment to purposeful invention place us at the center of the way the world interacts, works, learns, and lives.

Our sustained investment in research, development, and invention means we’re always ready to solve the toughest challenges alongside our customers. 

The global Information Technology (IT) Function is leading efforts to align IT and Business Strategy, leverage IT investments, and optimize end to end business processes and associated information integration technologies.  Through these efforts, IT helps to improve the competitive position of Corning's businesses through IT enabled processes.  IT also delivers Information Technology applications, infrastructure, and project services in a cost efficient manner to Corning worldwide.

The DevOps Engineer, Databricks Platform is responsible for the administration, maintenance, and provisioning of the Databricks-related elements of Corning’s AWS SaaS-based data lake and unified analytics service. Infrastructure as code (IaC) is a core aspect of this role, as you implement technical solutions to automatically provision compute cluster capacity, allocate compute and storage resources, monitor resource consumption, and assist with the migration of production interactive notebooks to more job-based compute capabilities. Assisting you in these efforts are “inner-sourced” implementation patterns your software engineering colleagues and the broader advanced analytics community contribute to as we continue to rapidly expand the use of machine and deep learning globally.

Responsibilities:

As a DevOps Engineer, Databricks Platform, your primary responsibilities will be to deploy and automate, administer, and monitor the core platform services provided by our AWS-hosted Databricks unified analytics platform:

• Work with our platform operations staff to oversee billing, subscription levels, workspaces, host AWS accounts, audit logs, and high-level usage monitoring of the hundreds of users on the platform

• Work closely with the Privacera/Ranger DevOps Engineer on data access governance policy administration

• Work closely with our AWS cloud devops team on compute, storage and network resource management, ensuring that Corning has a comprehensive view of resource consumption across AWS and Databricks

• Collaborate closely with our IDP team to ensure proper operation of our SAML and SCIM identity management integrations

• Work to implement Databricks delegated account administration with the various divisions and functions across the company

• Work with the primary Databricks workspace administrator and work to build out a team of workplace admins across the company

• In all admin use cases, leverage the Databricks platform APIs to build out the capability to work through IaC automation with Terraform

• Work collaboratively and proactively with software engineers and site reliability engineers across the platform engineering organization and the user community to anticipate, troubleshoot and resolve Databricks performance, connectivity, and security issues as they present themselves

• Research, adopt and relate new technologies to solve issues that arise while adhering to company security and standards.

Education & Experience

• Undergraduate degree in Computer Science, IT, IS, or equivalent area of technical study

• At least 12-18 months’ experience managing a production Databricks deployment as an account owner and workspace admin

• 2 years’ builder experience on AWS, preferably with recent certification as solution architect, devops engineer

• 3 years of hands-on production programming experience with agile software development in a variety of languages, such as Python, .NET/C#, Go, Java, JavaScript/Node.JS

• 1-2 years of production Terraform IaC development and deployment experience

• 2 years of production site reliability engineering experience across both cloud and on-premise environments

Technical Qualifications:

A minimum of 2 years hands-on staging and production experience in each of the following areas

• Working with a team of 5 or more software engineers using Gitlab, Jenkins, or Azure DevOps.

• Infrastructure and configuration management tools incliuding Terraform, Ansible or Chef

• Container orchestration services, especially Docker & Kubernetes

• Systems administration scripting methods such as Python, Bash or PowerShell.

In-depth operational expertise in the following areas:

• Cloud architecture best practices around operational excellence, security, reliability, performance efficiency, and cost optimization (e.g. AWS Well Architected Framework)

• Best practices and IT operations for dynamic, always-up, always-available services

Other Qualifications:

• Demonstrated systems perspective when analyzing problems, thinking about overall operation, failure modes and how to address these problems proactively

• A strong sense for the importance of documentation, and the importance of not having to learn things twice

• Ability to work in an agile product team environment and balance a diverse set of stakeholder requests

• Excellent oral and written communication skills with an ability to break down complex technical systems to help business partners understand the value

• Ability to collaborate with other teams across the company, defining technology roadmaps, sharing experiences and lessons learned for continual improvement

• Excellent problem-solving and troubleshooting skills

• Process-oriented with great documentation skills