Data Engineer

Responsibilities

 As a Data Engineer for our advanced analytics platforms, your main responsibilities will be to: 

• Design and implement patterns of practice for productized, portable, modular, instrumented, CI/CD automated and highly performant data ingestion pipelines that leverage structured streaming techniques, processing both batch and streamed data in unstructured, semi-structured and structure form, using Apache Spark, Delta lake, Delta Engine, Hive and other relevant tech stacks 
• Ensure that data ingestion pipelines built with these patterns validate and profile inbound data reliably, identify anomalous and trigger appropriate remediation actions by operations staff when needed 
• Filter, consolidate and contextualize ingested data and further aggregate according to data analytics and ML requirements.
• Use agile development practices, and continually improve development methods with the goal of automating the build, integration, deployment and monitoring of ingestion, enrichment and ML pipelines 
• Using your expertise and influence, help establish patterns of practice for the above, and encourage their adoption by software and data engineering teams across the company 
• Monitor and optimize the performance of data pipelines to ensure efficient data processing and storage.
   

Education, Experience, and Licensing:

• Advanced degree in computer science, but at a minimum a bachelor's degree in 
• 5 years of programming proficiency in, at least one modern programming language (e.g. Java, C#, JavaScript, etc.) and at least one other high-level programming language such as Python SQL
• Expert level proficiency with agile software development & continuous integration + continuous deployment methodologies along with supporting tools such as Git (Gitlab), Terraform
• 5+ years of experience in big data engineering roles, developing and maintaining ETL and ELT pipelines for data warehousing, on-premise and cloud datalake environments 
• 3+ years experience with AWS platform services, including AWS S3 & EC2, Data Migration Services (DMS), RDS, EMR, RedShi0ft, Lambda, DynamoDB, CloudWatch, CloudTrail 
• Strong technical collaboration and communication skills 
   

Technical Qualifications:

• Proficiency with functional programming methods and their appropriate use in distributed systems
• Expert proficiency with data management fundamentals and data storage principles
• Expert proficiency with AWS foundational compute services, including S3 and EC2, ECS and EKS,
• IAM and CloudWatch
• Experience with Databricks, including the use of Databricks notebooks, clusters, and jobs
   

Other Qualifications:

• Demonstrated curiosity and an ability to learn new skills on an ongoing, sustained basis 
• Demonstrated systems perspective when analyzing problems, thinking about overall operation, failure modes and how to address these problems proactively 
• A strong sense for the importance of documentation, and the importance of not having to learn things twice 
• Ability to work in an agile product team environment and balance a diverse set of stakeholder requests 
• Excellent oral and written communication skills with an ability to break down complex technical systems to help business partners understand the value 
• Strong technical collaboration and communication skills as well as the ability to drive cultural change and adoption of best practices through community participation 
• Ability to collaborate with other teams across the company, defining technology roadmaps, sharing experiences and lessons learned for continual improvement 
• Excellent problem-solving and troubleshooting skills 
• Process-oriented with great documentation skills 
• Experience with data visualization tools and techniques
• Familiarity with machine learning frameworks and libraries