Senior Big Data & DevOps Engineer

Role: Senior Big Data & DevOps Engineer.

Location: Pittsburgh PA (Remote).

Duration: Long Term Contract.

Position Summary:

• Seeking a Senior Big Data & DevOps Engineer with 8 years of expertise in HDFS Hive Impala PySpark Python and DevOps automation (uDeploy Jenkins). Responsible for end-to-end management of enterprise data platforms including HDFS table governance scalable ETL pipeline development CI/CD automation platform upgrades incident/change management and production support.

Key Responsibilities:

• Design manage and optimize HDFS directories tables and data partitioning strategies across large distributed clusters.
• Enforce data retention lifecycle policies and security compliance in high-volume Hadoop environments.
• Administer Hive and Impala platforms focusing on performance tuning capacity planning and troubleshooting.
• Develop scalable ETL pipelines and reusable data frameworks using PySpark Hive and Python.
• Implement and support CI/CD pipelines via Jenkins and uDeploy ensuring automated reliable multi-environment deployments.
• Manage platform upgrades patching and code promotion across DEV QA UAT and PROD stages.
• Perform advanced Linux administration including scripting system tuning disk management and disaster recovery.
• Drive incident and change management using ServiceNow including RCA investigation issue resolution and preventive actions.
• Mentor junior engineers foster technical leadership and document operational knowledge for internal sharing.

Required Qualifications:

• Bachelors degree in Computer Science or related field.
• 8 years in Big Data engineering and DevOps.
• Advanced proficiency in HDFS Hive Impala PySpark Python and Linux.
• Proven CI/CD experience with Jenkins and uDeploy.
• Strong ETL development orchestration and optimization skills.
• Experience with ServiceNow for incident/change/problem management.

Nice to Have:

• Experience with cloud-based data platforms (AWS EMR).
• Knowledge of Docker Kubernetes and infrastructure automation (Ansible Terraform).