Data engineer

Job Description

Our Healthcare Insurance Client is building a new 4 person Sr. Data Engineering team to support growth in 2 major portfolios within their Flagship Secure Member Portal. This team will help handle interrupt work and self service work. These positions are remote work from home and will be contract to hire.

As a Senior Data Engineer you'll be an integral part of our team, helping to design, build, and maintain the infrastructure and systems that support the effective collection, storage, and processing of large volumes of data. In this exciting role, you'll have the opportunity to develop robust data pipelines that effortlessly ingest, transform, and load (ETL/ELT) data from a variety of sources into our digital platforms. Your dedication to ensuring data quality, integrity, and security throughout the data lifecycle will be key, as will your efforts to optimize our data storage solutions for top-notch performance and scalability.

Key Responsibilities:

• Help design and implement scalable data pipelines and architectures that support low-latency and high-performance data processing.

• Support and enhance existing data processing platforms deployed in AWS, ensuring optimal performance and scalability.

• Collaborate closely with software engineers, product managers, and business analysts to understand data requirements and deliver effective solutions.

• Contribute to the development of best practices and standards for data engineering and machine learning, promoting efficiency and quality across projects.

• Implement backup, disaster recovery, and failover procedures for mission-critical data systems using AWS services like Amazon RDS.

• Help manage and scale Elasticsearch clusters for efficient indexing and retrieval of large datasets for micro-services.

• Ingest data from various sources, such a flat files, streaming systems, and RESTful APIs .

• Ensure compliance with data privacy regulations, such as HIPAA, to protect sensitive information.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to [email protected] learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy:

Skills and Requirements

• 7+ years of experience in a Data Engineering role, with a proven track record of successfully delivering complex, large-scale, multi-team data projects.

• 5+ years Proficiency in working with diverse data infrastructures, including relational databases (e.g., SQL), and column stores (e.g., Redshift).

• 3+ years of experience in building scalable data pipelines using scheduling tools like Airflow.

• 5+ years of proven ability to document processes, architecture designs, deployment pipelines, development methods, and troubleshooting techniques.

• 5+ years of proficiency in Python with a focus on scaling data pipelines.

Preferred Qualifications:

• Excellent communication skills, with the ability to maintain essential cross-team and cross-functional relationships.

• Familiarity with the latest technologies in the data engineering landscape, particularly AWS data services and Apache projects.

• Demonstrated ability to analyze complex problems, identify root causes, and implement effective solutions in a timely manner.

• AWS Cloud Infrastructure experience with core services including Amazon S3, EC2, RDS, Amazon Redshift, DynamoDB, and AWS Lambda.

• Experience designing and implementing resilient data platforms for mission critical services on cloud technologies.

• Knowledge of Master Data Management (MDM) and Customer Relationship Management (CRM) solutions.

• Familiarity with Agile development practices, including Scrum and Kanban, with a track record of participating in sprint planning, daily stand-ups, and retrospectives.