Position Summary:
We are seeking an experienced Data Engineer to join our dynamic technology consulting team. The ideal candidate will have 5+ years of Python expertise focused on ETL, and a strong background in AWS-based data ecosystems. This role centers around building scalable, secure data solutions, supporting advanced analytics, and contributing to cybersecurity initiatives, offers a very good opportunity to learn.
Responsibilities:
- Design, Build, and Maintain ETL Pipelines: Develop robust, efficient ETL processes using Python PySpark for diverse data sources and third-party integrations.
- Exploratory Data Analysis (EDA): Conduct EDA to drive insights, optimize data models, and support machine learning initiatives.
- DataLake Architecture: Architect and optimize DataLake solutions leveraging AWS technologies such as S3, Glue, and Athena.
- AWS Ecosystem Integration: Deploy, monitor, and maintain solutions on EC2; write and optimize Athena SQL queries; manage scalable cloud infrastructure for data engineering needs.
- Data Wrangling: Clean, transform, and standardize large heterogeneous data sets, enabling seamless analytics and reporting.
- Cybersecurity Analytics (Preferred): Engineer pipelines for security-related data, supporting threat detection, SIEM, and anomaly modeling.
- Cribl Experience (Preferred): Leverage Cribl solutions for data routing, enrichment, and log management within cybersecurity contexts.
Preferred Skills & Experience:
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field.
- 5+ years hands-on experience as a Data Engineer, working extensively with Python, ETL frameworks, and ML engineering concepts.
- Proven track record in building and optimizing ETL and DataLake solutions.
- Demonstrated expertise in AWS cloud data engineering (EC2, S3, Glue, Athena SQL).
- Strong skills in EDA, data wrangling, and building analytics-ready data sets.
- Exposure to cybersecurity analytics, log enrichment, or SIEM tools (Splunk, ELK Stack, etc.).
- Experience with Cribl or similar log data streaming frameworks.
- AWS Professional or Specialty certification (Data Analytics, Security, or Machine Learning).
- Strong written and verbal communication skills for client engagement and team collaboration.