Design and manage large-scale structured and unstructured datasets in Hadoop and cloud platforms (AWS S3, Azure Blob). Ingest and curate data using Sqoop, Spark, and ADF pipelines. Clean, preprocess, standardize, and validate datasets for analytical use. Perform exploratory data analysis using SQL, Spark SQL, and Python/Scala to identify trends and patterns. Develop data models, aggregations, and feature datasets for reporting and predictive analysis. Implement distributed data processing using Spark, optimizing performance through partitioning and parallelization. Build and maintain ETL/ELT pipelines integrating data from sources such as Azure SQL, Data Warehouse, and external APIs. Support statistical analysis and data-driven decision-making by preparing high-quality datasets. Monitor data quality, lineage, and governance. Collaborate with cross-functional teams to translate business problems into analytical datasets and scalable data solutions. May require providing services at unanticipated locations to various clients across the United States for short term and long- term durations. There is no international travel and travel frequency is unanticipated.
Standard Company Benefits.
