Applied Scientist / ML Engineer

GOSST T&L • AI/ML Role for LLM Accuracy & Model Development Job Description GOSST Turnover & Launch is looking for a hands-on ML/AI engineer (contractor, L5 equivalent) to join our team and drive LLM accuracy improvements across our product portfolio. This role sits at the intersection of applied machine learning and operational technology - you will fine-tune models, build evaluation frameworks, and improve AI-driven automation for our Turnover & Launch tools (OrderPad, UTP, Field Installation, and supporting agentic systems). Key Responsibilities: • Fine-tune and optimize Large Language Models (LLMs) for domain-specific tasks including code estimation, sprint planning, and operational automation. • Design and implement evaluation frameworks to measure model quality — including automated metrics, LLM-asjudge, and human evaluation workflows. • Build and maintain RAG (Retrieval-Augmented Generation) pipelines that ground model outputs in codebase and operational data. • Develop prompt engineering strategies and optimize prompt chains for multi-step agentic workflows. • Collaborate with SDEs and SDMs to integrate model improvements into production services (Amazon Bedrock, SageMaker, or equivalent). • Analyze model failure modes, identify accuracy gaps, and propose targeted improvements. • Establish baseline metrics and track accuracy improvements over time with dashboards and reporting. • Stay current on LLM advancements and recommend adoption of new techniques (LoRA, DPO, constitutional AI, etc.) where applicable. Basic QualificationsMS or PhD in Computer Science, Machine Learning, NLP, Statistics, or a related quantitative field (or equivalent industry experience).3+ years of hands-on experience with deep learning and NLP/LLM techniques.2+ years of experience fine-tuning language models (GPT, Claude, Llama, Mistral, or similar).2+ years of experience building ML pipelines end-to-end (data preparation, training, evaluation, and deployment).Strong Python skills.Experience with PyTorch, TensorFlow, or Hugging Face Transformers.Experience with model evaluation methodologies and metrics (BLEU, ROUGE, Accuracy, F1, Human Evaluation).Ability to work independently and deliver results with minimal supervision. Preferred QualificationsExperience with Amazon Bedrock, SageMaker, or AWS ML infrastructure.Hands-on experience with RAG architectures, vector databases (OpenSearch, Pinecone, FAISS), and embedding models.Experience with fine-tuning techniques: LoRA, QLoRA, RLHF, DPO, and PEFT.Experience with agentic AI systems (tool use, multi-step reasoning, and orchestration).Experience building LLM evaluation harnesses and automated testing for model quality.Familiarity with prompt engineering best practices and chain-of-thought techniques.Experience with CI/CD, agile development, and production deployment of ML models.Prior experience in supply chain, operations, or enterprise SaaS domains is a plus.Software / Programs / ToolsScience.Required Years of Experience3+ years. Basic QualificationsMS or PhD in Computer Science, Machine Learning, NLP, Statistics, or a related quantitative field (or equivalent industry experience).3+ years of hands-on experience with deep learning and NLP/LLM techniques.2+ years of experience fine-tuning language models (GPT, Claude, Llama, Mistral, or similar).2+ years of experience building ML pipelines end-to-end (data preparation, training, evaluation, and deployment).Strong Python skills.Experience with PyTorch, TensorFlow, or Hugging Face Transformers.Experience with model evaluation methodologies and metrics (BLEU, ROUGE, Accuracy, F1, Human Evaluation).Ability to work independently and deliver results with minimal supervision. Preferred QualificationsExperience with Amazon Bedrock, SageMaker, or AWS ML infrastructure.Hands-on experience with RAG architectures, vector databases (OpenSearch, Pinecone, FAISS), and embedding models.Experience with fine-tuning techniques: LoRA, QLoRA, RLHF, DPO, and PEFT.Experience with agentic AI systems (tool use, multi-step reasoning, and orchestration).Experience building LLM evaluation harnesses and automated testing for model quality.Familiarity with prompt engineering best practices and chain-of-thought techniques.Experience with CI/CD, agile development, and production deployment of ML models.Prior experience in supply chain, operations, or enterprise SaaS domains is a plus.Software / Programs / ToolsScience.Required Years of Experience3+ years.