Applied Scientist / ML Engineer

GOSST T&L - AI/ML Role for LLM Accuracy & Model DevelopmentJob Description GOSST Turnover & Launch is looking for a hands-on ML/AI engineer (contractor, L5 equivalent) to join our team and drive LLM accuracy improvements across our product portfolio. This role sits at the intersection of applied machine learning and operational technology - you will fine-tune models, build evaluation frameworks, and improve AI-driven automation for our Turnover & Launch tools (OrderPad, UTP, Field Installation, and supporting agentic systems).Key Responsibilities: • Fine-tune and optimize Large Language Models (LLMs) for domain-specific tasks including code estimation, sprint planning, and operational automation. • Design and implement evaluation frameworks to measure model quality — including automated metrics, LLM-asjudge, and human evaluation workflows. • Build and maintain RAG (Retrieval-Augmented Generation) pipelines that ground model outputs in codebase and operational data. • Develop prompt engineering strategies and optimize prompt chains for multi-step agentic workflows. • Collaborate with SDEs and SDMs to integrate model improvements into production services (Amazon Bedrock, SageMaker, or equivalent). • Analyze model failure modes, identify accuracy gaps, and propose targeted improvements. • Establish baseline metrics and track accuracy improvements over time with dashboards and reporting. • Stay current on LLM advancements and recommend adoption of new techniques (LoRA, DPO, constitutional AI, etc.) where applicable. Required qualifications/certifications: Basic Qualifications • MS or PhD in Computer Science, Machine Learning, NLP, Statistics, or a related quantitative field (or equivalent industry experience) • 3+ years of hands-on experience with deep learning and NLP/LLM techniques • 2+ years of experience fine-tuning language models (GPT, Claude, Llama, Mistral, or similar) • 2+ years of experience building ML pipelines end-to-end (data prep, training, evaluation, deployment) • Strong Python skills; experience with PyTorch, TensorFlow, or Hugging Face Transformers • Experience with model evaluation methodologies and metrics (BLEU, ROUGE, accuracy, F1, human eval) • Ability to work independently and deliver results with minimal supervision Preferred Qualifications • Experience with Amazon Bedrock, SageMaker, or AWS ML infrastructure • Hands-on experience with RAG architectures, vector databases (OpenSearch, Pinecone, FAISS), and embedding models • Experience with fine-tuning techniques: LoRA, QLoRA, RLHF, DPO, PEFT • Experience with agentic AI systems (tool use, multi-step reasoning, orchestration) • Experience building LLM evaluation harnesses and automated testing for model quality • Familiarity with prompt engineering best practices and chain-of-thought techniques • Experience with CI/CD, agile development, and production deployment of ML models • Prior experience in supply chain, operations, or enterprise SaaS domains is a plus • What You Will Work On • Improving LLM accuracy for code-grounded effort estimation across 6 product lines • Building evaluation pipelines to measure and track model quality across agentic ceremony workflows • Fine-tuning domain-specific models for sprint planning, prioritization, and operational automation • Reducing hallucination rates in RAG-based systems that reference internal codebases and documentation • Developing model quality dashboards and establishing accuracy baselines for continuous improvement Software / programs / tools: Science Required Years of Experience: 3+