article thumbnail

Map Earth’s vegetation in under 20 minutes with Amazon SageMaker

AWS Machine Learning Blog

We pick the first week of December 2023 in this example. By utilizing the search_raster_data_collection function from SageMaker geospatial, we identified 8,581 unique Sentinel-2 images taken in the first week of December 2023. These batches are then evenly distributed across the machines in a cluster. format("/".join(tile_prefix),

ML 118
article thumbnail

PEFT fine tuning of Llama 3 on SageMaker HyperPod with AWS Trainium

AWS Machine Learning Blog

The process of setting up and configuring a distributed training environment can be complex, requiring expertise in server management, cluster configuration, networking and distributed computing. To simplify infrastructure setup and accelerate distributed training, AWS introduced Amazon SageMaker HyperPod in late 2023.

AWS 106
professionals

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Racing into the future: How AWS DeepRacer fueled my AI and ML journey

AWS Machine Learning Blog

Within a year, we built a world-class inference platform processing over 2 billion video frames daily using dynamically scaled Amazon Elastic Kubernetes Service (Amazon EKS) clusters. Although I set another world record in London, it wasn’t enough for first place.

AWS 106
article thumbnail

Evaluating Long-Context Question & Answer Systems

Eugene Yan

2023) found that domain experts in fields like biology or economics preferred answers that were both comprehensive and faithful, particularly for long-form questions. 2023) covers documents ranging from 3,000 to 200,000 tokens and includes 20 diverse subtasks, 508 extensive documents, and over 2,000 human-annotated question-answer pairs.

article thumbnail

Innovations in Analytics: Elevating Data Quality with GenAI

Towards AI

Hype Cycle for Emerging Technologies 2023 (source: Gartner) Despite AI’s potential, the quality of input data remains crucial. GenAI can help by automatically clustering similar data points and inferring labels from unlabeled data, obtaining valuable insights from previously unusable sources.

article thumbnail

How climate tech startups are building foundation models with Amazon SageMaker HyperPod

Flipboard

SageMaker HyperPod is a purpose-built infrastructure service that automates the management of large-scale AI training clusters so developers can efficiently build and train complex models such as large language models (LLMs) by automatically handling cluster provisioning, monitoring, and fault tolerance across thousands of GPUs.

AWS 116
article thumbnail

Real value, real time: Production AI with Amazon SageMaker and Tecton

AWS Machine Learning Blog

The US nationwide fraud losses topped $10 billion in 2023, a 14% increase from 2022. Orchestrate with Tecton-managed EMR clusters – After features are deployed, Tecton automatically creates the scheduling, provisioning, and orchestration needed for pipelines that can run on Amazon EMR compute engines.

ML 101