Remove 2012 Remove Clustering Remove Python
article thumbnail

Integrate HyperPod clusters with Active Directory for seamless multi-user login

AWS Machine Learning Blog

Amazon SageMaker HyperPod is purpose-built to accelerate foundation model (FM) training, removing the undifferentiated heavy lifting involved in managing and optimizing a large training compute cluster. In this solution, HyperPod cluster instances use the LDAPS protocol to connect to the AWS Managed Microsoft AD via an NLB.

article thumbnail

Use LangChain with PySpark to process documents at massive scale with Amazon SageMaker Studio and Amazon EMR Serverless

AWS Machine Learning Blog

Cost optimization – The serverless nature of the integration means you only pay for the compute resources you use, rather than having to provision and maintain a persistent cluster. This same interface is also used for provisioning EMR clusters. The following diagram illustrates this solution.

AWS 120
professionals

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Fine-tune multimodal models for vision and text use cases on Amazon SageMaker JumpStart

AWS Machine Learning Blog

We cover two approaches: using the Amazon SageMaker Studio UI for a no-code solution, and using the SageMaker Python SDK. FMs through SageMaker JumpStart in the SageMaker Studio UI and the SageMaker Python SDK. Fine-tune using the SageMaker Python SDK You can also fine-tune Meta Llama 3.2 Vision models. WASHINGTON, D.

ML 110
article thumbnail

Bring legacy machine learning code into Amazon SageMaker using AWS Step Functions

AWS Machine Learning Blog

The term legacy code refers to code that was developed to be manually run on a local desktop, and is not built with cloud-ready SDKs such as the AWS SDK for Python (Boto3) or Amazon SageMaker Python SDK. The best practice for migration is to refactor these legacy codes using the Amazon SageMaker API or the SageMaker Python SDK.

AWS 144
article thumbnail

Build ML features at scale with Amazon SageMaker Feature Store using data from Amazon Redshift

Flipboard

If you are prompted to choose a kernel, choose Data Science as the image and Python 3 as the kernel, then choose Select. as the image and Glue Python [PySpark and Ray] as the kernel, then choose Select. Here we use RedshiftDatasetDefinition to retrieve the dataset from the Redshift cluster.

ML 123
article thumbnail

Schedule your notebooks from any JupyterLab environment using the Amazon SageMaker JupyterLab extension

AWS Machine Learning Blog

In addition to the IAM user and assumed role session scheduling the job, you also need to provide a role for the notebook job instance to assume for access to your data in Amazon Simple Storage Service (Amazon S3) or to connect to Amazon EMR clusters as needed. Prerequisites For this post, we assume a locally hosted JupyterLab environment.

AWS 101
article thumbnail

Machine learning with decentralized training data using federated learning on Amazon SageMaker

AWS Machine Learning Blog

Usually, if the dataset or model is too large to be trained on a single instance, distributed training allows for multiple instances within a cluster to be used and distribute either data or model partitions across those instances during the training process. Each account or Region has its own training instances.