SAN MATEO, Calif., Oct. 10, 2024 — Hammerspace, the company orchestrating the next data cycle, and Parallel Works, provider of the ACTIVATE control plane for AI and HPC resources, today unveiled a unified solution for compute and data orchestration.

This new solution enables IT infrastructure teams, research computing teams and AI teams to orchestrate compute, storage and data workloads seamlessly across sites and clouds, making it easy to scale cloud compute resources for burst capacity.

“Innovators need the agility to move complete workloads across different compute clusters – adding flexibility to use CPUs for data preparation, GPUs for processing, or accessing remote clusters for extra cycles or cost-efficient cloud regions. Seamlessly orchestrating both compute and data across locations and cloud regions is a game-changer,” said David Flynn, Founder and CEO of Hammerspace. “Our unified solution with Parallel Works simplifies operations, reduces overhead and accelerates innovation by ensuring data is always where it needs to be, with minimal latency.”

“This partnership marks a major advancement in managing compute resources and data across hybrid and multi-cloud environments,” said Matthew Shaxted, CEO of Parallel Works. “By integrating our expertise in HPC and AI workload orchestration with Hammerspace’s parallel file system, global namespace and data objectives, we’re enabling application portability and achieving agile task placement. This approach empowers organizations to efficiently allocate resources where needed, driving performance, innovation, and simplified hybrid infrastructure management.”

AI and HPC workloads demand significant compute resources and low-latency access to large datasets. However, both data and compute resources are often distributed across different locations, making it challenging to process data efficiently. Leveraging alternate data centers or cloud infrastructure provides a scalable, elastic option for burst compute capacity, enabling on-demand scaling. Yet, historically, orchestrating workloads across multiple, geographically dispersed clusters – especially when data is remote from the compute – has been highly complex. This complexity arises from the need to relocate the application while managing data transfer, network latency and synchronization, leading to error-prone and time-consuming setups that most teams have been unwilling to take on the burden of managing.

Hammerspace and Parallel Works have co-developed a unified, integrated solution for compute and data orchestration that automates the provisioning and management of local and cloud compute resources and orchestrates the flow of data to those compute resources.

In this solution, Parallel Works ACTIVATE acts as the unified control plane that creates, provisions, and manages the compute resources located in multiple data centers and clouds. Parallel Works also automates workflows, such as spinning up Slurm instances to manage job scheduling and the associated HPC nodes.

Hammerspace acts as the data platform that provides high-performance file and object storage, creates a Global Namespace that spans sites and clouds, and automates the flow of data within that namespace with Data Orchestration services. Hammerspace delivers the data to the HPC servers regardless of where they reside, with orchestration policies set at the share, directory, and even individual file levels.

This integrated solution enables administrators to create, configure, and manage storage clusters from the Parallel Works ACTIVE Control Plane console.

By making it easier to use elastic cloud resources for burst computing capacity, organizations can “right size” their infrastructure with the ease of use of a fully managed cloud service but at the cost point of standard cloud infrastructure services. This approach can deliver significant cloud cost savings, as shown in the chart below that compares the annual costs of Hammerspace on standard cloud virtual machines to Managed Lustre and shows the savings potential of the Hammerspace standards-based architecture.

Webinar

To dive deeper into the future of hybrid and multi-cloud orchestration, David Flynn and Matthew Shaxted will discuss this solution in a webinar hosted by The Register on Thursday, October 24, 2024. To register for the webinar, click on this link: Orchestrating the Future: Seamless Workload and Data Management in Distributed HPC Environments.

About Hammerspace

Hammerspace is radically changing how unstructured data is used and preserved. Our Global Data Platform unifies unstructured data across edge, data centers, and clouds. It provides extreme parallel performance for AI, GPUs, and high-speed data analytics, and orchestrates data to any location and any storage system. This eliminates data silos and makes data an instantly accessible resource for users, applications, and compute clusters, no matter where they are located.

About Parallel Works

Parallel Works ACTIVATE is the premiere control plane for HPC and AI resources, empowering teams with seamless provisioning, management, and sharing of compute resources at scale across on-premises, cloud, and hybrid environments with advanced cost control and budgeting features. ACTIVATE facilitates collaborative research and enhances productivity through intuitive interfaces and API-driven processes, enabling HPC cluster creation in the cloud.

---

Source: Hammerspace