Everything you need to know about NVIDIA H100 NVL

Interface: PCIe Gen 5.0 x16, equipped with three NVLink 4 bridges connecting dual GPUs.
Power Consumption: 700W to 800W total for both GPUs (350W to 400W per GPU).
Memory: 94GB HBM3 per GPU, totaling 188GB for the entire card. This offers 14GB more memory than standard H100 models.
Memory Bandwidth: 3.9TB/s per GPU, combining to 7.8TB/s.
Cooling Design: Dual-slot active cooling, designed for dense server environments.
Form Factor: PCIe, dual GPU configuration designed for maximum AI inference performance.
Architecture: NVIDIA Hopper, leveraging the latest advancements in AI and high-performance computing.
Compute Cores: Tensor Core performance parity with the SXM5 variant of the H100, supporting the latest AI model processing.
Compute Performance: Up to 3,341 TFLOPS of FP8 Tensor Core performance and 835 TFLOPS with TF32 sparsity.
MIG Technology: Supports multi-instance GPU (MIG) functionality for resource partitioning, allowing scalable usage across multiple AI tasks.
Special Features: Designed to supercharge large language model inference, particularly for GPT-3 and LLaMA-2 models, outperforming the A100 by up to 12x in inference.

AI and Deep Learning: Optimized for large language models, such as GPT-3 and LLaMA-2. It delivers up to 12x faster inference compared to the A100, making it highly effective for deploying large-scale AI models.
High-Performance Computing (HPC): Like the PCIe variant, it is also capable of performing advanced scientific simulations, but with significantly enhanced memory bandwidth and tensor core performance.
Data Analytics: Supports massive datasets with 94GB of HBM3 memory per GPU, ideal for real-time analytics requiring fast memory access.
Enterprise AI Workloads: Bundled with NVIDIA AI Enterprise, this card is designed to integrate seamlessly into AI infrastructure, enabling quick scaling for enterprise-level AI tasks.

Cooling and Power: Ensure your server infrastructure can handle the power demands (700W to 800W) and dual-slot cooling requirements for the dual GPU setup.
Infrastructure Compatibility: While the card uses PCIe Gen 5.0, its three NVLink bridges ensure high GPU-to-GPU bandwidth for tasks requiring maximum memory and tensor core performance.
Software Optimization: Leverage tools like CUDA, cuDNN, TensorRT, and NVIDIA AI Enterprise to maximize the performance of AI models, particularly for large-scale inference.

server-parts.eu Blog