NVIDIA and Siemens Healthineers Just Gave Ultrasound a Physics-Aware AI Upgrade

NVIDIA and Siemens Healthineers have been quietly working on something that could change how we think about ultrasound imaging. They just released a model called NV-Raw2Insights-US, and the approach is genuinely different from what we’ve seen before.

Instead of training on finished ultrasound images — the kind you’d see on a screen — this model learns directly from raw sensor data. The signals coming off the probe before any reconstruction happens. That might sound like a small technical detail, but it’s actually a pretty big deal.

The Problem with Traditional Ultrasound

For decades, ultrasound images have been built using a hand-engineered reconstruction pipeline. It works, but it makes assumptions. The biggest one: that sound travels at a constant speed through all tissue. We know that’s not true. Sound moves differently through fat, muscle, bone, and fluid. But the traditional beamforming process simplifies that away because it’s too computationally expensive to account for it per patient.

The result is an image that’s good enough for most diagnostic purposes, but it discards a lot of information along the way. Raw sensor data is rich. It contains details about how sound actually scattered and reflected. That data is normally thrown out after reconstruction.

What NV-Raw2Insights-US Actually Does

The model takes raw channel data — the closest thing to how sound truly interacts with the body — and estimates a personalized speed-of-sound map for each patient in a single AI pass. It then feeds that back into the imaging pipeline to correct the focus in real time.

What used to require complex, iterative computation now happens almost instantly. The shift is from processing ultrasound images to understanding the underlying physics of each individual patient. They’re calling this class of models Raw2Insights, and this is the first application.

Getting Raw Data Out of Clinical Scanners

There’s a practical problem here. Most clinical ultrasound scanners don’t expose raw channel data easily. It’s high-bandwidth and not designed for external access. NVIDIA solved this with something called the Holoscan Sensor Bridge — an open-source FPGA IP that streams data from the scanner’s DisplayPort outputs over Ethernet to a GPU.

They paired it with an Altera Agilex-7 FPGA development kit and an ACUSON Sequoia scanner from Siemens. The data goes from the scanner DisplayPort, gets packetized by the FPGA, transmitted over Ethernet to an NVIDIA IGX system, and then runs inference on a Blackwell-class GPU. The sound-speed estimate gets streamed back to the scanner to improve live imaging.

This is probably the most interesting part of the whole thing. They’re not building a new scanner. They’re retrofitting existing hardware by using DisplayPort outputs that are already there. That’s a pragmatic approach that actually has a chance of making it into clinical environments.

What This Unlocks

Three things stand out about this architecture:

First, it’s software-defined. That means improvements can roll out through updates rather than hardware swaps. Second, once raw data is in GPU memory, adding new AI models is straightforward. You’re not fighting to get access to the data again. Third, the system runs on NVIDIA’s Holoscan edge platform, which is designed for real-time medical workloads.

Is it perfect? No. This is investigational technology. There’s no regulatory approval yet. The infrastructure required — FPGA boards, high-end GPUs, specialized networking — is not trivial. But the direction is right.

Why This Matters Beyond Image Quality

The real win here isn’t just sharper ultrasound images, though that’s part of it. It’s that AI is finally being designed to work with the physics of the modality, not just polish the output. Most medical AI today operates on reconstructed images. It’s trying to find tumors in JPEGs. This approach works with the raw signal. That’s a fundamentally different starting point.

NVIDIA and Siemens Healthineers have released the model weights, dataset, and code on GitHub. If you’re working on ultrasound or medical imaging AI, it’s worth taking a look. The shift from processing images to understanding physics is going to be a theme across medical imaging in the next few years. This is one of the first practical examples of what that looks like.