[Jaskaran Singh Nagi (jnagi-intel)]

Description

Adds support for using a plugin EP device's own allocator for input/output buffers in onnxruntime_perf_test, instead of always using the CPU allocator.

• When a session uses plugin EP(s) (--plugin_eps), onnxruntime_perf_test now selects an allocator from the appended OrtEpDevice: it prefers the device's default (device-only) allocator, falls back to its host-accessible allocator, and falls back to the CPU allocator if neither is available or if more than one EP device was appended (no single unambiguous allocator to pick).
• Generated inputs (-r) and pre-allocated fixed-shape outputs now use this allocator instead of the CPU allocator.
• Real test data loaded from files is staged into the selected allocator's memory once at load time (via Ort::Env::CopyTensor), instead of being copied per-Run() implicitly.
• String tensors are always kept in CPU-accessible memory (inputs and outputs), since std::string is placement-constructed by ORT directly into the tensor's owned memory and can't safely live in device-only memory.
• Added [Plugin EP]-prefixed logging indicating which allocator was selected.

Motivation and Context

onnxruntime_perf_test previously always allocated input/output buffers on the CPU when running with plugin EPs. For plugin EPs whose kernels expect device-resident inputs/outputs, this meant every Run() iteration paid for an implicit host↔device copy that was entirely avoidable, skewing per-iteration latency measurements away from the EP's actual compute cost. Selecting and reusing the EP device's own allocator up front lets perf numbers better reflect steady-state inference cost for plugin EPs.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Updates perftest to select allocators exposed by plugin EP devices.

Changes:

• Returns selected plugin EP devices during session setup.
• Chooses a default or host-accessible plugin allocator.
• Falls back to the CPU allocator for ambiguous/unavailable devices.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
onnxruntime/test/perftest/utils.h	Declares plugin device and allocator-selection APIs.
onnxruntime/test/perftest/common_utils.cc	Implements selected-device tracking and allocator lookup.
onnxruntime/test/perftest/ort_test_session.cc	Applies the selected allocator to perftest buffers.

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[amd-rmotiram]

Should we have an option to make it configurable?
We may have situation where we want to use plugin EP but for some models we might want to use the allocator and for others we don't
Let me know if I missed something and there is a better way to account for this problem

[Jaskaran Singh Nagi (jnagi-intel)]

Should we have an option to make it configurable? We may have situation where we want to use plugin EP but for some models we might want to use the allocator and for others we don't Let me know if I missed something and there is a better way to account for this problem

Added an option to force cpu allocator, leaving plugin EP allocators as default as they would be best for performance.

[Chi Lo (chilo-ms)]

The issue mentioned above needs to be addressed.
#32244 (comment)

has_dynamic_output_shapes_ is global, and Run() clears every entry in outputs_ whenever any output is dynamic (lines 89–90). For a model with mixed fixed and dynamic outputs, the fixed outputs preallocated here with the plugin allocator are therefore discarded before the first run, so those buffers no longer use the selected allocator. Track dynamic outputs per index and reset only those entries (and use memory-info binding if dynamic outputs must also use plugin memory).

[Jaskaran Singh Nagi (jnagi-intel)]

The issue mentioned above needs to be addressed. #32244 (comment)

has_dynamic_output_shapes_ is global, and Run() clears every entry in outputs_ whenever any output is dynamic (lines 89–90). For a model with mixed fixed and dynamic outputs, the fixed outputs preallocated here with the plugin allocator are therefore discarded before the first run, so those buffers no longer use the selected allocator. Track dynamic outputs per index and reset only those entries (and use memory-info binding if dynamic outputs must also use plugin memory).

Chi Lo (@chilo-ms) Adressed it in may last commit.