model explorer example with more than single inputs.

[artificalaudio]

Hi,

I've just built the Onnx runtime and been trying to get to know the API, and how to use ORT in a C++ app.

I have got the model explorer example working, and works as expected with models with a single input and output.

I've disabled the check/assertion about model must have only one input/output, and I am trying to figure out how to enable setting of multiple inputs.

I have been able to print the architecture of the input and output as such:

    /* This is the input shape of our model
    Input Node Name/Shape (0):
        main_in : 1x-1x256
        main_in_lengths : 1
        track : 1x-1
        rnd : 1x192x-1
    Output Node Name/Shape (0):
        Out : -1x1x-1
    */ 

How would I pass through a dummy input for a model with multiple inputs. I'm also not sure about the -1, the dynamic axes, should I give them a fixed amount/size? I'm a tad confused here.

If anyone could clear up how you'd pass through dummy inputs to a multiple input kind of model I would very much appreciate the help.

I'm using this code, which is identical to the model explorer code:

int main(int argc, ORTCHAR_T* argv[]) {
  if (argc != 2) {
    std::cout << "Usage: ./onnx-api-example <onnx_model.onnx>" << std::endl;
    return -1;
  }

  std::basic_string<ORTCHAR_T> model_file = argv[1];

  // onnxruntime setup
  Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "example-model-explorer");
  Ort::SessionOptions session_options;
  Ort::Session session = Ort::Session(env, model_file.c_str(), session_options);

  // print name/shape of inputs
  Ort::AllocatorWithDefaultOptions allocator;
  std::vector<std::string> input_names;
  std::vector<std::int64_t> input_shapes;
  std::cout << "Input Node Name/Shape (" << input_names.size() << "):" << std::endl;
  for (std::size_t i = 0; i < session.GetInputCount(); i++) {
    input_names.emplace_back(session.GetInputNameAllocated(i, allocator).get());
    input_shapes = session.GetInputTypeInfo(i).GetTensorTypeAndShapeInfo().GetShape();
    std::cout << "\t" << input_names.at(i) << " : " << print_shape(input_shapes) << std::endl;
  }
//   // some models might have negative shape values to indicate dynamic shape, e.g., for variable batch size.
  for (auto& s : input_shapes) {
    if (s < 0) {
      s = 1;
    }
  }

  // print name/shape of outputs
  std::vector<std::string> output_names;
  std::cout << "Output Node Name/Shape (" << output_names.size() << "):" << std::endl;
  for (std::size_t i = 0; i < session.GetOutputCount(); i++) {
    output_names.emplace_back(session.GetOutputNameAllocated(i, allocator).get());
    auto output_shapes = session.GetOutputTypeInfo(i).GetTensorTypeAndShapeInfo().GetShape();
    std::cout << "\t" << output_names.at(i) << " : " << print_shape(output_shapes) << std::endl;
  }

[skottmckay]

The model having a symbolic dimension for an input means any value can be provided.

When you create the input, the shape for the input must have values that match the actual data sizes. i.e. all values in the shapes for the input data must be 1 or higher.

This example shows creating inputs and calling Ort::Session::Run. Whilst the code only creates one input, it's written in a generic way where more inputs could be added to the vectors without needing to change the call to Run.

https://github.com/microsoft/onnxruntime-inference-examples/blob/83abcb28e382be0ab42a9728dd4446550cb4be69/c_cxx/OpenVINO_EP/Windows/squeezenet_classification/squeezenet_cpp_app.cpp#L350-L356

[artificalaudio]

@skottmckay

Thank you very much. I think I was over complicating the process in my head.

I've got to a stage where I roughly know what's going on. In the python code I'm porting currently, there's a dynamic axis, which is fine, I can override like I'm doing below. When I use in python I can have any size input, the thing I can't figure out, is based on this input, one the dimensions of the output changes. I do not know how to account for this, what if the shape of the output is unknown, or based on the input size? Instead of me hardcoding 1, 4, 256, what if I didn't know the size/length of 4 in advance?

Here's my simple class for anyone that wants to explore and is getting a bit confused with how things should be set up.

class OnnxSimple
{
    public:
    
        OnnxSimple(const std::string& vec_path="/me/home/model.onnx")
            : env(ORT_LOGGING_LEVEL_WARNING, "ContentVecModel"),
              session(env, vec_path.c_str(), session_options),
              model_path(vec_path)
        {
            std::cout << "load model(s) from " << model_path << std::endl;
            
            auto info = Ort::MemoryInfo::CreateCpu(OrtDeviceAllocator, OrtMemTypeCPU);
            mInputShapes  = GetInputShapes();
            mOutputShapes = GetOutputShapes();
            mInputNames   = GetInputNames();
            mOutputNames  = GetOutputNames();

            mInputShapes[0][2]  = 512; //input 1 1 -1 // replacing -1
            
            //if you had more inputs then they would be on different indexs, here [1] meaning 2nd input
            //mInputShapes[1][2]  = 512; //input 1 1 -1 // replacing -1
            
            //output:
            mOutputShapes[0][0] = 1;
            mOutputShapes[0][1] = 4;
            mOutputShapes[0][2] = 256;
            
            for (auto n : mInputNames)
                DBG (n); // You print input names and output names
        
            for (auto n : mOutputNames)
                DBG (n);
    
            for (auto& shape : mInputShapes) {
                juce::String shapeStr;
                for (auto dim : shape) {
                    shapeStr += juce::String(dim) + " ";
                }
                DBG(shapeStr); // check input shapes
            }
    
            for (auto& shape : mOutputShapes) {
                juce::String shapeStr;
                for (auto dim : shape) {
                    shapeStr += juce::String(dim) + " ";
                }
                DBG(shapeStr); // check output shapes
            }

            // inputs + outputs need resizing:
            mInputs.resize(  1 * 512 );
            mOutputs.resize( 4 * 256  );
            
            // clear/fill inputs with zeroes
            std::fill(mInputs.begin(), mInputs.end(), 0.0f);
            std::fill(mOutputs.begin(), mOutputs.end(), 0.0f);
            // push back input data into ort value/create tensor
            mInputTensors.push_back(Ort::Value::CreateTensor<float>(info, mInputs.data(), mInputs.size(), mInputShapes[0].data(), mInputShapes[0].size()));

            mOutputTensors.push_back(Ort::Value::CreateTensor<float>(info, mOutputs.data(), mOutputs.size(), mOutputShapes[0].data(), mOutputShapes[0].size()));

            DBG("finished setting up, ready to prime the model");
            
            RunInference();
            
            DBG("finished priming the model"); // if you can get to this stage it's a good sign
        }
    
    void RunInference()
    {
        const char* inputNamesCstrs[] =
        {
          mInputNames[0].c_str(),
          //mInputNames[1].c_str(), // here you would put any additional inputs, and same with outputs
          //mInputNames[2].c_str() // etc, for example. 
        };
        
        const char* outputNamesCstrs[] =
        {
            mOutputNames[0].c_str(),
            //mOutputNames[1].c_str() // if you had additional outputs 
            
        };

        session.Run(mRunOptions, inputNamesCstrs, mInputTensors.data(), mInputTensors.size(), outputNamesCstrs, mOutputTensors.data(), mOutputTensors.size());
    }

    private:
    
        inline std::vector<std::vector<int64_t>> GetInputShapes() const {
          size_t node_count = session.GetInputCount();
          std::vector<std::vector<int64_t>> out(node_count);
          for (size_t i = 0; i < node_count; i++) out[i] = session.GetInputTypeInfo(i).GetTensorTypeAndShapeInfo().GetShape();
          return out;
        }

        inline std::vector<std::vector<int64_t>> GetOutputShapes() const {
          size_t node_count = session.GetOutputCount();
          std::vector<std::vector<int64_t>> out(node_count);
          for (size_t i = 0; i < node_count; i++) out[i] = session.GetOutputTypeInfo(i).GetTensorTypeAndShapeInfo().GetShape();
          return out;
        }
        
        inline std::vector<std::string> GetInputNames() const {
          Ort::AllocatorWithDefaultOptions allocator;
          size_t node_count = session.GetInputCount();
          std::vector<std::string> out(node_count);
          for (size_t i = 0; i < node_count; i++) {
            auto tmp = session.GetInputNameAllocated(i, allocator);
            out[i] = tmp.get();
          }
          return out;
        }

        inline std::vector<std::string> GetOutputNames() const {
          Ort::AllocatorWithDefaultOptions allocator;
          size_t node_count = session.GetOutputCount();
          std::vector<std::string> out(node_count);
          for (size_t i = 0; i < node_count; i++) {
            auto tmp = session.GetOutputNameAllocated(i, allocator);
            out[i] = tmp.get();
          }
          return out;
        }
    
    Ort::RunOptions mRunOptions {nullptr};

    std::vector<Ort::Value> mInputTensors;
    std::vector<std::vector<int64_t>> mInputShapes;

    std::vector<Ort::Value> mOutputTensors;
    std::vector<std::vector<int64_t>> mOutputShapes;
    std::vector<std::string> mInputNames;
    std::vector<std::string> mOutputNames;
    
    //inputs/outputs
    std::vector<float>   mOutputs;
    std::vector<float> mInputs;
    
    Ort::SessionOptions session_options;
    Ort::Env env;
    Ort::Session session;
    std::string model_path;
};