Intel AI Boost – Specialized AI Acceleration Instructions in C++

1. Understanding Intel AI Boost

Intel AI Boost, also known as Intel Deep Learning Boost (DL Boost), is a set of Intel AVX-512 instructions designed to enhance the performance of artificial intelligence (AI) applications, particularly in deep learning tasks. These instructions provide significant improvements in deep learning inference performance on modern Intel processors.

Through Vector Neural Network Instructions (VNNI), Intel AI Boost optimizes deep learning operations by reducing the number of instructions required for common tasks. Previously, executing a deep learning operation required three separate instructions; with VNNI, it can now be done in a single instruction. This results in major performance gains. Furthermore, VNNI supports INT8 inference, which reduces memory bandwidth requirements and power consumption while maintaining high model accuracy.

2. Using Intel AI Boost in C++ Applications

Developers can leverage Intel AI Boost in C++ applications by utilizing optimized libraries and frameworks that support these instructions. Intel provides various tools and libraries that facilitate the integration of these instructions into applications.

2.1 Using Intel® Extension for PyTorch* with C++

Intel offers an extension for PyTorch called Intel® Extension for PyTorch*, which includes a dynamic C++ library that facilitates AI inference acceleration.

To use this library, follow these steps:

• Download and install the cppsdk package: Ensure that you have downloaded and installed the appropriate cppsdk package.

• Write C++ code: You can write a C++ application using the PyTorch C++ API (known as LibTorch) with Intel’s extension. This involves loading the model, transferring data to the appropriate device (such as a CPU with Intel AI Boost support), and performing inference.

Example C++ Code Using PyTorch C++ API

​x
cppCopyEdit#include <torch/script.h>
#include <iostream>
#include <memory>

int main(int argc, const char* argv[]) {
  torch::jit::script::Module module;
  try {
    module = torch::jit::load(argv[1]);
  }
  catch (const c10::Error& e) {
    std::cerr << "Error loading the model\n";
    return -1;
  }
  module.to(at::kXPU);

  std::vector<torch::jit::IValue> inputs;
  torch::Tensor input = torch::rand({1, 3, 224, 224}).to(at::kXPU);
  inputs.push_back(input);

  at::Tensor output = module.forward(inputs).toTensor();
  std::cout << output.slice(/*dim=*/1, /*start=*/0, /*end=*/5) << std::endl;
  std::cout << "Execution successful" << std::endl;

  return 0;
}

This example demonstrates how to load a saved PyTorch model, transfer it to the appropriate device, and execute inference.

2.2 Using Intel® oneAPI Deep Neural Network Library (oneDNN)

Intel also provides the oneDNN library, an optimized performance library for deep learning operations such as convolutions, activations, and normalization. oneDNN supports Intel AI Boost instructions and can be used in C++ applications to improve performance.

Example Using oneDNN in C++

xxxxxxxxxx
cppCopyEdit#include <dnnl.hpp>
#include <vector>

int main() {
  using namespace dnnl;

  // Create engine and stream
  engine eng(engine::kind::cpu, 0);
  stream s(eng);

  // Define memory shapes
  memory::dims src_dims = {1, 1, 28, 28};
  memory::dims weights_dims = {16, 1, 3, 3};
  memory::dims bias_dims = {16};
  memory::dims dst_dims = {1, 16, 26, 26};

  // Create memory objects
  auto src_mem = memory({{src_dims}, memory::data_type::f32, memory::format_tag::nchw}, eng);
  auto weights_mem = memory({{weights_dims}, memory::data_type::f32, memory::format_tag::oihw}, eng);
  auto bias_mem = memory({{bias_dims}, memory::data_type::f32, memory::format_tag::x}, eng);
  auto dst_mem = memory({{dst_dims}, memory::data_type::f32, memory::format_tag::nchw}, eng);

  // Create convolution descriptor
  auto conv_desc = convolution_forward::desc(prop_kind::forward_inference,
                                             algorithm::convolution_direct,
                                             src_mem.get_desc(),
                                             weights_mem.get_desc(),
                                             bias_mem.get_desc(),
                                             dst_mem.get_desc(),
                                             {1, 1}, {0, 0}, {0, 0});

  // Create convolution primitive descriptor
  auto conv_prim_desc = convolution_forward::primitive_desc(conv_desc, eng);

  // Create convolution primitive and execute
  auto conv = convolution_forward(conv_prim_desc);
  conv.execute(s, {{DNNL_ARG_SRC, src_mem},
                   {DNNL_ARG_WEIGHTS, weights_mem},
                   {DNNL_ARG_BIAS, bias_mem},
                   {DNNL_ARG_DST, dst_mem}});

  // Wait for execution to complete
  s.wait();

  return 0;
}

This example demonstrates how to create and execute a convolution operation using oneDNN, leveraging Intel AI Boost instructions for better performance.

3. Benefits of Using Intel AI Boost in C++ Applications

• Improved Performance: Intel AI Boost provides significant performance enhancements for AI workloads.

• Optimized Memory and Power Consumption: By utilizing INT8 inference, these instructions reduce memory bandwidth requirements and power usage while maintaining accuracy.

• Seamless Integration: Intel provides tools and libraries such as oneDNN and PyTorch extensions that allow easy integration into existing AI workflows.

• Better Resource Utilization: By offloading computations to specialized CPU instructions, developers can optimize system resources effectively.

Intel AI Boost is an essential technology for C++ developers working with AI and deep learning, allowing them to maximize performance without requiring expensive hardware accelerators like GPUs. By leveraging optimized libraries and frameworks, developers can efficiently implement AI models on Intel CPUs, achieving high-speed inference with minimal resource overhead.