市值: $3.3401T -0.830%
體積(24小時): $100.8368B 22.900%
  • 市值: $3.3401T -0.830%
  • 體積(24小時): $100.8368B 22.900%
  • 恐懼與貪婪指數:
  • 市值: $3.3401T -0.830%
加密
主題
加密植物
資訊
加密術
影片
頭號新聞
加密
主題
加密植物
資訊
加密術
影片
bitcoin
bitcoin

$107768.358498 USD

-1.49%

ethereum
ethereum

$2529.161625 USD

-1.87%

tether
tether

$1.000169 USD

0.00%

xrp
xrp

$2.251414 USD

-1.08%

bnb
bnb

$658.434141 USD

-0.59%

solana
solana

$148.098376 USD

-2.76%

usd-coin
usd-coin

$1.000043 USD

0.00%

tron
tron

$0.287075 USD

-0.38%

dogecoin
dogecoin

$0.166579 USD

-3.89%

cardano
cardano

$0.574391 USD

-2.74%

hyperliquid
hyperliquid

$37.242063 USD

-6.85%

bitcoin-cash
bitcoin-cash

$495.213033 USD

-0.77%

sui
sui

$2.835127 USD

-3.41%

chainlink
chainlink

$13.230913 USD

-2.80%

unus-sed-leo
unus-sed-leo

$9.041547 USD

-0.05%

加密貨幣新聞文章

llama.cpp: Writing A Simple C++ Inference Program for GGUF LLM Models

2025/01/14 03:04

llama.cpp: Writing A Simple C++ Inference Program for GGUF LLM Models

This tutorial will guide you through the process of building a simple C++ program that performs inference on GGUF LLM models using the llama.cpp framework. We will cover the essential steps involved in loading the model, performing inference, and displaying the results. The code for this tutorial can be found here.

Prerequisites

To follow along with this tutorial, you will need the following:

A Linux-based operating system (native or WSL)

CMake installed

GNU/clang toolchain installed

Step 1: Setting Up the Project

Let's start by setting up our project. We will be building a C/C++ program that uses llama.cpp to perform inference on GGUF LLM models.

Create a new project directory, let's call it smol_chat.

Within the project directory, let's clone the llama.cpp repository into a subdirectory called externals. This will give us access to the llama.cpp source code and headers.

mkdir -p externals

cd externals

git clone https://github.com/georgigerganov/llama.cpp.git

cd ..

Step 2: Configuring CMake

Now, let's configure our project to use CMake. This will allow us to easily compile and link our C/C++ code with the llama.cpp library.

Create a CMakeLists.txt file in the project directory.

In the CMakeLists.txt file, add the following code:

cmake_minimum_required(VERSION 3.10)

project(smol_chat)

set(CMAKE_CXX_STANDARD 20)

set(CMAKE_CXX_STANDARD_REQUIRED ON)

add_executable(smol_chat main.cpp)

target_include_directories(smol_chat PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})

target_link_libraries(smol_chat llama.cpp)

This code specifies the minimum CMake version, sets the C++ standard and standard flag, adds an executable named smol_chat, includes headers from the current source directory, and links the llama.cpp shared library to our executable.

Step 3: Defining the LLM Interface

Next, let's define a C++ class that will handle the high-level interactions with the LLM. This class will abstract away the low-level llama.cpp function calls and provide a convenient interface for performing inference.

In the project directory, create a header file called LLMInference.h.

In LLMInference.h, declare the following class:

class LLMInference {

public:

LLMInference(const std::string& model_path);

~LLMInference();

void startCompletion(const std::string& query);

std::string completeNext();

private:

llama_model llama_model_;

llama_context llama_context_;

llama_sampler llama_sampler_;

std::vector _messages;

std::vector _formattedMessages;

std::vector _tokens;

llama_batch batch_;

};

This class has a public constructor that takes the path to the GGUF LLM model as an argument and a destructor that deallocates any dynamically-allocated objects. It also has two public member functions: startCompletion, which initiates the completion process for a given query, and completeNext, which fetches the next token in the LLM's response sequence.

Step 4: Implementing LLM Inference Functions

Now, let's define the implementation for the LLMInference class in a file called LLMInference.cpp.

In LLMInference.cpp, include the necessary headers and implement the class methods as follows:

#include "LLMInference.h"

#include "common.h"

#include

#include

#include

LLMInference::LLMInference(const std::string& model_path) {

llama_load_model_from_file(&llama_model_, model_path.c_str(), llama_model_default_params());

llama_new_context_with_model(&llama_context_, &llama_model_);

llama_sampler_init_temp(&llama_sampler_, 0.8f);

llama_sampler_init_min_p(&llama_sampler_, 0.0f);

}

LLMInference::~LLMInference() {

for (auto& msg : _messages) {

std::free(msg.content);

}

llama_free_model(&llama_model_);

llama_free_context(&llama_context_);

}

void LLMInference::startCompletion(const std::string& query)

免責聲明:info@kdj.com

所提供的資訊並非交易建議。 kDJ.com對任何基於本文提供的資訊進行的投資不承擔任何責任。加密貨幣波動性較大,建議您充分研究後謹慎投資!

如果您認為本網站使用的內容侵犯了您的版權,請立即聯絡我們(info@kdj.com),我們將及時刪除。

2025年07月08日 其他文章發表於