漏洞信息
尽管我们使用了先进的大模型技术,但其输出仍可能包含不准确或过时的信息。神龙努力确保数据的准确性,但请您根据实际情况进行核实和判断。
Vulnerability Title
tokenizers BpeBuilder Buffer Overflow via merge token
Vulnerability Description
tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.
CVSS Information
CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H
Vulnerability Type
跨界内存写
Vulnerability Title
Hugging Face Tokenizers 缓冲区错误漏洞
Vulnerability Description
Hugging Face Tokenizers是Hugging Face公司的一个将文本切分为词元的高性能工具。 Hugging Face Tokenizers 0.23.2及之前版本存在缓冲区错误漏洞,该漏洞源于tokenizers/src/models/bpe/model.rs中的BpeBuilder::build存在越界缓冲区访问,加载特制的tokenizer.json时,合并规则超过最长词汇键可能导致缓冲区溢出,造成拒绝服务,特定配置下还可能发生usize下溢,导致内存损坏。
CVSS Information
N/A
Vulnerability Type
N/A