packages/sentencepiece
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

add readme

Browse Source
This commit is contained in:
xiefangqi 2021-11-16 19:50:09 +08:00
parent bf542b1c4c
commit 22c62f617e
3 changed files with 332 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

171
README.en.md
View File

@ -8,15 +8,174 @@ Software architecture description
#### Installation
1. xxxx
2. xxxx
3. xxxx
1. Python module
SentencePiece provides Python wrapper that supports both SentencePiece training and segmentation. You can install Python binary package of SentencePiece with.
% pip install sentencepiece
2. Build and install SentencePiece command line tools from C++ source
The following tools and libraries are required to build SentencePiece:
* cmake
* C++11 compiler
* gperftools library (optional, 10-40% performance improvement can be obtained.)
On Ubuntu, the build tools can be installed with apt-get:
% sudo apt-get install cmake build-essential pkg-config libgoogle-perftools-dev
Then, you can build and install command line tools as follows.
% git clone https://github.com/google/sentencepiece.git
% cd sentencepiece
% mkdir build
% cd build
% cmake ..
% make -j $(nproc)
% sudo make install
% sudo ldconfig -v
On OSX/macOS, replace the last command with sudo update_dyld_shared_cache.
3. Build and install using vcpkg
You can download and install sentencepiece using the vcpkg dependency manager:
git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg integrate install
./vcpkg install sentencepiece
The sentencepiece port in vcpkg is kept up to date by Microsoft team members and community contributors. If the version is out of date, please create an issue or pull request on the vcpkg repository.
#### Instructions
1. xxxx
2. xxxx
3. xxxx
1. Train SentencePiece Model
% spm_train --input=< input > --model_prefix=<model_name> --vocab_size=8000 --character_coverage=1.0 --model_type=<type>
* --input: one-sentence-per-line raw corpus file. No need to run tokenizer, normalizer or preprocessor. By default, SentencePiece normalizes the input with Unicode NFKC. You can pass a comma-separated list of files.
* --model_prefix: output model name prefix. <model_name>.model and <model_name>.vocab are generated.
* --vocab_size: vocabulary size, e.g., 8000, 16000, or 32000
* --character_coverage: amount of characters covered by the model, good defaults are: 0.9995 for languages with rich character set like Japanese or Chinese and 1.0 for other languages with small character set.
* --model_type: model type. Choose from unigram (default), bpe, char, or word. The input sentence must be pretokenized when using word type.
2. Encode raw text into sentence pieces/ids
% spm_encode --model=<model_file> --output_format=piece < input > output
% spm_encode --model=<model_file> --output_format=id < input > output
Use --extra_options flag to insert the BOS/EOS markers or reverse the input sequence.
% spm_encode --extra_options=eos (add </s> only)
% spm_encode --extra_options=bos:eos (add <s> and </s>)
% spm_encode --extra_options=reverse:bos:eos (reverse input and add <s> and </s>)
SentencePiece supports nbest segmentation and segmentation sampling with --output_format=(nbest|sample)_(piece|id) flags.
% spm_encode --model=<model_file> --output_format=sample_piece --nbest_size=-1 --alpha=0.5 < input > output
% spm_encode --model=<model_file> --output_format=nbest_id --nbest_size=10 < input > output
3. Decode sentence pieces/ids into raw text
% spm_decode --model=<model_file> --input_format=piece < input > output
% spm_decode --model=<model_file> --input_format=id < input > output
Use --extra_options flag to decode the text in reverse order.
% spm_decode --extra_options=reverse < input > output
4. End-to-End Example
% spm_train --input=data/botchan.txt --model_prefix=m --vocab_size=1000
unigram_model_trainer.cc(494) LOG(INFO) Starts training with :
input: "../data/botchan.txt"
... <snip>
unigram_model_trainer.cc(529) LOG(INFO) EM sub_iter=1 size=1100 obj=10.4973 num_tokens=37630 num_tokens/piece=34.2091
trainer_interface.cc(272) LOG(INFO) Saving model: m.model
trainer_interface.cc(281) LOG(INFO) Saving vocabs: m.vocab
% echo "I saw a girl with a telescope." | spm_encode --model=m.model
▁I ▁saw ▁a ▁girl ▁with ▁a ▁ te le s c o pe .
% echo "I saw a girl with a telescope." | spm_encode --model=m.model --output_format=id
9 459 11 939 44 11 4 142 82 8 28 21 132 6
% echo "9 459 11 939 44 11 4 142 82 8 28 21 132 6" | spm_decode --model=m.model --input_format=id
I saw a girl with a telescope.
You can find that the original input sentence is restored from the vocabulary id sequence.
5. Export vocabulary list
% spm_export_vocab --model=<model_file> --output=<output file>
<output file> stores a list of vocabulary and emission log probabilities. The vocabulary id corresponds to the line number in this file.
6. Redefine special meta tokens
By default, SentencePiece uses Unknown (<unk>), BOS (<s>) and EOS (</s>) tokens which have the ids of 0, 1, and 2 respectively. We can redefine this mapping in the training phase as follows.
% spm_train --bos_id=0 --eos_id=1 --unk_id=5 --input=... --model_prefix=... --character_coverage=...
When setting -1 id e.g., bos_id=-1, this special token is disabled. Note that the unknow id cannot be disabled. We can define an id for padding (<pad>) as --pad_id=3.
7. Vocabulary restriction
spm_encode accepts a --vocabulary and a --vocabulary_threshold option so that spm_encode will only produce symbols which also appear in the vocabulary (with at least some frequency).
The usage is basically the same as that of subword-nmt. Assuming that L1 and L2 are the two languages (source/target languages), train the shared spm model, and get resulting vocabulary for each:
% cat {train_file}.L1 {train_file}.L2 | shuffle > train
% spm_train --input=train --model_prefix=spm --vocab_size=8000 --character_coverage=0.9995
% spm_encode --model=spm.model --generate_vocabulary < {train_file}.L1 > {vocab_file}.L1
% spm_encode --model=spm.model --generate_vocabulary < {train_file}.L2 > {vocab_file}.L2
shuffle command is used just in case because spm_train loads the first 10M lines of corpus by default.
Then segment train/test corpus with --vocabulary option
% spm_encode --model=spm.model --vocabulary={vocab_file}.L1 --vocabulary_threshold=50 < {test_file}.L1 > {test_file}.seg.L1
% spm_encode --model=spm.model --vocabulary={vocab_file}.L2 --vocabulary_threshold=50 < {test_file}.L2 > {test_file}.seg.L2
#### Contribution

170
README.md
View File

@ -9,15 +9,173 @@ An unsupervised text tokenizer and detokenizer.
#### 安装教程
1. xxxx
2. xxxx
3. xxxx
1. Python模块
SentencePiece 提供了支持 SentencePiece 训练和分割的 Python 包装器。你可以安装 SentencePiece 的 Python 二进制包。
% pip install sentencepiece
2. 从 C++ 源代码构建和安装 SentencePiece 命令行工具
构建 SentencePiece 需要以下工具和库：
* make
* C++11编译器
* gperftools库（可选，可以获得 10-40% 的性能提升。）
在 Ubuntu 上，可以使用 apt-get 安装构建工具：
% sudo apt-get install cmake build-essential pkg-config libgoogle-perftools-dev
然后，您可以按如下方式构建和安装命令行工具。
% git clone https://github.com/google/sentencepiece.git
% cd sentencepiece
% mkdir build
% cd build
% cmake ..
% make -j $(nproc)
% sudo make install
% sudo ldconfig -v
在 OSX/macOS 上，将最后一个命令替换为 sudo update_dyld_shared_cache
3. 用 vcpkg 构建和安装
您可以使用vcpkg依赖项管理器下载并安装句子：
git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
./bootstrap-vcpkg.sh
./vcpkg integrate install
./vcpkg install sentencepiece
vcpkg 中的sentencepiece端口由 Microsoft 团队成员和社区贡献者保持最新。
#### 使用说明
1. xxxx
2. xxxx
3. xxxx
1. 训练句子模型
% spm_train --input=<input> --model_prefix=<model_name> --vocab_size=8000 --character_coverage=1.0 --model_type=<type>
* --input：每行一个句子的原始语料库文件。无需运行分词器、规范器或预处理器。默认情况下，SentencePiece 使用 Unicode NFKC 规范化输入。您可以传递逗号分隔的文件列表。
* --model_prefix: 输出模型名称前缀。<model_name>.model并<model_name>.vocab生成。
* --vocab_size: 词汇量，例如 8000、16000 或 32000
* --character_coverage: 模型覆盖的字符数，好的默认值是：0.9995对于具有丰富字符集的语言，如日语或中文，1.0以及其他具有小字符集的语言。
* --model_type: 型号。从unigram（默认）bpe、char、 或 中选择word。使用wordtype时，输入的句子必须预先标记。
2. 将原始文本编码为句子片段/id
% spm_encode --model=<model_file> --output_format=piece < input > output
% spm_encode --model=<model_file> --output_format=id < input > output
使用--extra_optionsflag 插入 BOS/EOS 标记或反转输入顺序。
% spm_encode --extra_options=eos (add </s> only)
% spm_encode --extra_options=bos:eos (add <s> and </s>)
% spm_encode --extra_options=reverse:bos:eos (reverse input and add <s> and </s>)
SentencePiece 支持 nbest 分割和带--output_format=(nbest|sample)_(piece|id)标志的分割采样。
% spm_encode --model=<model_file> --output_format=sample_piece --nbest_size=-1 --alpha=0.5 < input > output
% spm_encode --model=<model_file> --output_format=nbest_id --nbest_size=10 < input > output
3. 将句子片段/id 解码为原始文本
% spm_decode --model=<model_file> --input_format=piece < input > output
% spm_decode --model=<model_file> --input_format=id < input > output
使用--extra_options标志以相反的顺序解码文本。
% spm_decode --extra_options=reverse < input > output
4. 端到端示例
% spm_train --input=data/botchan.txt --model_prefix=m --vocab_size=1000
unigram_model_trainer.cc(494) LOG(INFO) Starts training with :
input: "../data/botchan.txt"
... <snip>
unigram_model_trainer.cc(529) LOG(INFO) EM sub_iter=1 size=1100 obj=10.4973 num_tokens=37630 num_tokens/piece=34.2091
trainer_interface.cc(272) LOG(INFO) Saving model: m.model
trainer_interface.cc(281) LOG(INFO) Saving vocabs: m.vocab
% echo "I saw a girl with a telescope." | spm_encode --model=m.model
▁I ▁saw ▁a ▁girl ▁with ▁a ▁ te le s c o pe .
% echo "I saw a girl with a telescope." | spm_encode --model=m.model --output_format=id
9 459 11 939 44 11 4 142 82 8 28 21 132 6
% echo "9 459 11 939 44 11 4 142 82 8 28 21 132 6" | spm_decode --model=m.model --input_format=id
I saw a girl with a telescope.
可以发现原来的输入句是从词表id序列中还原出来的。
5. 导出词汇表
% spm_export_vocab --model=<model_file> --output=<output file>
<output file>存储词汇表和排放日志概率列表。词汇 id 对应于该文件中的行号。
6. 重新定义特殊元标记
By default, SentencePiece uses Unknown (<unk>), BOS (<s>) and EOS (</s>) tokens which have the ids of 0, 1, and 2 respectively. 我们可以在训练阶段重新定义这个映射如下。
% spm_train --bos_id=0 --eos_id=1 --unk_id=5 --input=... --model_prefix=... --character_coverage=...
例如，设置 -1 id 时，bos_id=-1将禁用此特殊标记。请注意，无法禁用未知 ID。我们可以将填充的 id (<pad>) 定义为--pad_id=3.
7. 词汇限制
spm_encode接受 a--vocabulary和 a--vocabulary_threshold选项，这样spm_encode只会产生也出现在词汇表中的符号（至少有一些频率）。
用法与subword-nmt. 假设 L1 和 L2 是两种语言（源/目标语言），训练共享的 spm 模型，并为每个得到结果词汇：
% cat {train_file}.L1 {train_file}.L2 | shuffle > train
% spm_train --input=train --model_prefix=spm --vocab_size=8000 --character_coverage=0.9995
% spm_encode --model=spm.model --generate_vocabulary < {train_file}.L1 > {vocab_file}.L1
% spm_encode --model=spm.model --generate_vocabulary < {train_file}.L2 > {vocab_file}.L2
shuffle命令只是为了以防万一，因为spm_train默认情况下加载语料库的前 10M 行。
然后使用--vocabulary选项分割训练/测试语料库
% spm_encode --model=spm.model --vocabulary={vocab_file}.L1 --vocabulary_threshold=50 < {test_file}.L1 > {test_file}.seg.L1
% spm_encode --model=spm.model --vocabulary={vocab_file}.L2 --vocabulary_threshold=50 < {test_file}.L2 > {test_file}.seg.L2
#### 参与贡献

4
sentencepiece.spec
View File

@ -1,6 +1,6 @@
Name: sentencepiece
Version: 0.1.92
Release: 4
Release: 5
Summary: An unsupervised text tokenizer and detokenizer
License: Apache-2.0
URL: https://github.com/google/sentencepiece
@ -49,6 +49,8 @@ make install
%{_includedir}/sentencepiece_*.h
%changelog
* Tue Nov 16 2021 xiefangqi <xiefangqi2@huawei.com> - 0.1.92.5
- add README.md/README.en.md
* Tue Nov 2 2021 xiefangqi <xiefangqi2@huawei.com> - 0.1.92-4
- fix of an unattainable condition
* Tue Nov 2 2021 xiefangqi <xiefangqi2@huawei.com> - 0.1.92-3