EnglishRobertaTokenizer
FilterUnsupportedChars- Normalizer
- PadIndex
- PreTokenizer
- SymbolsCount
- Vocabulary
AddMaskSymbol(String)- ConvertIdsToOccurrenceRanks(IReadOnlyList<Int32>)
- ConvertIdsToOccurrenceValues(IReadOnlyList<Int32>)
- ConvertOccurrenceRanksToIds(IReadOnlyList<Int32>)
- CountTokens(String, ReadOnlySpan<Char>, EncodeSettings)
- Create(Stream, Stream, Stream)
- Create(String, String, String)
- Create(Stream, Stream, Stream, PreTokenizer, Normalizer, Boolean)
- Create(String, String, String, PreTokenizer, Normalizer, Boolean)
- Decode(IEnumerable<Int32>)
- Decode(IEnumerable<Int32>, Span<Char>, Int32, Int32)
- EncodeToIds(String, ReadOnlySpan<Char>, EncodeSettings)
- EncodeToTokens(String, ReadOnlySpan<Char>, EncodeSettings)
- GetIndexByTokenCount(String, ReadOnlySpan<Char>, EncodeSettings, Boolean, String, Int32)
- IsSupportedChar(Char)
net5.0
namespace Microsoft.ML.Tokenizers
{
public sealed class EnglishRobertaTokenizer : Tokenizer
{
public static EnglishRobertaTokenizer! Create(Stream! vocabularyStream, Stream! mergeStream, Stream! highestOccurrenceMappingStream, PreTokenizer? preTokenizer = null, Normalizer? normalizer = null, bool filterUnsupportedChars = true);
}
}
| nuget.org | 0.0 %
Reference this API |
|---|---|
| .NET | 5.06.07.08.09.0 |
| .NET Core | 2.02.12.23.03.1 |
| .NET Framework | 4.6.14.6.24.74.7.14.7.24.84.8.1 |
| .NET Standard | 2.02.1 |
Information specific to net5.0 | |
| Assembly | Microsoft.ML.Tokenizers , Version=1.0.0.0, PublicKeyToken=cc7b13ffcd2ddd51 |
| Referencing | Your project needs a package reference to |
| Package | Microsoft.ML.Tokenizers netstandard2.0 |
| Preview | This API is contained in a prerelease package. |
| Platform Restrictions | This API is supported on all platforms. |
- Built-in API
- Package-provided API