Skip to main content
.NET API Catalog
  • Browse
  • Search
  • Diff
  • Others
    Feature Usage Stats
    Version Source
  • .NET Upgrade Planner
  1. Catalog
  2. Microsoft.ML.Tokenizers
  3. EnglishRobertaTokenizer
  4. Vocabulary
  • EnglishRobertaTokenizer
    • FilterUnsupportedChars
    • Normalizer
    • PadIndex
    • PreTokenizer
    • SymbolsCount
    • Vocabulary
    • AddMaskSymbol(String)
    • ConvertIdsToOccurrenceRanks(IReadOnlyList<Int32>)
    • ConvertIdsToOccurrenceValues(IReadOnlyList<Int32>)
    • ConvertOccurrenceRanksToIds(IReadOnlyList<Int32>)
    • CountTokens(String, ReadOnlySpan<Char>, EncodeSettings)
    • Create(Stream, Stream, Stream)
    • Create(String, String, String)
    • Create(Stream, Stream, Stream, PreTokenizer, Normalizer, Boolean)
    • Create(String, String, String, PreTokenizer, Normalizer, Boolean)
    • Decode(IEnumerable<Int32>)
    • Decode(IEnumerable<Int32>, Span<Char>, Int32, Int32)
    • EncodeToIds(String, ReadOnlySpan<Char>, EncodeSettings)
    • EncodeToTokens(String, ReadOnlySpan<Char>, EncodeSettings)
    • GetIndexByTokenCount(String, ReadOnlySpan<Char>, EncodeSettings, Boolean, String, Int32)
    • IsSupportedChar(Char)
monoandroid
namespace Microsoft.ML.Tokenizers
{
    public sealed class EnglishRobertaTokenizer : Tokenizer
    {
        public IReadOnlyDictionary<string!, int>! Vocabulary { get; }
    }
}
.NET
5.06.07.08.09.011.0
.NET Core
2.02.12.23.03.1
.NET Framework
4.6.14.6.24.74.7.14.7.24.84.8.1
.NET Standard
2.02.1
Information specific to monoandroid
Assembly

Microsoft.ML.Tokenizers , Version=1.0.0.0, PublicKeyToken=cc7b13ffcd2ddd51

Referencing

Your project needs a package reference to Microsoft.ML.Tokenizers.

Package Microsoft.ML.Tokenizers (3.0.0) netstandard2.0
Platform Restrictions

This framework does not have platform annotations.

  • Built-in API
  • Package-provided API
Search

11/16/2026
This site is managed by Microsoft | © 2026 Microsoft. All rights reserved.
  • Privacy Statement
  • Consumer Health Privacy
  • Terms of Use
  • Contact Us
  • Hosted on Microsoft Azure
  • Manage cookies
  • Your Privacy Choices
Oops, we couldn't process your request. Reload 🗙