Skip to content

v3.1.1

Choose a tag to compare

View all tags
@umarbutler umarbutler released this 18 Feb 05:02
· 22 commits to main since this release
v3.1.1
b164036

Added

  • Added a note to the quickstart section of the README advising users to deduct the number of special tokens automatically added by their tokenizer from their chunk size. This note had been removed in version 3.0.0 but has been readded as it is unlikely to be obvious to users.
Assets 2
Loading