Skip to content

Tokenizer fixes and span_tokenize method #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 267 commits into
base: master
Choose a base branch
from
Open

Tokenizer fixes and span_tokenize method #20

wants to merge 267 commits into from

Conversation

chekunkov
Copy link
Contributor

@chekunkov chekunkov commented Jun 7, 2014 •
edited
Loading

Tokenizer from #15 had issues like not splitting a dot at the end of a sentence as a separate token

40006,40007c40017
< community
< .

---
> community.
41148,41149c41158
< Reserved
< .

---
> Reserved.

Now this issue should be fixed.

Also I've refactored code and added span_tokenize method (@kmike I remember you said it would be nice to have this method)

Performance wasn't hurt

X, y = webstruct.HtmlTokenizer().tokenize(trees)

CPU times: user 3.42 s, sys: 32 ms, total: 3.46 s
Wall time: 3.45 s

kmike and others added 30 commits July 31, 2013 07:27
@kmike
change tokenization rules again: don't split : and don't handle contr…
8392112 
…actions
@kmike
always clean html before feature extraction
c9cbedc
@kmike
let +34 and -8 be numbers
a59c4d1
@kmike
more features
4a44842
@kmike
more ideas for tags
113a5c9
@kmike
more annotated data
feeb392
@kmike
finished annotating US contacts pages corpus
9792d0e
@kmike
utility for grouping IOB-encoded entities
58de2dd
@kmike
discourage usage of preprocess.to_features_and_labels
1cd4ed6
@kmike
utility for substrings extraction
cf1877b
@kmike
small cleanup
cd40b8d
@kmike
WapitiTagger class
4badffc
@kmike
add IDEA files to gitignore
4f26c07
@kmike
handle an edge case for feature extraction
43314ba
@kmike
WapitiChunker is a better name
1ea5656
@kmike
more training data
caa97de
@kmike
more training data
2c37a10
@kmike
remove nltk dependency
66c13d9
@kmike
clarify requirements
9cac9ed
@kmike
fix default value in cleaning script
e42cfd0
@kmike
reannotated corpus (2/3 so far)
5b2e22f
@kmike
annotation guidelines
3be853f
@kmike
finish reannotation
71ba386
@kmike
new tags
d43da01
@kmike
simple docs
5ac1800
@kmike
add IPython temp files to gitignore
6e13b51
@tpeng
prepare html pages for NL
0e494e4
@tpeng
annotate NL pages
5a03eab
@tpeng
fix encoding
91cef1e
@tpeng
add notebook to train NL open hours parser
acf4519
kmike and others added 26 commits May 21, 2014 14:59
@kmike
annotation fixes
223d8f1
@kmike
more annotation fixes
8875d3c
@kmike
+100 pages
146ad5e
@kmike
annotation fixes
448048e
@kmike
BUG fix an issue with WebAnnotatorLoader: it shouldn't add extra "Non…
87279df 
…e" tokens
@kmike
fix a test after annotation fix
2150bda
@kmike
easier Trainer customization for CRFsuiteCRF
79d81c5
@kmike
X_dev and y_dev support for webstruct.crfsuite
a98431e
@kmike
+100 pages
1c47f9e
@kmike
doctests (failing) for some tokenization gotchas
e9ebeaa
@kmike
expose LongestMatchGlobalFeature
f80c382
@kmike
annotations fix
1c17e7c
@kmike
one more failing tokenization example
17a5d4e
@kmike
webstruct.gazetteers.geonames.read_geonames_zipped: try to handle geo…
9d8fcdc 
…names_filename automatically
@kmike
DAWG gazetteers support (they are much faster than MARISA-based, but …
ce775e6 
…less memory efficient)
@kmike
more annotated data
6ee718f
@kmike
CRFsuiteFeatureEncoder is not needed with python-crfsuite==0.6
ed40e3e 
Dropping it gives a nice speedup because computations are now in Cython.
@kmike
Undocumented HtmlFeatureExtractor post-processing step is removed to …
b2cb0e7 
…simplify code and make it faster.

If needed, it can be implemented as a global feature.
@kmike
bias feature
649c814
@kmike
tiny speedup for BestMatch._find_matches
12be72e
@kmike
NER.extract_groups_from_url
727f61b
@kmike
export webstruct.smart_join
cd1860d
@kmike
annotation fixes (more locations for about 70 pages)
56cd57e
@chekunkov
tokenizer - dot regex fix. WordTokenizer refactoring to be able to re…
33e638d 
…use code. span_tokenize method.
@chekunkov
Merge branch 'master' into tokenizer_additional_fixes_and_span_method
960bc7b 
Conflicts:
	webstruct/text_tokenizers.py
@chekunkov
fixed broken doctests
a010b00
@kmike
Copy link
Member

kmike commented Nov 25, 2016

@chekunkov do you by chance recall why wasn't this PR merged?

@chekunkov
Copy link
Contributor Author

@kmike nope, have no idea why.

@kmike kmike mentioned this pull request Dec 23, 2016
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@chekunkov @kmike @stav @tpeng @redapple