EN TN Fixes for Issue 166 (#207)

* Rebases the updated main

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* Passes Pynini fails SP

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* Adjustst the weights on the domain graph

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* Enables semiotic classes for SP tests

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Reweights the tokenizer

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* Updates test cases

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updates test cases

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* Cleans up ELECTRONIC tagger

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updates test cases

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* Updates Jenkins

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* Enables all CI tests

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

* Updates EN TN Cache

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>

---------

Signed-off-by: Simon Zuberek <szuberek@nvidia.com>
Co-authored-by: Simon Zuberek <szuberek@nvidia.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: zoobereq

Jenkinsfile

@@ -13,7 +13,7 @@ pipeline {
 
     AR_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/04-24-24-0'
     DE_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/06-03-24-0'
-    EN_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/08-02-24-0'
+    EN_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/08-15-24-0'
     ES_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/08-14-24-0'
     ES_EN_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/08-14-24-0'
     FR_TN_CACHE='/home/jenkinsci/TestData/text_norm/ci/grammars/06-04-24-0'

nemo_text_processing/text_normalization/en/data/electronic/domain.tsv

@@ -1,6 +1,7 @@
 .com	dot com
 .org	dot org
 .gov	dot gov
+.edu	dot edu
 .uk	dot UK
 .fr	dot FR
 .net	dot net

nemo_text_processing/text_normalization/en/taggers/electronic.py

@@ -1,4 +1,4 @@
-# Copyright (c) 2021, NVIDIA CORPORATION.  All rights reserved.
+# Copyright (c) 2024, NVIDIA CORPORATION.  All rights reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -22,7 +22,6 @@
     NEMO_DIGIT,
     NEMO_NOT_SPACE,
     NEMO_SIGMA,
-    NEMO_SPACE,
     NEMO_UPPER,
     TO_UPPER,
     GraphFst,
@@ -55,6 +54,7 @@ def __init__(self, cardinal: GraphFst, deterministic: bool = True):
         cc_cues = pynutil.add_weight(pynini.string_file(get_abs_path("data/electronic/cc_cues.tsv")), MIN_NEG_WEIGHT)
 
         accepted_symbols = pynini.project(pynini.string_file(get_abs_path("data/electronic/symbol.tsv")), "input")
+
         accepted_common_domains = pynini.project(
             pynini.string_file(get_abs_path("data/electronic/domain.tsv")), "input"
         )
@@ -115,16 +115,19 @@ def __init__(self, cardinal: GraphFst, deterministic: bool = True):
             username + domain_graph_with_class_tags,
         )
 
-        # abc.com, abc.com/123-sm
-        # when only domain, make sure it starts and end with NEMO_ALPHA
-        graph |= (
+        full_stop_accep = pynini.accep(".")
+        dollar_accep = pynini.accep("$")  # Include for the correct transduction of the money graph
+        excluded_symbols = full_stop_accep | dollar_accep
+        filtered_symbols = pynini.difference(accepted_symbols, excluded_symbols)
+        accepted_characters = NEMO_ALPHA | NEMO_DIGIT | filtered_symbols
+        domain_component = full_stop_accep + pynini.closure(accepted_characters, 2)
+        graph_domain = (
             pynutil.insert('domain: "')
-            + pynini.compose(
-                NEMO_ALPHA + pynini.closure(NEMO_NOT_SPACE) + accepted_common_domains + pynini.closure(NEMO_NOT_SPACE),
-                domain_graph,
-            ).optimize()
+            + (pynini.closure(accepted_characters, 1) + pynini.closure(domain_component, 1))
             + pynutil.insert('"')
-        )
+        ).optimize()
+
+        graph |= pynutil.add_weight(graph_domain, MIN_NEG_WEIGHT)
 
         # www.abc.com/sdafsdf, or https://www.abc.com/asdfad or www.abc.abc/asdfad
         graph |= protocol + pynutil.insert(" ") + domain_graph_with_class_tags

nemo_text_processing/text_normalization/en/taggers/tokenize_and_classify.py

@@ -53,7 +53,7 @@ class ClassifyFst(GraphFst):
     Final class that composes all other classification grammars. This class can process an entire sentence including punctuation.
     For deployment, this grammar will be compiled and exported to OpenFst Finite State Archive (FAR) File.
     More details to deployment at NeMo/tools/text_processing_deployment.
-    
+
     Args:
         input_case: accepting either "lower_cased" or "cased" input.
         deterministic: if True will provide a single transduction option,
@@ -78,11 +78,11 @@ def __init__(
             os.makedirs(cache_dir, exist_ok=True)
             whitelist_file = os.path.basename(whitelist) if whitelist else ""
             far_file = os.path.join(
-                cache_dir, f"en_tn_{deterministic}_deterministic_{input_case}_{whitelist_file}_tokenize.far"
+                cache_dir, f"en_tn_{deterministic}_deterministic_{input_case}_{whitelist_file}_tokenize.far",
             )
         if not overwrite_cache and far_file and os.path.exists(far_file):
             self.fst = pynini.Far(far_file, mode="r")["tokenize_and_classify"]
-            logger.info(f'ClassifyFst.fst was restored from {far_file}.')
+            logger.info(f"ClassifyFst.fst was restored from {far_file}.")
         else:
             logger.info(f"Creating ClassifyFst grammars.")
 
@@ -107,7 +107,7 @@ def __init__(
             logger.debug(f"fraction: {time.time() - start_time: .2f}s -- {fraction_graph.num_states()} nodes")
 
             start_time = time.time()
-            measure = MeasureFst(cardinal=cardinal, decimal=decimal, fraction=fraction, deterministic=deterministic)
+            measure = MeasureFst(cardinal=cardinal, decimal=decimal, fraction=fraction, deterministic=deterministic,)
             measure_graph = measure.fst
             logger.debug(f"measure: {time.time() - start_time: .2f}s -- {measure_graph.num_states()} nodes")
 
@@ -157,7 +157,7 @@ def __init__(
             time_final = pynini.compose(time_graph, v_time_graph)
             date_final = pynini.compose(date_graph, v_date_graph)
             range_graph = RangeFst(
-                time=time_final, date=date_final, cardinal=cardinal, deterministic=deterministic
+                time=time_final, date=date_final, cardinal=cardinal, deterministic=deterministic,
             ).fst
             logger.debug(f"range: {time.time() - start_time: .2f}s -- {range_graph.num_states()} nodes")
 
@@ -171,10 +171,10 @@ def __init__(
                 | pynutil.add_weight(ordinal_graph, 1.1)
                 | pynutil.add_weight(money_graph, 1.1)
                 | pynutil.add_weight(telephone_graph, 1.1)
-                | pynutil.add_weight(electonic_graph, 1.1)
+                | pynutil.add_weight(electonic_graph, 1.11)
                 | pynutil.add_weight(fraction_graph, 1.1)
                 | pynutil.add_weight(range_graph, 1.1)
-                | pynutil.add_weight(serial_graph, 1.1003)  # should be higher than the rest of the classes
+                | pynutil.add_weight(serial_graph, 1.12)  # should be higher than the rest of the classes
             )
 
             # roman_graph = RomanFst(deterministic=deterministic).fst
@@ -187,7 +187,8 @@ def __init__(
             punct = pynutil.insert("tokens { ") + pynutil.add_weight(punct_graph, weight=2.1) + pynutil.insert(" }")
             punct = pynini.closure(
                 pynini.compose(pynini.closure(NEMO_WHITE_SPACE, 1), delete_extra_space)
-                | (pynutil.insert(" ") + punct),
+                | (pynutil.insert(" ") + punct)
+                | punct,
                 1,
             )
 

tests/nemo_text_processing/en/data_text_normalization/test_cases_electronic.txt

@@ -4,7 +4,7 @@ a@hotmail.fr~a at hotmail dot FR
 a@hotmail.it~a at hotmail dot IT
 a@aol.it~a at aol dot IT
 a@msn.it~a at msn dot IT
-cdf@abc.edu~cdf at abc dot EDU
+cdf@abc.edu~cdf at abc dot edu
 abc@gmail.abc~abc at gmail dot ABC
 abc@abc.com~abc at abc dot com
 asdf123@abc.com~asdf one two three at abc dot com
@@ -13,9 +13,9 @@ ab3.sdd.3@gmail.com~ab three dot sdd dot three at gmail dot com
 email is abc1@gmail.com~email is abc one at gmail dot com
 abs@nvidia.com~abs at NVIDIA dot com
 email is a-1.b3_c&-d@gma4i-l.com test~email is a dash one dot b three underscore c ampersand dash d at gma four i dash l dot com test
-nvidia.com~NVIDIA dot com
+nvidia.com~nvidia dot com
 test.com~test dot com
-test.abc~test.abc
+test.abc~test dot abc
 http://www.ourdailynews.com.sm~HTTP colon slash slash WWW dot ourdailynews dot com dot SM
 https://www.ourdailynews.com.sm~HTTPS colon slash slash WWW dot ourdailynews dot com dot SM
 www.ourdailynews.com.sm~WWW dot ourdailynews dot com dot SM
@@ -24,8 +24,8 @@ sdf@gmail.com.sm~sdf at gmail dot com dot SM
 sdf@gmail.com/123-sm~sdf at gmail dot com slash one two three dash SM
 sdf@gmail.abc/123-sm~sdf at gmail dot ABC slash one two three dash SM
 sdf@gmail.abc/123456-sm~sdf at gmail dot ABC slash one two three four five six dash SM
-ourdailynews.com/12-sm~ourdailynews dot com slash one two dash SM
-ourdailynews.abc~ourdailynews.abc
+ourdailynews.com/12-sm~ourdailynews dot com slash one two dash sm
+ourdailynews.abc~ourdailynews dot abc
 file:///c/code/NeMo/docs/build/html/processing/intro.html~file colon slash slash slash c slash code slash nemo slash docs slash build slash html slash processing slash intro dot HTML
 file:///photos/image.jpg~file colon slash slash slash photos slash image dot jpeg
 electronic test.com and test2.uk~electronic test dot com and test two dot UK
@@ -38,4 +38,5 @@ rtxprohelp@exchange.nvidia.com~RTX pro help at exchange dot NVIDIA dot com
 enterpriseservices@nvidia.com~enterprise services at NVIDIA dot com
 enterprise-services@nvidia.com~enterprise dash services at NVIDIA dot com
 https://www.nvidia.com/dgx-basepod/~HTTPS colon slash slash WWW dot NVIDIA dot com slash DGX dash BASEPOD slash
-i can use your card ending in 8876~i can use your card ending in eight eight seven six
+i can use your card ending in 8876~i can use your card ending in eight eight seven six
+here is mail.nasa.gov.~here is mail dot nasa dot gov.

tests/nemo_text_processing/en/data_text_normalization/test_cases_punctuation.txt

@@ -39,7 +39,6 @@ a: ``b``, ``c``, ``d``, ``e``, ``f`` or ``g``.~a: "b", "c", "d", "e", "f" or "g"
 "And now," he said, "He,~"And now," he said, "He,
 80's~eighty's
 test .b@gmail.com~test. b at gmail dot com
-ourdailynews.abc/123-sm~ourdailynews. abc/one hundred twenty three-sm
 123,000012~one hundred and twenty three, zero zero zero zero one two
 9000,000th~nine thousand, zero zero zero th
 16 July, 1943~the sixteenth of july, nineteen forty three
@@ -59,6 +58,6 @@ dr. Evil~dr. Evil
 1!!!!~one!!!!
 (1)Hello~(one) Hello
 ÀÁÂÃ check §- and ƛ, also ɧ~ÀÁÂÃ check section - and ƛ, also ɧ
-Hi it's 5pm,4A.M.?-34. Hi,no,yes,34! 12,again,4 and NO?17 and $.01,here & there--0.004kg~Hi it's five PM, four AM.? minus thirty four. Hi,no,yes, thirty four! twelve, again, four and NO? seventeen and one cent, here and there - minus zero point zero zero four kilograms
+Hi it's 5pm,4A.M.?-34. Hi,no,yes,34! 12,again,4 and NO?17 and $.01,here & there -0.004kg~Hi it's five PM, four AM.? minus thirty four. Hi,no,yes, thirty four! twelve, again, four and NO? seventeen and one cent, here and there minus zero point zero zero four kilograms
 1°C.~one degree Celsius.
 my email is myemail@gmail.com!~my email is myemail at gmail dot com!

tests/nemo_text_processing/en/data_text_normalization/test_cases_punctuation_match_input.txt

@@ -10,4 +10,4 @@ a: ``b``,``c``, ``d``, ``e``, ``f`` or ``g``.~a: "b","c", "d", "e", "f" or "g".
 1 ,~one ,
 (1)Hello~(one)Hello
 ÀÁÂÃ check §- and ƛ, also ɧ~ÀÁÂÃ check section- and ƛ, also ɧ
-Hi it's 5pm,4A.M.?-34. Hi,no,yes,34! 12,again,4 and NO?17 and $.01,here & there--0.004kg~Hi it's five PM,four AM.?minus thirty four. Hi,no,yes,thirty four! twelve,again,four and NO?seventeen and one cent,here and there - minus zero point zero zero four kilograms
+Hi it's 5pm,4A.M.?-34. Hi,no,yes,34! 12,again,4 and NO?17 and $.01,here & there -0.004kg~Hi it's five PM,four AM.?minus thirty four. Hi,no,yes,thirty four! twelve,again,four and NO?seventeen and one cent,here and there minus zero point zero zero four kilograms