Usage
Matcher
The simplest example is to search a list of keywords in a document. By default, the Matcher performs exact match only.
With a list of words (keywords)
from iamsystem import Matcher
labels = ["acute respiratory distress syndrome", "diarrrhea"]
text = "Pt c/o Acute Respiratory Distress Syndrome and diarrrhea"
matcher = Matcher()
matcher.add_labels(labels=labels)
annots = matcher.annot_text(text=text)
for annot in annots:
print(annot)
# Acute Respiratory Distress Syndrome 7 42 acute respiratory distress syndrome
# diarrrhea 47 56 diarrrhea
The matcher outputs a list of Annotation. To add attributes, create a Keyword subclass. The Term class shown below associates a unique identifier to each label.
With a list of terms
Often, keywords are derived from a knowledge graph that associates a label with a unique identifier. The Term has a code attribute to store an identifier.
from iamsystem import Matcher, Term
term1 = Term(label="acute respiratory distress syndrome", code="J80")
term2 = Term(label="diarrrhea", code="R19.7")
text = "Pt c/o acute respiratory distress syndrome and diarrrhea"
matcher = Matcher()
matcher.add_keywords(keywords=[term1, term2])
annots = matcher.annot_text(text=text)
for annot in annots:
print(annot)
# acute respiratory distress syndrome 7 42 acute respiratory distress syndrome (J80)
# diarrrhea (R19.7) 47 56
Context window (w)
iamsystem algorithm tries to match a sequence of tokens in a document to a sequence of tokens in a keyword/term. The w parameter determines how much discontinuous the sequence of tokens can be. By default, w=1 means that the sequence must be continuous.
Let’s say we want to detect the keyword “calcium level” in a document. With w=1, the matcher wouldn’t find the keyword in “calcium blood level” since the sequence of tokens in the document is discontinuous. One solution would be to add “blood” to the Stopwords list, however if “blood” is used by another keyword it would be a bad solution. Another solution is to set w=2 that lets the algorithm searches 2 words after token “calcium”.
1 from iamsystem import Matcher
2 labels = ["calcium level"]
3 matcher = Matcher()
4 matcher.add_labels(labels=labels)
5 annots = matcher.annot_text(text="calcium blood level", w=2)
6 for annot in annots:
7 print(annot)
# calcium level 0 7;14 19 calcium level
The semicolon indicates that the sequence is discontinuous. The first token “calcium” starts at character 0 and ends at character 6 (7-1). The second token “level” starts at character 14 and ends at character 18 (19-1).
Unidirectional detection
Word order is important. When the sequence of words in the document is not the same as the words sequence of the keyword, the algorithm fails to detect it. For example:
from iamsystem import Matcher
labels = ["calcium level"]
matcher = Matcher()
matcher.add_labels(labels=labels)
annots = matcher.annot_text(text="level calcium", w=1)
print(len(annots)) # 0
This problem can be solved by changing the order of the tokens in a sentence which is the responsibility of the tokenizer. See Tokenizer section on Change tokens order.
Tokenizer
The iamsystem matcher is highly dependent on how documents and keywords are tokenized and normalized. The ITokenizer is responsible for turning text into tokens. To do so, the TokenizerImp class performs tokenization with two inner functions:
split the text into (start,end) offsets
normalize each token
The english_tokenizer and french_tokenizer are concrete implementations. To use another library to perform the tokenization you can build an adapter by creating a new implementation of the a ITokenizer class. For example, this package provides a spaCy custom component that consumes spaCy’s tokenizer.
Default split function
By default, the Matcher class calls the french_tokenizer that splits a document by word character (a letter or digit or underbar [a-zA-Z0-9_]).
I recommend that you check the generated tokens to verify it matches your needs. For example:
from iamsystem import english_tokenizer
tokenizer = english_tokenizer()
tokens = tokenizer.tokenize("SARS-CoV+")
for token in tokens:
print(token)
# Token(label='SARS', norm_label='sars', start=0, end=4)
# Token(label='CoV', norm_label='cov', start=5, end=8)
The ‘+’ sign is ignored even though it is important. The split function can be modified as follow :
from iamsystem import english_tokenizer, split_find_iter_closure
tokenizer = english_tokenizer()
tokenizer.split = split_find_iter_closure(pattern=r"(\w+|\+)")
tokens = tokenizer.tokenize("SARS-CoV+")
for token in tokens:
print(token)
# Token(label='SARS', norm_label='sars', start=0, end=4)
# Token(label='CoV', norm_label='cov', start=5, end=8)
# Token(label='+', norm_label='+', start=8, end=9)
Change default Tokenizer
To change Matcher’s default tokenizer, pass it to the constructor.
1 from iamsystem import Matcher, Term, split_find_iter_closure, english_tokenizer
2 term1 = Term(label="SARS-CoV+", code="95209-3")
3 text = "Pt c/o acute respiratory distress syndrome. RT-PCR sars-cov+"
4 tokenizer = english_tokenizer()
5 tokenizer.split = split_find_iter_closure(pattern=r"(\w+|\+)")
6 matcher = Matcher(tokenizer=tokenizer)
7 matcher.add_keywords(keywords=[term1])
8 annots = matcher.annot_text(text=text)
9 for annot in annots:
10 print(annot)
# sars cov + 51 60 SARS-CoV+ (95209-3)
Default normalize function
You can override the normalize function of a tokenizer to suit your needs. The english_tokenizer normalizes each token by doing lowercasing. The french_tokenizer performs lowercasing and remove accents. The only difference between the french_tokenizer and the english_tokenizer is the removal of diacritics done with the unidecode library that tries to transform the label in ASCII characters. Using the french_tokenizer for english documents adds very little overhead.
Change tokens order
Word order is important for iamsystem. In the example below, the keyword “blood calcium level “ is mentioned but the tokens are discontinuous and not in the right order. One solution is to order the tokens alphabetically. By doing this, the tokens of the document and the keyword are in the same order. Given a wide window, the keyword can be found.
1 from iamsystem import Matcher, english_tokenizer, tokenize_and_order_decorator
2 text = "the level of calcium can measured in the blood."
3 tokenizer = english_tokenizer()
4 tokenizer.tokenize = tokenize_and_order_decorator(tokenizer.tokenize)
5 matcher = Matcher(tokenizer=tokenizer)
6 matcher.add_labels(labels=["blood calcium level"])
7 tokens = matcher.tokenize(text=text)
8 annots = matcher.annot_tokens(tokens=tokens, w=len(tokens))
9 for annot in annots:
10 print(annot)
11 # level calcium blood 4 9;13 20;41 46 blood calcium level
Note that the window size is calculated with the number of tokens. This approach is not suitable if the document is very long or the number of keywords is big.
Stopwords
Default Stopwords
It can be useful to remove stopwords, i.e. words that are not relevant to find a match. For example, the words ‘unspecified’ or ‘NOS’ (Not Otherwise Specified) is frequently used in medical terminologies to denote an entity that has been incompletely characterized.
1 from iamsystem import Matcher, Term, english_tokenizer
2 tokenizer = english_tokenizer()
3 matcher = Matcher(tokenizer=tokenizer)
4 matcher.add_stopwords(words=["unspecified"])
5 term = Term(label="Essential hypertension, unspecified", code="I10.9")
6 matcher.add_keywords(keywords=[term])
7 text = "Medical history: essential hypertension"
8 annots = matcher.annot_text(text=text)
9 for annot in annots:
10 print(annot)
# essential hypertension 17 39 Essential hypertension, unspecified (I10.9)
NegativeStopwords
Sometimes it’s useful to ignore all the words but those of the keywords. For example, we want to find the label “calcium blood” whatever the words between calcium and blood are as long as the order is kept. One solution would be to change the Context window (w). Another solution is to use NegativeStopwords to ignore all words except those that the user wants to keep:
1 from iamsystem import Matcher, Terminology, NegativeStopwords, english_tokenizer, Keyword, NoStopwords
2 text = "the level of calcium can be measured in the blood."
3 termino = Terminology()
4 termino.add_keywords(keywords=[Keyword(label="calcium blood")])
5 neg_stopwords = NegativeStopwords()
6 tokenizer = english_tokenizer()
7 neg_stopwords.add_words(words_to_keep=termino.get_unigrams(tokenizer=tokenizer, stopwords=NoStopwords()))
8 matcher = Matcher(tokenizer=tokenizer, stopwords=neg_stopwords)
9 matcher.add_keywords(keywords=termino)
10 annots = matcher.annot_text(text=text, w=1)
11 for annot in annots:
12 print(annot)
# calcium blood 13 20;44 49 calcium blood
Note that you can use the Terminology class to retrieve all the unigrams of your keywords.
Annotation
A Matcher outputs instances of Annotation. iamsystem algorithm tries to match a sequence of tokens in a document to a sequence of tokens in a keyword/term. An Annotation instance stores the sequence of tokens of a document matched to one or multiple keywords. Also, the name of the fuzzy algorithm that matched a token in a document is stored for machine learning or debugging purposes.
Annotation’s format
The to_string method returns a string representation containing three tabulated fields:
A concatenation of tokens label as they appear in the document.
The start-end offsets in the Brat format (start and end are separated by a space, a semicolon is used to separate offsets of discontinuous tokens).
A string representation of detected Keywords.
For example:
1 from iamsystem import Matcher, Abbreviations, Term
2 matcher = Matcher()
3 abb = Abbreviations(name="abbs")
4 abb.add(short_form="infect", long_form="infectious", tokenizer=matcher)
5 matcher.add_fuzzy_algo(abb)
6 term = Term(label="infectious disease", code="D007239")
7 matcher.add_keywords(keywords=[term])
8 text = "Infect mononucleosis disease"
9 annots = matcher.annot_text(text=text, w=2)
10 for annot in annots:
11 print(annot)
12 print(annot.to_string(text=text))
13 print(annot.to_string(text=text, debug=True))
# Infect disease 0 6;21 28 infectious disease (D007239)
# Infect disease 0 6;21 28 infectious disease (D007239) Infect mononucleosis disease
# Infect disease 0 6;21 28 infectious disease (D007239) Infect mononucleosis disease infect(abbs);disease(exact)
Passing the document to the to_string function adds the document substring that begins at the first token start offset and ends at the last token end offset. If debug equals True, it adds each token’s normalized label and the name(s) of the fuzzy algorithm(s) that detected it.
The method to_dict returns a dictionary representation of an annotation.
Multiple keywords per annotation
An Annotation has multiple keywords if and only if these keywords have the same tokenization output, i.e. the same sequence of tokens. This happens if two terms have the same label but also if the normalization process removes punctuation or if stopwords are ignored. In the example below, only one annotation is produced and it has 3 keywords:
from iamsystem import Matcher, english_tokenizer, Term
term1 = Term(label="Infectious Disease", code="J80")
term2 = Term(label="infectious disease", code="C0042029")
term3 = Term(label="infectious disease, unspecified", code="C0042029")
tokenizer = english_tokenizer()
matcher = Matcher(tokenizer=tokenizer)
matcher.add_stopwords(words=["unspecified"])
matcher.add_keywords(keywords=[term1, term2, term3])
text = "History of infectious disease"
annots = matcher.annot_text(text=text)
annot = annots[0]
for keyword in annot.keywords:
print(keyword)
# Infectious Disease (J80)
# infectious disease (C0042029)
# infectious disease, unspecified (C0042029)
Overlapping and ancestors
In a knowledge base, labels can share a same prefix. For example keywords “lung” and “lung cancer” have the same prefix “lung”. “lung” is called an ancestor of “lung cancer” because iamsystem algorithm constructs a graph representation of keywords. Note that ancestor is not defined by a binary relation (e.g. subsomption) that could exist in the knowledge base but only when two keywords have a common prefix.
Full overlapping
Definition: let a1 and a2 two annotations. If a1.start <= a2.start and a1.end > a2.end then we say that a1 fully overlaps a2. Furthermore, if a1 has all the tokens of a2 then a2 is called a nested annotation. By default, the Matcher removes nested annotation. For example:
1 from iamsystem import Matcher
2 matcher = Matcher()
3 matcher.add_labels(labels=["lung", "lung cancer"])
4 text = "Presence of a lung cancer"
5 annots = matcher.annot_text(text=text, w=1)
6 for annot in annots:
7 print(annot)
8 # lung cancer 14 25 lung cancer
9 self.assertEqual("lung cancer 14 25 lung cancer", str(annots[0]))
10 matcher.remove_nested_annots = False
11 annots = matcher.annot_text(text=text, w=1)
12 for annot in annots:
13 print(annot)
14 # lung 14 18 lung
15 # lung cancer 14 25 lung cancer
Another example where the first annotation fully overlaps the second but the latter is not a nested annotation:
from iamsystem import Matcher
matcher = Matcher()
matcher.add_labels(labels=["North America", "South America"])
text = "North and South America"
annots = matcher.annot_text(text=text, w=3)
for annot in annots:
print(annot)
# North America 0 5;16 23 North America
# South America 10 23 South America
The first annotation, starting at offset 0 and ending at offset 23, fully overlaps the second. However, it doesn’t have all the tokens of the second annotation, thus the second annotation is not a nested annotation and it’s not removed. The brat format shows that North America keyword is a discontinuous sequence of tokens in the document.
Under the hood, the rm_nested_annots function is called to remove nested annotations. Ancestors are a frequent cause of nested annotations but not the only one. This function allows to remove nested annotations but to keep ancestors. Removing or keeping ancestors depends on your use case. In a semantic annotation task, only the longest terms must be kept so the ancestors need to be removed. In an information retrieval task, ancestors should be kept.
Partial overlapping
Definition: let a1 and a2 two annotations. If a1.start < a2.start and a2.start < a1.end then we say that a1 partially overlaps a2.
from iamsystem import Matcher
matcher = Matcher()
matcher.add_labels(labels=["lung cancer", "cancer prognosis"])
annots = matcher.annot_text(text="lung cancer prognosis")
for annot in annots:
print(annot)
# lung cancer 0 11 lung cancer
# cancer prognosis 5 21 cancer prognosis
The first annotation partially overlaps the second because it ends after the second starts. In this example, both annotations share the “cancer” token.
Similarly the the rm_nested_annots function has no effect here.
Fuzzy Algorithms
Introduction
iamsystem algorithm tries to match a sequence of tokens in a document to a sequence of tokens in a keyword. The default fuzzy algorithm of the Matcher class is the exact match algorithm. In general, in entity linking tasks, exact matching has high precision but low recall since a single character difference in a token can lead to a miss.
In this package, a fuzzy algorithm is an algorithm that is a called for each token in a document and can return one or more synonym, i.e. another string with the same meaning. The combination of several fuzzy algorithms offers great flexibility in the matching strategy, it increases recall but can also decrease precision.
This package doesn’t contain any implementation of approximate string matching algorithms, it relies on and wraps external libraries to do so. Some external libraries are not in the requirement file of this package, so you will need to install them manually depending on the fuzzy algorithm you wish to add.
Which fuzzy algorithm to choose
The set of fuzzy algorithms is configured by the user. Which one to add depends heavily on your documents and the keywords you want to detect.
If your documents contain a lot of typos, String Distance algorithms can help. If your documents contain a lot of abbreviations, it’s useful to have a sense inventory and add abbreviations to the Abbreviations class. If your documents and keywords contain inflected forms (singular, plurial, conjugated form), it is useful to add a normalization method (lemmatization, stemming) with the WordNormalizer class. If your keywords contain regular expressions, the FuzzyRegex class takes care of that.
Remember that for each token in the document, all fuzzy algorithms added to the Matcher will be called, so the more algorithms you add, the slower iamsystem. However, algorithms that are context independant can be cached to avoid calling them multiple times. See CacheFuzzyAlgos.
Abbreviations
The Abbreviations class allows you to provide a sense inventory of abbreviations to the matcher.
1 from iamsystem import Matcher, Abbreviations, english_tokenizer, Term
2 tokenizer = english_tokenizer()
3 abbs = Abbreviations(name="abbs")
4 abbs.add(short_form="Pt", long_form="patient", tokenizer=tokenizer)
5 abbs.add(short_form="PT", long_form="physiotherapy", tokenizer=tokenizer)
6 abbs.add(short_form="ARD", long_form="Acute Respiratory Distress", tokenizer=tokenizer)
7 matcher = Matcher(tokenizer=tokenizer)
8 term1 = Term(label="acute respiratory distress", code="J80")
9 term2 = Term(label="patient", code="D007290")
10 term3 = Term(label="patient hospitalized", code="D007297")
11 term4 = Term(label="physiotherapy", code="D007297")
12 matcher.add_keywords(keywords=[term1, term2, term3, term4])
13 matcher.add_fuzzy_algo(fuzzy_algo=abbs)
14 annots = matcher.annot_text(text="Pt hospitalized with ARD. Treament: PT")
15 for annot in annots:
16 print(annot.to_string(debug=True))
# Pt hospitalized 0 15 patient hospitalized (D007297) pt(abbs);hospitalized(exact)
# ARD 21 24 acute respiratory distress (J80) ard(abbs)
# PT 36 38 patient (D007290) pt(abbs)
# PT 36 38 physiotherapy (D007297) pt(abbs)
Note the following:
The first word “Pt” is associated with a single annotation.
Since “hospitalized” comes after the abbreviation and since the matcher removes nested keywords by default (See Full overlapping), the ambiguity is removed.
The last word “PT” has two annotations
The Abbreviations is context independent and cannot resolve the ambiguity here. To solve this problem, the annotations could be post-processed to identify the correct long form. A second solution would be to create a custom FuzzyAlgo instance which would be context dependent and which would return the most likely long.
In the case where two abbreviations have different string cases (Pt stands only for patient and PT for physiotherapy), the Abbreviations class can be configured to be case sensitive.
The Abbreviations class can be configured with a method that checks if the document’s token is an abbreviation or not:
1 from iamsystem import Matcher, Abbreviations, english_tokenizer, Term, TokenT
2
3 def upper_case_only(token: TokenT) -> bool:
4 """ Return True if all token's characters are uppercase."""
5 return token.label.isupper()
6
7 def first_letter_capitalized(token: TokenT) -> bool:
8 """ Return True if the first letter is uppercase."""
9 return token.label[0].isupper() and not token.label.isupper()
10
11 tokenizer = english_tokenizer()
12 abbs_upper = Abbreviations(name="upper case abbs", token_is_an_abbreviation=upper_case_only)
13 abbs_upper.add(short_form="PT", long_form="physiotherapy", tokenizer=tokenizer)
14 abbs_upper.add(short_form="ARD", long_form="Acute Respiratory Distress", tokenizer=tokenizer)
15 abbs_capitalized = Abbreviations(name="capitalized abbs", token_is_an_abbreviation=first_letter_capitalized)
16 abbs_capitalized.add(short_form="Pt", long_form="patient", tokenizer=tokenizer)
17 matcher = Matcher(tokenizer=tokenizer)
18 term1 = Term(label="acute respiratory distress", code="J80")
19 term2 = Term(label="patient", code="D007290")
20 term3 = Term(label="patient hospitalized", code="D007297")
21 term4 = Term(label="physiotherapy", code="D007297")
22 matcher.add_keywords(keywords=[term1, term2, term3, term4])
23 matcher.add_fuzzy_algo(fuzzy_algo=abbs_upper)
24 matcher.add_fuzzy_algo(fuzzy_algo=abbs_capitalized)
25 annots = matcher.annot_text(text="Pt hospitalized with ARD. Treament: PT")
26 for annot in annots:
27 print(annot.to_string(debug=True))
# Pt hospitalized 0 15 patient hospitalized (D007297) pt(capitalized abbs);hospitalized(exact)
# ARD 21 24 acute respiratory distress (J80) ard(upper case abbs)
# PT 36 38 physiotherapy (D007297) pt(upper case abbs)
Notice that TokenT is a generic token type, so if you use a custom tokenizer (i.e. from an external library like spaCy) you can access custom attributes.
String Distance
This package utilizes the spellwise python library to access string distance algorithms. In the example below, iamsystem is configured with two spelling algorithms: Levenshtein distance which measures the number of edits needed to transform one word into another, and Soundex which is a phonetic algorithm.
1 from iamsystem import ESpellWiseAlgo
2 from iamsystem import Matcher
3 from iamsystem import SpellWiseWrapper
4 from iamsystem import Term
5
6 levenshtein = SpellWiseWrapper(
7 ESpellWiseAlgo.LEVENSHTEIN, max_distance=1, min_nb_char=5
8 )
9 soundex = SpellWiseWrapper(ESpellWiseAlgo.SOUNDEX, max_distance=1)
10 term1 = Term(label="acute respiratory distress", code="J80")
11 matcher = Matcher()
12 matcher.add_keywords(keywords=[term1])
13 for algo in [levenshtein, soundex]:
14 algo.add_words(words=matcher.get_keywords_unigrams())
15 matcher.add_fuzzy_algo(algo)
16 annots = matcher.annot_text(text="acute resiratory distresssss")
17 for annot in annots:
18 print(annot.to_string(debug=True))
# acute resiratory distresssss 0 28 acute respiratory distress (J80) acute(exact,LEVENSHTEIN,SOUNDEX);resiratory(LEVENSHTEIN);distresssss(SOUNDEX)
The get_unigrams function retrieve all the single words (excluding stopwords) form the keywords. Spellwise algorithms need to get the keywords’words to return a suggestion. For a list of available Spellwise algorithms, see ESpellWiseAlgo. See also SpellWiseWrapper for configuration.
When the number of keywords is large, these algorithms can be slow. Since their output doesn’t depend on the context, I recommend using the CacheFuzzyAlgos class to store them.
CacheFuzzyAlgos
Fuzzy algorithms that are not context depend can be cached to avoid calling them multiple times. The CacheFuzzyAlgos stores fuzzy algorithms, calls them once and then stores their results.
1 from iamsystem import Abbreviations
2 from iamsystem import CacheFuzzyAlgos
3 from iamsystem import ESpellWiseAlgo
4 from iamsystem import Matcher
5 from iamsystem import SpellWiseWrapper
6 from iamsystem import Term
7
8 matcher = Matcher()
9 abbs = Abbreviations(name="abbs")
10 abbs.add(short_form="a", long_form="acute", tokenizer=matcher)
11 levenshtein = SpellWiseWrapper(
12 ESpellWiseAlgo.LEVENSHTEIN, max_distance=1, min_nb_char=5
13 )
14 soundex = SpellWiseWrapper(ESpellWiseAlgo.SOUNDEX, max_distance=1)
15 term1 = Term(label="acute respiratory distress", code="J80")
16 matcher.add_keywords(keywords=[term1])
17 cache = CacheFuzzyAlgos()
18 for algo in [levenshtein, soundex]:
19 algo.add_words(words=matcher.get_keywords_unigrams())
20 cache.add_algo(algo=algo)
21 # cache.add_algo(algo=abbs) ## no need to be this one in cache
22 matcher.add_fuzzy_algo(fuzzy_algo=cache)
23 matcher.add_fuzzy_algo(fuzzy_algo=abbs)
24 annots = matcher.annot_text(text="a resiratory distresssss")
25 for annot in annots:
26 print(annot.to_string(debug=True))
# acute resiratory distresssss 0 28 acute respiratory distress (J80) acute(exact,LEVENSHTEIN,SOUNDEX);resiratory(LEVENSHTEIN);distresssss(SOUNDEX)
Note that although we could have put the Abbreviations instance in the cache, it’s not necessary to do so since this algorithm is a fast as the cache because it stores the abbreviations in a dictionary.
FuzzyRegex
Regular expressions are very useful and can be used with iamsystem. For example, if you want to detect blood test results in electronic health records, such as calcium levels in blood, you can have a regular expression in your keyword: “calcium (^d*[.,]?d*$) mmol/L”. The class FuzzyRegex allows you to do this. The regular expression (^d*[.,]?d*$) is placed in the FuzzyRegex instance, with a patter name (ex: numval), and the pattern name is placed in your keyword (“calcium numval mmol/L”).
1 from iamsystem import Matcher, FuzzyRegex, split_find_iter_closure, english_tokenizer
2 fuzzy = FuzzyRegex(algo_name="regex_num", pattern=r"^\d*[.,]?\d*$", pattern_name="numval")
3 split = split_find_iter_closure(pattern=r"(\w|\.|,)+")
4 tokenizer = english_tokenizer()
5 tokenizer.split = split
6 detector = Matcher(tokenizer=tokenizer)
7 detector.add_labels(labels=["calcium numval mmol/L"])
8 detector.add_stopwords(words=["level", "is", "normal"])
9 detector.add_fuzzy_algo(fuzzy_algo=fuzzy)
10 annots = detector.annot_text(text="the blood calcium level is normal: 2.1 mmol/L", w=1)
11 for annot in annots:
12 print(annot)
13 # calcium 2.1 mmol L 10 17;35 45 calcium numval mmol/L
14 self.assertEqual(1, len(annots))
# calcium 2.1 mmol L 10 17;35 45 calcium numval mmol/L
Note that the Default split function must be modified to detect decimal values. Also note that the label of the keyword “calcium numval mmol/L” (line 7) contains the same pattern name numval. When the fuzzy algorithm receives the token value 2.1, it finds that it matches its regular expression and returns the pattern name numval.
In the example above, stopwords have been added, otherwise the algorithm wouldn’t have found the keyword with a context window of 1. It’s often the case that intermediate words are not known in avance, so this method wouldn’t work. Another way to do exactly the same annotation is to use the NegativeStopwords class which ignores all unigrams that are not in the keywords:
from iamsystem import FuzzyRegex
from iamsystem import Keyword
from iamsystem import Matcher
from iamsystem import NegativeStopwords
from iamsystem import NoStopwords
from iamsystem import Terminology
from iamsystem import english_tokenizer
from iamsystem import split_find_iter_closure
fuzzy = FuzzyRegex(
algo_name="regex_num",
pattern=r"^\d*[.,]?\d*$",
pattern_name="numval",
)
split = split_find_iter_closure(pattern=r"(\w|\.|,)+")
tokenizer = english_tokenizer()
tokenizer.split = split
keyword = Keyword(label="calcium numval mmol/L")
termino = Terminology()
termino.add_keywords(keywords=[keyword])
stopwords = NegativeStopwords(
words_to_keep=termino.get_unigrams(
tokenizer=tokenizer, stopwords=NoStopwords()
)
)
stopwords.add_fun_is_a_word_to_keep(fuzzy.token_matches_pattern)
matcher = Matcher(tokenizer=tokenizer, stopwords=stopwords)
matcher.add_keywords(keywords=termino)
matcher.add_fuzzy_algo(fuzzy_algo=fuzzy)
annots = matcher.annot_text(
text="the blood calcium level is normal: 2.1 mmol/L", w=1
)
for annot in annots:
print(annot)
# calcium 2.1 mmol L 10 17;35 45 calcium numval mmol/L
WordNormalizer
Word normalization is a common pre-processing step in NLP. The idea is to group words that have the same normalized form; for example “eating”, “eats”… have the same canonical form “eat”.
The WordNormalizer offers the possibility to add a normalization function. A token in a document will match a token in a keyword if they have the same normalized form.
In the example below, nltk is used to access a French stemmer. The stemming function is given to the WordNormalizer class:
1 from nltk.stem.snowball import FrenchStemmer
2
3 from iamsystem import Matcher
4 from iamsystem import Term
5 from iamsystem import WordNormalizer
6 from iamsystem import french_tokenizer
7
8 tokenizer = french_tokenizer()
9 matcher = Matcher(tokenizer=tokenizer)
10 matcher.add_stopwords(words=["de", "la"])
11 stemmer = FrenchStemmer()
12 fuzzy_stemmer = WordNormalizer(
13 name="french_stemmer", norm_fun=stemmer.stem
14 )
15 term1 = Term(label="cancer de la prostate", code="C61")
16 matcher.add_keywords(keywords=[term1])
17 fuzzy_stemmer.add_words(words=matcher.get_keywords_unigrams())
18 matcher.add_fuzzy_algo(fuzzy_stemmer)
19 annots = matcher.annot_text(text="cancer prostatique")
20 for annot in annots:
21 print(annot)
# cancer prostatique 0 18 cancer de la prostate (C72)
Abstract Base classes
You might be interested in the fuzzy algorithms abstract base classes if you want to create a new custom fuzzy algorithm. The hierarchy is the following:
Implements this class to create a context dependent algorithm. For each token for which a synonym is expected, the context words and the algorithm’s states are available.
Implements this class to create a context-free algorithm that depends only on the current token. The class has access to the generic token for which a synonym is expected. Examples of such algorithms: FuzzyRegex, Abbreviations.
Implements this class to create a context-free algorithm that depends only on the normalized form of the token. The class has access to the normalized label of the token for which a synonym is expected. These algorithms can be cached with CacheFuzzyAlgos. Examples of such algorithms: String Distance, WordNormalizer.
Brat
Brat is an open source text annotation tool. This package provides a Brat adapter to generate Brat annotation files (.ann extension) in order to visualise iamsystem’s annotations in the Brat web interface.
Brat Document
The class BratDocument can store Brat entities and Brat notes. Each entity corresponds to an annotation:
An ID
A Brat type that should be declared in Brat’s configuration file (annotation.conf)
start-end offsets
text substring
1 from iamsystem import Matcher, Term, BratDocument
2 matcher = Matcher()
3 term1 = Term(label="North America", code="NA")
4 matcher.add_keywords(keywords=[term1])
5 text = "North and South America"
6 annots = matcher.annot_text(text=text, w=3)
7 brat_document = BratDocument()
8 brat_document.add_annots(annots, text=text, brat_type="CONTINENT", keyword_attr=None)
9 print(str(brat_document))
# T1 CONTINENT 0 5;16 23 North America
# #1 IAMSYSTEM T1 North America (NA)
The first line is the brat entity, the second is the brat note. T1 is the ID of the brat entity. Each note is linked to a brat entity by its ID, here T1. In the brat note, ‘North America (NA)’ is the comment related to this entity. By default, this comment is generated by calling the __str__ method of the Keyword. Here the __str__ method of the Term class concatenated the label ‘North America’ and the code ‘(NA)’. You can modify this last value by overriding the get_note function of the BratDocument class.
Also note that in the above example, the Brat type “CONTINENT” is passed as a parameter and applies to all annotations. If you have multiple Brat types, a better way to do this is to store the Brat type in a Keyword subclass attribute and to pass the attribute name to the add_annots function:
1 from iamsystem import Term
2 class Entity(Term):
3 def __init__(self, label: str, code: str, brat_type: str):
4 super().__init__(label, code)
5 self.brat_type = brat_type
6
7 from iamsystem import Matcher, BratDocument
8 matcher = Matcher()
9 term1 = Entity(label="North America", code="NA", brat_type="CONTINENT")
10 matcher.add_keywords(keywords=[term1])
11 text = "North and South America"
12 annots = matcher.annot_text(text=text, w=3)
13 brat_document = BratDocument()
14 brat_document.add_annots(annots=annots, text=text, keyword_attr='brat_type')
15 print(str(brat_document))
# T1 CONTINENT 0 5;16 23 North America
# #1 IAMSYSTEM T1 North America (NA)
Brat Writer
This package provides an utility class to write a BratDocument.
1 from iamsystem import Matcher, Term, BratDocument, BratWriter
2 matcher = Matcher()
3 term1 = Term(label="North America", code="NA")
4 matcher.add_keywords(keywords=[term1])
5 text = "North and South America"
6 annots = matcher.annot_text(text=text, w=3)
7 brat_document = BratDocument()
8 brat_document.add_annots(annots, text=text, brat_type="CONTINENT")
9 filename = "./doc.ann"
10 with(open(filename, 'w')) as f:
11 BratWriter.saveEntities(brat_entities=brat_document.get_entities(), write=f.write)
12 BratWriter.saveNotes(brat_notes=brat_document.get_notes(), write=f.write)
spaCy
This package provides a stateful spaCy component to add iamsystem algorithm in a spaCy pipeline. Since a Matcher configuration is not JSON serializable, matcher’s parameters are passed in registered functions:
from typing import Iterable
from typing import List
import spacy
from spacy.lang.fr import French
from iamsystem import Abbreviations
from iamsystem import FuzzyAlgo
from iamsystem.spacy import IAMsystemSpacy # noqa
from iamsystem.spacy import IsStopSpacy
from iamsystem.spacy import TokenSpacyAdapter
from iamsystem import IKeyword
from iamsystem import IStopwords
from iamsystem import Term
from iamsystem import Terminology
from iamsystem import french_tokenizer
@spacy.registry.misc("umls_terms.v1")
def get_termino_umls() -> Iterable[IKeyword]:
"""An imaginary set of umls terms."""
termino = Terminology()
term1 = Term("Insuffisance Cardiaque", "I50.9")
term2 = Term("Insuffisance Cardiaque Gauche", "I50.1")
termino.add_keywords(keywords=[term1, term2])
return termino
@spacy.registry.misc("fuzzy_algos_short_notes.v1")
def get_fuzzy_algos_short_notes() -> List[FuzzyAlgo]:
"""An imaginary set of fuzzy algorithms for medical short notes."""
tokenizer = french_tokenizer()
abbs = Abbreviations(name="French medical abbreviations")
abbs.add(short_form="ins", long_form="insuffisance",
tokenizer=tokenizer)
abbs.add(
short_form="ic",
long_form="insuffisance cardiaque",
tokenizer=tokenizer,
)
return [abbs]
@spacy.registry.misc("stopwords_spacy.v1")
def get_stopwords_short_notes() -> IStopwords[TokenSpacyAdapter]:
"""Use spaCy stopword list."""
stopwords = IsStopSpacy()
return stopwords
nlp = French()
nlp.add_pipe(
"iamsystem",
name="iamsystem",
last=True,
config={
"keywords": {"@misc": "umls_terms.v1"},
"stopwords": {"@misc": "stopwords_spacy.v1"},
"fuzzy_algos": {"@misc": "fuzzy_algos_short_notes.v1"},
"w": 1,
"remove_nested_annots": True,
},
)
doc = nlp("ic gauche")
spans = doc.spans["iamsystem"]
for span in spans:
print(span._.iamsystem)
# ic gauche 0 9 Insuffisance Cardiaque Gauche (I50.1)
See IAMsystemSpacy to configure this component.