Usage

Installation

To use iamsystem, first install it using pip:

(.venv) $ pip install iamsystem

Matcher

The simplest example is to search a list of words in a document. To do so, Matcher is the main public API of this package. I recommend to use the Matcher build method to simplify its construction:

With a list of words (keywords)

from iamsystem import Matcher

matcher = Matcher.build(
    keywords=["acute respiratory distress syndrome", "diarrrhea"]
)
annots = matcher.annot_text(
    text="Pt c/o Acute Respiratory Distress Syndrome and diarrrhea"
)
for annot in annots:
    print(annot)
# Acute Respiratory Distress Syndrome	7 42	acute respiratory distress syndrome # noqa
# diarrrhea	47 56	diarrrhea

The matcher outputs a list of Annotation. By default, it performs exact match only. A limitation of passing words to the matcher is that no attributes are associated.

With a list of entities

Often, keywords are derived from a knowledge graph that associates a label with a unique identifier. The Entity class has a kb_id attribute to store an identifier.

from iamsystem import Entity
from iamsystem import Matcher

ent1 = Entity(label="acute respiratory distress syndrome", kb_id="J80")
ent2 = Entity(label="diarrrhea", kb_id="R19.7")
text = "Pt c/o acute respiratory distress syndrome and diarrrhea"
matcher = Matcher.build(keywords=[ent1, ent2])
annots = matcher.annot_text(text=text)
for annot in annots:
    print(annot)
# acute respiratory distress syndrome	7 42	acute respiratory distress syndrome (J80) # noqa
# diarrrhea (R19.7)	47	56

With a custom of keyword subclass

If you need to add other attributes to a keyword, you can create your own IKeyword implementation.

from iamsystem import Entity
from iamsystem import IEntity
from iamsystem import Matcher

class MyKeyword(IEntity):
    def __init__(
        self, label: str, category: str, kb_name: str, uri: str
    ):
        """label is the only mandatory attribute."""
        self.label = label
        self.kb_name = kb_name
        self.category = category
        self.kb_id = uri

    def __str__(self):
        """Called by print(annot)"""
        return f"{self.kb_id}"

ent1 = MyKeyword(
    label="acute respiratory distress syndrome",
    category="disease",
    kb_name="wikipedia",
    uri="https://www.wikidata.org/wiki/Q344873",
)
ent2 = Entity(label="diarrrhea", kb_id="R19.7")
text = "Pt c/o acute respiratory distress syndrome and diarrrhea"
matcher = Matcher.build(keywords=[ent1, ent2])
annots = matcher.annot_text(text=text)
for annot in annots:
    print(annot)
# acute respiratory distress syndrome	7 42	https://www.wikidata.org/wiki/Q344873 # noqa
# diarrrhea	47 56	diarrrhea (R19.7)

Note you can add different keyword types.

Context window (w)

iamsystem algorithm tries to match a sequence of tokens in a document to a sequence of tokens in a keyword/term. The w parameter determines how much discontinuous the sequence of tokens can be. By default, w=1 means that the sequence must be continuous.

Let’s say we want to detect the keyword “calcium level” in a document. With w=1, the matcher wouldn’t find the keyword in “calcium blood level” since the sequence of tokens in the document is discontinuous. One solution would be to add “blood” to the Stopwords list, however if “blood” is used by another keyword it would be a bad solution. Another solution is to set w=2 that lets the algorithm searches 2 words after token “calcium”.

from iamsystem import Matcher

matcher = Matcher.build(keywords=["calcium level"], w=2)
annots = matcher.annot_text(text="calcium blood level")
for annot in annots:
    print(annot)
# calcium level	0 7;14 19	calcium level

The semicolon indicates that the sequence is discontinuous. The first token “calcium” starts at character 0 and ends at character 6 (7-1). The second token “level” starts at character 14 and ends at character 18 (19-1).

Unidirectional detection

Word order is important. When the sequence of words in the document is not the same as the words sequence of the keyword, the algorithm fails to detect it. For example:

from iamsystem import Matcher

matcher = Matcher.build(keywords=["calcium level"], w=2)
annots = matcher.annot_text(text="level calcium")
print(len(annots))  # 0

This problem can be solved by changing the order of the tokens in a sentence which is the responsibility of the tokenizer. See Tokenizer section on Change tokens order.

Tokenizer

The iamsystem matcher is highly dependent on how documents and keywords are tokenized and normalized. The ITokenizer is responsible for turning text into tokens. To do so, the TokenizerImp class performs alphanumeric tokenization with two inner functions:

• split the text into (start,end) offsets

• normalize each token

The english_tokenizer and french_tokenizer are concrete implementations.

Other libraries offer more elaborate tokenizers, I recommend you use them. To use the tokenizer of another library you can build an adapter by creating a new implementation of the a ITokenizer interface. For example, this package provides a spaCy custom component that consumes spaCy’s tokenizer.

Default split function

By default, the Matcher class calls the french_tokenizer that splits a document by word character (a letter or digit or underbar [a-zA-Z0-9_]).

I recommend that you check the generated tokens to verify it matches your needs. For example:

from iamsystem import english_tokenizer

tokenizer = english_tokenizer()
tokens = tokenizer.tokenize("SARS-CoV+")
for token in tokens:
    print(token)
# Token(label='SARS', norm_label='sars', start=0, end=4, i=0)
# Token(label='CoV', norm_label='cov', start=5, end=8, i=1)

The ‘+’ sign is ignored even though it is important. The split function can be modified as follow :

from iamsystem import english_tokenizer
from iamsystem import split_find_iter_closure

tokenizer = english_tokenizer()
tokenizer.split = split_find_iter_closure(pattern=r"(\w+|\+)")
tokens = tokenizer.tokenize("SARS-CoV+")
for token in tokens:
    print(token)
# Token(label='SARS', norm_label='sars', start=0, end=4, i=0)
# Token(label='CoV', norm_label='cov', start=5, end=8, i=1)
# Token(label='+', norm_label='+', start=8, end=9, i=2)

Change default Tokenizer

To change Matcher’s default tokenizer, pass it to the constructor.

from iamsystem import Entity
from iamsystem import Matcher
from iamsystem import english_tokenizer
from iamsystem import split_find_iter_closure

ent1 = Entity(label="SARS-CoV+", kb_id="95209-3")
text = "Pt c/o acute respiratory distress syndrome. RT-PCR sars-cov+"
tokenizer = english_tokenizer()
tokenizer.split = split_find_iter_closure(pattern=r"(\w+|\+)")
matcher = Matcher.build(keywords=[ent1], tokenizer=tokenizer)
annots = matcher.annot_text(text=text)
for annot in annots:
    print(annot)
# sars cov +	51 60	SARS-CoV+ (95209-3)

Default normalize function

You can override the normalize function of a tokenizer to suit your needs. The english_tokenizer normalizes each token by doing lowercasing. The french_tokenizer performs lowercasing and remove accents. The only difference between the french_tokenizer and the english_tokenizer is the removal of diacritics done with the unidecode library that tries to transform the label in ASCII characters. Using the french_tokenizer for english documents adds very little overhead.

Change tokens order

Word order is important for iamsystem. In the example below, the keyword “blood calcium level “ is mentioned but the tokens are discontinuous and not in the right order. One solution is to order the tokens alphabetically. By doing this, the tokens of the document and the keyword are in the same order. Given a wide window, the keyword can be found.

from iamsystem import Matcher
from iamsystem import english_tokenizer

text = "the level of calcium can measured in the blood."
tokenizer = english_tokenizer()
matcher = Matcher.build(
    keywords=["blood calcium level"],
    tokenizer=tokenizer,
    order_tokens=True,
    w=5,
)
annots = matcher.annot_text(text=text)
for annot in annots:
    print(annot)
# level calcium blood	4 9;13 20;41 46	blood calcium level

order_tokens parameter changes iamsystem’s matching strategy but it doesn’t change the document’s tokens order. This approach is not suitable if the document is very long or the number of keywords is large.

Stopwords

Default Stopwords

It can be useful to remove stopwords, i.e. words that are not relevant to find a match. For example, the words ‘unspecified’ or ‘NOS’ (Not Otherwise Specified) is frequently used in medical terminologies to denote an entity that has been incompletely characterized.

from iamsystem import Entity
from iamsystem import Matcher
from iamsystem import english_tokenizer

ent = Entity(
    label="Essential hypertension, unspecified", kb_id="I10.9"
)
matcher = Matcher.build(
    keywords=[ent],
    tokenizer=english_tokenizer(),
    stopwords=["unspecified"],
)
text = "Medical history: essential hypertension"
annots = matcher.annot_text(text=text)
for annot in annots:
    print(annot)
# essential hypertension	17 39	Essential hypertension, unspecified (I10.9) # noqa

NegativeStopwords

Sometimes it’s useful to ignore all the words but those of the keywords. For example, we want to find the label “calcium blood” whatever the words between calcium and blood as long as the order is kept. One solution would be to change the Context window (w). Another solution is to use NegativeStopwords to ignore all words except those that the user wants to keep:

from iamsystem import Matcher

text = "the level of calcium can be measured in the blood."
matcher = Matcher.build(keywords=["calcium blood"], negative=True)
annots = matcher.annot_text(text=text)
for annot in annots:
    print(annot)
# calcium blood	13 20;44 49	calcium blood

Annotation

A Matcher outputs instances of Annotation. iamsystem algorithm tries to match a sequence of tokens in a document to a sequence of tokens in a keyword/term. An Annotation instance stores the sequence of tokens of a document matched to one or multiple keywords. Also, the name of the fuzzy algorithm that matched a token in a document is stored for machine learning or debugging purposes.

Annotation’s format

The to_string method returns a string representation containing three tabulated fields:

• A concatenation of tokens label as they appear in the document.

• The start-end offsets in the Brat format (start and end are separated by a space, a semicolon is used to separate offsets of discontinuous tokens).

• A string representation of detected Keywords.

For example:

from iamsystem import Entity
from iamsystem import Matcher

ent = Entity(label="infectious disease", kb_id="D007239")
matcher = Matcher.build(
    keywords=[ent], abbreviations=[("infect", "infectious")], w=2
)
text = "Infect mononucleosis disease"
annots = matcher.annot_text(text=text)
for annot in annots:
    print(annot)
    print(annot.to_string(text=text))
    print(annot.to_string(text=text, debug=True))
# Infect disease	0 6;21 28	infectious disease (D007239) # noqa
# Infect disease	0 6;21 28	infectious disease (D007239)	Infect mononucleosis disease # noqa
# Infect disease	0 6;21 28	infectious disease (D007239)	Infect mononucleosis disease	infect(abbs);disease(exact) # noqa

Passing the document to the to_string function adds the document substring that begins at the first token start offset and ends at the last token end offset. If debug equals True, it adds each token’s normalized label and the name(s) of the fuzzy algorithm(s) that detected it.

The method to_dict returns a dictionary representation of an annotation.

Multiple keywords per annotation

An Annotation has multiple keywords if and only if these keywords have the same tokenization output, i.e. the same sequence of tokens. This happens if two terms have the same label but also if the normalization process removes punctuation or if stopwords are ignored. In the example below, only one annotation is produced and it has 3 keywords:

from iamsystem import Entity
from iamsystem import Matcher
from iamsystem import english_tokenizer

ent1 = Entity(label="Infectious Disease", kb_id="J80")
ent2 = Entity(label="infectious disease", kb_id="C0042029")
ent3 = Entity(
    label="infectious disease, unspecified", kb_id="C0042029"
)
matcher = Matcher.build(
    keywords=[ent1, ent2, ent3],
    tokenizer=english_tokenizer(),
    stopwords=["unspecified"],
)
text = "History of infectious disease"
annots = matcher.annot_text(text=text)
annot = annots[0]
for keyword in annot.keywords:
    print(keyword)
# Infectious Disease (J80)
# infectious disease (C0042029)
# infectious disease, unspecified (C0042029)

Overlapping and ancestors

In a knowledge base, labels can share a same prefix. For example keywords “lung” and “lung cancer” have the same prefix “lung”. “lung” is called an ancestor of “lung cancer” because iamsystem algorithm constructs a graph representation of keywords. Note that ancestor is not defined by a binary relation (e.g. subsomption) that could exist in the knowledge base but only when two keywords have a common prefix.

Full overlapping

Definition: let a1 and a2 two annotations. If a1.start <= a2.start and a1.end > a2.end then we say that a1 fully overlaps a2. Furthermore, if a1 has all the tokens of a2 then a2 is called a nested annotation. By default, the matcher removes nested annotation. For example:

from iamsystem import Matcher

matcher = Matcher.build(keywords=["lung", "lung cancer"], w=1)
text = "Presence of a lung cancer"
annots = matcher.annot_text(text=text)
for annot in annots:
    print(annot)
# lung cancer	14 25	lung cancer
matcher.remove_nested_annots = False
annots_2 = matcher.annot_text(text=text)
for annot in annots_2:
    print(annot)
# lung	14 18	lung
# lung cancer	14 25	lung cancer

Another example where the first annotation fully overlaps the second but the latter is not a nested annotation:

from iamsystem import Matcher

matcher = Matcher.build(
    keywords=["North America", "South America"], w=3
)
text = "North and South America"
annots = matcher.annot_text(text=text)
for annot in annots:
    print(annot)
# North America	0 5;16 23	North America
# South America	10 23	South America

The first annotation, starting at offset 0 and ending at offset 23, fully overlaps the second. However, it doesn’t have all the tokens of the second annotation, thus the second annotation is not a nested annotation and it’s not removed. The brat format shows that North America keyword is a discontinuous sequence of tokens in the document.

Under the hood, the rm_nested_annots function is called to remove nested annotations. Ancestors are a frequent cause of nested annotations but not the only one. This function allows to remove nested annotations but to keep ancestors. Removing or keeping ancestors depends on your use case. In a semantic annotation task, only the longest terms must be kept so the ancestors need to be removed. In an information retrieval task, ancestors could be kept in the index.

Partial overlapping

Definition: let a1 and a2 two annotations. If a1.start < a2.start and a2.start < a1.end then we say that a1 partially overlaps a2.

from iamsystem import Matcher

matcher = Matcher.build(keywords=["lung cancer", "cancer prognosis"])
annots = matcher.annot_text(text="lung cancer prognosis")
for annot in annots:
    print(annot)
# lung cancer	0 11	lung cancer
# cancer prognosis	5 21	cancer prognosis

The first annotation partially overlaps the second because it ends after the second starts. In this example, both annotations share the “cancer” token.

Similarly the rm_nested_annots function has no effect here.

Fuzzy Algorithms

Introduction

iamsystem algorithm tries to match a sequence of tokens in a document to a sequence of tokens in a keyword. The default fuzzy algorithm of the Matcher class is the exact match algorithm. In general, in entity linking tasks, exact matching has high precision but low recall since a single character difference in a token can lead to a miss.

In this package, a fuzzy algorithm is an algorithm that is a called for each token in a document and can return one or more synonym, i.e. another string with the same meaning. The combination of several fuzzy algorithms offers great flexibility in the matching strategy, it increases recall but can also decrease precision.

This package doesn’t contain any implementation of approximate string matching algorithms, it relies on and wraps external libraries to do so. Some external libraries are not in the requirement file of this package, so you will need to install them manually depending on the fuzzy algorithm you wish to add.

Which fuzzy algorithm to choose

The set of fuzzy algorithms is configured by the user. Which one to add depends heavily on your documents and the keywords you want to detect.

If your documents contain a lot of typos, String Distance algorithms can help. If your documents contain a lot of abbreviations, it’s useful to have a sense inventory and add abbreviations to the Abbreviations class. If your documents and keywords contain inflected forms (singular, plurial, conjugated form), it is useful to add a normalization method (lemmatization, stemming) with the WordNormalizer class. If your keywords contain regular expressions, the FuzzyRegex class takes care of that.

Remember that for each token in the document, all fuzzy algorithms added to the Matcher will be called, so the more algorithms you add, the slower iamsystem. However, algorithms that are context independant can be cached to avoid calling them multiple times.

Abbreviations

The Abbreviations class allows you to provide a sense inventory of abbreviations to the matcher.

from iamsystem import Entity
from iamsystem import Matcher

ent1 = Entity(label="acute respiratory distress", kb_id="J80")
ent2 = Entity(label="patient", kb_id="D007290")
ent3 = Entity(label="patient hospitalized", kb_id="D007297")
ent4 = Entity(label="physiotherapy", kb_id="D007297")
matcher = Matcher.build(
    keywords=[ent1, ent2, ent3, ent4],
    abbreviations=[
        ("Pt", "patient"),
        ("PT", "physiotherapy"),
        ("ARD", "Acute Respiratory Distress"),
    ],
)
annots = matcher.annot_text(
    text="Pt hospitalized with ARD. Treament: PT"
)
for annot in annots:
    print(annot.to_string(debug=True))
# Pt hospitalized	0 15	patient hospitalized (D007297)	pt(abbs);hospitalized(exact) # noqa
# ARD	21 24	acute respiratory distress (J80)	ard(abbs)
# PT	36 38	patient (D007290)	pt(abbs)
# PT	36 38	physiotherapy (D007297)	pt(abbs)

Note the following:

• The first word “Pt” is associated with a single annotation.

Since “hospitalized” comes after the abbreviation and since the matcher removes nested annotation by default (See Full overlapping), the ambiguity is removed.

• The last word “PT” has two annotations

The Abbreviations is context independent and cannot resolve the ambiguity here. To solve this problem, the annotations need to be post-processed (rules, language models…) to identify the most likely long form.

In the case where two abbreviations have different string cases (Pt stands only for patient and PT for physiotherapy), the Abbreviations class can be configured to be case sensitive. The Abbreviations class can be configured with a method that checks if the document’s token is an abbreviation or not:

from iamsystem import Abbreviations
from iamsystem import Entity
from iamsystem import Matcher
from iamsystem import TokenT
from iamsystem import english_tokenizer

def upper_case_only(token: TokenT) -> bool:
    """Return True if all token's characters are uppercase."""
    return token.label.isupper()

def first_letter_capitalized(token: TokenT) -> bool:
    """Return True if the first letter is uppercase."""
    return token.label[0].isupper() and not token.label.isupper()

tokenizer = english_tokenizer()
ent1 = Entity(label="acute respiratory distress", kb_id="J80")
ent2 = Entity(label="patient", kb_id="D007290")
ent3 = Entity(label="patient hospitalized", kb_id="D007297")
ent4 = Entity(label="physiotherapy", kb_id="D007297")
matcher = Matcher.build(
    keywords=[ent1, ent2, ent3, ent4], tokenizer=tokenizer
)

abbs_upper = Abbreviations(
    name="upper case abbs", token_is_an_abbreviation=upper_case_only
)
abbs_upper.add(
    short_form="PT", long_form="physiotherapy", tokenizer=tokenizer
)
abbs_upper.add(
    short_form="ARD",
    long_form="Acute Respiratory Distress",
    tokenizer=tokenizer,
)
abbs_capitalized = Abbreviations(
    name="capitalized abbs",
    token_is_an_abbreviation=first_letter_capitalized,
)
abbs_capitalized.add(
    short_form="Pt", long_form="patient", tokenizer=tokenizer
)
matcher.add_fuzzy_algo(fuzzy_algo=abbs_upper)
matcher.add_fuzzy_algo(fuzzy_algo=abbs_capitalized)
annots = matcher.annot_text(
    text="Pt hospitalized with ARD. Treament: PT"
)
for annot in annots:
    print(annot.to_string(debug=True))
# Pt hospitalized	0 15	patient hospitalized (D007297)	pt(capitalized abbs);hospitalized(exact) # noqa
# ARD	21 24	acute respiratory distress (J80)	ard(upper case abbs)
# PT	36 38	physiotherapy (D007297)	pt(upper case abbs)

Notice that TokenT is a generic token type, so if you use a custom tokenizer (i.e. from an external library like spaCy) you can access custom attributes.

String Distance

This package utilizes the spellwise and pysimstring libraries to access string distance algorithms.

Spellwise

In the example below, iamsystem is configured with two spellwise algorithms: Levenshtein distance which measures the number of edits needed to transform one word into another, and Soundex which is a phonetic algorithm.

from iamsystem import Entity
from iamsystem import ESpellWiseAlgo
from iamsystem import Matcher

ent1 = Entity(label="acute respiratory distress", kb_id="J80")
matcher = Matcher.build(
    keywords=[ent1],
    spellwise=[
        dict(
            measure=ESpellWiseAlgo.LEVENSHTEIN,
            max_distance=1,
            min_nb_char=5,
        ),
        dict(measure=ESpellWiseAlgo.SOUNDEX, max_distance=1),
    ],
)
annots = matcher.annot_text(text="acute resiratory distresssss")
for annot in annots:
    print(annot.to_string(debug=True))
# acute resiratory distresssss	0 28	acute respiratory distress (J80) acute(exact,LEVENSHTEIN,SOUNDEX);resiratory(LEVENSHTEIN);distresssss(SOUNDEX) # noqa

The spellwise parameter of the build function expects an iterable of dictionary. The key-value pairs of a dictionary are passed to the SpellWiseWrapper init function. Since a string distance algorithm is context independent, the Matcher build function placed them in a CacheFuzzyAlgos to avoid calling them multiple times. For a list of available Spellwise algorithms, see ESpellWiseAlgo.

String distance algorithms are often used to detect typos in a document. False positives are common since two words could have a short string distance. To avoid calling a string distance algorithm on common words of a language, you can set string_distance_ignored_w parameter:

from iamsystem import ESpellWiseAlgo
from iamsystem import Matcher

matcher = Matcher.build(
    keywords=["poids"],
    spellwise=[
        dict(
            measure=ESpellWiseAlgo.LEVENSHTEIN,
            max_distance=1,
            min_nb_char=4,
        )
    ],
)
annots = matcher.annot_text(text="Absence de poils.")
for annot in annots:
    print(annot)
# poils	11 16	poids
matcher = Matcher.build(
    keywords=["poids"],
    spellwise=[
        dict(
            measure=ESpellWiseAlgo.LEVENSHTEIN,
            max_distance=1,
            min_nb_char=4,
        )
    ],
    string_distance_ignored_w=["poils"],
)
annots_2 = matcher.annot_text(text="Absence de poils.")
for annot in annots_2:
    print(annot)  # 0

Since poils is one substitution from poids, the algorithm returns a false positive. By adding poils to string_distance_ignored_w, the string distance algorithm is not called.

I recommend to pass all common words of a language to string_distance_ignored_w parameter, it will make iamsystem faster since all string distance algorithms will be called only for unknown words and this will reduce false positives.

SimString

The pysimstring library provides an API to the fast simstring algorithm implemented in C++. The simstring parameter of the Matcher build function expects an iterable of dictionary. The key-value pairs of a dictionary are passed to the SimStringWrapper init function. Since a string distance algorithm is context independent, the build function placed them in a CacheFuzzyAlgos to avoid calling them multiple times.

from iamsystem import Entity
from iamsystem import Matcher
from iamsystem.fuzzy.simstring import ESimStringMeasure

ent1 = Entity(label="acute respiratory distress", kb_id="J80")
matcher = Matcher.build(
    keywords=[ent1],
    simstring=[dict(measure=ESimStringMeasure.COSINE, threshold=0.7)],
)
annots = matcher.annot_text(text="acute respiratori disstress")
for annot in annots:
    print(annot)
# acute respiratori disstress	0 27	acute respiratory distress (J80)

Using the cosine similarity and a threshold of 0.7, the tokens respiratori matched to respiratory and disstress matched to distress.

CacheFuzzyAlgos

Fuzzy algorithms that are not context depend can be cached to avoid calling them multiple times. The CacheFuzzyAlgos stores fuzzy algorithms, calls them once and then stores their results.

from iamsystem import Abbreviations
from iamsystem import CacheFuzzyAlgos
from iamsystem import Entity
from iamsystem import ESpellWiseAlgo
from iamsystem import Matcher
from iamsystem import SpellWiseWrapper

ent1 = Entity(label="acute respiratory distress", kb_id="J80")
matcher = Matcher.build(keywords=[ent1])
abbs = Abbreviations(name="abbs")
abbs.add(short_form="a", long_form="acute", tokenizer=matcher)
test = dict(
    measure=ESpellWiseAlgo.LEVENSHTEIN, max_distance=1, min_nb_char=5
)
levenshtein = SpellWiseWrapper(**test)
soundex = SpellWiseWrapper(ESpellWiseAlgo.SOUNDEX, max_distance=1)
cache = CacheFuzzyAlgos()
for algo in [levenshtein, soundex]:
    algo.add_words(words=matcher.get_keywords_unigrams())
    cache.add_algo(algo=algo)
# cache.add_algo(algo=abbs)  ## no need to be this one in cache
matcher.add_fuzzy_algo(fuzzy_algo=cache)
matcher.add_fuzzy_algo(fuzzy_algo=abbs)
annots = matcher.annot_text(text="a resiratory distresssss")
for annot in annots:
    print(annot.to_string(debug=True))
# a resiratory distresssss	0 24	acute respiratory distress (J80)	a(abbs);resiratory(LEVENSHTEIN);distresssss(SOUNDEX) # noqa

Note that although we could have put the Abbreviations instance in the cache, it’s not necessary to do so since this algorithm is as fast as the cache. If you use the Matcher build function, string distance algorithms are automatically cached.

FuzzyRegex

Regular expressions are very useful and can be used with iamsystem. For example, if you want to detect blood test results in electronic health records, such as calcium levels in blood, you can have a regular expression in your keyword: “calcium (^d*[.,]?d*$) mmol/L”. The fuzzy_regex parameter expects an iterable of dictionary. Key-value pairs of the dictionary correspond to FuzzyRegex init function parameters.

The regular expression (^d*[.,]?d*$) is placed in the FuzzyRegex instance, with a patter name (ex: numval), and the pattern name is placed in the keyword (“calcium numval mmol/L”).

from iamsystem import Matcher
from iamsystem import english_tokenizer
from iamsystem import split_find_iter_closure

tokenizer = english_tokenizer()
tokenizer.split = split_find_iter_closure(pattern=r"(\w|\.|,)+")
matcher = Matcher.build(
    keywords=["calcium numval mmol/L"],
    tokenizer=tokenizer,
    stopwords=["level", "is", "normal"],
    fuzzy_regex=[
        dict(
            name="regex_num",
            pattern=r"^\d*[.,]?\d*$",
            pattern_name="numval",
        )
    ],
)
annots = matcher.annot_text(
    text="the blood calcium level is normal: 2.1 mmol/L"
)
for annot in annots:
    print(annot)
# calcium 2.1 mmol L	10 17;35 45	calcium numval mmol/L

Note that the Default split function must be modified to detect decimal values. Also note that the label of the keyword “calcium numval mmol/L” (line 7) contains the same pattern name numval. When the fuzzy algorithm receives the token value 2.1, it finds that it matches its regular expression and returns the pattern name numval.

In the example above, stopwords have been added, otherwise the algorithm wouldn’t have found the keyword with a context window of 1. It’s often the case that intermediate words are not known in avance, so this method wouldn’t work. Another way to do exactly the same annotation is to use the NegativeStopwords class which ignores all unigrams that are not in the keywords:

from iamsystem import Matcher
from iamsystem import english_tokenizer
from iamsystem import split_find_iter_closure

tokenizer = english_tokenizer()
tokenizer.split = split_find_iter_closure(pattern=r"(\w|\.|,)+")
matcher = Matcher.build(
    keywords=["calcium numval mmol/L"],
    tokenizer=tokenizer,
    negative=True,
    fuzzy_regex=[
        dict(
            name="regex_num",
            pattern=r"^\d*[.,]?\d*$",
            pattern_name="numval",
        )
    ],
)
annots = matcher.annot_text(
    text="the blood calcium level is normal: 2.1 mmol/L"
)
for annot in annots:
    print(annot)
# calcium 2.1 mmol L	10 17;35 45	calcium numval mmol/L

WordNormalizer

Word normalization is a common pre-processing step in NLP. The idea is to group words that have the same normalized form; for example “eating”, “eats”… have the same canonical form “eat”.

The WordNormalizer offers the possibility to add a normalization function. A token in a document will match a token in a keyword if they have the same normalized form.

In the example below, nltk is used to access a French stemmer. The stemming function is given to the WordNormalizer class:

from nltk.stem.snowball import FrenchStemmer

from iamsystem import Entity
from iamsystem import Matcher
from iamsystem import french_tokenizer

ent1 = Entity(label="cancer de la prostate", kb_id="C61")
stemmer = FrenchStemmer()
matcher = Matcher.build(
    keywords=[ent1],
    tokenizer=french_tokenizer(),
    stopwords=["de", "la"],
    normalizers=[dict(name="french_stemmer", norm_fun=stemmer.stem)],
)
annots = matcher.annot_text(text="cancer prostatique")
for annot in annots:
    print(annot)
# cancer prostatique	0 18	cancer de la prostate (C72)

Abstract Base classes

You might be interested in the fuzzy algorithms abstract base classes if you want to create a new custom fuzzy algorithm. The hierarchy is the following:

• FuzzyAlgo

Implements this class to create a context dependent algorithm. For each token for which a synonym is expected, the context words and the algorithm’s states are available.

• ContextFreeAlgo

Implements this class to create a context-free algorithm that depends only on the current token. The class has access to the generic token for which a synonym is expected. Examples of such algorithms: FuzzyRegex, Abbreviations.

• NormLabelAlgo

Implements this class to create a context-free algorithm that depends only on the normalized form of the token. The class has access to the normalized label of the token for which a synonym is expected. These algorithms can be cached with CacheFuzzyAlgos. Examples of such algorithms: String Distance, WordNormalizer.

Brat

Brat is an open source text annotation tool. This package provides a Brat adapter to generate Brat annotation files (.ann extension) in order to visualise iamsystem’s annotations in the Brat web interface.

Brat Formatter

Given a sequence of tokens, there are several ways of creating a Brat annotation. The default Brat formatter groups continuous sequence of tokens:

from iamsystem import Matcher

matcher = Matcher.build(keywords=["North America"])
annots = matcher.annot_text(text="North America")
for annot in annots:
    print(annot)
# North America	0 13	North America

Indeed, “North America” has two tokens, “North” and “America” but a continuous annotation (0 13) is created.

In order to have one Brat span for each token, you can use the IndividualTokenFormatter:

from iamsystem import IndividualTokenFormatter
from iamsystem import Matcher

matcher = Matcher.build(keywords=["North America"])
annots = matcher.annot_text(text="North America")
formatter = IndividualTokenFormatter()
for annot in annots:
    annot.brat_formatter = formatter
    print(annot)
# North America	0 5;6 13	North America

If you have stopwords in your matching sequences, you can include them in the Brat annotation using TokenStopFormatter. Stopwords are included if and only if they form a continuous sequence of tokens. Check the differences:

from iamsystem import Entity
from iamsystem import Matcher
from iamsystem import TokenStopFormatter

matcher = Matcher.build(
    keywords=[Entity(label="cancer de prostate", kb_id="C61")],
    stopwords=["de", "la"],
)
annots = matcher.annot_text(text="cancer de la prostate")
formatter = TokenStopFormatter()
for annot in annots:
    print(f"Default formatter: {annot}")
    annot.brat_formatter = formatter
    print(f"TokenStop formatter: {annot}")
# Default formatter: cancer prostate	0 6;13 21	cancer de prostate (C61) # noqa
# TokenStop formatter: cancer de la prostate	0 21	cancer de prostate (C61) # noqa

If your match is a discontinuous sequence of tokens and you want a continuous Brat annotation from the start offsets of the first token and end offsets of the last token, you can use the SpanFormatter. Check the differences:

from iamsystem import Matcher
from iamsystem import SpanFormatter

matcher = Matcher.build(
    keywords=["North America"], stopwords=["and"], w=2
)
text = "North and South America"
annots = matcher.annot_text(text=text)
formatter = SpanFormatter(text=text)
for annot in annots:
    print(f"Default formatter: {annot}")
    annot.brat_formatter = formatter
    print(f"Span formatter: {annot}")
# Default formatter: North America	0 5;16 23	North America
# Span formatter: North and South America	0 23	North America

Brat Document

The class BratDocument can store Brat entities and Brat notes. Each entity corresponds to an annotation:

• An ID

• A Brat type declared in Brat’s configuration file (annotation.conf)

• start-end offsets

• text substring

from iamsystem import BratDocument
from iamsystem import Entity
from iamsystem import Matcher

ent1 = Entity(label="North America", kb_id="NA")
matcher = Matcher.build(keywords=[ent1], w=3)
annots = matcher.annot_text(text="North and South America")
brat_document = BratDocument()
brat_document.add_annots(
    annots, brat_type="CONTINENT", keyword_attr=None
)
print(str(brat_document))
# T1	CONTINENT 0 5;16 23	North America
# #1	IAMSYSTEM T1	North America (NA)

The first line is the brat entity, the second is the brat note. T1 is the ID of the brat entity. Each note is linked to a brat entity by its ID, here T1. In the brat note, ‘North America (NA)’ is the comment related to this entity. By default, this comment is generated by calling the __str__ method of the Keyword. Here the __str__ method of the Entity class concatenated the label ‘North America’ and the code ‘(NA)’. You can modify this last value by overriding the get_note function of the BratDocument class.

Also note that in the above example, the Brat type “CONTINENT” is passed as a parameter and applies to all annotations. If you have multiple Brat types, a better way to do this is to store the Brat type in a Keyword subclass attribute and to pass the attribute name to the add_annots function:

from iamsystem import Entity

class Entity(Entity):
    def __init__(self, label: str, code: str, brat_type: str):
        super().__init__(label, code)
        self.brat_type = brat_type

from iamsystem import BratDocument
from iamsystem import Matcher

ent1 = Entity(label="North America", code="NA", brat_type="CONTINENT")
matcher = Matcher.build(keywords=[ent1], w=3)
annots = matcher.annot_text(text="North and South America")
brat_document = BratDocument()
brat_document.add_annots(annots=annots, keyword_attr="brat_type")
print(str(brat_document))
# T1	CONTINENT 0 5;16 23	North America
# #1	IAMSYSTEM T1	North America (NA)

Brat Writer

This package provides an utility class to write a BratDocument.

import os
import tempfile

from iamsystem import BratDocument
from iamsystem import BratWriter
from iamsystem import Entity
from iamsystem import Matcher

ent1 = Entity(label="North America", kb_id="NA")
matcher = Matcher.build(keywords=[ent1], w=3)
annots = matcher.annot_text(text="North and South America")
doc = BratDocument()
doc.add_annots(annots=annots, brat_type="CONTINENT")
temp_path = tempfile.mkdtemp()
os.makedirs(temp_path, exist_ok=True)
filename = os.path.join(temp_path, "docs.ann")
with (open(filename, "w")) as f:
    BratWriter.saveEntities(
        brat_entities=doc.get_entities(), write=f.write
    )
    BratWriter.saveNotes(brat_notes=doc.get_notes(), write=f.write)

spaCy

With a list of words

This package provides a spaCy component to add iamsystem algorithm in a spaCy pipeline.

from spacy.lang.fr import French

from iamsystem.spacy.component import IAMsystemBuildSpacy  # noqa

nlp = French()
nlp.add_pipe(
    "iamsystem_matcher",
    name="iamsystem",
    last=True,
    config={
        "build_params": {
            "keywords": [
                "North America",
                "South America",
            ],
            "abbreviations": [("amer", "America")],
            "stopwords": ["and"],
            "w": 2,
            "remove_nested_annots": True,
            "spellwise": [dict(max_distance=1, measure="Levenshtein")],
        },
    },
)
doc = nlp("Northh and South Amer.")
self.assertEqual(2, len(doc.spans["iamsystem"]))
spans = doc.spans["iamsystem"]
for span in spans:
    print(span._.iamsystem)
# Northh Amer	0 6;17 21	North America
# South Amer	11 21	South America

The build_params expects serializable Matcher build parameters. See IAMsystemBuildSpacy to configure this component.

With a list of keywords

Since Keyword implementation is not JSON serializable, you will have an error passing keywords instance to the keywords parameter. You have three options:

• Create your own IKeyword implementation that is JSON serializable.

• Pass a registered function:

@spacy.registry.misc("umls_ents.v1")
def get_termino_umls() -> Iterable[IKeyword]:
    """An imaginary set of umls ents."""
    termino = Terminology()
    ent1 = Entity("Insuffisance Cardiaque", "I50.9")
    ent2 = Entity("Insuffisance Cardiaque Gauche", "I50.1")
    termino.add_keywords(keywords=[ent1, ent2])
    return termino

"build_params": {
    "keywords": {"@misc": "umls_ents.v1"},
}

Note that if you call nlp.to_disk your keywords will not be serialized.

• Pass the Keyword as a dictionary with asdict() function and pass the module and classname of the Keyword dataclass:

config={
    "serialized_kw": {
        "module": "iamsystem",
        "class_name": "Keyword",
        "kws": [Keyword(label="insuffisance cardiaque").asdict()],
    },
    "build_params": {"w": 1},
},

Note that if you call nlp.to_disk your keywords will be serialized.