Как преобразовать данные XML NER из корпуса CRAFT в формат JSON spaCy?

Вот код, который поможет вам начать работу. Это не полное решение, но проблема, которую вы поставили, очень сложная, и у вас не было стартового кода.

Он не отслеживает свойства identifier или NCBI Homologene, но я думаю, что они могут храниться в словаре отдельно.

import xml.etree.cElementTree as ET

import spacy

nlp = spacy.load('en_core_web_sm')

# this is one child of the XML doc
# https://www.ncbi.nlm.nih.gov/research/pubtator-api/publications/export/biocxml?pmcids=PMC6207735
passage_string = """
<passage>
<infon key="section_type">ABSTRACT</infon>
<infon key="type">abstract</infon>
<offset>141</offset>
<text>
Breast cancer is the most frequent tumor in women, and in nearly two-thirds of cases, the tumors express estrogen receptor alpha (ERalpha, encoded by ESR1). Here, we performed whole-exome sequencing of 16 breast cancer tissues classified according to ESR1 expression and 12 samples of whole blood, and detected 310 somatic mutations in cancer tissues with high levels of ESR1 expression. Of the somatic mutations validated by a different deep sequencer, a novel nonsense somatic mutation, c.2830 C>T; p.Gln944*, in transcriptional regulator switch-independent 3 family member A (SIN3A) was detected in breast cancer of a patient. Part of the mutant protein localized in the cytoplasm in contrast to the nuclear localization of ERalpha, and induced a significant increase in ESR1 mRNA. The SIN3A mutation obviously enhanced MCF7 cell proliferation. In tissue sections from the breast cancer patient with the SIN3A c.2830 C>T mutation, cytoplasmic SIN3A localization was detected within the tumor regions where nuclear enlargement was observed. The reduction in SIN3A mRNA correlates with the recurrence of ER-positive breast cancers on Kaplan-Meier plots. These observations reveal that the SIN3A mutation has lost its transcriptional repression function due to its cytoplasmic localization, and that this repression may contribute to the progression of breast cancer.
</text>
<annotation id="38">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="246" length="23"/>
<text>estrogen receptor alpha</text>
</annotation>
<annotation id="39">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="271" length="7"/>
<text>ERalpha</text>
</annotation>
<annotation id="40">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="291" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="41">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="392" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="42">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="512" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="43">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="720" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="44">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="868" length="7"/>
<text>ERalpha</text>
</annotation>
<annotation id="45">
<infon key="identifier">2099</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">47906</infon>
<location offset="915" length="4"/>
<text>ESR1</text>
</annotation>
<annotation id="46">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="930" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="47">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1048" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="48">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1087" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="49">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1201" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="50">
<infon key="identifier">25942</infon>
<infon key="type">Gene</infon>
<infon key="NCBI Homologene">32124</infon>
<location offset="1331" length="5"/>
<text>SIN3A</text>
</annotation>
<annotation id="51">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="185" length="5"/>
<text>women</text>
</annotation>
<annotation id="52">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="762" length="7"/>
<text>patient</text>
</annotation>
<annotation id="53">
<infon key="identifier">9606</infon>
<infon key="type">Species</infon>
<location offset="1031" length="7"/>
<text>patient</text>
</annotation>
<annotation id="54">
<infon key="identifier">29278</infon>
<infon key="type">Species</infon>
<location offset="397" length="10"/>
<text>expression</text>
</annotation>
<annotation id="55">
<infon key="identifier">29278</infon>
<infon key="type">Species</infon>
<location offset="517" length="10"/>
<text>expression</text>
</annotation>
<annotation id="56">
<infon key="identifier">c.2830C>T</infon>
<infon key="type">DNAMutation</infon>
<location offset="1054" length="10"/>
<text>c.2830 C>T</text>
</annotation>
<annotation id="57">
<infon key="identifier">CVCL:0031</infon>
<infon key="type">CellLine</infon>
<location offset="964" length="4"/>
<text>MCF7</text>
</annotation>
<annotation id="58">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1494" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="59">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="346" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="60">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="743" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="61">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1017" length="13"/>
<text>breast cancer</text>
</annotation>
<annotation id="62">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="477" length="6"/>
<text>cancer</text>
</annotation>
<annotation id="63">
<infon key="identifier">p.Q944*</infon>
<infon key="type">ProteinMutation</infon>
<location offset="642" length="9"/>
<text>p.Gln944*</text>
</annotation>
<annotation id="64">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="1130" length="5"/>
<text>tumor</text>
</annotation>
<annotation id="65">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="176" length="5"/>
<text>tumor</text>
</annotation>
<annotation id="66">
<infon key="identifier">c.2830C>T</infon>
<infon key="type">DNAMutation</infon>
<location offset="630" length="10"/>
<text>c.2830 C>T</text>
</annotation>
<annotation id="67">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="1258" length="14"/>
<text>breast cancers</text>
</annotation>
<annotation id="68">
<infon key="identifier">MESH:D009369</infon>
<infon key="type">Disease</infon>
<location offset="231" length="6"/>
<text>tumors</text>
</annotation>
<annotation id="69">
<infon key="identifier">MESH:D001943</infon>
<infon key="type">Disease</infon>
<location offset="141" length="13"/>
<text>Breast cancer</text>
</annotation>
</passage>"""

# turn into an object
passage = ET.fromstring(passage_string)

# these 3 definitions are per-passage
passage_annotations = passage.findall('./annotation')
passage_offset = int(passage.find('offset').text)
passage_text = passage.find('text').text

def get_entity_offset(offset_dict, passage_offset):
    """
    XML given offset_dict gives offset relative to the start of the document
    So subtract the passage offset (where passage starts relative to document beginning)
    """
    start = int(offset_dict['offset']) - passage_offset
    end = int(offset_dict['offset']) + (int(offset_dict['length']) + 1) - passage_offset
    return start, end

# collect entities as a list of tuples of the form
# (start, end, entitiy_type)
passage_entities = []
for ann in passage_annotations:
    entity_type = ann.find('./infon[@key="type"]').text
    od = ann.find('./location').attrib
    start, end = get_entity_offset(od, passage_offset)
    passage_entities.append((start, end, entity_type))

# this is one entry in the spacy NER format
# you would want many entries
spacyd_passage = (passage_text, {"entities": passage_entities})

# prove this worked
for ent in passage_entities:
    print(ent, passage_text[ent[0]:ent[1]])

# prints:
# (105, 129, 'Gene')  estrogen receptor alpha
# (130, 138, 'Gene') (ERalpha
# (150, 155, 'Gene')  ESR1
# (251, 256, 'Gene')  ESR1
# (371, 376, 'Gene')  ESR1
# (579, 585, 'Gene') (SIN3A
# (727, 735, 'Gene')  ERalpha
# (774, 779, 'Gene')  ESR1
# (789, 795, 'Gene')  SIN3A
# (907, 913, 'Gene')  SIN3A
# (946, 952, 'Gene')  SIN3A
# (1060, 1066, 'Gene')  SIN3A
# (1190, 1196, 'Gene')  SIN3A
# (44, 50, 'Species')  women
# (621, 629, 'Species')  patient
# (890, 898, 'Species')  patient
# (256, 267, 'Species')  expression
# (376, 387, 'Species')  expression
# (913, 924, 'DNAMutation')  c.2830 C>T
# (823, 828, 'CellLine')  MCF7
# (1353, 1367, 'Disease')  breast cancer
# (205, 219, 'Disease')  breast cancer
# (602, 616, 'Disease')  breast cancer
# (876, 890, 'Disease')  breast cancer
# (336, 343, 'Disease')  cancer
# (501, 511, 'ProteinMutation')  p.Gln944*
# (989, 995, 'Disease')  tumor
# (35, 41, 'Disease')  tumor
# (489, 500, 'DNAMutation')  c.2830 C>T
# (1117, 1132, 'Disease')  breast cancers
# (90, 97, 'Disease')  tumors
# (0, 14, 'Disease')  Breast cancer

Итак, первое, что я замечаю, это то, что некоторые из заданных смещений немного смещены, захватывая (. Вы можете найти if passage_text[ent[0]] == "(" и сместить начало объекта на 1, чтобы очистить его, или очистить вручную.

Кроме того, этот код использует один дочерний узел, passage связанного документа. Вам нужно будет загрузить этот документ локально, и вместо passage = ET.fromstring(passage_string) вы создадите tree = ET.parse('path_to_file'):

Что-то вроде

import xml.etree.cElementTree as ET

tree = ET.parse('path_to_file')
root = tree.getroot()
passages = root.findall('./passages')

spacy_data = []

for passage in passages:
    passage_annotations = passage.findall('./annotation')
    passage_offset = int(passage.find('offset').text)
    passage_text = passage.find('text').text

    passage_entities = []
    for ann in passage_annotations:
        entity_type = ann.find('./infon[@key="type"]').text
        od = ann.find('./location').attrib
        start, end = get_entity_offset(od, passage_offset)
        passage_entities.append((start, end, entity_type))

        spacyd_passage = (passage_text, {"entities": passage_entities})
        spacy_data.append(spacyd_package)

Это все еще можно улучшить. Вы захотите разделить эти passage.text отрывки, используя

import spacy

nlp = spacy.load('en_core_web_sm')

doc = nlp(passage_text)
sents = list(doc.sents)

Но сложность заключается в том, что вам нужно выполнять арифметические операции, чтобы индексы смещения были правильными. И вы также захотите посмотреть на начало и конец каждого объекта, чтобы убедиться, что он остается в одном предложении - он, вероятно, может быть разделен границей предложения, хотя, вероятно, нет.