I want to train the word2vec model on english wikipedia using python with gensim. I watched closely https://groups.google.com/forum/#!topic/gensim/MJWrDw_IvXw for this.
This works for me, but what I don't like about the resulting word2vec model is that named objects are separated, which makes the model unsuitable for my particular application. The model I need should represent named objects as a single vector.
That's why I planned to analyze Wikipedia articles with open spaces and combine entities such as North Carolina into North Carolina so that word2vec represents them as one vector. So far so good.
Spatial analysis should be part of the pre-processing that I initially did, as recommended in the related discussion, using:
...
wiki = WikiCorpus(wiki_bz2_file, dictionary={})
for text in wiki.get_texts():
article = " ".join(text) + "\n"
output.write(article)
...
This removes punctuation, stop words, numbers and capital letters and saves each article on a separate line in the final output file. The problem is that the spacious NER doesn't actually work with this pre-processed text, as I believe it is based on punctuation and capitalization for NER (?).
Does anyone know if I can "disable" gensim preprocessing so that it does not remove punctuation, etc., But does it analyze Wikipedia articles in text directly from a compressed Wikipedia dump? Or does anyone know a better way to do this? Thanks in advance!