Prose Similarities

The question

Every novel in English shares most of its vocabulary with every other novel in English. Said, man, little, know. That common layer is the bulk of the text and it tells you nothing, precisely because everyone uses it.

So throw it away. What is left after you subtract the words everybody uses is the vocabulary that actually distinguishes one writer from another — and two books that still overlap after that subtraction have something real in common.

How it works

1. POS-tag every text with TextBlob and keep only content words — nouns, verbs, adjectives, adverbs. Determiners and prepositions are structure, not voice.
2. Rank what survives by frequency, per text.
3. Subtract the vocabulary shared by every text in the set. This is the whole idea, and it is an intersection across all pairs, not a stop-word list — so the set of words thrown away is decided by the corpus rather than by a dictionary someone else wrote.
4. Keep each text's top N of what remains. Those are its distinctive words.
5. Edge weight is the overlap between each pair of texts. Every novel is a node; every pair is an edge; the weight is how much distinctive vocabulary they still share.
6. Lay it out with networkx, and run Dijkstra between any two writers.

def remove_all_overlaps(graph_edges, novel_most_used):
    all_overlap = None
    for edge in graph_edges:
        set_1 = set(novel_most_used[edge[0]].keys())
        set_2 = set(novel_most_used[edge[1]].keys())
        curr_overlap = set_1 & set_2

        if all_overlap is None:
            all_overlap = curr_overlap
        else:
            all_overlap = all_overlap & curr_overlap

    for novel in novel_most_used:
        for i in list(all_overlap):
            novel_most_used[novel].pop(i)

Read the two loops as one sentence: the first builds the set of words present in every book by intersecting pair after pair; the second deletes that set from all of them. A word has to be missing from at least one novel to survive. That is a much harsher filter than a stop-word list, and it is corpus-relative — run it on four Fitzgerald novels and it strips Fitzgerald's own habits, which is what makes within-author comparison work at all.

The part worth keeping

Because the edges carry weight, you can ask for the route between two writers who share almost nothing directly — and get back the chain of intermediate books that connects them. The graph doesn't just say who is near whom; it says how you would get from one to the other.

Also in here

The same file carries the fix for Novel Sentiment Analysis: sentiment_by_parts(novel, scale, jump) scores overlapping windows rather than paragraphs — scale sets the window, jump the stride. Same book, same library, readable curve. And sentiment_multiple_novels normalises several books to the same number of parts so they can share an axis.

Where it went

Two of the choices here are the ones Literature Mutations went back and replaced. The hard intersection became TF-IDF with a document-frequency cut (drop vocabulary common to more than 40% of books), which is the same instinct with a dial on it; and the all-pairs edge set became k-nearest-neighbours, because over long English prose a global similarity threshold gives you one undifferentiated blob.

Prose-Similarities on GitHub ↗