Web Search Engine

UCI CS 121 (Information Retrieval) · Spring 2025 · Team of four

An inverted-index search engine over about 50,000 UCI ICS web pages, ranked by TF-IDF, PageRank, HITS and title matches

Stack

Python, NLTK (stemming), BeautifulSoup, NetworkX (PageRank), Requests, Streamlit

Goal and outcome

The course provides a corpus of UCI ICS web pages, one JSON file per page with its URL and HTML. The assignment was to build a complete search engine over it, from indexing and retrieval to ranking and a search interface.

The final system parses, tokenizes and stems each page into postings that store the document ID, term frequency, term positions and the HTML tags a term appeared in. The index is written to disk as four partial files by first letter (A–I, J–R, S–Z, other), merged per token, and a bookkeeping file records each token's byte offset so a query can seek straight to its postings. Queries run as Boolean AND over stemmed terms and fall back to N-gram fuzzy matching (capped at 200 ms) when a term has no postings. Results are ranked by a weighted sum of min-max-normalized TF-IDF, PageRank and HITS authority, plus a title-match term. Exact duplicate pages (MD5) are dropped while indexing. A Streamlit UI highlights matched terms, paginates and keeps a viewing history.

We submitted two versions: one indexing the full corpus, and a clean-corpus version that first runs my filter to drop pages that can't be opened or have no title.

My role

Team: Claire Li, Emily Sun, Richard Youn and me. Neither the report nor the README lists individual contributions; “My work” below is based on the commit history of our team repository. Everything else was built by the team together.

My work

  • Index and queries: added term positions to every posting (in preparation for proximity analysis) and stemmed query terms so they match the stemmed index.
  • TF-IDF: moved document-frequency and TF-IDF computation from before each chunk is written to after all chunks are merged per token (Problem 1).
  • Clean corpus: wrote filter_dev_fast.py, which drops pages that can't be opened or have no title before indexing (Problem 2), and gave title matches more weight in the final score.
  • Code and delivery: reorganized the code into app / core / data / utils packages and prepared the final submission.

Built together by the team

  • PageRank (NetworkX) and HITS link analysis; score fusion with min-max normalization; exact-duplicate detection; N-gram fuzzy matching; filtering of untitled pages; the Streamlit search UI.

Technical problems

Problem 1: TF-IDF computed per chunk used the wrong document frequency

Problem
While indexing, the program wrote the in-memory index to disk after each batch of documents and then cleared it, document-frequency (df) counts included. TF-IDF was computed just before each clear, so a term's df only counted documents in the current batch, and idf used the running document count instead of the corpus size. The same token's postings also ended up on several lines across the partial files.
What I did
  • Stop computing TF-IDF during indexing. In the merge step, first concatenate every posting list for a token across the partial files.
  • Then take df as the length of the merged list, compute idf = log(N / df) with N the total number of documents, and weight each posting by (1 + log tf) × idf.
Why this approach
df is a corpus-wide statistic, and the merge step is the first point where a token's complete posting list exists. Computing it there costs no extra pass, because the merge already reads every line.
df = len(postings)
idf = math.log(total_docs / df) if df > 0 else 0.0

for posting in postings:
    tf = posting.get("tf", 1)
    tf_norm = 1 + math.log(tf) if tf > 0 else 0.0
    posting["tfidf"] = tf_norm * idf
Excerpt from utils/merge_jsonl.py: df and TF-IDF are computed only after a token's postings are merged (comments removed).

Problem 2: Pages nobody could open were in the index

Problem
In the first test round, the top result for both “machine learning” and “Software Engineering Course” was an untitled directory listing (maven-contents.txt), and other results were 404 pages or download links. Every file in the corpus was being indexed, whether or not it was a real, reachable web page.
What I did
  • Wrote filter_dev_fast.py to run before indexing. It skips files with invalid JSON, an empty URL or empty content, and HTML with no <html>/<body> or no non-empty <title>.
  • For pages that pass, it sends an HTTP HEAD request to the live URL (following redirects, 10 s timeout) and keeps the page only on status 200, pausing a random 50–150 ms between requests.
  • Surviving pages are copied to DEV_filtered, and the clean-corpus version indexes that folder instead. I also gave title matches more weight in the final score.
Why this approach
Filtering once before indexing keeps every downstream signal (TF-IDF, the link graph behind PageRank and HITS, and the index size) free of pages no user could open, instead of patching results at query time. Requests go out one at a time with a short random pause so the check doesn't hit the university's servers in bursts.

Results and verification

Pages in the full-corpus index
50,281
Pages in the clean-corpus index
8,694
Four standard test queries
2.6–146 ms
  • Standard queries on the final full-corpus engine: “cristina lopes” 58 results in 2.6 ms, “machine learning” 2,622 in 37.0 ms, “ACM” 2,480 in 16.3 ms, “master of software engineering” 993 in 146.4 ms.
  • How it was evaluated: the team wrote 20 test queries (10 that should work well and 10 hard ones: too broad, misspelled, stopwords only, nonexistent terms), ran them in three rounds, and recorded result counts, times and the top results.
  • Round 1 exposed untitled file listings, duplicates and dead links near the top. Round 3 reran all 20 queries on the clean corpus from my filter, with the higher title weight.
  • Both page counts come from each version's url_to_docid.json in the submission (exact duplicates already removed).

Limitations

  • The HEAD check only reflects which links were alive when the filter ran (June 2025), and any non-200 status drops a page, including servers that reject HEAD requests. The clean index has 8,694 pages versus 50,281 in the full index.
  • Filter requests run one at a time with a pause between them, so a pass over the whole corpus takes a long time.
  • Ranking weights (α, β, γ and the title weight) were tuned by hand on the test queries; there are no labeled relevance judgments to measure precision.
  • Broad and stopword-only queries remain the slowest (“the and or but”: 287 ms on the full corpus in round 2).
  • Tried but not in the final version: a proximity score and SimHash near-duplicate detection were both implemented, then commented out during testing.

Evidence

The code lives in our team's private repository and isn't published, because this is a graded course assignment.