The course provides a corpus of UCI ICS web pages, one JSON file per page with its URL and HTML. The assignment was to build a complete search engine over it, from indexing and retrieval to ranking and a search interface.
The final system parses, tokenizes and stems each page into postings that store the document ID, term frequency, term positions and the HTML tags a term appeared in. The index is written to disk as four partial files by first letter (A–I, J–R, S–Z, other), merged per token, and a bookkeeping file records each token's byte offset so a query can seek straight to its postings. Queries run as Boolean AND over stemmed terms and fall back to N-gram fuzzy matching (capped at 200 ms) when a term has no postings. Results are ranked by a weighted sum of min-max-normalized TF-IDF, PageRank and HITS authority, plus a title-match term. Exact duplicate pages (MD5) are dropped while indexing. A Streamlit UI highlights matched terms, paginates and keeps a viewing history.
We submitted two versions: one indexing the full corpus, and a clean-corpus version that first runs my filter to drop pages that can't be opened or have no title.