Multithreaded File Indexing and Search Engine

A high-performance search engine built in C++17 that indexes text files and ranks search results using TF-IDF scoring algorithm.

🚀 Features

• Inverted Index: Efficient word-to-document mapping data structure
• TF-IDF Ranking: Statistical ranking algorithm for result relevance
• Text Tokenization: Intelligent word extraction with stop-word filtering
• File Processing: Automatic discovery and indexing of text files
• Interactive Search: Real-time query interface with sub-millisecond response
• Performance Metrics: Detailed timing and throughput statistics

📊 Technical Highlights

• Language: C++17
• Data Structures: Hash maps (unordered_map), sets, vectors
• Algorithms: TF-IDF scoring, tokenization, ranking
• Time Complexity: O(N×M) indexing, O(K×D) search
• Space Complexity: O(V×D) where V=vocabulary, D=documents

🏗️ Architecture

┌─────────────────────────────────────┐
│         Search Engine               │
├─────────────────────────────────────┤
│                                     │
│  File Crawler → Tokenizer          │
│       ↓             ↓               │
│  Inverted Index ← Indexer          │
│       ↓                             │
│  Searcher (TF-IDF Ranker)          │
│                                     │
└─────────────────────────────────────┘

📁 Project Structure

search-engine/
├── include/
│   ├── tokenizer.h          # Text tokenization & preprocessing
│   ├── inverted_index.h     # Core index data structure
│   ├── file_crawler.h       # File discovery & reading
│   └── searcher.h           # Search & ranking logic
├── src/
│   └── main.cpp             # Main application entry point
├── data/                    # Text documents to index
├── README.md
└── search_engine.exe      #Compiledexecutable

### Compilation
```bash
# Windows (MinGW)
g++ -std=c++17 -Wall -Iinclude src\main.cpp -o search_engine.exe

# Linux/Mac
g++ -std=c++17 -Wall -Iinclude src/main.cpp -o search_engine

🎮 Usage

1. Prepare Documents

Create text files in the data/ directory:

data/
├── ai.txt
├── programming.txt
└── databases.txt

2. Run the Search Engine

.\search_engine.exe

3. Search Interface

Search> machine learning
Query: "machine learning"
Found 2 document(s) in 245 μs

=== Top 2 Results ===

1. ai.txt
   Score: 1.2164
   Matches: 3 term occurrences
   Terms found: [learning:2] [machine:1]

2. programming.txt
   Score: 0.8109
   Matches: 2 term occurrences
   Terms found: [learning:1] [machine:1]

   ....


   ## 📈 Performance

**Sample Dataset**: 3 documents, ~65 tokens

- **Indexing Time**: 5-10 ms
- **Search Latency**: < 1 ms (sub-millisecond)
- **Throughput**: ~10,000+ tokens/sec
- **Memory Usage**: Minimal (< 5 MB for small datasets)

## 🧮 TF-IDF Algorithm

### Term Frequency (TF)
Number of times a term appears in a document.

### Inverse Document Frequency (IDF)

IDF(term) = log(Total Documents / Documents Containing Term)

Final Score

TF-IDF = TF × IDF

Why it works: Balances term frequency with rarity—frequent but unique terms rank highest.

🎯 Key Implementation Details

Inverted Index Structure

unordered_map> index;

struct DocumentInfo {
    string doc_path;
    int frequency;
    vector positions;
};

### Tokenization Pipeline
1. Convert text to lowercase
2. Extract alphanumeric tokens
3. Remove stop words ("the", "a", "is", etc.)
4. Store positions for phrase search (future enhancement)

### Ranking Algorithm
1. Tokenize query
2. Calculate TF-IDF for each term in matching documents
3. Aggregate scores per document
4. Sort by total score (descending)

**Built with ❤️ in C++**