Read Me
# Academic Meta-Search MVP Runnable MVP of a deep web search app for academic sources. It performs **live meta-search** across: - arXiv - PubMed - CrossRef - Semantic Scholar - OpenAlex The backend fans out each query to the upstream APIs, normalizes and de-duplicates results, and returns a unified list. The UI is a simple responsive web page optimized for desktop and modern mobile browsers (portrait orientation). ## Tech stack - Backend: Node.js (>= 18), Express - Frontend: Vanilla HTML/CSS/JS, served by Express - Caching: In-memory, short-lived (per-process) No database and no authentication. ## Features - Public search page (no login) - Live meta-search over the 5 major academic APIs - Filter by source and publication year range - Sort by relevance (default) or date (newest first) - Unified normalized results with: - title - authors - year - source - abstract snippet - DOI / external IDs when available - links to full text or landing pages when available - Detail view per result (full abstract & metadata) entirely from the search response ## Running the project ### Prerequisites - Node.js **18 or newer** (uses the built-in `fetch` API) ### Install dependencies ```bash npm install ``` ### Configure environment (optional) Create a `.env` file (or export variables in your shell) to adjust defaults: - `PORT` – server port (default: `3000`) - `SEMANTIC_SCHOLAR_API_KEY` – optional API key to increase Semantic Scholar limits Example `.env`: ```bash PORT=3000 SEMANTIC_SCHOLAR_API_KEY=your_key_here ``` ### Start the server ```bash npm start ``` Then open http://localhost:3000 in a desktop browser. ## Architecture overview - `src/server.js` – Express app & HTTP server - `src/config.js` – central configuration (ports, timeouts, per-source settings) - `src/cache.js` – small in-memory TTL cache for search results - `src/connectors/` – pluggable connectors for each external source - `arxiv.js` - `pubmed.js` - `crossref.js` - `semanticScholar.js` - `openAlex.js` - `src/search/` – normalization, de-duplication, aggregation logic - `aggregate.js` - `normalize.js` - `dedupe.js` - `src/routes/searchRoute.js` – `/api/search` HTTP endpoint - `public/` – static frontend (HTML/CSS/JS) Connectors follow a simple interface: ```js export const id = 'arxiv'; export const displayName = 'arXiv'; export async function search(query, options) { // options: { yearFrom, yearTo, limit, signal } // return an array of normalized result objects (see normalize.js) } ``` The aggregator imports all connectors and runs them in parallel, making it easy to add institutional repositories later. ## Limitations - No persistent storage or indexing; all results are live from upstream APIs with short-lived in-memory caching. - Rate limits from upstream APIs still apply. The code is written to degrade gracefully if a source is temporarily unavailable. - UI is optimized for desktop and modern mobile browsers in portrait orientation; tablet and landscape layouts are best-effort only. ## Security & compliance - Only uses **public, openly accessible APIs** (no scraping of paywalled content). - No user accounts or authentication implemented.
