College Essay Prompts API
Send a college's name, get its essay prompts back as clean JSON. A FastAPI service scraped each school once, cached it in MongoDB, and served it to developers behind API keys. A small Next.js dashboard called EssayDash handled key signups and usage.
Three months from a script to 500 accounts
The whole thing ran from August to November 2024. The first prototype is in a private repo; everything after it is public.
-
Aug 13, 2024Scraper prototype
A 110-line Selenium and BeautifulSoup script that walked a public list of colleges, opened each school's prompt page, and wrote everything to a CSV. No API yet.
-
Sep 2, 2024Turned it into an API
Wrapped the scraper in FastAPI with two POST endpoints, one for a single school and one for a comma-separated list. MongoDB sits in front of the scraper as a cache, so each school only gets scraped the first time someone asks for it.
-
Sep 10, 2024Deployed to Azure
Set up GitHub Actions to build and deploy to Azure App Service, and pinned the dependencies. Getting the deploy working took four commits that day.
-
Oct 18, 2024API keys and rate limits
Added an
x-api-keyheader checked against a MongoDB collection of issued keys, plus SlowAPI rate limiting at five requests a minute per IP address. Also added a startup script that activates the virtualenv and starts uvicorn on Azure. -
Nov 3, 2024Last deployment change
The final update to the Azure workflow, and the end date on my résumé. I shut the API down after this because collecting the data was costing more than the project was worth.
-
Jun 12–14, 2026Archived for the portfolio
Rewrote the README and published the EssayDash dashboard as its own repo.
Everyone wanted the prompts as data, and nobody had them that way
Counselors, advising firms, and student apps all want college essay prompts in a structured form they can query. The prompts change every application cycle and live on web pages in inconsistent formats, so every tool that needed them ended up writing and maintaining its own scraper against the same pages.
I figured one shared service could do that scraping once and hand everyone the same clean JSON.
Scrape on the first request, serve from cache after that
A request comes in with a school name and an API key. The service checks the key, then looks for that school in MongoDB. On a hit it returns immediately. On a miss it launches headless Chrome, searches for the school, opens its prompt page, parses each prompt card with BeautifulSoup (title, required or optional, word count, text, and any sub-options), stores the result, and returns it.
5 req/min limit
→
Each school is scraped once. Every request after that comes straight from MongoDB.
def get_prompts_for_college(college_name): cached_data = collection.find_one({"college_name": college_name}) if cached_data: return {"college_name": college_name, "prompts": cached_data['prompts']} driver = setup_driver() try: scraped_college_name, prompts = fetch_essay_prompts_from_selenium(driver, college_name) if not prompts: return None collection.insert_one({"college_name": scraped_college_name, "prompts": prompts}) return {"college_name": scraped_college_name, "prompts": prompts} finally: driver.quit() @app.post("/get_essay_prompts/", dependencies=[Depends(verify_api_key), Depends(limiter.limit("5/minute"))]) def get_essay_prompts(request: CollegeRequest): ...
From main.py, with logging and a duplicate-name check removed.
The part developers used directly
An API with 500 accounts needs somewhere to hand out keys. EssayDash is a Next.js 14 dashboard where an organization signs in with Firebase Auth, creates an API key (stored in MongoDB), checks its access status, and sees recent requests and monthly usage on Recharts graphs. It also has a browser for the list of supported college names, so people could check coverage before writing any code.
Key creation per organization, lookup of existing orgs, usage charts, and the supported-colleges list. Database credentials stayed server-side in a Next.js API route.
Hash keys at rest and show each one only once, protect the admin-level lookups, and feed the charts from real request logs.
500 accounts, then a deliberate shutdown
The API reached 500+ developer accounts, with response times between 269 and 1,006 ms depending on how many schools a request asked for. Then I shut it down. Keeping the data fresh meant scraping an outside site every cycle, and that cost was growing faster than the project's value.
If I picked it back up, I'd add tests around the parser that run against a saved copy of the page, so a markup change upstream breaks a test right away instead of returning empty results. I'd also upgrade FastAPI and Pydantic so it runs on current Python.
The API and the dashboard.