To create an API for fetching the latest videos from YouTube, sorted in reverse chronological order of their publishing date-time, for the search query "F1". The API should provide paginated responses.
- Programming Language: Go (Golang)
- Web Framework: GoFiber
- Database: PostgreSQL
- ☑️ Server continuously calls the YouTube API asynchronously with a 10-second interval to fetch and store the latest videos for the search query "F1". The video data, including fields like video title, description, publishing datetime, thumbnail URLs, and more, is stored in a database with proper indexes.
- ☑️ A GET API returns stored video data in a paginated response, sorted in descending order of published datetime.
- ☑️ The solution should be scalable and optimized for performance.
- ☑️ Add support for multiple API keys to automatically switch to the next available key when the quota is exhausted.
- ☑️ Create a dashboard to view stored videos with filter and sorting options (optional).
-The backend of this project is deployed on an AWS EC2 instance. The deployment ensures that the API is accessible and functional in a production environment.
-To enhance scalability, the Docker Compose configuration can be adjusted to increase the number of replicas in the web service. This helps in balancing the load and ensuring efficient handling of incoming requests.
https://documenter.getpostman.com/view/16178117/2s9Y5R37FY
- To build and run
docker-compose up -d --buildIn this project, the PostgreSQL database is optimized for effective text-based searches using tsvector and the GIN index.
-
tsvector Data Type: The
tsvectordata type processes text, transforming words into normalized lexemes. It enables efficient searches by generating processed text representations. -
GIN Index: The
GINindex is a specialized index for full-text searches. It's applied to thetsvectorcolumn, allowing PostgreSQL to swiftly retrieve matching documents.
In our implementation:
- Video titles and descriptions are processed using
to_tsvectorand stored in thesearch_weightscolumn. - User queries are converted into
tsqueryusingplainto_tsquery. - Searches are conducted using the
@@operator on the indexedtsvector. - The
search__weights_idxGIN index optimizes search performance.
These techniques ensure rapid and precise search results based on title and description content.