Scalable Hashtag Recommender
Distributed image-based hashtag recommendation pipeline using CNN features, clustering, Apache Spark, and AWS EMR.
A university project on distributed machine learning: given an image, recommend likely hashtags by combining pretrained visual features, clustering, and a Spark-based execution pipeline.
Approach
- Feature extraction: images are passed through a pretrained CNN to obtain high-level feature vectors rather than relying on hand-written visual descriptors.
- Clustering: those vectors are grouped with k-means and mini-batch k-means so visually similar images land in the same region of the feature space.
- Recommendation: for a new image, the system finds the nearest cluster and suggests hashtags based on the tags most associated with that cluster.
Scalability
The part I liked most was the systems side. The pipeline was built around Apache Spark and deployed on AWS EMR, with Flintrock-based cluster setup and batch-oriented training / inference scripts. That’s what turned it from a notebook experiment into a small distributed ML system.
There is also a later experimental branch in the HashtagRecommenderAIVersion repo, which captures follow-up work and rougher iterations. I treat that as an extension of the original academic prototype rather than a separate polished product.
This project is best described as a distributed content-based recommendation experiment: computer vision, clustering, and cloud execution brought together to make the pipeline workable at larger scale.