A six-node cluster for live social sentiment
Streaming Mastodon and Bluesky data through serverless functions and local LLMs into live Kibana dashboards, on Kubernetes across Melbourne Research Cloud.
- Client
- Cloud & Cluster Computing, University of Melbourne
- Role
- Infrastructure and data pipeline
- Timeline
- Mar 2025 – Jun 2025
- Status
- Delivered
The build
I provisioned and managed a scalable Kubernetes cluster across six Melbourne Research Cloud VMs to handle distributed workloads and keep uptime high under continuous ingestion.
On top of it, resilient serverless functions on Fission stream data from Mastodon and Bluesky, extract and label sentiment, and index the results into Elasticsearch for live analysis in Kibana.
Local models next to hosted ones
The pipeline runs a local Gemma 2B alongside hosted inference through Groq. Splitting the work that way keeps cost and latency controllable: the cluster handles volume locally, and the API handles the cases where quality matters more than throughput.
It is the same trade-off that shows up in every production LLM system I have built since — how much can you serve yourself before paying per token becomes the bottleneck.
Why it stuck with me
Social streams do not stop when a node does. Building for continuous ingest meant treating failure as normal rather than exceptional, and it is the clearest example I have of operating infrastructure rather than just deploying to it.
Stack
Need something like this built?
I take on freelance briefs and part-time engineering work from Melbourne, Australia.