Managing rickyraymond.dev Chatbot

My explorations with LLMS.

This post is part of an ongoing series where I dive into my research and experimentation with large language models (LLMs), particularly in the context of the chatbot running on my site. The chat bot is currently powered by gpt-4o from OpenAI, which is a strong choice for a low-cost assistant with state-of-the-art capabilities.

At the moment, the chatbot’s routing logic simply appends my resume and some supplementary information to the system prompt. It uses function calling to retrieve these blog posts and other relevant content. This gives the bot a degree of contextual awareness when responding to messages. I will be extending its capabilities with more tools over time.

Recently, I completed a GCP bootcamp focused on LLM deployment, which introduced me to several libraries I have not implemented yet but plan to explore soon. One key concern I am addressing is availability. Many LLMs now share compatible APIs, and libraries exist for load balancing and routing between models. This opens the door for a more dynamic backend where model selection could be based on message classification or availability rather than relying on a single provider.

A shifting model backend raises questions about observability. Built-in logging and analytics from model providers are not sufficient for deep QA. I plan to introduce observability tooling to evaluate responses more closely, with the goal of fine-tuning smaller models in the future for increased accuracy and performance.

On the protective side, I sanitize incoming messages, implement Google CAPTCHA, and use a lightweight classifier to ensure questions are relevant and to avoid abuse, such as farming tokens. More security layers are on the roadmap to safeguard my site, the bot, and my OpenAI budget.

This blog serves as a snapshot of version 0.1 of the chatbot and a foundation for ongoing development. I have learned a great deal about what LLMs can do, and I am looking forward to showcasing their agentic potential directly on my site.