Richard Song from Epsilla
In this episode, Richard from Epsilla discusses the journey of building AI applications with the hosts. He shares his background in database and infrastructure, and how the emergence of conversational AI inspired him to start Epsilla. Richard explains the importance of marrying large language models (LLMs) with data and introduces the concept of a vector database as a missing piece in the AI space. He discusses Epsilla's platform for building production-ready LLM applications and the value it brings to citizen builders and enterprises. Richard also shares his vision for the future of AI and offers advice for newcomers in the field.
Takeaways
- Marrying large language models (LLMs) with data is a key opportunity in the AI space.
- Epsilla provides a platform for building production-ready LLM applications connected with existing data.
- The target users of Epsilla are citizen builders and enterprises, who can benefit from the platform's ease of use and scalability.
- The future of AI involves higher levels of abstraction, AI-native compilers, and AI operating systems.
Episode Transcript
Introduction and Background
Rod: Welcome to a new episode of our show. I'm your host Rod, joined by co-host Max. Today, we're excited to have Richard from Epsilla with us. Welcome, Richard!
Richard: Thank you for having me, Rod and Max. I'm thrilled to be here.
Rod: Before we dive in, could you tell us how you got started in the AI engineering space?
Richard: Certainly. Prior to founding Epsilla, I spent eight years at TigerGraph, a graph database startup. I'm essentially a database and infrastructure specialist. When ChatGPT was released in late 2022, it completely blew my mind. I had imagined this level of human-like conversational AI wouldn't emerge until around 2045, near the technological singularity. I never expected it to happen so soon.
However, I quickly realized ChatGPT's limitations. It lacked access to the latest knowledge and couldn't handle private data from individuals or companies. This made it challenging to use large-scale models for specific business use cases. Given our enterprise database background, we saw a huge opportunity to combine Large Language Models (LLMs) with data.
The Birth of Epsilla
Richard: We discovered vector database technology in the market, which proved to be the missing link between LLMs and data. It allows for augmenting large language models with the most semantically relevant information. We now call this RAG (Retrieval-Augmented Generation), though the term didn't exist back then.
The secret sauce to making vector databases perform well is a technology called graph indexing. Given our background in graph technology, there was a natural founder-market fit. That's how we started Epsilla as a high-performance, open-source vector database company.
We joined Y Combinator in the summer of 2023. Despite being early in the game, we developed technology that showed a tenfold improvement in quality-performance compared to other leaders in the field. We were also the first in the industry to build a fully serverless vector database service, launching three months earlier than PineCone's serverless offering.
Challenges and Opportunities
Richard: Initially, like other database companies, we focused on top-down enterprise sales. As you know, this approach takes time, especially in a crowded market with many options. We realized that for large enterprises, while performance and scalability are key decision-making factors when choosing a database, these only become critical when moving from prototype to production systems.
As of today, most companies are still in the early stages of adoption. Scalability and performance aren't yet bottlenecks. It can take months for companies to transition from prototype to production. Why is this so challenging? There are numerous tools in the market helping companies build quick prototypes, and plenty of online tutorials for creating basic RAG systems. However, there's a lack of resources on how to evolve from prototype to production-ready RAG systems.
We saw this as a significant opportunity. We decided to build upon our core vector database technology and move up the value chain, providing a RAG-assist service platform to accelerate this process. That's how we entered the AI space.
Y Combinator Experience
Rod: Wow, that's a lot to digest. Taking a step back, you mentioned you were recently part of a Y Combinator cohort, which is notoriously competitive. How did you get in? What do you think Y Combinator liked about Epsilla, especially given the crowded market?
Richard: I believe it was a combination of luck and our background. Coming from the database industry, we had a unique perspective. We believe that our deep understanding of database technology, combined with our vision for its application in the AI space, resonated with Y Combinator.
Philosophy and Approach
Max: That's fascinating. I noticed something interesting on your LinkedIn profile. You wrote, "Technology is the skeleton, art is the flesh and blood, and business is the soul." Could you expand on that? How does this philosophy translate into your current business?
Richard: Absolutely. As engineers transitioning to entrepreneurship, it's a common mistake to start with brilliant technology and then try to find a problem it can solve. I believe this approach is backwards. We should always start by identifying a real, existing problem and then work backwards to determine the best technology to solve it.
This perspective shift is crucial. Instead of spending time building a "perfect" product before approaching customers, we assume the product already exists. We talk to our target audience and ask, "If we had this product, how would you use it? Is this solving a critical pain point for you, or is it just a 'nice-to-have'?"
This "fake it till you make it" mindset allows us to use customer feedback to shape the product. It's often surprising to discover that our original idea might not be addressing a real problem, but customer conversations can guide us towards solving genuine issues. This is the biggest mindset change when transitioning from engineer to business owner: build something people actually want.
Epsilla's Product Tiers
Rod: Looking at your pricing page, I see three tiers or product types: self-hosted (free, community, open-source), cloud solution, and Enterprise. These seem to target very different profiles. How do you manage addressing these diverse needs?
Richard: These three tiers are complementary to each other. The open-source offering is the core of our vector database, serving as a foundation for all different user personas. The cloud service abstracts away infrastructure management, making it super easy for users to start and see value quickly. The enterprise tier builds on top of the cloud and open-source offerings, providing additional enterprise-level features.
While the core problem we're solving remains the same – making GenAI application building super fast – for enterprises, we add features like enhanced data security, compliance, authentication, authorization, and data governance. These are additional layers on top of our existing offerings, creating a tiered structure that caters to different needs.
The Concept of RAG as a Service
Rod: I notice you call it "RAG as a Service." Is this already a widely understood concept, or is it something new you're introducing?
Richard: The "as a Service" pattern is widely used by many companies, and we're seeing more businesses adopt this terminology. We believe it's a trend, and we're excited to see many innovative players tackling this problem from different angles. There's a huge market, and as they say, "a rising tide lifts all boats." We believe that in the near future, together with all these players, we'll make something significant in this market.
Future Vision for Epsilla
Rod: Looking ahead, where do you see Epsilla in the future?
Richard: In the next few years, we envision Epsilla becoming the Salesforce for GenAI. We aim to build a platform-as-a-service for GenAI applications, empowering builders from different industries to democratize the value of GenAI and use AI to accelerate business value. That's our vision for the coming years.
Advice for Newcomers in AI
Rod: You've been in this space for a long time. What advice would you give to those just starting their careers or graduating, considering the current wave of generative AI?
Richard: Regardless of new trends, GenAI is already transforming our work and life. It's hard to imagine anyone not using GenAI in their daily activities. Start with simple use cases, like using AI to improve communication. As a non-native English speaker myself, I use AI to polish emails, messages, and documentation. This doesn't require significant technological investment but can greatly improve work outcomes.
Use AI to make your code efficiency 10 times higher. Here's a secret: more than 50% of Epsilla's codebase is either written by AI or polished by AI, including our vector database engine. We believe there's still huge potential to further improve AI integration in our day-to-day work.
No matter what industry you're in, start using AI to help you work faster. Automate ad-hoc tasks like Python scripts, runtime data analytics, and generating plots and charts. You don't need to write these yourself anymore. Use state-of-the-art models like GPT-4; they can do a great job. This allows you to focus on the most interesting and significant aspects of your work and learn really fast. This advice applies to anyone building a career in the tech industry.
The Future of AI in Daily Life
Rod: If you had a super AI, Richard, what would you ask it to do for you?
Richard: There are so many possibilities! For example, I'd like an AI assistant to remind me of my daily agenda, highlighting things that need attention, and perhaps engaging in a conversation to see what needs adjusting. Later, when the time comes, the agent could notify me of necessary actions or help process tedious work behind the scenes.
When I'm about to leave, AI could inform me when the next train is coming and help me avoid heavy traffic. There are countless ways AI could improve our daily lives. The opportunities are endless.
Closing Thoughts
Rod: Is there any final message you'd like to leave our audience with?
Richard: We're in an incredibly exciting period. While I wouldn't say it's once in a lifetime – because technological iterations are accelerating and the singularity is approaching – it's definitely a unique time. There are endless opportunities enabled by large language models and GenAI.
However, it's still early days, and the GenAI landscape is rapidly evolving across all layers – infrastructure, dev tools, and applications. Most developments are still in the prototype phase. We'd love to work with AI engineers and data scientists to figure out how to elevate these prototypes into production-grade GenAI outcomes and deliver real business value. It's going to be a very rewarding journey.
Rod: If someone wants to reach out, where can they find you?
Richard: You can reach us through our website at Epsilla.com. Our GitHub is epsilon-cloud, we have a LinkedIn page under Epsilla, and our Twitter handle is @epsilon_inc. We also have a Discord channel with a link on our website.
Rod: Thank you so much for being here today, Richard. It's been a great discussion. For everyone listening, check out Epsilla.com. As companies start building RAG applications and chatbots with data, they're realizing that while it seems easy in theory, it's much more complicated in practice. Having a plug-and-play service that allows you to start building robust applications with everything ready to go is incredibly beneficial for many companies. I hope some of them will start using Epsilla soon. Thank you for being here today, Richard.
Richard: Thank you all, and thanks for having me here.