AI Agent

Evolving from MVP to AI Agent: Steps for Implementation

I built an end-to-end automated workflow through AI agent design and a new ecosystem.

95%
System Integration to Leverage Data
+30%
Information Accuracy
3X
Traffic Handling Capacity
Evolving from MVP to AI Agent: Steps for Implementation

Background

- The leasing process became increasingly complex, incorporating multiple features and requiring more data to support decision-making.
- There was significant ambiguity as LLMs were new to me, the team, and the traditional rental and real estate industry. I had to lead the team in exploring and developing solutions incrementally.

Pain Point

- Pre-Designed CRM and Database: Existing CRM systems and databases were designed for various human workflows and were not tailored to integrate AI features effectively.
- Limitations of AI Integration: Even with AI features in place, certain functions cannot be fully automated, making it essential to ensure a smooth transition and seamless switching between AI-driven processes and manual workflows.
- System Reliability: Previous system outages posed challenges, and as more features are added, it’s crucial to implement robust measures to prevent future disruptions.

Goal & Achievement

- System Integration: Achieved 95% integration with existing systems to leverage proprietary data
- Accuracy Improvement: Enhanced information accuracy by 30%
- Traffic Capacity: Increased traffic-handling capacity by 3x

Product Planning & Solution

Stage 1: MVP

I built a MVP to test out the concept of AI leasing assistant using only one single prompt to chat with potential customers about a brand-new property, monitored in real-time by humans. While this approach was not scalable, it confirmed that timely responses significantly increased engagement. Customers often had numerous options when property hunting, and responding promptly helped stand out and refresh their memory.

Stage 2: State Machine
technical structure
visualization log to show the generation process

- High-Level System Design with State Machine
1. Horizontally, the system was centered around a state machine with additional modules, as shown in the technical structure above, tailored to business needs such as preference collection and recommendation, with each state having its own conditions and actions.
2. Vertically, each response generated by the LLM would be processed through trigger categorization, intention parsing, status checking, and final actions and validations, as shown in the visualization log above.

- Connected with Internal Systems via APIs
As a significant amount of customer and property data was required, and given that internal systems predated the AI initiative, I collaborated with upstream and downstream teams to revamp customer status, multiple APIs, and configs in CRM and property database to ensure compatibility.

- Developed Knowledge Base (RAG)
Recognizing that many customer inquiries involve specific leasing policies beyond basic property details (e.g., "Are birds like parrots allowed as pets?"), I proposed creating a dedicated knowledge base with RAG. This centralized repository would act as a valuable resource for the AI leasing assistant, enabling it to provide more accurate and comprehensive responses to questions and enhance the overall customer experience.

- Fine-Tuned LLM
Since some tasks, such as parsing the budget, were very specific and straightforward, I collaborated with the data team to fine-tune models, thereby reducing costs and improving their adaptability to edge cases.

- Solved LLM Mathematical Restriction
LLM didn’t specialized in numbers and math, which always calculated the wrong rent with promotions. Usually a team can train the model or combine LLMs with specialized computational tools or algorithms that are designed for mathematical operations. I implemented 3 ways to resolve the issue in a relative lite way:
1. Designed prompt engineering with more specific rules, improved formatting, and simplified promotion details to ensure clarity.
2. Utilized APIs to calculate actual prices and insert accurate numbers.
3. Leveraged hardcoded messages to provide examples with additional context for the model.

Stage 3: AI Agent
AI agent design based on Lilian Weng's work

As the project and leasing flow became more complex, the rule-based state machine struggled to handle certain cases effectively. To address this, I led the team in exploring AI agent design, incorporating LLMs, tool use, planning, and memory components to create a more AI-native solution with multi-modality, greater scalability and extensibility. For example, I replaced the traditional programming logic with LLM-controlled function calls and query/NL2SQL methods to execute actions.

- Planning
1. Routing as the foundation of the control flow, because the major stages of leasing flow is very distinct and straightforward, such as preference collection, recommendation, tour booking and rental application assistance.
2. Still need to add reflection and error correction modules.

- Memory
1. Short-term memory for the current session's interaction, such as the customer's last question and the property currently being discussed.
2. Long-term memory for information that can be leveraged across sessions, such as previous chats and customer tour notes from weeks ago.

- Tool Use
Limited tools will be used in the different stages of the leasing flow, and here are some examples:
1. Read-only: API to check human agents' available time slots; API to query the database for the price of a unit; API to review the knowledge base for amenities of a property.
2. Write: API to update CRM and Google calendar to book or reschedule tours; API to modify the info in the rental application form.

(multi-agent orchestration) Another way to think about it is as advanced tools or multi-functional agents. Each tool is compound and comprehensive, such as recommenders or booking assistants, incorporating multiple steps. For example, a recommender can identify user requirements, translate them into queries or embeddings, retrieve relevant results, rank them, return recommendations in the required format, and perform self-checks.

Learning

  1. Future Vision Drives Present Actions
    What I aim to build in the future shapes what I choose to do today. Envisioning the future of the product is always valuable, especially when working on AI products that are rich with possibilities for imagination and innovation.
  2. Data as the Core Asset in AI Startups
    Data is the most valuable asset for AI application startups that rely on existing models: In my case at the rental startup, property data forms the foundation of the applications, while chat history helps improve AI performance.
  3. Tailoring AI Design to Industry Needs
    The focus of AI agent design varies across industries and use cases: For example, the chat feature for dasher support at DoorDash required Retrieval-Augmented Generation (RAG) with extensive additional documents and scenarios. In contrast, for the AI leasing assistant I worked on, the information was shorter, simpler, and more distinct, with less interdependence between the data, relying more on memory rather than RAG.
  4. I also wrote an article to summarize my other learnings here!
    1. Problem comes before the solution
    2. Find the most suitable use case to maximize the value of AI
    3. Always maintain an ecosystem mindset
Back to Homepage