AI Orchestration Meetup [April 2024] - Recap

With the emergence and fast-growing pace of Generative Artificial Intelligence (GenAI) technologies impacting industries from entertainment to healthcare, recent research suggests that the AI orchestration market size is projected to reach approximately $35 billion by 2031. This growth highlights the increasing need for advanced AI systems capable of effectively handling various AI algorithms, data streams, and workflows. This, in turn, is driving significant progress in different sectors worldwide, marking a transformative shift in the global economy.

So, what exactly is AI orchestration? AI orchestration is the process of managing the interaction, deployment, and integration of various AI models within a workflow or system. These AI models add a human touch to the system when it comes to making complicated decisions. It can help rebalance IT/LOB (Line of business) in decision-making roles, reorganize operational systems, redefine customer acquisition and sales processes, and more.

Yet, with so many AI models entering the market, how can we effectively utilize these models in our applications? Integrating the LLM models and prompts into the application as code by conventional methods eventually creates new monolithic modules that are difficult to deploy, debug, and maintain. However, AI orchestration decouples models, prompts, and vector databases to bring a customized RAG (Retrieval Augmented Generation) framework that can be set up in minutes.

How do you seamlessly integrate AI orchestration into your application? That’s where Orkes, the leading workflow orchestration platform, comes into play. Recently, we organized a developer meet-up to share more about AI orchestration and how to leverage it while building distributed applications. This blog recaps the key takeaways from the meetup and then covers some basic concepts on GenAI, AI orchestration, and its implementation with Orkes Conductor.

Recap from AI Orchestration Bengaluru Meetup

In collaboration with Google for Developers, the Orkes DevRel team hosted the meetup in Bengaluru, India. The event attracted attendees from diverse backgrounds, including software engineering, system administrators, AI research, backend engineering, data science, and more. They represented companies like ANZ, Accenture, HP, IBM, Dell, Wipro, Amazon, Slice, and Gojek.

The meetup featured 3 sessions:

1. Manan Bhatt (Founding Engineer at Orkes) delivered a session on seamlessly integrating Google Vertex AI with Orkes Conductor to build RAG frameworks.
2. Riza Farheen (Developer Advocate at Orkes) demonstrated the power of Orkes Conductor's GenAI capabilities through a demo on automating document classification.
3. Nikhil Rana (Cloud Consultant at Google Cloud) explored the latest updates from Google Gemini, emphasizing its powerful models and features.

A big thank you to Sangeeta Gupta, Community Programs Manager at Orkes, and Srividhya S Subramanian, Solution Architect at Orkes, for their efforts in hosting and networking with attendees.

One of the attendees from the meetup, Prasoon Kumar, quoted:

And here’s what our community shared about the meetup:

Now, let’s look into some of the basic GenAI and AI Orchestration concepts covered in the meetup.

Understanding AI Orchestration

Before delving into the specifics, let's grasp the significance of AI.

Do you recall the internet buzzing with excitement when ChatGPT was announced in November 2022? During that period, we experimented with creating Conductor workflows using ChatGPT, and the results were amazing.

And now, in May 2024, how often do you use tools like ChatGPT? For instance, finance professionals rely on AI tools to aggregate and summarize financial news, providing timely insights and market trends. Similarly, travelers utilize them to generate personalized travel ideas and itineraries based on their preferences, helping them plan trips efficiently.

Whether it's assisting with research, generating creative content, or aiding decision-making processes, tools like ChatGPT are increasingly integrated into daily processes across diverse sectors. The technology behind tools like ChatGPT is called Generative AI (GenAI).

What is Generative AI (GenAI)?

Generative AI, or GenAI for short, is a type of artificial intelligence capable of producing human-like content from pre-trained models in response to prompts. It can generate content such as text, images, audio, videos, and more.

Given the vast array of AI technologies emerging, why should we specifically focus on GenAI? To understand this, let's examine the key differences between traditional/conventional AI and GenAI.

In conventional AI systems, algorithms are designed to follow a set of predefined rules. In contrast, GenAI is the next generation of artificial intelligence, capable of creating something new from the inputs.

Consider the example of traditional AI, where you play chess with a computer. The computer knows the rules; it can predict the moves and make its own based on a predefined strategy. However, it doesn’t invent new ways to play chess; it chooses from strategies it was programmed with.

With GenAI, you can provide the input, and the AI model generates something new with similar characteristics, surpassing the capabilities of traditional AI models.

What is AI Orchestration & its Benefits?

Do you recall Kevin Scott’s (Microsoft's CTO) keynote at MSBuild 2023? (The Orkes team were also at MSBuild 2023—check out the highlights if interested.) In his keynote, Scott shared the importance of orchestration in building GenAI apps.

Scott remarked: “We really are setting the pace of innovation in the field of AI right now. And I think even for us, it’s been surprising to see how much of the zeitgeist is being captured by things like ChatGPT and applications that people are building on top of these large foundation models.”

Integrating AI models and prompts into the applications by code can result in monolithic modules that are difficult to deploy, debug, and maintain. However, AI orchestration allows easy management of such interaction, deployment, and integration. AI orchestration can decouple models, prompts, and vector databases to create a framework that can be integrated into the application in minutes.

AI orchestration offers a multitude of benefits, including:

• Scalability - Makes it easier to scale applications to handle increasing user demands. This is useful in scenarios where AI models need to process a lot of data.
• Efficiency - Streamlines the deployment and management of AI components, increasing efficiency in development and operation processes.
• Flexibility - Enables the integration of diverse AI models and components, allowing developers to leverage the best tools and algorithms for specific tasks.
• Consistency - Provides standardized processes for deploying and managing AI models, helping maintain consistency across different projects and teams.
• Optimization - AI orchestration tools often include features for monitoring performance metrics and optimizing resource utilization, which helps identify improvement opportunities.
• Cost Savings - Reduce operational costs associated with AI development and deployment by efficiently utilizing resources and automating repetitive tasks. This can help companies achieve higher ROI on their AI investments.

Let’s look at a recent Gartner poll from Oct 2023:

One notable highlight from the survey is that software development stands out as the area with the highest adoption or investment rate in GenAI.

Now, the question arises: How can you integrate GenAI capabilities into your application development strategy? That’s where a tool like Orkes Conductor can be leveraged. Orkes is the leading workflow orchestration platform built over the popular open-source Conductor OSS.

With Orkes Conductor introducing AI agents in Nov 2023, it’s now easier to bridge the gap between AI models and distributed applications.

AI Orchestration with Orkes Conductor

Let’s say you've got an existing application and want to add AI capabilities to it.

With Orkes Conductor, it's easy to build a workflow using AI agents and seamlessly integrate it as a new feature into your application.

If you're starting from scratch, Orkes Conductor can be your go-to tool for building the entire application while leveraging its GenAI capabilities.

Now, let’s look at the major steps involved in creating a workflow with AI agents in Orkes Conductor:

1. Integrate LLM models with Orkes Conductor
2. Create AI Prompts
3. Create Workflow including AI Agents
4. Execute Workflow

Step 1 - Integrating LLM Models with Orkes Conductor

The advancements in GenAI shed new prominence on large language models (LLMs), now equipped with billions and trillions of parameters, opening up a new era in the GenAI landscape.

Orkes Conductor supports integration with the following LLM providers and their hosted models:

For Retrieval Augmented Generation (RAG), you need to integrate Orkes Conductor with vector database providers as well.

Vector databases play a crucial role in RAG by indexing and storing large amounts of data in vector form, enabling efficient retrieval at the required flow within the application. Whether it's answering user queries, providing personalized recommendations, or generating content, the integration of Orkes Conductor with vector database providers enhances the capabilities of AI-powered applications, delivering more accurate and engaging results to users. We currently support integration with providers such as Pinecone and Weaviate.

The added integrations can be shared widely with the required team through role-based access control (RBAC) functionality within Orkes Conductor.

Step 2 - Creating AI Prompts

The next step is to create AI prompts in Orkes Conductor that interact with the integrated LLM models.

Prompts are the instructions provided to the LLM models, and a Prompt Engineer creates them to elicit the most optimal response. With Orkes Conductor, prompt engineers can design prompts, which app developers can utilize while creating their application flows.

Step 3 - Creating Workflows with AI Agents

With LLM model integration and prompts in place, the next step is to create a workflow with the AI agents.

In Orkes Conductor, the workflows can be built using code in your preferred language, JSON, or the Conductor UI. The AI Agents currently supported are:

Depending on the application requirements, the LLM tasks can be added to build our GenAI-powered application.

Step 4 - Executing Workflows

The final step is to execute the workflow. The workflow can be invoked using different methods such as code, APIs, UI, scheduler, and Webhook events or in response to events from external eventing systems such as Kafka, SQS, and more.

And voila! Your GenAI-powered application is up and running. Check our documentation if you are curious about a basic example of a translation workflow covering all these steps.

Summary

We wrapped up a fantastic developer meetup in India on AI Orchestration & Distributed Applications. To check out Orkes Conductor’s GenAI functionality and capabilities, explore our free trial today.

Get Orkes Cloud Free Trial!

Stay updated on our upcoming events, and if you'd like to participate as a speaker, sponsor, or collaborator at Orkes events, contact us at devrel@orkes.io.