A failed integration usually does not visibly disrupt a system, but it likely does leave your overall organisation in more subtle state of error. How do we become aware that a failed integration has happened? How do we investigate and remedy the situation?

Today we would usually build a very technical integration dashboard. An operations engineer would use the dashboard to dig into the issue, understand what had caused the failure, and then pass that information on to a business user where necessary.

Can we do better?

Very often when an integration fails, it is a data problem. The receiving system may have different validation requirements on a field compared to the sending system, and what was acceptable to one system is not acceptable to the other. In my experience data isses are for more common than more technical issues, such as networking or authenitcation errors.

Given that these errors are often better resolved by a business user than a technical person, I wondered if there is there a way to surface them in a more suitable fashion.

I envisioned a solution where a business user could ask a question such as “Why didn’t employee Bob Smith make it to System X?” and receive an answer that was actionable for them.

Using Azure AI Services, Azure Integration Services, and a (very) simple web application, we can achieve this vision!

So how does it work?

The implementation uses the popular Retrieval Augmented Generation (RAG) pattern, utilising the Azure AI Search and Azure OpenAI services. This pattern is widely used in many “chat with your own data” style applications.

In this pattern, a company’s data is initially loaded into a vector database, which is a kind of database that excels at searching for things that are ‘similar’. Then, when a user’s query is received, a database lookup is performed to look for similar content, and then the user’s query and the database response is fed into a ChatGPT-style Generative AI to create a user-friendly response.

We generally build integrations using Azure Logic Apps. The Logic App team recently announced new connectors for Azure AI Search and Azure OpenAI. This opened the door to easily extending current integration implementations for our customers, without needing to build entirely new AI solutions.

I created a new, reusable ingestion workflow. The workflow receives a small payload containing the request and response information from an attempted system integration, as well as some accompanying metadata that provides context to help improve the quality of the results. This new workflow leveraged Azure OpenAI to ‘vectorise’ the payload, and Azure AI Search to store the payload and its vector.

This workflow can then easily be called from any existing Logic App integrations.

Below is a common integration pattern where a workflow is picking up a pub/sub style message (in this instance from an Azure Service Bus Topic) and attempting to send the message to an operational system. The only addition needed to this workflow to enable this solution is the final action to call the new ingestion workflow.

The other half of the solution is the retrieval workload. This flow takes the user’s question, vectorises it, retrieves data from Azure AI Search, then passes it all in to Azure OpenAI to generate that user-friendly content. This flow did not need to be built using a Logic App, but here I wanted to test out the new connectors.

The retrieval flow can then be called from the simple web app shown above, enable citizen users to investigate integration issues!

I find it interesting to explore the new approaches to building solutions that have arrived with AI services, especially being able to leverage so much out-of-the-box with Azure AI Services.

Placing Azure API Management (APIM) in front of an Open AI service was already a good idea - It provides a centralised point to manage your Azure OpenAI credentials without having to distriute them across your organisation, and it provides load-balancing capabilities to manage traffic across different models, regions, or pricing models.

Recently, Microsoft announced some new Generative AI capabilities in Azure API Management that make the service even more useful for Generative AI workloads.

Pricing in Azure OpenAI is fundamentally constructed around ‘tokens’. Every request in and response out of an Azure Open AI service has a token count, and the more tokens you consume, the more you have to pay.

Out of the box Azure OpenAI lets you set a token-per-minute limit for a model deployment and provides metrics on token consumption - however, you cannot drill down on these metrics to see either per-user, or per-application details. This lack of ability to drill down can make it hard to understand how your workloads are consuming Generative AI, and hard to enable sensible controls.

The chart below shows some sample traffic to an Azure OpenAI service over a 30 minute period. We can see overall token consumption, but not much else.

The first new Generative AI capability in Azure APIM is simply enhanced metrics. You can now easily view token usage at either the user or the application level.

Looking at the chart below, we can now drill down and see service usage per user. In this example, we can now see that over half of our OpenAI usage can be attributed to one user.

Taking this to the next level, there is now the ability to apply rate-limiting, also at either the user or the application level. This can help ensure that your total tokens assigned to your model are being shared around, and not being entirely consumed by one hungry user.

In this chart, the Azure APIM service is enacting a per-user rate limit of 1000 tokens/minute. We can see that the user with the largest usage has had their consumption limited, while the other users consumption levels are unaffected.

The next capability is a little more advanced, but fundamentally involves the ability to cache common requests in Azure API Management to avoid needing to use the Open AI service as much.

This may initially sound counter-intuitive for GenAI, as a lot of our individual experiences to date have been blank-page open-world style discussions across wide ranging topics with ChatGPT, but I think it is more relevant when you consider the types of Generative AI use cases that organisations generally implement.

Consider, for example, a public facing chatbot on a company’s website. The chatbot allows a customer to search over the company’s data, using the popular Retrieval Augmented Generation (RAG) pattern. This pattern takes a user’s request, uses the request to look up relevant data in a database, and then passes the user’s request and the database result into OpenAI to generate a user-friendly response.

If you have a lot of customers making very similar requests, such as store locations and opening times, or event information such as start times, then this style of caching is perfect. The capability even allows a ‘similarity threshold’, enabling you to fine-tune what is and isn’t considered a similar request.

The chart below shows the overall traffic with caching enabled, and we can see that the total token usage within the OpenAI service is now only half of what it originally was.

This caching comes with some warnings tho - firstly it requires the use of an Azure Redis Enterprise cache, which is not a cheap resource, and secondly it uses your OpenAI service as part of the caching process, so could possibly increase your overall cost.

Overall these new Generative AI capabilities in Azure API Management make it much easier for organisations to govern their usage of Azure OpenAI. The new metrics and rate limiting capabilities allow you to control access per user or per application, and with the right use case the caching capability could allow you to heavily reduce your token consumption.

But, as with most things in architecture, there are trade-offs. App Service has somewhat of a trade-off around cost optimisation. For established workloads that are worthy of their own (or multiple) compute units, the trade-off disappears, but for new, small things, App Service can be an expensive option.

Microsoft somewhat alleviates this by letting you group multiple services together. The scale unit for an App Service is called an App Service Plan. An App Service Plan is the thing you pay for, and once you have created it you can deploy many different App Services to it. You can even deploy your Function Apps there side-by-side with App Services, to stop you needing to pay for both App Service Plans and Function App Plans.

So what might a typical lifecycle look like? Initially we have our App Service Plan hosting an App Service or two, but what happens as we grow?

When you create an app in App Service, it is put into an App Service plan. When the app runs, it runs on all the VM instances configured in the App Service plan. If multiple apps are in the same App Service plan, they all share the same VM instances.

We aren’t going to be able to keep adding App Services to our App Service Plan for ever, sooner or later it is going to start getting full - i.e. run out of memory or CPU. So what are our options?

We could scale out if we are hitting CPU limits and just need to balance the traffic across some extra nodes, but this is unlikely to help us if we are hitting memory limits.

We could scale up, but this is just kicking the problem down the road. App Service Plans come in a few sizes (i.e. Small, Medium, Large), but there is a limit and we can’t scale up forever.

We could create a second App Service Plan, and start putting new services on there, but this creates a whole bunch of other problems that we’d prefer to avoid. How to decide which App Services go on which App Service Plan? We could just put the new ones on the new App Service Plan, but wouldn’t we prefer to scale certain services together, etc. Then there are other concerns such as virtual network integration… wouldn’t it be much nicer if we could stay within the one App Service Plan?

Then there is one final option, something Microsoft call “High-density hosting on Azure App Service using per-app scaling”!

It is somewhat buried away in the docs (red flag!), not mentioned at all in the Microsoft Learn AZ-204 training (red flag!), and not actually configurable via the portal, only Powershell/ARM (red flag!), but I decided to try it out anyways.

The general idea is that when enabled, an App Service no longer runs on all the instances configured in the App Service Plan, instead running on the number of instances that you configure that particular App Service to run on.

Sounds straightforward, right? Well, I decided to try it out and see if it does what it says on the box.

Experiment #1 - Creation

The docs provide an example powershell statement on how to create an App Service Plan with Per Site Scaling.

New-AzAppServicePlan -ResourceGroupName $ResourceGroup 
                            -Name $AppServicePlan
                            -Location $Location
                            -Tier Premium 
                            -WorkerSize Small
                            -NumberofWorkers 2 
                            -PerSiteScaling $true

The number of instances for an App Service is again controlled by a NumberOfWorkers setting. The default appears to be 1, which suits me just fine for this experiment.

I started created App Services, observing the instance they were being deployed to, expecting them to be evenly spread across the two instances.

The first App Service went to a certain instance. Then the second App Service went to the same instance (hmm). Then the third App Service went to the same instance (hmmmmm). Then the fourth, then the fifth, and so on.

Looking at the App Service Plan in Azure Monitor showed that Azure had only created one instance in the App Service Plan, despite the number of workers being set to two.

Something was clearly wrong.

Digging back into the docs, there is a section at the bottom labelled “Recommended configuration for high-density hosting” that states

Follow these steps to configure high-density hosting for your apps:

1. Designate an App Service plan as the high-density plan and scale it out to the desired capacity.
2. Set the PerSiteScaling flag to true on the App Service plan.

Following these steps did lead to having all instances activated, and App Services were then spread evenly across instances. However, needing to know the desired capacity up front is a pretty big turn-off, I don’t think that that is how this cloud thing is meant to work at all! How would it handle scaling out my App Service Plan? Time to find out!

Experiment #2 - Scaling Out

Next I wanted to see if scaling-out an App Service Plan would result in the existing App Services being rebalanced across all instances.

Confidence was low after Experiment #1, but the following line in the docs gave me hope -

Applications are rebalanced only when instances are added or removed from the App Service plan.

I started with an App Service Plan with two instances, each hosting three App Services. I then scaled out the App Service Plan to three instances, hoping that an App Service from each of the previously existing instances would be moved over to the new instance.

What happened? Absolutely nothing. The services stayed exactly where they were.

I did find ways to get my services to rebalance, but these were beyond what I feel I can reasonably ask of a team.

Conceptually, I think this capability could further increase the surface area where App Service is suitable as a compute option. However due to the manner in which it has been implemented, at least for my use cases, that potential value is not able to be utilised.

A core part of this migration, especially for customer-facing organisations, is moving users. Good customer experience demands a frictionless approach, a-la ‘Seamless Migration’, but migrating user passwords that you (hopefully!) do not have in plain text, or do not know the hashing algorithm for, presents a challenge.

Azure AD B2C

Azure AD B2C is Azure’s Identity as a Service (IDaaS) customer identity access management service.

AD B2C uses standards-based authentication protocols including OpenID Connect and OAuth 2.0. By serving as the central authentication authority for your web applications, mobile apps, and APIs, Azure AD B2C enables you to build a single sign-on (SSO) solution for them all.

Its competitors in the IDaaS space are Okta, probably the leader in the space (at least according to Gartner), but more expensive, and Ping. This Okta blog post, comparing themselves to Azure AD B2C is insightful, but should of course be read with a grain of salt.

There are a variety of open-source options in the self-hosted space, including Duende IdentityServer which is popular in the .NET ecosystem.

Importantly however, a lot of cloud migrations have core principles of utilising managed services, and of leveraging as much of their chosen cloud vendor as possible, so for anyone leaning towards Azure it makes sense to start by evaluating AD B2C.

Seamless Migration

The AD B2C docs does have a page on migrating users to AD B2C which discusses Seamless Migration.

There are two main components - first, an initial ‘pre-migration’ of all user accounts into AD B2C is performed. This is followed by creating a callback to your existing identity solution to validate a user’s password.

Then, using an AD B2C ‘custom policy’, each time a user logs in a call is triggered to the legacy system. If it returns that the password is correct then AD B2C stores the password, marks the user as migrated, and no longer needs to callback for that user.

If your legacy solution is not accessible via an API call then you are out of luck.

For me, the existence of this page in the offical documentation initially implied that Seamless Migration was ‘built-in’ functionality. However the docs eventually link you to a GitHub repo which has the rather ominous disclaimer.

“The migration application is developed and managed by the open-source community in GitHub. The application is not part of Azure AD B2C product and it’s not supported under any Microsoft standard support program or service. This migration app is provided AS IS without warranty of any kind.”

Let’s try it out

With the exception of being slightly out of date, the repository readme does an excellent job of guiding you through what you need to do to perform Seamless Migration.

In order to track whether a user has been migrated, a custom attribute is created in Azure AD B2C. It is great that there exists the ability to create custom attributes, however it would be really helpful to be able to see the value of a custom attribute for a user from within the portal, and to be able to sort/filter all users on a custom attribute. I ended up creating my own management web app to display this value.

The pre-migration step was relatively straightforward, utilising the user-friendly Microsoft Graph APIs to create a bunch of users. The name of the custom attribute used when accessing it programatically is a bit odd, but not a big deal.

The biggest challenge came with Azure AD B2C Custom Policies. I understand that adding extensibility points within products is hard, but this felt like customising a service that was not designed for customisation.

B2C is explicit that you are entering ‘identity pro’ land, but I am not convinced that slapping a warning on something gets you out of delivering a good product experience! This admission is tellingly called out in the above-mentioned Okta blog post.

There is a large amount of boilerplate point-and-click to enable custom policies in B2C, outside of authoring the policies themselves. There is a community app to automate this process, but it would be great to see this process automated within B2C itself.

Likewise, there is a (somewhat out of date) community repo full of custom policy samples, but no canonical ‘starting point’, which I believe would be useful. Taking the sample policies and updating them according to the readme results in policies that fail validation, which is not great.

Authoring the policies was challenging. The policies are written in xml, and while there are fairly strong docs, a lot of the terms used are not particularly user-friendly. Being honest, I would say I more cut-and-pasted sections from blogs/tutorials/samples than actually wrote much policy.

Lastly, uploading policies triggers a validation process, that in the case of failure (i.e. quite often) resulted in strangely formatted error messages.

Combining determination and my knowledge of the domain eventually resulted in policies that both passed validation and worked as desired! I wouldn’t say I deeply understand how they are working - I may return and figure it out, I may not.

In summary, I achieved what I was looking to achieve, but spent a little longer to do so than I would have liked. I don’t particularly value time spent learning how to configure SaaS tools, and feel Azure AD B2C would benefit from a little more investment in the customer experience for some of these more advanced scenarios.