You. Hello and welcome to the let's talk.
Azure podcast with your hosts, Sam Foote and Alan Armstrong.
If you're new here, we're a pair of Azure and Microsoft three, six, five focused it security professionals. It's episode eight of season five. Alan and I had a recent discussion around Azure event grid, a managed event routing service for event driven architectures. During our discussion, we covered what is Azure event grid? What are the common use cases for Azure event grid? How can you get started and what are the licensing options? We've noticed a large number of you aren't subscribed. If you do enjoy our podcast, please do consider subscribing. It would mean a lot to us for you to show your support to the show. It's a really great episode. So without further delay, let's get started. Hey, Alan, how are you doing this week?
Hey, Sam. Not doing too bad. It's been a busy week. How about you?
Yeah, I feel like this week has just flown by. We seem to run out of time to record these episodes every week. I assume that's a good thing. Are you getting excited for MVP summit coming up?
Yeah, definitely. Yeah. Really looking forward to it. All flights are booked and everything. The scheduled planner is out there, so I've started scheduling all the sessions and. Yeah, looking forward to it and meeting up with some of the other mvps out there.
This is your first time in person, right?
First time for the MVP summit? Yeah. I just missed it last time because I've technically been an MVP for a year now. Maybe. No, April. It was almost right.
When's the renewal?
June, July time. I got to submit by the end of this month. End of March.
End of March. Okay, nice. I'm sure you'll be fine. Anybody from Microsoft listening, please.
Anyway, we've made loads of content.
Yeah. We have to be fair. Yeah, no, I'm sure it'll be absolutely fine.
Cool. Okay. Shall we dive into this episode?
Yeah, let's get started.
So, Sam, we're kind of talking about events.
I guess.
So what is an event driven architecture?
Okay. Yeah. It's worth sort of laying the groundwork, I think, for event grid and the sort of solutions, how it works and what it's aiming to achieve. In traditional computing systems, people would write code that in a very basic sense, they wrote programs of code which included modules and functions of code which executed synchronously. You know, you. You gave the computer a command and then it, you know, well, a bunch of commands, and it sort of followed them start to finish. We then got sort of parallelism, so to say multithreaded cpus where we could offload tasks that could be done in parallel. And cpus then grew and grew and sort of scaled horizontally and vertically with Moore's law. But essentially we packed more and more computing power into a single cpu. I use this analogy because it shows you sort of how we went from sort of a synchronous programming flow to a parallel programming flow. I say now that we have the Internet, we've had the Internet for ages, but now we have distributed systems that need to communicate with each other. A traditional way that you would communicate with another system is by using an API, an application programming interface. If you've ever built an API before, you've probably heard of a rest API or a soap API. And these are effectively protocols which allow, it's probably best to say, synchronous type communication. I mean, there is an element of. But let's pretend you send a message and you receive a response that underpins a lot of the Internet. But that's a very simplistic data flow. Nowadays we have lots and lots of different devices, sensors, systems that emit events that work on an event driven model. I'll use the example of a, let's use the example of a doorbell. Every single time you press an Internet connected doorbell, you need to send an event that the doorbell has been rung somewhere, right? And it may be that a lot of doorbells are programmed in this way that they will just call a rest API endpoint with, hey, I'm being pressed. But what instead you could build is an event driven architecture. So instead of needing a synchronous response from that endpoint to say, hey, yeah, I'm ready to open that door. There might be multiple different systems that you have to pass through in order to make the decision about whether you should open the door or not. In this example, you might have biometric, like facial recognition, a keypad, a second sort of factor authentication that you need to go through. Maybe that's a different system. Maybe you've got a partner system which then verifies the messages. You could have multiple interconnected systems even inside your own organization. So asynchronous flow, where you go and talk to one sort of responder, one API might not really work for certain more distributed systems nowadays with more complexity. And this is where sort of an event driven architecture and durability comes in. So what an event driven, in simple senses is messages are created in event driven systems. There are event brokers or aggregators which take those events and put them on some sort of queue to be processed. And in that queue, different applications can publish and subscribe to sort of notifications of events that are going on there. And what that allows you to do is that message then has its own lifecycle, it's its own durable entity. So alan's pressed the doorbell. So there's a doorbell event that happens. If there's a system that needs to react or validate that, they can subscribe to those types of events, like those topic of events. So every time the doorbell is pressed, I want to know about that and they can modify that event, push it onto another sort of queue topic to subscribe to, and you can pass this message around or the results of these messages to lots of different systems. It is effectively an asynchronous way for multiple distributed systems to communicate with each other. A good example as well is IoT, IoT, especially if you do any sort of home automation, you'll know about MQTT. And MQTT is a type of brokering protocol. But if you imagine like a temperature sensor, right? A temperature sensor sitting on your wall, all that's effectively doing is emitting that current temperature that it sees every x interval, that might be every minute, might be every hour, whatever it's configured to do, it doesn't really care where that data is going. It's effectively a really dumb device. So if you have a messaging system that can then pick that up and take it forward, those devices have to do a lot less programming, a lot less sort of logic and understanding of what they're connecting to. Also, if you send it to an event broker, it can then do all the distribution of those messages, sort of fan out to different systems. They're just a couple of examples. But event messaging systems are used really widely in lots of different applications. You may notice that some websites that you use, some websites that you use might live update in front of you. Have you ever used a website where you've got a new message in your inbox and it just pops in and a notification shows, right, your browser could just be refreshing all the time and it could be going, give me the information. Or your browser session could have set up a connection to a web server which is pushing events down to you in an asynchronous manner so your browser doesn't have to wake up every 30 seconds and send a message going, hey, have I got more emails? Hey, have I got more emails? As an email comes in to your messaging provider, they can fire a message down to your browser. An event down to your browser to say, hey, there's a new email here, you should refresh and show the user that it's an email there. So it isn't just used in distributed systems Iot, it is actually used on the web as well. In reverse. There's many different messaging protocols, proprietary open source protocols, but essentially event driven systems are an architecture which will stand the test of time. They are known to be massively distributed, massively scalable, and actually a good way to represent sort of flow logic in complex systems.
Cool. Okay, so yeah, it's sort of using, like you said, events going into a queue or something like that, and then systems being told about that, them going into there and then them consuming them, I guess, and vice versa, like you said, going through multiple systems, like the example that you said about a doorbell being rung. But that could be anything. Like you said, it could be. And I guess at this point providing whatever that mechanism is in the middle that's looking after those events. Like you said, the system's calling upon it, then they can be everything and anything, can't they? Because it's all detached on that part. Okay, so let's talk about Azure grid's capability then.
Okay, so to start off with, if you wanted to roll and host your own system, you would have to find a vendor or an open source application to help you with that. You would have to host it somewhere, be that on bare metal and sort of manage it yourself. Event grid is completely managed by Microsoft. It's an Azure product. Now. It's quite a sort of multifaceted solution really, because the way that I was sort of introduced to Azure event grid is actually events inside of azure triggering and sort of messaging out. And then the more I started to use it really sort of highlighted its other capabilities. So there's quite a few different bits to go through. So as I've mentioned before, it's fully MQTT compatible. So if you are utilizing applications and devices which support MQTT, it can be used as a broker for that, which is pretty powerful in its own right. It's effectively there ready to go. It's fully built in with cloud integration, it has TLS support, it's got a lot of cloud access control methods. If you've ever integrated MQTT yourself, some of those features are lacking or hard to configure in open source tooling. No criticism there. It's just there's a level of knowledge and expertise that are required there. If MQTT isn't sort of your flavor, you can do event messaging via HTTP as well. So there's effectively a HTTP endpoint there available for you to integrate with. It's worth just noting that you can integrate with sdks. So there's Azure CLI Powershell Rest APIs, which is the HTP element, Net, Java, Python node, JS. So there's a lot of if you are building in arguably the most popular languages or very closely most popular languages, there's really good support there. And there is just raw rest API support if that's what you want to do. MQTT, just to jump back to that, usually MQTT is like a glue between a lot of your client devices and applications, so you effectively publish and subscribe to multiple devices and applications from both sides, and it can effectively act as a broker and a queuing system for that. We've talked about ingesting IoT telemetry via MQTT and what's really powerful is that cloud integration, passing on those MQTT sort of basic MQTT protocols and messages from those sort of, I'm going to call them dumb IoT devices. You'll buy a lot of name brand generic IoT devices which only support MQTT, or sometimes they support proprietary protocols and integrations which aren't so great, but it allows you to sort of contain all of that and then broker it into cloud applications a lot more simplistic. It's also probably worth talking about in reverse. So MQTT can push data back to end devices via broadcasting alerts to sort of fleet of MQTT listening devices. So that's definitely worth sort of understanding. Okay, I've talked about MQTT and let's talk about various different concepts. It's probably worth going through a bit of a taxonomy of different things here. Events are essentially the messages that are passed through Azure event grid, which is effectively your payload of data that you want to send through. There's different content modes that you can send through, you can send through binary data, structured JSON data. There's definitely a set of flexibility there dependent on your data types. Let me just give me 2 seconds. I've just got some notes because I want to talk about, just give me 1 second. I want to talk about event grid basic because we've talked quite heavily around IoT and how that's sort of integrated. But also there's the concept of event grid Basic, which allows sort of pub sub events and cloud events to different topics and partner topics as well. This is more the Azure integrated mode. I will talk about it later in a bit more depth of how you integrate in certain solutions. But this is where the multi sort of faceted element of event grid comes in, where you do have these standardized protocols like MQTT, but there are also sort of Microsoft's interpretation of events queuing. So if you do want a more standards orientated end to end protocol and system that is supported, but there is also a more, I would say simplistic cloud sort of focused queuing system built in as well.
Okay, cool. So yeah, lots of sdks as always. I think event grid has been around for quite some time now, hasn't it? So it's had the time to mature from that side of things. So there's a lot of documentation as well as like you said, all the sdks out there to help build. So I guess how do you deploy event grid?
Yeah, so as is probably kind of obvious, I suppose from an Azure perspective, it's fully managed in the Azure portal. You can obviously deploy it, you don't have to deploy it via the portal if you don't want to, but it's effectively a resource that you deploy into Azure. So it's fully sort of azure native. It's very similar to any other resource that you would deploy there.
Okay, that's good to know. Like you said, if you had to build it yourself, you'd have to configure it, manage it, resource it, et cetera. So for it to be potentially a couple of clicks away to have something to start using is really good. Okay, you kind of talked lightly about it earlier, but how do you integrate event grid?
Okay, so I think what I want to sort of focus really on is sort of cloud events really. I have spoken about sort of custom event driven methodologies if you wanted to integrate that into one of your applications. But I think it's worth highlighting the power of event grid integrated into Microsoft and Azure's services and different resources there. I might be doing it a bit of a disjustice talking about it here, but I do think it is a really good and powerful automation platform inside of Azure. So let's start with where did we start on this topic? Let's talk about partners different event sources. So in event grid you can have a number of different sources that you could get events from. Partners are one of those. So partner events effectively allow partners which are like third party SaaS applications to register different events that can then be emitted into and through event grid. So things like SAP auth Zero is integrated, Graph API is also integrated as well. Even though you would think that, that's not really like a partner integration because it's Microsoft, but it's effectively a different set of APIs kind of outside of Azure, isn't it?
Different SAS app, isn't it?
It's a different SAS app, yeah, exactly. So let's talk about Graph API because I think a lot of people think about graph in the Graph API in a sort of consumption mode. You send a request, you get a response. It's very sort of simplistic in that sense. But what if you wanted to subscribe to an event, say when a new record is added into graph or available in graph? Well you can do that. There's a system called change notifications within side of Graph API. Not all objects are supported, but I'll run through a few that are sort of in our space as well. So security alerts which are listed, listed in graph, you can detect changes to a specific alert and you could subscribe to them there. And you can also respond to changes in filtered alerts from graph as well, which I thought was pretty powerful. Teams chats, you can respond to any changes in any teams chats sort of as they happen in real time. Any changes to a user object inside of graph you can subscribe to. So there's a lot of different functionality there. There's a big long list on learn so I would sort of urge you to sort of go and check that out. But if you are looking to respond to certain events inside of graph grid can pick those up and then trigger other automations further on down the line because it's worth talking about. Let's keep on the subject of topics and event producers because what we've just talked about there with those partner events is they are creating the messages, right? And then you're just consuming them in event grid. There is also a concept of system topics and the system being in question here is azure. So other Azure resources can actually emit events which can be captured within event grid. So let's use the example. I'll read through the list. I won't read through the whole list, I'll just read through some of the ones because there's quite a few. API management is in there as your app service, Azure blob storage, Azure communication services, as your IoT hub, as your key vault, as your resource groups, as your service bus, as your subscriptions. So I'll pick one of those just to dive into some of the events for them. So key Vault, we have an episode on Keyvault, is that right? Or do we not have one yet?
I don't think we've done one yet. We've talked.
Holy moly. Okay, that'll be my next episode then. If we haven't done. Yeah. Key Vault, if you haven't used it before, it's a place to store your secrets, your keys, your certificates. There you go. That's one single sentence basically for it. But key Vault can be a challenge sometimes, because you can put time limited sort of access and expiration on a secret in Keyvault. As an example, Keyvault might not know what's inside of that secret, but there could be an expiration date in a separate system which is linked to that secret. So you'd want to expire it from key vault at the same time. So as an example, there's an event that can trigger from key vault on expiry of a secret. So once a secret expires, it goes past that expiration date. It will call event grid and start a messaging, a message off, and then you can follow through with that and start an automation from there. And there's absolutely loads. It's not just when they've expired. When a secret is near expiry, 30 days before the expiration date, you can get it to trigger. It might be nice that you would email the resource owner if some secrets are nearing their expiry, because we know how challenging that can be. To refresh your secrets and your certificates especially. I'll pick another example. Blob storage. Now, blob storage is quite, well, it's highly integrated with a lot of logic apps, function, apps flows. It's a trigger for a lot of different systems. But again, if you want to integrate that into your own third party application, or you want to do some sort of other custom logic there, and let's say you didn't want to use like a logic app, you have things like Blob created, blob deleted events that you can trigger there as well. There's a big long. I've obviously been through the sort of resources, but there's definitely a lot of messaging and different events that can create those alerts. So that's really talking about the actual events that created. Right. But then what do you do with them? Right, because it's all well and good, like creating these events. But if you're not going to do anything with them, then what's the point, so to speak? So in event grid, there's the concept of an event handler. This is effectively where you then chain in logic further down the sort of track. I'll use the example of azure functions, because it's quite a common use case to have an event trigger and then to run a function off of that, so that you don't have any sort of long running infrastructure that you've got to really sort of care about. So you create your function app as normal, and then what you can effectively do is create like a grid topic to subscribe to and you can then subscribe your function basically to that topic. And effectively when you post messages into that topic. I'm not going to go into topicking. Is that even a word? Probably not. Subscriptions Pub sub, different sort of pushing scenarios and sort of distribution scenarios. But essentially you're saying this function app is waiting for messages in this queue that relate to this topic. A topic is like a label, basically you could have a queue. Let's say if we go back to the doorbell, is a doorbell a good idea? Okay, let's go with the doorbell. Imagine if you press a doorbell once, it will go into a queue. So maybe it's like the doorbell rings queue or something like that. If you press it once, you might put a message in there with the topic single button press as an example. If somebody double taps the button, you might still put it into the same queue, but you might give it a different label, a different topic, and say one system is only looking for single presses, one system is only looking for double presses. So you can do some element of filtering. Even if you've just got one sort of queue in system. You effectively connect these to the subscription in and then anytime you post a message into Eventgrid, the data will then be emitted through into the function app completely automatically. That wiring you do not have to do via a CLI or some sort of config file or integration. Basically you can just create it in the portal in a first party sort of integration, which is pretty powerful really, because these systems are usually hung together in a very distributed manner. You would have somebody that is like your event and queuing engineer who would create all the topics, all of the subscriptions, all of the queues, and manage all of the architecture in and around it and how it's all managed. Whereas Microsoft is effectively saying connect these two systems together and tell us how to connect these two systems together and we'll just carry out everything in between. Any sort of automation and sort of messaging engineers out there. There is a lot of different publishing and subscription functionality there for filtering messages, durability of messages, and how all of those messages fan out. I'm not going to go into that in this podcast, because that's actually looking at integrating it and it's a bit deeper. The fact is that there's a powerful event driven system there that Microsoft will happily manage for you.
Okay, cool. There's loads there, isn't there? And I think when you were talking about the use of logic caps and things like that, it might be that you have a logic cap that has multiple functionality that says looking at a queue and maybe it's determining, like you said, it might be doing blob storage events and maybe it's something else. I can't think of anything else right now. But yeah, you're making it sort of generic at that point. And I guess really some of those connectors in the background like you're talking about graph things about events happening when a new file is appeared in this sharepoint site. It kind of feels like the connector is basically backing onto event grid that we don't see.
It could well be right, because what is the raw underlying event driven system that is there? Right. The thing that sort of intrigues me about event grid is when you're doing a lot of logic apps, as an example, a lot of the time you are sometimes just like connecting two systems, right? You're connecting like something being created and then you're putting it somewhere else or just maybe transposing the data in a small way, I don't know. But instead of having to sort of do that in a logic app, you could potentially just do that in an event grid subscription if it's possible to do that. So that's why I thought that for really basic messaging scenarios, it can be obviously pretty powerful to sort of handle that.
Yeah. Okay, so the question that we always love to do, how much does it cost? Is there any tiers, et cetera?
Okay, so if you use event grid basic, which is not really that basic, I wouldn't say this is what we're talking about when we're talking about Azure events, these Microsoft sort of managed events and partner events, you get 100,000 operations free per month. So go off and consume those because that's a decent amount of operations. And then it is mean in UK south, but in dollars anyway, let's just go with it. Few. It's quite a few. If you go up to the standard tier, this is where you get your MQTT broker capability, pull and push delivery of events because you are effectively hosting sort of broker there, you purchase throughput units. So a throughput unit is, let me just run the monthly rate on that is $29 per month. And then on top of that you pay one dollars per million MQTT operations, but you do get a million operations again for free usage per month. And then if you're doing event operations, which I believe are the HTTP related non MQTT ones, it's a million operations again for free and as well. So again like other sort of PaaS solutions in Azure free tiers, to get you going to validate your sort of hypothesis of the solution, get it integrated in maybe a dev staging environment potentially. And then when you actually roll out into production, I can't tell you if that's expensive versus actually hosting something yourself, but I assume once you get to the scale of spending a lot of money on event grid, you can probably afford based on your business model to be able to build out your own system. So in terms of accelerating your sort of adoption of this type of technology, it's a really great way to sort of accelerate that process.
Okay, great. Is that all of the tiers, things like that?
Yeah, that's it. Cool.
Seems fairly simple in some way. We've talked about some of those other services and there's multiple ways to consume it.
Yeah, I haven't really gone into sort of delivery mechanisms in different sort of broker and queuing mechanisms because I think I would say the true benefits, because they are benefits obviously of the system, but they are features that are common to event driven systems, not specifically to the sort of magic of Azure event grid, if that makes sense. The magic to me of Azure event grid is that you just literally start running with it and you get a million operations free per month and you can get started with it. You don't have to learn all these different things and bring up all these different systems. If you are looking to make sure that you have good event durability filtering retries and delivery mechanisms, it is there. It's just definitely go and have a look at it because there is maturity there in terms of its sort of process.
Okay, great. So is there anything else around this side of things you might have missed or anything else you want to. I suppose we've done quite way through sort of our episode timing I guess.
Yeah, exactly. I think I'm all good there with what we've covered but yeah, definitely give it a try. And if you've got any feedback on how you found it, if you've used it for sort of a production app, which I don't think I've got to the stage of calling it that, it's more automation in and around Azure. Definitely interesting to see how you use it.
Okay, great. Well, thanks, Sam. That was a great run through on so our next episode will be sort of the first week of the month, so we're going to make it into our news episode. So we'll find out what was released in February and we'll run through that.
Yeah. It always surprises me when we prep for that episode and then you read through all of the new updates that there have been, and you're like, this organization is huge. Do these people ever sleep right? There just must be an army of people building and deploying. Right? It's pretty scary.
Yeah, definitely. Okay, so did you enjoy this episode? If so, please do consider leaving us a review on Apple. Spotify helps us reach out to more people like yourselves. If you have any specific feedback or suggestions, we have a link in our show notes to get in contact with us.
Yeah. And if you've been listening for this long, thank you ever so much and we'll catch you on the next one.
Yeah, thanks all. Bye.