Managing your API

So you have developed a brilliant new web based API, the world will be fighting to get access to it, its the most useful thing ever.  But how do you manage access to it?  How do you protect it, how do you monitor calls to it, how do you restrict access based on who the consumer is?

Microsoft offer a solution to this as part of their Azure offering.  This blog will look at what Microsoft Azure's API Management product offers.

With APIM you can

• Ensure access to the API is only via APIM
• Set throttling policies based on users/groups
• Make users sign-up for access
• Generate and publish developer documentation easily
• Monitor traffic

And probably a whole lot more that I have not explored yet.

And all this can be done for multiple APIs all through a single endpoint.  You simply need to have an azure account and create an APIM instance to get started.

I'll look at how to set up some of the things mentioned above, starting with establishing an APIM instance and pointing it to you already published API.

To create an APIM instance, go to the management portal for azure (I use the old style portal at https://manage.windowsazure.com/), select the APIM charm on the left

You will be presented with a list of your instances (but I guess you dont have any), and in the bottom left is the new button to create your first instance.  Clicking this will open a banner with the options of what you want to create, there are no options, so just click create in third column

You will be asked for a URL, this is the endpoint that the world will use to gain access to your API, and a few other questions about where the instance will be hosted, who you are and your contact details, and if you select the advanced settings you can select a pricing tier, but that as the name suggests is an advanced topic, and this is a basic intro blog, so leave the pricing tier as Developer for now.

After that it will go away and spin up an instance for you which will take some time.  At the end of this the list of APIM instances will be populated with a grand total of 1!!!

Next select the name of your new APIM instance and you will be presented with the quick start page, the only page that I honestly use within the azure portal in relation to the APIM instance.

The 'Publisher Portal' is the gateway to everything to do with the APIM instance.  And out of interest the 'Developer Portal' is what the public will see if you tell them where to look for information on you API, i.e. where you can provide documentation.

To finish setting up your vanilla first APIM instance, go into the publisher portal and you will be presented with a dashboard with not a lot of data

The next thing to do is connect APIM to you existing API, which you do by clicking the add API button, and providing the details required.  You need a name for the API, the URL for the API you made earllier, a suffix which will be used to distinguish between the APIs associated with the APIM instance and provide whether you want the access to the API (via APIM) to be allowed with or without TLS.

There are stil two steps to go. firstly defining the operations that can be accessed via the API.  I.e. what verbs are allowed and for what URLs.  Add operations such a GET users using the intuitive dialog

and finally you need to associate a product with the API.  Products are in my opinion badly named.  They represent levels of subscription to the API.  By default 2 products are preconfigured, Starter and Unlimited, you can associate these or any other products with the API using the Products tab on the right of the screen.

After this your new APIM is ready to go.

The next thing you may wish to do is add a throttling policy to the API (or more specifically the product).  You do this by selecting the policies option on the menu in the left, pick the combination of options you want to set the policy for (product, api and operation) and click the add policy option in the text box below. This will add a blank policy, and you can add the details of the policy using the code snippets on the right, so for a simple throttling policy select the limit call rate option, and this will add code to set a limit on the number of calls within a given time window.  By default the starter product is limited to 5 calls in any 60 seconds, and 100 calls in a week

This gives you a flavour of what can be controlled with this.

The use of the products and policies in conjunction allows you to fine grain the access to the API and its operations in a way that is best fitted to you and your users.

The next thing I would look at is securing your API so that the rules set up in APIM must be followed.  If people can go to the API directly, bypassing APIM, then these policies and rules are meaningless.

The simplest way to do this is to use mutual certificates between APIM and your API, and add code into your API to ensure that all requests have come from a source with that certificate.  This can be done by going to the security tab on the API section of the publisher portal

then pick the mutual certificates option in the drop down.  You will need to upload the certificate to the APIM instance, which can be done by clicking the manage certificates button.  In terms of ensuring the request has come from a trusted source, that is a coding question, but for completeness, within a c# MVC webAPI system, add a message handler to the pipeline for the incoming requests by editing the WebAPIConfig class

 public static class WebApiConfig  
 {  
      public static void Register(HttpConfiguration config)  
      {  
           // require client certificate on all calls  
           config.MessageHandlers.Add(new ClientCertificateMessageHandler());  
      }  
 }  

And then add a class to check the certificate:

 public class ClientCertificateMessageHandler : DelegatingHandler  
 {  
      protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)  
      {  
           bool isValid = false;  
           X509Certificate2 cert = request.GetClientCertificate();  
           if (cert != null)  
           {  
                if (cert.Thumbprint.Equals(RoleEnvironment.GetConfigurationSettingValue("ClientCertThumbprint1"), StringComparison.InvariantCultureIgnoreCase)  
                 || cert.Thumbprint.Equals(RoleEnvironment.GetConfigurationSettingValue("ClientCertThumbprint2"), StringComparison.InvariantCultureIgnoreCase))  
                {  
                     isValid = true;  
                }  
           }  
           if (!isValid)  
           {  
                throw new HttpResponseException(request.CreateResponse(HttpStatusCode.Forbidden));  
           }  
           return base.SendAsync(request, cancellationToken);  
      }  
 }  

This allows you to define two certificate thumbprints in the web.config of the API to compare against the incoming request.  All requests via APIM will include the certificate that you upload in the publisher portal, so if you ensure that this cert is not made public, then you can be assured that all requests hitting the API have come through APIM.

I mentioned making users sign up for access.  Well if they want to use your API we have just ensured tha their requests must be directed via APIM.  The rest is built in.  The products that you configured earlier have associated with them a subscription key that the consumer of the API must supply with every request.  This ensures that every consumer of the API must have the subscription key.  The developer portal provides a way to subscribe to your APIM and thus get a key.  You can if you wanted restrict this so that you need to manually give access to every subscriber before they get a key, and that way you could monetize the access, but that is way beyond the scope of this article. Suffice to say they need to sign up to get the key or APIM will reject the requests.

The documentation aspect of APIM is probably something best explored yourself, I'm a techy and not best placed to explain this, however in summary you can document each operation on the page where the operation is created/configured, and the content of the description can take the form of html/javascript so you may wish to use some client side script to retrieve the documentation and manage the content externally.

 <div id="dynamicContent"></div>
<script src="https://myserver/scripts/documentation-ang.js"> </script>  

would call the javascript file which could provide the content required based on some data scraped from the rendered page.

The final thing to look at is the analysis of traffic.  Good news, you don't need to do a thing to configure this.  Basic analytics are provided out of the box.  If you require anything more complex than APIM offers for free you may wish to look at other products that could be bolted on to the system, but in general the data that is captured will provide you with a lot.

So in summary to that all, APIM offers an easy to set up and easy to configure interface to your API when you want to publish it to the world.  It gives easy access to some very good security features and access control.  There are many platforms out there that will host an API for you, but if you want to host you own somewhere, APIM offers a lot of what you need to consider before exposing your delicate little flower to the horror of the web.

Any questions or comments are welcome, and I will try to answer anything you ask.  And please share this with everyone you know, and even those you don't know.  Spread the word, the cloud is fluffy and nice!

Dedication, Motivation and Achieving

It has been said many times that to become an expert in anything you need to practice for 10,000 hours.  This theory has been professed for sporting prowess (see the book Bounce by Matthew Syed) and for technical skills (see John Sonmez's podcast) and many people argue against it.

I see the logic behind the idea, if you do something consistently for that long you will surely iron out a lot of the problems you have with it and become better.  You may not become the best, there may be some innate talent involved that you do not posses.  For sporting prowess, anatomically you may not be able to surpass some other people.  But practice anything for 10,000 hours and you will become a whole lot better than you are right now. One important point is that you probably do need some guidance (training) during these 10,000 hours, otherwise you may develop bad habits that will hold you back and will be difficult to break once they are deeply ingrained.

So coming to the technical skills of programming, Sonmez argues that practicing, repeatedly trying to build systems and applications, whilst trying to extend you skill set  is simply a matter of perseverance and habit.  That you must get yourself into the habit of practicing (coding for instance) on a regular basis (daily?), and that motivation and drive have little to do with it. I disagree with this.

In order to be able to commit and persevere with the practice you need some form of motivation, something to keep you going.  The simplest thing that motivates most people is greed.  Sounds dirty, but its true.  We all live at a level beyond necessity (materially speaking) so we need money to do this, we may not all strive for more more more, but living above the level of having food and shelter could be perceived as greed (I'm not saying we are all greedy, but we all like comfort and the nice things that are accepted as normal in modern living, and these all cost money.  So we all look to have an income, and as programmers/developers/coders writing code is a way of getting this.  So that is a base motivation achieved.  But how does that equate to practicing skills and getting better?  Most of us work for someone else, we have fairly clear objectives of what our employer needs us to deliver in order to maintain that employment and be paid.  This is often not enough to drive us to get better.  We can drift along, doing enough to get by without getting better.  10,000 hours of this will not get you very far.

So how do you get better?  What form will the 10,000 hours of practice take?  Well you could potentially use this employment time, if you use it wisely.  This is easy when you are starting out, all of your time will be spent on new things and so will be useful, but give it a year or so and your day-to-day work will stop stretching you.

10,000 hours equates to 1,250 8 hour days, or nearly 5 years of normal employment (8 hour days, 260 days per year).  So it is not a quick thing.  And if only about the 1st year will really stretch you and be useful practice then these 5 years will not actually amount to the 10,000 hours. So how do you build it up?

Side projects, personal learning, pushing your employer to embrace new ideas and technologies.  That is how.  If the project you are working on is stagnating, look for how it could be moved forward to the benefit of your employer and push for them to do it.  It will benefit you from a skills and proficiency perspective, and benefit them where they like it, in their wallet.

But for side projects and personal learning, the problem is often one of motivation, drive and time.  How do you drive yourself to put the time in?  You have just worked a 40 hour week for the man, where is the motivation to put in more time on top of this?  Yes if you put in the hard yards you will see a payback eventually, but 10,000 hours is a long time, and we are simple beasts, we want dividends now.  Something perceivable and measurable.  That is where the motivation aspect comes in.  That is the motivation.

You need some sort of short term perceivable and measurable metric of your progress and success.  Look for what it is in programming that you find most rewarding.  It could be a wider question than that, what do you find rewarding in life?  Personally I like the feeling of seeing a task completed.  The sense of satisfaction that something I set out to do happened and I succeeded in completing it.  I love hiking, and specifically hiking up mountains, and the greatest sense of achievement in this is to get to the summit.  But reaching a high summit can be a hard slog.  It might only take 5 hours rather than 10,000, but physically I still feel that fatigue in reaching my final goal en route.  What I do is to set intermediate goals along the way.  The next 3 miles, the next false summit, where I want to be before 11am.  Anything to give me a sense that I am doing well and on track for my final goal, the little pile of stones that marks the top.

In motivating myself to do some personal development it is the same.  I set short term goals that will get me to the big prize.  The big prize is the 10,000 hours mark?  No its becoming what you want to be personally.  Pinning what that is is difficult, and right now I can't easily express what that is, but I can set short term goals, and the final goal will come into sight at some point.

For side projects this short term goal setting is easy (although the side project itself should probably be one of your short term goals, but breaking this down further is a good idea) try to work in an Agile manner within the project.  It might be a personal project, with a team of 1, but breaking the development down into short iterations, with a goal of delivering a functioning system at the end of each iteration is a very valid approach still.

If you do this, and your psyche works anything like mine, then at the end of each iteration you will feel a sense of achievement and that will help you to motivate yourself for your next iteration.  You will surprise yourself by how easy it becomes to motivate yourself to complete the tasks of the iteration, because you will want that sense of achievement at the end of the iteration, and quickly the side project will take shape and maybe start to deliver on that base motivation, hard cash!  The great side benefit is that you will push yourself towards that 10,000 hours mark, which is a notional mark to indicate you are becoming a much better developer.

Motivation is important, its not just about turning the wheel, but you must find your own motivation by setting achievable milestones and rewarding yourself in some way that helps to keep you focused on the next step, the next step and eventually the big prize.