tag:blogger.com,1999:blog-27003018657508357502024-03-21T18:47:07.897-07:00Communication and Maths Problems In C#A blog exploring topics of software development with specific focus on effective communication between the disciplines involved in the software lifecycle; and ways to solve some select mathematical problems using the .NET framework as a base, specifically coding in C#. This will document a learning journey shared by the author (me) and the readers (you and all your lovely friends) alikeAnonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.comBlogger19125tag:blogger.com,1999:blog-2700301865750835750.post-47287904661078783362016-04-06T12:29:00.000-07:002016-04-06T12:29:16.684-07:00Polymorphic extension methods The title is an exaggeration in some ways, but in the way the final code reads you would think it is exactly what we have.<br />
<br />
Let me set the scene for the problem. I have a problem where I am writing a client library that will be used to call a web service. The web service will return an enumerable collection of objects. These objects will all derive from a common base type (we'll call it BaseType for want of a anything less descriptive) and each type will need to be processed in a different way. I know that the web service will return only three types at the time I write the client library, but in the future more types will be introduced.<br />
<br />
The client library will be used by a large number of applications, so to reduce the need to redeploy these applications when the extra types are introduced, the library must be able to dynamically extend its type handling capabilities.<br />
<br />
My first thoughts went straight to a loop through the collection, calling a method on each item to handle that item. Basic polymorphism. However, to do this the base type must expose the method and each derived type override it, neither of which I the case. <br />
<br />
Next I thought about extension methods. The loop would look the same<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public void HandleValues(IEnumerable<BaseType> values)
{
foreach (var item in values)
{
item.Handle();
}
}
</code></pre>
<br />
but the extension methods would differ based upon type.<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public static class ExtensionMethods
{
public static void Handle(this DerivedType1 item)
{
//do the work
}
public static void Handle(this DerivedType2 item)
{
//do the work
}
}
</code></pre>
<br />
This does not work however, the type of item in the loop is BaseType, so no extension method is found.<br />
<br />
A big nasty if block or switch statement could be introduced to either cast the item or call the correct handler method. But this is not very extensible, and certainly doesn't satisfy the requirement to be able to extend the set of types that can be handled without cracking open the code, recompiling and redeploying.<br />
<br />
That is when dependency injection patterns came to mind. The idea is that an extension method is created to extend BaseType which will dynamically select a handler and call it.<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public static void Handle(this BaseType item)
{
var handler = DIContainer.Instance.GetType<IHandler>(item.GetType().ToString());
handler.Handle(item);
}
</code></pre>
<br />
and the loop is unchanged<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public void HandleValues(IEnumerable<BaseType> values)
{
foreach (var item in values)
{
item.Handle();
}
}
</code></pre>
<br />
The container I chose was MEF, as it allows very simple plugin extension. I define the handlers I a class library,drop this into a folder and simply point the MEF container to that folder to load all handlers. If i want to handle a new type, simply create the handler in a new assembly, drop the assembly into the correct folder, and the next time the app starts it will have the handler.<br />
<br />
To load the correct handler requires that each handler is attributed with the type it is to handle. Also, all handlers must either derive from a common type, or implement a common interface, I chose the second<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public interface IHandler
{
void Handle(BaseType item);
}
public interface IHandler<T>: IHandler where T:BaseType
{
void Handle(T item);
}
public abstract class BaseHandler<T> : IHandler<T> where T : BaseType
{
public void Handle(BaseType item)
{
Handle((T)item);
}
public abstract void Handle(T item);
}
</code></pre>
<br />
This takes things a little further, the generic interface IHandler<T> has a method that takes an object of the derived type,but this cannot be called directly as we only have an object reference by the BaseType. As such I created a non generic interface with a method that takes a BaseType as it parameter, then I created an abstract BaseHandler class that implements both the generic andnon generic interfaces. The implementation of the non generic method casts the item to the derived type,which it knows by virtue of its generic definition, then calls the generic method with the derived type instance. This approach allow me to create concrete handlers that dont need to performany casting and can deal directly with instances of the derived type they are designed to handle.<br />
<br />
an example of a handler would be<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public class DerivedType1Handler : BaseHandler<DerivedType1>
{
public override void Handle(DerivedType1 item)
{
//do the work
}
}
</code></pre>
<br />
While I have chosen MEF to implement the dynamic loading of the handlers and the selection of these, any number of other DI containers could be used. The use of MEF allows me to simply decorate the handlers with the type that they are to handle as their exported contract name:<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #cccccc; color: black; font-family: "arial"; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> [Export(contractName:"MyNamespace.DerivedType1", contractType:typeof(IHandler)]
public class DerivedType1Handler : BaseHandler<DerivedType1>
{
public override void Handle(DerivedType1 item)
{
//do the work
}
}
</code></pre>
<br />Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-45123551453291866192016-04-02T14:42:00.003-07:002016-04-02T14:45:59.727-07:00Composition with MEFI was working on a project recently where I was asked to write a nice standalone greenfield app. I thought 'brilliant, I can do this the way I like. Nice clean code, follow patterns I choose' etc,etc. Only restrictions were that I should follow the corporate model for things like logging, storage access and the corporate line was that as few third party dependencies should be introduced as possible.<br />
<br />
The application was a desktop app,so my choice was to use WPF, follow a MVVM pattern and obviously write the code following the SOLID principles. Well here came the first challenge. Dependency injection and inversion of control are things that were not practiced in the business that I was working in. This mean that I needed to introduce the concept. Eliminating 3rd party, and most specifically, open source technologies, I was left with a choice of MEF or possible Unity as my DI technology (or IOC container if you prefer to think of it that way).<br />
<br />
Now whilst I had used Unity in the past, I had not fond it the easiest to work with, but with the restrictions listed above it left me with only MEF as an alternative. Many moons ago I had investigated MEF,and I liked the concept, but at the time, having only used it in practice tutorials and never in a production system (we all remember the calculator example that was around when MEF first came to prominenece) I wasnt sure it was the best choice. Despite my reservations, I was sat next to a MEF evangelist, and I was convinced it was the way to go. Little did I know at that stage that it would prove an inspired choice.<br />
<br />
Soon I came to my first real challenge that MEF came to rescue me from. I mentioned above the corporate model for logging,this involved simply using a library written in-house that wraps log4net and adds some custom logic before logging. The pattern adopted to perform this was to instantiate a static member in each class of the system to perform any logging required by that class:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijctKqrlJ8osqp4lBlQe97prDcvbONYRYDZ3UE4bVqOLBk27LINLWUSvHHSivZPildDmTs2wqwUyH1w9P4Ful3YxWeqBXZJ1Iw8C2eemIzOexHxO8ttq3BQxrvQdhQPffdAUQzpmxQ9Tbt/s1600/staticconstruction.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="45" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijctKqrlJ8osqp4lBlQe97prDcvbONYRYDZ3UE4bVqOLBk27LINLWUSvHHSivZPildDmTs2wqwUyH1w9P4Ful3YxWeqBXZJ1Iw8C2eemIzOexHxO8ttq3BQxrvQdhQPffdAUQzpmxQ9Tbt/s400/staticconstruction.JPG" width="400" /></a></div>
<i><br /></i>
The problem was that I didnt want to instantiate it directly in this way. I wanted to compose each object using DI. I wanted to be able to unit test all classes without the need for concrete implementations of the dependencies of the class. That meant to me that I needed to only have dependencies of class that were defined by an interface. A simple contract between the objects. Assumptions would be made in the unit tests that all logic external to the class under test was flawless, and as such could be mocked from the interface alone using any given mocking framework.<br />
<br />
So the question was how do I do this when the logging library does not expose an interface, and furthermore the constructor of the logging class requires the type of the object that is to contain it. The first thing to do was extract an interface from the logging class. By luck there was in fact only one such class, so the extraction of an interface was very straightforward. And the organisation had adopted a policy of creating nuget packages for any shared libraries used in the organisation, so use of this new library wold be adopted for any project undergoing development.<br />
<br />
Next came the problem of injecting the logging object. How could I inject an instance of the logger class to the constructor of an object and know the type of the object in advance to construct the logger? I could configure my DI container with <i>n</i> logger instances, each constructed with a given type as its constructor argument for the <i>n</i> types I have in my system. This seemed stupid and a maintenance nightmare. I would need to change my DI container setup for each class I wrote. I cold create a new constructor for the logger class, a default one, and pass the type in via a method or property. But this wold involve the developer always remembering to perform this extra step of setting the type on the logger class. An unnecessary additional step that wold invariably be missed and lead to countless bugs.<br />
<br />
In steps MEF with it great range of possibilities, Not only does it allow you to inject objects via a constructor (or to a property, although I am not a fan of this. I feel if a class needs another object, this should be clearly conveyed to the developer by appearing in the constructor argument list. I prefer the external dependencies of my classes to be immutable. Property injection to me seems to result in a class that will do something that is not at all clear to the developer, almost like the person writing the class wants to hide something. As a side note it can be used to be round circular construction dependencies, but to me if you have such a circular dependency your design is wrong and you need to rethink your architecture at that low class level.),and back to my point, it allows you to inject anything to the constructor. You can inject a simple value type. <br />
<br />
You can inject a delegate, and that is where the solution to my problem came from. I chose to inject a function that takes a type as a parameter and returns an object in the form of a reference to the new logging interface:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBbCG3fwjviyO5N7E983aS-_iaNsWnmwovOhEcEhYMRjdGI72yMNpjhaoPD3sak74Sd8iYwaibw_d8A345XeEOdKiAv3MfwKRmmMCRwi6M0uLbicEW_PYd50gS55mbtkbi-JZ1U3h3MIBB/s1600/ImportConstructor.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="113" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBbCG3fwjviyO5N7E983aS-_iaNsWnmwovOhEcEhYMRjdGI72yMNpjhaoPD3sak74Sd8iYwaibw_d8A345XeEOdKiAv3MfwKRmmMCRwi6M0uLbicEW_PYd50gS55mbtkbi-JZ1U3h3MIBB/s400/ImportConstructor.JPG" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLLWgKw7B4V6r7dZh0kF9cQmNMwel1tU4rZ247jrija_W2FESQo_pNuT2VLe6DGEOyNj9k6GCkwjakUDnqMEM8jaaS_snl6ibKTsui5v8ZKCWNBsqB5en5KAKfR8Fzms1UErL1B5Ttdexp/s1600/ExportFactory.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="132" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhLLWgKw7B4V6r7dZh0kF9cQmNMwel1tU4rZ247jrija_W2FESQo_pNuT2VLe6DGEOyNj9k6GCkwjakUDnqMEM8jaaS_snl6ibKTsui5v8ZKCWNBsqB5en5KAKfR8Fzms1UErL1B5Ttdexp/s400/ExportFactory.JPG" width="400" /></a></div>
<br />
<i><br /></i>
But how would I set up my MEF container to have such an entity (for want of a better term)? WellI created a static class in the logging library that exports the function. This wold be picked up by the population of the MEF container, and injected into any class the requires it.<br />
<br />
This is nothing revolutionary in terms of how to construct an object. It is simply the factory pattern, here the factory is the function. But in terms of injecting an object that needs to know something about the target of the injection I feel it is very neat. If you want to inject a different implementation of the function, you could, if you want to introduce a new implementation of the logging interface this would simply involve changing the factory function. Clearly I have written this code in the simplest way,and to make it more extensible I might choose to pass more parameters to the factory function, which by changing it wold break any current use of it, however I don't foresee a need for this any time soon, and I believe in only building for what the requirements need now. I don't try to cater for every 'what if' that might rear up in the future as I could never think of them all, never mind code for them all.Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-57753849782329019752016-03-24T13:53:00.001-07:002016-03-24T13:53:43.698-07:00Writing quality code good for the soulI recently found myself without paid employment. Not a great situation at first thought, but it actually turned out to be a blessing. Having been made redundant I was forced to think about what I really want to do. <br />
<br />
I enjoy writing code and solving problems. Making things, you might say coding is my outlet for my creative side given my lack of artistic talent. The thing is I had spent years in the corporate machine, writing the code I was told to write, in the style I was told to, to do things that other people said it needed to do, the way they said it should do it. The lack of creative input was stifling. I wanted control. Don't we all. Without my own software house how could I do that? Well I couldn't. Not entirely, but I did have the power to only accept a job that felt right. <br />
<br />
I know from experience that what you expect of a job from the advert and interview is seldom the reality. This left me with the quandary of how to choose a role that would fulfill my needs. I couldn't guarantee that I would choose a role that delivered what it promised in reality, so the best decision was to not tie myself to a single role long term. As a result I have become a contract software developer.<br />
<br />
This leads me to my current role, one that on the face of it doesn't tick many of the boxes I was looking for. It's a role advertised as vb.net with vb6 developer, which after more investigation looked to be a job of migrating old applications that only run in Windows XP to work on Windows 8. Not a shiny sexy role, but a safe role. At the interview however things started to look better. Yes the legacy apps are vb.net (1.1 and 2.0) and vb6, but the technical vision was heading towards C# and .net 4.x. Also it was clear that the dev team, with the architect involved in the interview, had a good amount of control of the technical direction. It felt like the devs were valued as experts in their craft and opinions mattered.<br />
<br />
Due to business reasons the technology was a little mired in the past, but the appetite to move forward was clear amongst the devs.<br />
<br />
After a couple of months doing the necessary, taking a legacy app up to .net 4, and making the required changes to keep the app working in XP and now also in win8, I have the opportunity to work on a 'greenfield' project. It isn't a brand new app, rather a rewrite of an existing one, one that had bugs and required significant changes for the 'migration', so the decision to rewrite made sense.<br />
<br />
I have been given almost full control of technical decisions for the app, as a result I am writing it using WPF, with a MVVM structure. The control of technical decisions has also allowed me to develop the code in a fully TDD manner, with near 100% unit test coverage using mocking through Moq to isolate classes for testing. The code I have written adheres to the SOLID principles to an extent no code I have written before did. The system is built up using dependency injection, and the code coupling is minimal. All in all its a pleasure to work with. <br />
<br />
The thing I had in mind when I envisaged this post was that I now feel reinvigorated about coding. No longer is it a slog. When I add functionality, it goes in cleanly without making the existing code messy. When I refactor some sections I do it with confidence. I actually look forward to touching this codebase. It's amazing what clean code can mean, and I know better now than ever before why many clean coding evangelists preach so much about the pros of clean code. Its not just about a functional, maintainable system. Its about a functional, happy development team.Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-72358156662620735182015-07-13T06:28:00.004-07:002015-07-13T06:28:45.941-07:00Managing your APISo you have developed a brilliant new web based API, the world will be fighting to get access to it, its the most useful thing ever. But how do you manage access to it? How do you protect it, how do you monitor calls to it, how do you restrict access based on who the consumer is?<br />
<br />
Microsoft offer a solution to this as part of their Azure offering. This blog will look at what Microsoft Azure's API Management product offers.<br />
<br />
With APIM you can<br />
<br />
<ul>
<li>Ensure access to the API is only via APIM</li>
<li>Set throttling policies based on users/groups</li>
<li>Make users sign-up for access</li>
<li>Generate and publish developer documentation easily</li>
<li>Monitor traffic</li>
</ul>
<div>
And probably a whole lot more that I have not explored yet.</div>
<div>
<br /></div>
<div>
And all this can be done for multiple APIs all through a single endpoint. You simply need to have an azure account and create an APIM instance to get started.</div>
<div>
<br /></div>
<div>
I'll look at how to set up some of the things mentioned above, starting with establishing an APIM instance and pointing it to you already published API.</div>
<div>
<br /></div>
<div>
To create an APIM instance, go to the management portal for azure (I use the old style portal at https://manage.windowsazure.com/), select the APIM charm on the left</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglpvDGh3jgNAJsBxG224pSLz3GxPZYwWIGcBVC_msGXkeBwtoTp3NnrhcMwKG1W66Ruva2Rpu5I5sk22ZZ45H6-X81qo7MOJFA1e1d-w2ajloRyLa_p_-fUk3RcnIfzgQ-ehIvrvZ6Gx1V/s1600/APIM.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglpvDGh3jgNAJsBxG224pSLz3GxPZYwWIGcBVC_msGXkeBwtoTp3NnrhcMwKG1W66Ruva2Rpu5I5sk22ZZ45H6-X81qo7MOJFA1e1d-w2ajloRyLa_p_-fUk3RcnIfzgQ-ehIvrvZ6Gx1V/s1600/APIM.PNG" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
You will be presented with a list of your instances (but I guess you dont have any), and in the bottom left is the new button to create your first instance. Clicking this will open a banner with the options of what you want to create, there are no options, so just click create in third column</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGKQIMFhaOafo3L7J0aEA2ZdwiSedoC633HYfj16vscCr0Sg5H_hTFVlF4ZG1ByJFokKa3CHgvdrSvOAY1O6_2WU0iaI-GEEj9ehgzDREAmROGz-GH-bIP19A3PXIUVyjv4mtd05VOAFqE/s1600/APIM+create.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="265" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGKQIMFhaOafo3L7J0aEA2ZdwiSedoC633HYfj16vscCr0Sg5H_hTFVlF4ZG1ByJFokKa3CHgvdrSvOAY1O6_2WU0iaI-GEEj9ehgzDREAmROGz-GH-bIP19A3PXIUVyjv4mtd05VOAFqE/s400/APIM+create.PNG" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
You will be asked for a URL, this is the endpoint that the world will use to gain access to your API, and a few other questions about where the instance will be hosted, who you are and your contact details, and if you select the advanced settings you can select a pricing tier, but that as the name suggests is an advanced topic, and this is a basic intro blog, so leave the pricing tier as Developer for now.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
After that it will go away and spin up an instance for you which will take some time. At the end of this the list of APIM instances will be populated with a grand total of 1!!!</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Next select the name of your new APIM instance and you will be presented with the quick start page, the only page that I honestly use within the azure portal in relation to the APIM instance.</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYjtkyPs_bO2MmyFhbD-vPBpkFfC_LYwfdwdL0nTmtCqly7qR6nmT0HG0OnhQLUXBAb2NtkpQYiKBn-cep_UNiFMtrLvz1YIjlKeEpbz1S9D_Ya_gBeyiqZG8Ya9m7iVIfvCFUcrNNGeYK/s1600/quickstart.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="249" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYjtkyPs_bO2MmyFhbD-vPBpkFfC_LYwfdwdL0nTmtCqly7qR6nmT0HG0OnhQLUXBAb2NtkpQYiKBn-cep_UNiFMtrLvz1YIjlKeEpbz1S9D_Ya_gBeyiqZG8Ya9m7iVIfvCFUcrNNGeYK/s320/quickstart.PNG" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div>
The 'Publisher Portal' is the gateway to everything to do with the APIM instance. And out of interest the 'Developer Portal' is what the public will see if you tell them where to look for information on you API, i.e. where you can provide documentation.</div>
<div>
<br /></div>
<div>
To finish setting up your vanilla first APIM instance, go into the publisher portal and you will be presented with a dashboard with not a lot of data</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqCtLmlDb7tEoMD9TD5Ffjep-BUEp_yhGXO4E00vws00cteXegS22CeRJ8IMrslY9kd02g7D3dS8AB-RqNKr1KUygO5j01FYf3Fa41rmayPpQ7cfkvluBLZnRGMtZ__vNMqvZP2_7yVxcx/s1600/ppDashboard.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="306" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqCtLmlDb7tEoMD9TD5Ffjep-BUEp_yhGXO4E00vws00cteXegS22CeRJ8IMrslY9kd02g7D3dS8AB-RqNKr1KUygO5j01FYf3Fa41rmayPpQ7cfkvluBLZnRGMtZ__vNMqvZP2_7yVxcx/s320/ppDashboard.PNG" width="320" /></a></div>
<div>
The next thing to do is connect APIM to you existing API, which you do by clicking the add API button, and providing the details required. You need a name for the API, the URL for the API you made earllier, a suffix which will be used to distinguish between the APIs associated with the APIM instance and provide whether you want the access to the API (via APIM) to be allowed with or without TLS.</div>
<div>
<br /></div>
<div>
There are stil two steps to go. firstly defining the operations that can be accessed via the API. I.e. what verbs are allowed and for what URLs. Add operations such a GET users using the intuitive dialog</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCAbV1UHQcDk8IRFOa9GGkqQ-RzwqJrbt2JWkOyzIS_jmLyqhevVqpAQmf9b2KdIahGBfd-yNQymYxQ3vmIGtjOoenwbSNaMRrQf4xVXw-E_Z86sC_d7466O4GmOrmmi3hk-lhKqat26tD/s1600/operation.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="235" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCAbV1UHQcDk8IRFOa9GGkqQ-RzwqJrbt2JWkOyzIS_jmLyqhevVqpAQmf9b2KdIahGBfd-yNQymYxQ3vmIGtjOoenwbSNaMRrQf4xVXw-E_Z86sC_d7466O4GmOrmmi3hk-lhKqat26tD/s320/operation.PNG" width="320" /></a></div>
<div>
and finally you need to associate a product with the API. Products are in my opinion badly named. They represent levels of subscription to the API. By default 2 products are preconfigured, Starter and Unlimited, you can associate these or any other products with the API using the Products tab on the right of the screen.</div>
<div>
<br /></div>
<div>
After this your new APIM is ready to go.</div>
<div>
<br /></div>
<div>
The next thing you may wish to do is add a throttling policy to the API (or more specifically the product). You do this by selecting the policies option on the menu in the left, pick the combination of options you want to set the policy for (product, api and operation) and click the add policy option in the text box below. This will add a blank policy, and you can add the details of the policy using the code snippets on the right, so for a simple throttling policy select the limit call rate option, and this will add code to set a limit on the number of calls within a given time window. By default the starter product is limited to 5 calls in any 60 seconds, and 100 calls in a week</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnuW7KJ4uPjvOGJzGCe_F2mqJbaxAIuwG2MDKDkrPl1xMlfxlX-_scANy3f2iomW_HLIULCbsfvvA5O1vN51nxJYuaX4541D5ExQ2XTL9opV_H5xVCzQh3CEIFitqKoDVyy2km74EQiB9R/s1600/policies.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="123" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnuW7KJ4uPjvOGJzGCe_F2mqJbaxAIuwG2MDKDkrPl1xMlfxlX-_scANy3f2iomW_HLIULCbsfvvA5O1vN51nxJYuaX4541D5ExQ2XTL9opV_H5xVCzQh3CEIFitqKoDVyy2km74EQiB9R/s400/policies.PNG" width="400" /></a></div>
<div>
This gives you a flavour of what can be controlled with this.</div>
<div>
<br /></div>
<div>
The use of the products and policies in conjunction allows you to fine grain the access to the API and its operations in a way that is best fitted to you and your users.</div>
<div>
<br /></div>
<div>
The next thing I would look at is securing your API so that the rules set up in APIM must be followed. If people can go to the API directly, bypassing APIM, then these policies and rules are meaningless.</div>
<div>
<br /></div>
<div>
The simplest way to do this is to use mutual certificates between APIM and your API, and add code into your API to ensure that all requests have come from a source with that certificate. This can be done by going to the security tab on the API section of the publisher portal</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9riUmmpiswq-GBLr4p7kLkfNjaeU4nN8WeKbG6xCEz5n3mZEwOT-JFak-CiAIVdv2NZhxeWRgsCzzxlYkdbAINqRCCtile779rslhWwAtVIDsgduqYVArUIvgZko27Jq8UpK2oOqeOvHy/s1600/security.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="187" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9riUmmpiswq-GBLr4p7kLkfNjaeU4nN8WeKbG6xCEz5n3mZEwOT-JFak-CiAIVdv2NZhxeWRgsCzzxlYkdbAINqRCCtile779rslhWwAtVIDsgduqYVArUIvgZko27Jq8UpK2oOqeOvHy/s400/security.PNG" width="400" /></a></div>
<div>
then pick the mutual certificates option in the drop down. You will need to upload the certificate to the APIM instance, which can be done by clicking the manage certificates button. In terms of ensuring the request has come from a trusted source, that is a coding question, but for completeness, within a c# MVC webAPI system, add a message handler to the pipeline for the incoming requests by editing the WebAPIConfig class</div>
<div>
<br /></div>
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public static class WebApiConfig
{
public static void Register(HttpConfiguration config)
{
// require client certificate on all calls
config.MessageHandlers.Add(new ClientCertificateMessageHandler());
}
}
</code></pre>
And then add a class to check the certificate:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public class ClientCertificateMessageHandler : DelegatingHandler
{
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
bool isValid = false;
X509Certificate2 cert = request.GetClientCertificate();
if (cert != null)
{
if (cert.Thumbprint.Equals(RoleEnvironment.GetConfigurationSettingValue("ClientCertThumbprint1"), StringComparison.InvariantCultureIgnoreCase)
|| cert.Thumbprint.Equals(RoleEnvironment.GetConfigurationSettingValue("ClientCertThumbprint2"), StringComparison.InvariantCultureIgnoreCase))
{
isValid = true;
}
}
if (!isValid)
{
throw new HttpResponseException(request.CreateResponse(HttpStatusCode.Forbidden));
}
return base.SendAsync(request, cancellationToken);
}
}
</code></pre>
This allows you to define two certificate thumbprints in the web.config of the API to compare against the incoming request. All requests via APIM will include the certificate that you upload in the publisher portal, so if you ensure that this cert is not made public, then you can be assured that all requests hitting the API have come through APIM.<br />
<br />
I mentioned making users sign up for access. Well if they want to use your API we have just ensured tha their requests must be directed via APIM. The rest is built in. The products that you configured earlier have associated with them a subscription key that the consumer of the API must supply with every request. This ensures that every consumer of the API must have the subscription key. The developer portal provides a way to subscribe to your APIM and thus get a key. You can if you wanted restrict this so that you need to manually give access to every subscriber before they get a key, and that way you could monetize the access, but that is way beyond the scope of this article. Suffice to say they need to sign up to get the key or APIM will reject the requests.<br />
<br />
The documentation aspect of APIM is probably something best explored yourself, I'm a techy and not best placed to explain this, however in summary you can document each operation on the page where the operation is created/configured, and the content of the description can take the form of html/javascript so you may wish to use some client side script to retrieve the documentation and manage the content externally.<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <div id="dynamicContent"></div>
<script src="https://myserver/scripts/documentation-ang.js"> </script>
</code></pre>
would call the javascript file which could provide the content required based on some data scraped from the rendered page.<br />
<br />
The final thing to look at is the analysis of traffic. Good news, you don't need to do a thing to configure this. Basic analytics are provided out of the box. If you require anything more complex than APIM offers for free you may wish to look at other products that could be bolted on to the system, but in general the data that is captured will provide you with a lot.<br />
<br />
So in summary to that all, APIM offers an easy to set up and easy to configure interface to your API when you want to publish it to the world. It gives easy access to some very good security features and access control. There are many platforms out there that will host an API for you, but if you want to host you own somewhere, APIM offers a lot of what you need to consider before exposing your delicate little flower to the horror of the web.<br />
<br />
Any questions or comments are welcome, and I will try to answer anything you ask. And please share this with everyone you know, and even those you don't know. Spread the word, the cloud is fluffy and nice!Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-24785735040422324752015-07-06T01:31:00.002-07:002015-07-06T02:46:43.009-07:00Dedication, Motivation and Achieving It has been said many times that to become an expert in anything you need to practice for 10,000 hours. This theory has been professed for sporting prowess (see the book <a href="http://www.amazon.co.uk/gp/product/B003P2WJ18?keywords=bounce&qid=1436167778&ref_=sr_1_1&sr=8-1">Bounce by Matthew Syed</a>) and for technical skills (see <a href="https://www.youtube.com/watch?v=v-60HSFWaPo">John Sonmez's podcast</a>) and many people argue against it.<br />
<br />
I see the logic behind the idea, if you do something consistently for that long you will surely iron out a lot of the problems you have with it and become better. You may not become the best, there may be some innate talent involved that you do not posses. For sporting prowess, anatomically you may not be able to surpass some other people. But practice anything for 10,000 hours and you will become a whole lot better than you are right now. One important point is that you probably do need some guidance (training) during these 10,000 hours, otherwise you may develop bad habits that will hold you back and will be difficult to break once they are deeply ingrained.<br />
<br />
So coming to the technical skills of programming, Sonmez argues that practicing, repeatedly trying to build systems and applications, whilst trying to extend you skill set is simply a matter of perseverance and habit. That you must get yourself into the habit of practicing (coding for instance) on a regular basis (daily?), and that motivation and drive have little to do with it. I disagree with this.<br />
<br />
In order to be able to commit and persevere with the practice you need some form of motivation, something to keep you going. The simplest thing that motivates most people is greed. Sounds dirty, but its true. We all live at a level beyond necessity (materially speaking) so we need money to do this, we may not all strive for more more more, but living above the level of having food and shelter could be perceived as greed (I'm not saying we are all greedy, but we all like comfort and the nice things that are accepted as normal in modern living, and these all cost money. So we all look to have an income, and as programmers/developers/coders writing code is a way of getting this. So that is a base motivation achieved. But how does that equate to practicing skills and getting better? Most of us work for someone else, we have fairly clear objectives of what our employer needs us to deliver in order to maintain that employment and be paid. This is often not enough to drive us to get better. We can drift along, doing enough to get by without getting better. 10,000 hours of this will not get you very far.<br />
<br />
So how do you get better? What form will the 10,000 hours of practice take? Well you could potentially use this employment time, if you use it wisely. This is easy when you are starting out, all of your time will be spent on new things and so will be useful, but give it a year or so and your day-to-day work will stop stretching you.<br />
<br />
10,000 hours equates to 1,250 8 hour days, or nearly 5 years of normal employment (8 hour days, 260 days per year). So it is not a quick thing. And if only about the 1st year will really stretch you and be useful practice then these 5 years will not actually amount to the 10,000 hours. So how do you build it up?<br />
<br />
Side projects, personal learning, pushing your employer to embrace new ideas and technologies. That is how. If the project you are working on is stagnating, look for how it could be moved forward to the benefit of your employer and push for them to do it. It will benefit you from a skills and proficiency perspective, and benefit them where they like it, in their wallet.<br />
<br />
But for side projects and personal learning, the problem is often one of motivation, drive and time. How do you drive yourself to put the time in? You have just worked a 40 hour week for the man, where is the motivation to put in more time on top of this? Yes if you put in the hard yards you will see a payback eventually, but 10,000 hours is a long time, and we are simple beasts, we want dividends now. Something perceivable and measurable. That is where the motivation aspect comes in. That is the motivation.<br />
<br />
You need some sort of short term perceivable and measurable metric of your progress and success. Look for what it is in programming that you find most rewarding. It could be a wider question than that, what do you find rewarding in life? Personally I like the feeling of seeing a task completed. The sense of satisfaction that something I set out to do happened and I succeeded in completing it. I love hiking, and specifically hiking up mountains, and the greatest sense of achievement in this is to get to the summit. But reaching a high summit can be a hard slog. It might only take 5 hours rather than 10,000, but physically I still feel that fatigue in reaching my final goal en route. What I do is to set intermediate goals along the way. The next 3 miles, the next false summit, where I want to be before 11am. Anything to give me a sense that I am doing well and on track for my final goal, the little pile of stones that marks the top.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZBBKnsCsfOtTE78MAoA36UpwqMbc3N2UeWHr5X4Zpk8KLfs_855lxDikoMMdcz4CmzS1s8UWqQmGaxBVU2p_vz6x4IM_p3KblPKGmyCmiZ4q58VEP_Qw-_mGrWzVZeXRoDHXXUEq-5Jje/s1600/summit+cairn.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="220" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZBBKnsCsfOtTE78MAoA36UpwqMbc3N2UeWHr5X4Zpk8KLfs_855lxDikoMMdcz4CmzS1s8UWqQmGaxBVU2p_vz6x4IM_p3KblPKGmyCmiZ4q58VEP_Qw-_mGrWzVZeXRoDHXXUEq-5Jje/s320/summit+cairn.jpg" width="320" /></a></div>
In motivating myself to do some personal development it is the same. I set short term goals that will get me to the big prize. The big prize is the 10,000 hours mark? No its becoming what you want to be personally. Pinning what that is is difficult, and right now I can't easily express what that is, but I can set short term goals, and the final goal will come into sight at some point.<br />
<br />
For side projects this short term goal setting is easy (although the side project itself should probably be one of your short term goals, but breaking this down further is a good idea) try to work in an Agile manner within the project. It might be a personal project, with a team of 1, but breaking the development down into short iterations, with a goal of delivering a functioning system at the end of each iteration is a very valid approach still.<br />
<br />
If you do this, and your psyche works anything like mine, then at the end of each iteration you will feel a sense of achievement and that will help you to motivate yourself for your next iteration. You will surprise yourself by how easy it becomes to motivate yourself to complete the tasks of the iteration, because you will want that sense of achievement at the end of the iteration, and quickly the side project will take shape and maybe start to deliver on that base motivation, hard cash! The great side benefit is that you will push yourself towards that 10,000 hours mark, which is a notional mark to indicate you are becoming a much better developer.<br />
<br />
Motivation is important, its not just about turning the wheel, but you must find your own motivation by setting achievable milestones and rewarding yourself in some way that helps to keep you focused on the next step, the next step and eventually the big prize.Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-66331462550865846422015-06-15T12:00:00.000-07:002015-06-15T12:00:03.364-07:00Can Adding an API Save Your System?Creating an api as a publicly accessible interface to your system can actually make your system better than it currently is!<br />
<br />
A bold claim I know, but bear with me and I will try to justify this claim. Lets begin by imagining a large system which has evolved over time into something your customers like. It fits the purpose, it earns you money, it continues to grow and evolve. All sounds well. Now look behind the curtain, in the box, under the surface. Those little duck legs are paddling like mad just to keep the serene appearance on the surface. Your system has evolved into something if a mess beneath the surface. You're not at fault, its a fact of any software systems that over time commercial pressures mean that technical debt accrues and the system architecture is corrupted into a mess. Often UI and business logic boundaries are blurred, code duplication becomes rife, maintainability is generally reduced.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheckiZ_UA9eZv7y_3xeidWli3XevZ9fnZd945sZjSCZLqLsIKufo_6H-KviPqtQ1OEQp4ggbSXPqsuHZBMTlyK_t5-YqOfdW-7qGfH78KZb0jzt9aroi-pyauCkIK3FxLv2O_3hm4eQHcA/s1600/mixedlayers.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="160" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheckiZ_UA9eZv7y_3xeidWli3XevZ9fnZd945sZjSCZLqLsIKufo_6H-KviPqtQ1OEQp4ggbSXPqsuHZBMTlyK_t5-YqOfdW-7qGfH78KZb0jzt9aroi-pyauCkIK3FxLv2O_3hm4eQHcA/s400/mixedlayers.PNG" width="400" /></a></div>
<br />
<br />
A nice clean architecture would have clear separation between the layers of the system in terms of the responsibilities, but more importantly in terms of the code base. In reality many systems blur these boundaries over time and code that should be in a business layer may be forced into the UI layer by time and commercial pressures. For a system with one UI this does not surface as a problem, but consider adding a second UI, quickly the business rules that you think are robust and well tested are being violated. How come? Well they were implemented in the previously sole UI, but are now being bypassed by the new UI. This highlights the build up of technical debt that putting the code in the wrong place causes. But as I say, you could live with this happily for years unless you introduce a new UI.<br />
<br />
In a clean system the abstraction between layers should be such that any layer could be replaced an the system still function. If there is an overlap in responsibilities between layers this is not so straightforward.<br />
<br />
Given the evolution of the technological landscape to a much more mobile, flexible one, with the desire to access everything from everywhere, there is an obvious drive towards supporting multiple interfaces to a system to accommodate this. Take a system that was written for a standard office of the past. It may have a big thick client with maybe just a shared database behind, or a thickish client with a server side engine that does the grunt work and a central database. To evolve either of these systems a web front end may be produced utilizing the same data. Where a server side engine existed this may be reused for consistency of functionality and minimal effort to create the new UI. If however any of the business logic existed in the client this will need to be replicated in the new web service. If we extend further and add mobile applications the logic will need to be placed in these too. And what about integrations with third party systems? Where does the logic sit there? We need to contain all the logic in a common business layer, just as our clean system architecture planned.<br />
<br />
This is a problem I have seen many times over the years, and often when creating this one new UI the decision was taken to do the 'easy' thing at the time and duplicate the logic. Often this resulted in the logic being implemented inconsistently in the new UI. And worse, any bug found in either flavour of the system would only be fixed in that one flavour. <br />
<br />
Recently I have been working on the addition of a rich RESTful API for a mature system, and where we find a hole in the business logic due to the API bypassing all UI layers the decision has been taken to do the right thing. Move the business logic into the business logic layer so all UIs have the same logic implemented by the common layer below.<br />
<br />
All this sounds like a lot of bad coding has happened in the past, and that bad decisions have been made as to where to put code logic. But this is not the case in reality. Image a form in a system that allows you to enter data. A numeric field is entered, and the form restricts the format of the data. E.g. non negative, with enforced bounds and precision. The business layer of the system may well be the place that all the data validation is being performed, but what if some of the boundary conditions that should be guarded against are missed when writing this validation code? No-one made the decision to miss this validation. The testers thought of the boundary cases and tested them. The system held up robustly as the UI did not allow the user to enter the invalid data. But if the UI had not enforced these restriction then the invalid data may have gotten through. There was no way to test this. We thought the system was robust, and it was, but only via the single UI that was available to the testers to explore/exploit the system. <br />
<br />
If in the scenario of the ever evolving technological landscape above we add a second UI, for whatever reason these restrictions may not be possible and the system becomes just that bit more fragile.<br />
<br />
With an API, we can decide to implement no restrictions (other than type) on the data input and thus force the (common) business layer to take the responsibility for all the data validation.<br />
<br />
The decision made to do this on the system I am currently writing the API for means that the API will be a little slower to produce, but more importantly the overall system will end up in a much better state of technical health. And the biggest benefit from this is that if a new UI is needed in the future that maybe does not even use the API but communicates directly with the business layer, we can be confident that the logic will be intact and robust.<br />
<br />
So the addition of an API will not directly save your system, but can give you confidence that the system is fit and healthy enough to evolve further into a rapidly changing world that may put ever more challenging requirements on the core of the system.Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-31320747737613614652015-06-02T03:39:00.001-07:002015-06-08T04:08:46.221-07:00OAuth in WinformsThe junior dev I am mentoring at my job was recently given the task of extending his knowledge of APIs and asked to demonstrate his newly gained knowledge by producing an application that consumes a third party API. More specifically a RESTful API.<br />
<br />
In itself this does not seem the most taxing of tasks, but bear in mind that the junior dev has no web development experience, his only dev exposure in his current role has been on a winforms application. So to make this a targeted task, aimed at learning about RESTful APIs, it was decided a simple winforms application using the 4 main verbs would demonstrate sufficient understanding.<br />
<br />
This however did raise a question, how to interact with an OAuth provider from a winforms application. This should be a simple matter, but it is something that is not well documented, especially in the documentation of most of the more famous APIs. There are plenty of tutorials for how to authenticate with an OAuth provider from a web site, and most of the APIs the junior dev looked at provided their own OAuth.<br />
<br />
The final choice of the API to consume was Instagram, which provides great documentation for its OAuth when being consumed in a web site, but nothing for Winforms. This is not surprising, Winforms is an old technology, not something that you would expect to be used with a service like Intstagram, but why not? It should be possible (and is). But it is understandable that Instagram have not invested time in providing detailed documentation on how to do this. So here we go on how it was accomplished:<br />
<br />
Firstly, the method of validating the user's claim of access to Instagram is via a web page hosted by Instagram. The documentation states<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfpp8EZqtd4H0rapQJzH49GfioZbXEZNYWVHAAsQj5tM3r_RzSmxYprtVazwiQ2dJEFep_vy5pDQtCdNeLlbmTby87qnQNEzKlMNWnna2hxHVtQLSGyaZaWu3pW4v-sQUKMyxF50fcW7fu/s1600/Auth.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="105" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfpp8EZqtd4H0rapQJzH49GfioZbXEZNYWVHAAsQj5tM3r_RzSmxYprtVazwiQ2dJEFep_vy5pDQtCdNeLlbmTby87qnQNEzKlMNWnna2hxHVtQLSGyaZaWu3pW4v-sQUKMyxF50fcW7fu/s640/Auth.PNG" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
which is fairly straightforward in a web app, but how do you do this in a winform application?</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
The answer is to host a web browser control within your application which will display the url above and be redirected upon completion of the authorization process. We found some code with a quick trawl of the search engines to perform this action in a pop up window:</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> string authorizationCode = StartTaskAsSTAThread(() => RunWebBrowserFormAndGetCode()).Result;</code></pre>
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">
private static Task<T> StartTaskAsSTAThread<T>(Func<T> taskFunc)
{
TaskCompletionSource<T> tcs = new TaskCompletionSource<T>();
Thread thread = new Thread(() =>
{
try
{
tcs.SetResult(taskFunc());
}
catch (Exception e)
{
tcs.SetException(e);
}
});
thread.SetApartmentState(ApartmentState.STA);
thread.Start();
return tcs.Task;
}
private static string RunWebBrowserFormAndGetCode()
{
Form webBrowserForm = new Form();
WebBrowser webBrowser = new WebBrowser();
webBrowser.Dock = DockStyle.Fill;
var uri = new Uri(@"https://api.instagram.com/oauth/authorize/?client_id=CLIENT_ID&redirect_uri=REDIRECT_URI&response_type=code");
webBrowser.Url = uri;
webBrowserForm.Controls.Add(webBrowser);
string code = null;
WebBrowserDocumentCompletedEventHandler documentCompletedHandler = (s, e1) =>
{
string[] parts = webBrowser.Url.Query.Split(new char[] { '?', '&' });
foreach (string part in parts)
{
if (part.StartsWith("code="))
{
code = part.Split('=')[1];
webBrowserForm.Close();
}
else if (part.StartsWith("error="))
{
Debug.WriteLine("error");
}
}
};
webBrowser.DocumentCompleted += documentCompletedHandler;
Application.Run(webBrowserForm);
webBrowser.DocumentCompleted -= documentCompletedHandler;
return code;
}
</code></pre>
<div class="separator" style="clear: both; text-align: left;">
which gets you the code included in the redirect URL. The CLIENT_ID you need to get from Instagram, register an application with them and it will be provided, and the REDIRECT_URL must match, but the location is unimportant, as the web browser will close on completion it will not be seen.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
There is still one problem, when using Fiddler to inspect the successful API calls from the test bed of Apigee the access token has three parts period delimited, the user_id, an unknown part, and the code returned from the OAuth authentication stage. This is not well documented, and at this stage we are unable to generate the full access token.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
All this highlights that the documentation of even well established APIs can at times be lacking for non-standard development paths.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-33641674416598095602015-05-26T02:25:00.000-07:002015-06-02T03:36:58.099-07:00Consuming Free APIsMy day job has recently thrown up a nice new challenge, producing a public API for the company's core system to enable simpler access to the ecosystem for 3rd party developers. Historically 3rd party developers have been able to interface with and customize the system by consuming an SDK, but that was in the days of an on premise installation of the system. We are pushing towards a multi-tennented, single instance, cloud based offering, and this cannot be extended or customized by the use of an SDK. How could the multiple customers using the system each have their own extensions and customizations, purchased from 3rd parties, when there is only a single instance of the service centrally hosted on the cloud?<BR><BR>As a result the decision was made to replace the SDK with a RESTful API which allows access to the data of the individual customer and thus multiple extensions could all coexist, simply accessing the system via the new API.<BR><BR>This new track of development got me thinking. I had produced APIs in the past, non-web based APIs (using such legacy technologies as COM+ among others), and web based APIs, one using a SignalR interface performing a large amount of business logic and data persistence. However, none of these APIs had been produced entirely as a public API for consumption by 3rd parties. All had a target consuming application, be it a cooperating 3rd party (with the potential to resale to other customers) or an internal partner dev team in the case of the SignalR interfaced system. These systems were developed as APIs as a way of separating concerns, allowing the multiple teams to develop in isolation, concentrating on their own area of expertise, with an agreed interface. The key advantage of an API.<BR><BR>So having not really thought in depth about the creation of an API for general public use, equally I had not thought about this from the other side, consuming a publicly accessible API produced by someone else who I have no professional relationship with.<BR><BR><IMG src="http://www.artofclick.com/wp-content/uploads/2014/07/mobile-apps.jpg"><BR><BR>I have for some time been toying with the idea of producing a mobile app, but have not yet come up with an idea of something I want to create for the public to see. I am of the opinion that creating something rather than nothing is a better idea, be it imperfect or not, I should simply create something and release it, and that is what I will do. This is a way of thinking that is a little new to me, previously I was of the opinion that if it was not a 'great' idea there was no point in trying it. But <A href="http://simpleprogrammer.com/">John Sonmez's</A> blog post on the subject of simply <A href="http://simpleprogrammer.com/2015/04/08/quantity-versus-quality-is-an-illusion/">creating something regularly</A> is better than sitting on an idea until a perfect idea comes along.<BR><BR>So where do I start? Years ago I thought that indie games would be the best approach, I still think this would be a good thing to do, potentially very profitable, but an approach that could take a lot of effort to get something out. And more importantly something I have little experience in, particularly the design and aesthitics aspect. If I want to create something fairly rapidly in my spare time (which is scarce) I need to find a short-cut. This is where the use of APIs come in.<BR><BR>There are thousands of publicly accessible, freely accessible, APIS out there. Web sites like <A href="https://www.mashape.com/">https://www.mashape.com</A> <A href="http://www.programmableweb.com/">http://www.programmableweb.com/</A> and <A href="http://freeapi.net/">http://freeapi.net/</A> offer a range of APIs, and are just the tip of the iceberg for what is available. Creating a mobile application that consumes one of these APIs should be fairly straightforward, and as I have no great ideas of my own, my plan is to look through the lists of readily available APIs, and try to come up with an idea of how to present its functionality in a new way within a mobile app.<BR><BR>This approach has the massive advantage that someone else has done all the hard work of creating the API, hopefully thought through all the logic and created something that will do what is claims to do. So if I can present a good human interface to use such an API, then I will have a useful app, and something that the public may want to consume.Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-62717953826611525592015-05-18T06:07:00.000-07:002015-05-18T06:07:47.940-07:00The Wrong Path?I read an email post today regarding the benefits of doing something rather than nothing, even if it might be the wrong thing to do. This post by <a href="http://simpleprogrammer.com/">John Sonmez</a> uses the quote<br />
<blockquote class="tr_bq" style="background-color: white; font-family: Helvetica, Arial, sans-serif; font-size: 20px; margin: 0px 0px 15px; padding: 0px;">
<strong>Sometimes you can only discover the right path by taking the wrong path 50 times.</strong></blockquote>
This struck a real chord with me on a number of levels. I have a current desire to enter into the world of mobile apps, the problem here being that I need and idea for an app to create. I have written at some length here about the use of genetic algorithms to solve difficult mathematical problems, an approach based entirely in exploring the 'wrong path'. I started a SaaS project with some friends a while back that has stalled somewhat due to a number of reasons, not least a reluctance to expose the beta version to wider public scrutiny. And lastly, in my day job I am working on a public API to the companies core system. This is a new venture, and we are finding ourselves galloping down the wrong path on a daily basis, and returning to try the left fork rather than the right very quickly. It has been a very refreshing approach from a software house that is steeped in protracted design and analysis phases before development. An approach that is producing some considerable discomfort in a majority of the dev team (testers included), but one that I am relishing.<br />
<br />
<img height="265" src="https://patcegan.files.wordpress.com/2014/03/fork-in-the-road.jpg" width="400" /><br />
<br />
A big part of the problem is that we are attempting to created an automated testing suite as we develop, a great idea in itself, however the testers who are defining the content of this test suite are struggling a little with the idea of the expectations of the system changing faster than the tests are being produced.<br />
<br />
For example, when requesting the list of entities related to another entity using an http get request with the url /entityA/{id}/entityB to get all the instances of entityB which has an entityA with id = {id}, if there is no entityA with id ={id} the testers argue that the system should return an error, most likely a 404 not found error. This assumption was backed up with the rationale that if the request /entityA/{id} was used then a 404 error would be returned. In contrast if entityA with id = {id} does exist, but has no entityB children, then an empty list would be valid.<br />
<br />
A compromise was made that for the example we looked at, it was not valid to have an entityA with no entityB children, so if an empty list was retrieved from the database, then a 404 would be returned. The developers were happy, as we could simply return a 404 for 'routed' get all requests resulting in a zero length list, successful blanks lists for 'direct' get all requests (/entityB/) and 404 for a direct get single (/entityA/{id}) with no match.<br />
<br />
This meant that no validation of {id} was required, so a single database query will give all the information we need.<br />
<br />
A couple of days later this became more complex. An example was found with entityC and entityD, where entityC could exist with no children of type entityD, so the request /entityC/{id}/entityD could produce no results for two situations, entityC with id = {id} may not exist, or it may exist with no children. The code as it stood returned a successful result with a blank list in both circumstances. The testers did not like this as we had made the decision to go this way with analysis of only the A,B case, without consideration of the C,D case.<br />
<br />
The final decision of what we will do is yet to be made, however we have a system that supports the query of both A,B and C,D working right now. If we had started analyzing all the possible combinations of entity types in the system (100s) before we decided on a way to go for any routed queries we would have nothing for a good month.<br />
<br />
We may have taken the wrong path, the testers would say that we have, we may have chosen the right one, the architects are arguing that side based on reducing database hits to a minimum (checking if entityA with id = {id} would require another db call after we know that the full query returns nothing, and for more complex queries with multiple routings this may amount to a significant increase in DB traffic and therefore cost as its a cloud based DB).<br />
<br />
Whether this will get us to Eldorado in the end only time will tell. We are due to engage early with a consumer of the API, and their preference on what we return will be the deciding factor, but at least we are in a position very early to show them something.Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-39908438740016226002015-04-22T06:58:00.000-07:002015-07-07T07:06:16.583-07:00Never Lose an IdeaIt may be a bit off topic for this blog, but its something that has plagued me so I thought I would document some things I have done to help myself. I, like many a dev, am forever looking for the next great idea that I cant develop to potentially create a secondary source of income, and the dream is this secondary source become great enough to stop the primary source of working for someone else.<br />
<br />
The problem is getting the idea. Well we all have ideas, the <a href="http://www.efficientprogrammer.com/the-only-way-to-come-up-with-ideas-tonnes-of-them/">efficient programmer blog</a> talks about the fact we can all come up with ideas, but generally it does not happen when we sit down and try to have an idea. This was the case for me last week, I had an idea for an SaaS system whilst in the car on the commute home. Fortunately I was able to explore the idea, and it turned out not to be a particularly viable idea as someone already has a system offering the same service very well at a cheap price. But what if I had not been in a position to explore the idea straight away? What if I had been too busy to use the idea right away? What if I had been on a roll and came up with too many ideas to deal with at once. How could I keep track of them all? How could I organise them for a later date? Well the efficient programmer blog offered a good solution to this, and I will look to document my progress through implementing this for myself.<br />
<br />
The system they proposed is the use of <a href="http://trello.com/">Trello</a> as a repository for the ideas, and the use of and orchestration system to forward emails into the Trello board. I am looking to implement a simialr system, using only Trello with no orchestration software to set up.<br />
<br />
I created 3 boards in Trello, one to contain ideas for content matter (blogs, videos etc) one for mobile app ideas, and one for SaaS system ideas. On each of these boards I added 5 lists: Ideas; To Develop; In Progress; Done; and Rejected. The plan being that new ideas go into the first. Once I have done a little background investigation and decided its a feasible idea it will move to the second. When work begins for real, into the third, and when something goes live the card goes to Done. Anything noted as not feasible goes into the Rejected list.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgacBzFu1e-iFF8oHQ6gyueHPYklnc0GdFLWg5MnBUef7J4O3CWJnJ2On-PDJHpoOQ78vEM9LoQx4CgsRdhk88SzOyVdIhmobu97WqE6-ch0P6D4OYb4rUAa1a93x1rkJe__OYRsbHiQn58/s1600/TrelloBoard.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="89" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgacBzFu1e-iFF8oHQ6gyueHPYklnc0GdFLWg5MnBUef7J4O3CWJnJ2On-PDJHpoOQ78vEM9LoQx4CgsRdhk88SzOyVdIhmobu97WqE6-ch0P6D4OYb4rUAa1a93x1rkJe__OYRsbHiQn58/s1600/TrelloBoard.PNG" width="640" /></a></div>
<br />
<br />
This gives me the ability to organise and file the ideas, categorising between content/app/saas and progressing the ideas. It also give the ability to track rejected ideas so I dont waste time coming up with the same idea every 2 months. The advantage the efficient programmer approach was the ability to email in ideas, so as long as you have access to a device with email capability you can record ideas. I also added this ability, but my approach does not require any other software, it uses inbuilt Trello functionality.<br />
<br />
To hook up the ability to email a card into Trello, go to the Menu and select 'Email to Board Settings'. This will bring up a panel with an email address, and the options of which list to add the cards generated to. You can also choose to add cards to the bottom or the top of the list. I selected 'Ideas' list and 'Bottom', so the oldest ideas will always be at the top for sanitising.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1ytGVc33eq5t9OYV_qa1WXxVQ2Fy0QEcuQW5ZxoWDceQRD14QHdjk1_hxXQSTvHFdxjn0P-sycgr228o9m2JvVbQXcjc4YDUQF2NLFeChTRLafkO5Y2IO8I4ZuLipGauMQid_LbGiqcxt/s1600/emailsettings.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1ytGVc33eq5t9OYV_qa1WXxVQ2Fy0QEcuQW5ZxoWDceQRD14QHdjk1_hxXQSTvHFdxjn0P-sycgr228o9m2JvVbQXcjc4YDUQF2NLFeChTRLafkO5Y2IO8I4ZuLipGauMQid_LbGiqcxt/s1600/emailsettings.PNG" width="248" /></a></div>
<br />
The approach used by efficient programmer, on the face of it, allows you to have your different categories (content/app/SaaS) on a single Trello board as multiple lists, where as my approach requires one board per category with the multiple lists used as a progress status indicator.<br />
<br />
Using the email address for a test produces a card:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHh09B3TwsWxVo7873IwjDuvPCYlQRVtczCcfYQRqcjLedhquVK7Ik5pMHY2z3vFs_le4ycISvcBuM5k2B93BZ1tBbfO_HLfBv6_KC7iz_2AtbhVh7CzTZ_obb7fhqLrZqp7zPLln7SrGa/s1600/with+card.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="86" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHh09B3TwsWxVo7873IwjDuvPCYlQRVtczCcfYQRqcjLedhquVK7Ik5pMHY2z3vFs_le4ycISvcBuM5k2B93BZ1tBbfO_HLfBv6_KC7iz_2AtbhVh7CzTZ_obb7fhqLrZqp7zPLln7SrGa/s1600/with+card.PNG" width="640" /></a></div>
and the card contains the subject of the email as its title and the body of the email as the card description:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5pUU-Kf_46su2rd2dgnRLeh16eWyrIvyjIkn-OGpY1nHe-Ek2ItEawsS-owClmzoY_yZz94sQ2YnBzFZug076OMoZyObBxmVgTsj-0HtFQTRnJC4Td8H39HcUTgE4wURSTQdtrr8-XvdQ/s1600/the+card.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="266" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5pUU-Kf_46su2rd2dgnRLeh16eWyrIvyjIkn-OGpY1nHe-Ek2ItEawsS-owClmzoY_yZz94sQ2YnBzFZug076OMoZyObBxmVgTsj-0HtFQTRnJC4Td8H39HcUTgE4wURSTQdtrr8-XvdQ/s1600/the+card.PNG" width="320" /></a></div>
<br />
The one problem I have found is the inability to configure the email address associated with the board, but other than that, this approach gives a solution to the problem of recording and organising ideas that are generated at inopportune moments with little set up. I set up this entire system in under 20 minutes including taking screen shots along the way.<br />
<br />
Finally, if you found this blog useful and interesting please follow me and share this with anyone and everyone.Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-24751615350067248932015-04-20T07:15:00.002-07:002015-04-20T07:15:52.297-07:00How Automated Integration Testing Can Breakdown<br />
Automated integration testing to me means being able to run tests against a system that test the way some or all of the parts integrate to make the whole, fulfilling the requirements of the system, in a way that can be repeated without manual intervention. This may be as part of a continuous integration type build process, or simply a human initiating the test run.<br />
<div>
In order to perform this task, the tests must be able to be run in any order, as any subset of the entire suite, and in parallel with no implications on the result of any test.</div>
<div>
<br /></div>
<div>
For the purposes of automated integration testing, I am an advocate of a BDD style approach using a test definition syntax such as that of <a href="https://cukes.info/">cucumber</a>, and I will explore a case where I have worked on this approach to integration testing only to be bitten by some inadequacies of the way it was implemented. I will begin by saying that the choice of using cucumber style tests for this project was not necessarily wrong, but greater pre-planning of the approach was needed due to some specific technical issues.</div>
<div>
<br /></div>
<div>
<b>System under Test</b></div>
<div>
<b><br /></b></div>
<div>
To give some background around the system that we were building, it consisted of a small tool that ran a series of stored procedures on a database, these output data in xml format. This data was then transformed using a xslt, and the result was uploaded to an API endpoint. The scope of the testing was the stored procedures and xslts, so ensuring that the data extracted and transformed conformed to the schema required by the API and that it was comprised of the data expected from the database being used.</div>
<div>
The database itself was the back end of a large, mature system, the structure and population of which was the responsibility of another team. Additionally the system that populates this database has a very thick and rich business layer, containing some very complex rules around the coupling within the data model which are not represented by the simple data integrity rules of the data model itself. The data model is held only as a series of sql scripts (original creation plus a plethora of update scripts) and as such proves difficult to integrate into a new system.</div>
<div>
During the day to day use of the existing, mature system the data is changed in various ways, and it is the job of the new system to periodically grab a subset of this data and upload it to the API using an incremental change model. So the tests are required to validate that the stored procedures can run for a first time to get all data, and after some modification to get the incremental changes.</div>
<div>
<br /></div>
<div>
<b>How Integration Tests were Set Up</b></div>
<div>
<b><br /></b></div>
<div>
The tests themselves followed the format:</div>
<div>
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> Given a blank database
And data in table "mytable"
|field1|field2|field3|
|value1|value2|value3|
When the data is obtained for upload
the output will be valid
</code></pre>
</div>
<div>
With the variation<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> Given a blank database
And data in table "mytable"
|field1|field2|field3|
|value1|value2|value3|
And the data is obtained for upload
And data in table "mytable" is updated to
|field1|field2|field3|
|value1a|value2a|value3a|
the output will be valid
</code></pre>
</div>
<div>
<br />
for an incremental change.<br />
This required that an unique blank database be created for that test (and every test in practice), the data be added to the database, the process of running the stored procedures and transforms be performed and the resulting data for upload be used in the validation step. Creation of a blank database is simple enough, and can be done either by the scripts used by the main system, or as we chose, by creating an Entity Framework code-first model to represent the data model. The first and most obvious problem you will have guessed is when the core data model changes, the EF model will need to be updated. This problem was swept under the carpet as the convention is to not delete or modify anything in the data model, only to extend, but still it is a flaw in the approach taken for the testing.<br />
<br />
The second and in my opinion most problematic area of this approach comes from data integrity. The EF model contains the data integrity rules of the core data model and enforces them, however if for example in the test above 'mytable' contained a foreign key constrain from 'field1' to some other table ('mytable2'), this other table would also need populating with some data to maintain data integrity. The data in 'mytable2' is not of interest to the test as it is not extracted by the stored procedures, so any data could be inserted so long as the constraints of the data model are met. To this end a set of code was written to auto-populate any data required for integrity of the model. This involved writing some backbone code to cover the general situation, and one class for each table<br />
(for example<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> class AlternativeItemHelper : EntityHelper<AlternativeItem>
{
public AlternativeItemHelper(S200DBBuilder dbbuilder)
: base(dbbuilder)
{
PrimaryKeyField = "AlternativeItemID";
ForeignKeys.Add(new ForeignKeySupport.ForeignKeyInfo<AlternativeItem, StockItem> { LocalField = "ItemID", ForeignField = "ItemID", Builder = dbbuilder });
ForeignKeys.Add(new ForeignKeySupport.ForeignKeyInfo<AlternativeItem, StockItem> { LocalField = "ItemAlternativeID", ForeignField = "ItemID", Builder = dbbuilder });
}
}
</code></pre>
will create data in table 'StockItem' to satisfy constraints on 'ItemID' and 'ItemAlternativeID' fields of the 'AlternativeItem' table.)<br />
that was to be populated as part of a test scenario. As you can imagine, if 'mytable2' contains a similar foreign key relation, then 3 tables need to be populated, and with the real data model, this number grows very large for some tables due to multiple constraints on each table. In one test scenario the addition of one row of data to one table resulted in over 40 tables being populated. This problem was not seen early, as the first half dozen tables populated did not have any such constraints, so the out of the box EF model did not need any additional data to be satisfied.<br />
One advantage of the use of an EF model that I should highlight at this stage is the ability to set default values for all fields, this means that non-nullable fields can be given a value without it being defined in the test scenario (or when auto populating the tables for data integrity reasons alone).<br />
If the data in the linked tables was of interest to the test scenario, the the tables could be populated in the pattern used in the example scenario, and so long as the ordering of the data definition was correct the integrity would be maintained with the defined data.<br />
<br />
The third problem was the execution time for the tests. Even the simplest of tests had a minimum execution time of over one minute. This is dominated by the database creation step. In itself this is not a show stopper if tests are to be run during quiet time, e.g. overnight, but if developers and testers want to run tests in real time, and multiple tests are of interest, this meant a significant wait for results.<br />
<br />
<b>Summary</b><br />
<b><br /></b>
The biggest problem with the approach taken was the time required to write a seemingly simple test. The addition of data to a single table may require developer time to add in the data integrity rules, a large amount of tester time to defined what data needs to be in each table required by the integrity rules of the model and if the default values are sufficient. Potentially dev time to define the default values for the additional tables. In the end a suite of around 200 test was created which takes over 3 hours to run, but due to a lack of testing resource, full coverage was never achieved and manual testing was decided as the preferred approach by management.<br />
<br />
<b>Supporting Code Examples</b><br />
<b><br /></b>
The example entity helper for auto populating additional tables is derived from a generic class for help with all entity types, this takes to form<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public abstract class EntityHelper<T> : IEntityHelper<T>
where T : class, new()
{
protected S200DBBuilder dbbuilder;
protected DbSet<T> entityset;
protected long _id = 0;
protected string PrimaryKeyField { get; set; }
protected Lazy<GetterAndSetter> PkFieldProp;
public Lazy<List<PropertySetter>> PropertySetters { get; protected set; }
public EntityHelper(S200DBBuilder dbbuilder):this()
{
Initialize(dbbuilder);
}
protected EntityHelper()
{
}
public object GetRandomEntity()
{
return GetRandomEntityInternal();
}
protected T GetRandomEntityInternal()
{
T entity = new T();
//need to set all the properties to random values - and cache a way to create them faster
PropertySetters.Value.ForEach(ps => ps.SetRandomValue(entity));
return entity;
}
public virtual void Initialize(S200DBBuilder dbbuilder)
{
this.dbbuilder = dbbuilder;
this.entityset = dbbuilder.s200.Set<T>();
ForeignKeys = new List<IForeignKeyInfo<T>>();
PkFieldProp = new Lazy<GetterAndSetter>(() =>
{
var type = typeof(T);
var prop = type.GetProperty(PrimaryKeyField);
return new GetterAndSetter { Setter = prop.GetSetMethod(true), Getter = prop.GetGetMethod(true) };
});
//initialise the PropertySetters
PropertySetters = new Lazy<List<PropertySetter>>(() =>
{
var list = new List<PropertySetter>();
list.AddRange(typeof(T)
.GetProperties()
.Where(p => !p.Name.Equals("OpLock", StringComparison.OrdinalIgnoreCase))
.Where(p => !(p.GetGetMethod().IsVirtual))
.Select(p => PropertySetterFactory.Get(dbbuilder.s200, p, typeof(T)))
);
return list;
});
}
protected virtual T AddForeignKeys(T ent)
{
UpdatePKIfDuplicate(ent);
ForeignKeys.ForEach(fk => CheckAndAddFK(fk, ent));
return ent;
}
protected void UpdatePKIfDuplicate(T ent)
{
//assumes all keys are longs
var pk = (long)PkFieldProp.Value.Getter.Invoke(ent, new object[] { });
var allData = entityset.AsEnumerable().Concat(entityset.Local);
var X = allData.Count();
while (allData.Where(e => PkFieldProp.Value.Getter.Invoke(e, new object[] { }).Equals(pk)).Count() >0)
{
pk++;
PkFieldProp.Value.Setter.Invoke(ent, new object[] {pk });
}
}
protected T ReplicateForeignKeys(T newent, T oldent)
{
ForeignKeys.ForEach(fk => fk.CopyFromOldEntToNew(oldent, newent));
return newent;
}
public void AddData(IEnumerable<T> enumerable)
{
entityset.AddRange(enumerable.Select(ent => AddForeignKeys(ent)));
}
public void UpdateData(IEnumerable<T> enumerable)
{
foreach (var newent in enumerable)
{
var oldent = GetCorrespondingEntityFromStore(newent);
UpdateEntityWithNewData(oldent, newent);
dbbuilder.s200.Entry(oldent).State = EntityState.Modified;
}
}
protected void UpdateEntityWithNewData(T oldent, T newent)
{
foreach (var prop in typeof(T).GetProperties())
{
//todo - change this line to be a generic check on the prop being a primary key field
if (prop.Name.Equals("SYSCompanyID")) continue;
var newval = prop.GetGetMethod().Invoke(newent, new object[] { });
// Not sure if this is the correct place to do this, will check with Mike W
if (newval != null)
{
var shouldUpdateChecker = UpdateCheckers.Get(prop.PropertyType);
shouldUpdateChecker.Update(newval, oldent, prop.GetSetMethod());
}
}
}
public void Delete(T entity)
{
var storeentity = GetCorrespondingEntityFromStore(entity);
DeleteEntity(entity);
}
private void DeleteEntity(T entity)
{
entityset.Remove(entity);
}
public void Delete(long id)
{
var entity = GetById(id);
DeleteEntity(entity);
}
public void DeleteAll()
{
var all = entityset.ToList();
entityset.RemoveRange(all);
}
public long AddSingle(T entity)
{
var id = Interlocked.Increment(ref _id);
SetId(entity, id);
AddData(new[] { entity });
return id;
}
protected void SetId(T entity, object id) { PkFieldProp.Value.Setter.Invoke(entity, new[] { id }); }
protected T GetCorrespondingEntityFromStore(T newent) { return GetById(PkFieldProp.Value.Getter.Invoke(newent, new object[] { })); }
protected T GetById(object id) { return entityset.AsEnumerable().Single(ent => PkFieldProp.Value.Getter.Invoke(ent, new object[] { }).Equals(id)); }
public void UpdateAllEntities(Action<T> act)
{
entityset.ToList().ForEach(act);
}
public void UpdateEntity(int id, Action<T> act)
{
var entity = GetById(id);
act(entity);
}
public IEnumerable GetDataFromTable(Table table)
{
return table.CreateSet<T>();
}
public void AddData(IEnumerable enumerable)
{
var data = enumerable.Cast<T>();
AddData(data);
}
public void UpdateData(IEnumerable enumerable)
{
var data = enumerable.Cast<T>();
UpdateData(data);
}
protected List<IForeignKeyInfo<T>> ForeignKeys { get; set; }
protected void CheckAndAddFK(IForeignKeyInfo<T> fk, T ent)
{
//first get the value on the entitity and check if it exists in the model already
fk.CreateDefaultFKEntityAndSetRelation(ent);
}
public void CreateDefaultEntity(out long fkID)
{
var entity = new T();
fkID = AddSingle(entity);
}
public void CreateDefaultEntityWithID(long fkID)
{
var entity = new T();
SetId(entity, fkID);
AddData(new[] { entity });
//the _id field needs to be greater than the id used here, so
fkID++;
if (fkID >= _id)
Interlocked.Exchange(ref _id, fkID);
}
</code></pre>
<br />
To create default values for any entity we created a partial class to extend the entity class that has one method:<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public partial class BinItem
{
partial void OnCreating()
{
DateTimeCreated = DateTime.Now;
BinName = string.Empty;
SpareText1 = string.Empty;
SpareText2 = string.Empty;
SpareText3 = string.Empty;
}
}
</code></pre>
this method being called as part of the constructor of the entity class<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> [Table("BinItem")]
public partial class BinItem
{
public BinItem()
{
...
OnCreating();
}
partial void OnCreating();
...
} </code></pre>
The entity helpers rely upon foreign key information to know about the constraints on the table, these are supported by a class
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public class ForeignKeyInfo<T, T2> : IForeignKeyInfo<T>
where T : class,new()
where T2 : class,new()
{
public ForeignKeyInfo()
{
BuildIfNotExists = true;
LocalFieldSetter = new Lazy<MethodInfo>(() =>
{
var type = typeof(T);
var prop = type.GetProperty(LocalField);
if (prop == null)
prop = type.GetProperty(LocalField+"s");
return prop.GetSetMethod(true);
});
LocalFieldGetter = new Lazy<MethodInfo>(() =>
{
var type = typeof(T);
var prop = type.GetProperty(LocalField);
if (prop == null)
prop = type.GetProperty(LocalField + "s");
return prop.GetGetMethod(true);
});
ForeignFieldGetter = new Lazy<MethodInfo>(() =>
{
var type = typeof(T2);
var prop = type.GetProperty(ForeignField);
if (prop == null)
prop = type.GetProperty(ForeignField + "s");
return prop.GetGetMethod(true);
});
ForeignTableGetter = new Lazy<MethodInfo>(()=>
{
var type = typeof(S200DataContext);
var prop = type.GetProperty(typeof(T2).Name);
if (prop == null)
{
prop = type.GetProperty(typeof(T2).Name+"s");
if (prop == null && typeof(T2).Name.EndsWith("y"))
{
var currentName = typeof(T2).Name;;
prop = type.GetProperty(currentName.Substring(0,currentName.Length-1) + "ies");
}
if (prop == null && typeof(T2).Name.EndsWith("eau"))
{
prop = type.GetProperty(typeof(T2).Name + "x");
}
if (prop == null && typeof(T2).Name.EndsWith("s"))
{
prop = type.GetProperty(typeof(T2).Name + "es");
}
}
var getter = prop.GetGetMethod(true);
return getter;
});
}
public string LocalField { get; set; }
public S200DBBuilder Builder { get; set; }
public string ForeignField { get; set; }
public bool DoesFKExist(T ent)
{
//check the foeign table to see is an entry exists which matches the ent
var lf = LocalFieldGetter.Value.Invoke(ent, new object[] { });
return GetForeignEnts(lf).Count()> 0;
}
public void CreateDefaultFKEntityAndSetRelation(T ent)
{
if (DoesFKExist(ent))
{
return;
}
var lf = LocalFieldGetter.Value.Invoke(ent, new object[] { });
if (lf == null)
{
if (BuildIfNotExists)
{
//the test did not define the FK ID to use, so just default it to the next in the sequence
long fkID = 0;
Builder.WithDefaultEntity(typeof(T2), out fkID);
//now set the FK relation
LocalFieldSetter.Value.Invoke(ent, new object[] { fkID });
}
}
else
{
//create the FK entity using the id that has been passed in
Builder.WithDefaultEntityWithID(typeof(T2), (long)lf);
}
}
private T2 GetForieignEnt(object fkID)
{
return GetForeignEnts(fkID).FirstOrDefault();
}
private IEnumerable<T2> GetForeignEnts(object fkID)
{
var castData = (DbSet<T2>)(ForeignTableGetter.Value.Invoke(Builder.s200, new object[] { }));
var allData = castData.AsEnumerable().Concat(castData.Local);
var fes = allData.Where(fe => ForeignFieldGetter.Value.Invoke(fe, new object[] { }).Equals(fkID));
return fes;
}
private Lazy<MethodInfo> LocalFieldSetter;
private Lazy<MethodInfo> LocalFieldGetter;
private Lazy<MethodInfo> ForeignFieldGetter;
private Lazy<MethodInfo> ForeignTableGetter;
public T CopyFromOldEntToNew(T oldent, T newent)
{
if (DoesFKExist(newent))
{
return newent;
}
var value = LocalFieldGetter.Value.Invoke(oldent, new object[] { });
LocalFieldSetter.Value.Invoke(newent, new object[] { value });
return newent;
}
public bool BuildIfNotExists { get; set; }
}
</code></pre>
<ul>
</ul>
</div>
Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-32565590878563114202015-04-13T05:29:00.002-07:002015-04-13T05:34:36.609-07:00BDD to Break Communication BarriersIn my years of software development the one underlying theme that has caused the most problems when it comes to delivering quality software that fits the needs of the customer is misunderstanding and miscommunication of requirements. This is not a problem that is due to any individual doing their job badly, not down to poor communication skills, not down to a lack of effort of diligence by anyone. It is simply due to the number of times the same information is written, spoken, heard, interpreted and misinterpreted. Its a tale of Chinese whispers, a term that is probably not politically correct in this day and age, but one that fits the bill. In an idealised scenario:<br />
<br />
<ol>
<li>Customer dreams up something they want a system to do</li>
<li>Business analyst and sales team talk to them to nail down chargeable and isolated items (stories if you like)</li>
<li>Business analysts document the stories for the technical team(s)</li>
<li>Architects and developers read these, interpret and develop a solution</li>
<li>Testers read and interpret the BA's docs and create a test suite</li>
</ol>
<div>
In this scenario the requirements are 'written' 2 times by 2 people, 'read' 3 times by 3 people but reinterpreted at each step, so what the 3 layers of this process (customer; BA and sales; dev and test) understand the system needing to do can be vastly different, especially in the detail. A situation can occur where the BA slightly misunderstands the customers needs, then does not communicate their understanding thoroughly. The dev and test teams pick this up and interpret in 2 further subtly different ways. All three layers of the process think they understand everything fully, the system 'works' in so much as it performs a function, all of the tests are passed so the system is delivered or demoed to the customer, but the system does not do what the customer really wanted it to do. Where did the failure happen? Everyone in the process has performed their own individual part successfully, and something has been built that everyone will be to some extent satisfied with, but its not what the customer wants.</div>
<div>
<br /></div>
<div>
What is missing here is a single item that all players can refer to and agree upon as fully defining the requirements. Conventional wisdom would say that the document produced by the BA is that item, and ideally that would work, one document, all parties agree to its contents, all code is built to its specifications, all tests validate these specifications and all the desires of the customer are met. In practise the 3 ways this document is interpreted mean the one single document conveys three messages. So how can we overcome this?</div>
<div>
<br /></div>
<div>
A document that can be 'read' by a machine, interpreted only in one way so code to make the system work and tests to validate this have the same interpretation, and so long as this can be read easily in a non-technical manner by the customer and agreed upon the loop should be closed. Customer agrees to the specification when written into the document, the tests prove this happens and works, and the code is written to satisfy the tests. No ambiguity remains.</div>
<div>
<br /></div>
<div>
This forms something of the basis for the concept of Behaviour Driven Development (<a href="http://en.wikipedia.org/wiki/Behavior_Driven_Development">BDD</a>) where the desired behaviour of the system is defined and forms the specifications, the tests and drives the development to meet these, akin to test driven development but where overall behaviour of the system is the driver, not the technical test specifications, which in general do not over the overall system, but isolated units of the system. The core advantage of BDD is that the behaviour specification is written in a language that can both be read by non-technical personnel (customers) and interpreted by the computer without translation.</div>
<div>
<br /></div>
<div>
The syntax of defining specifications is a well established one, and one that has many flavors (GWT, AAA etc). A DSL was developed to encapsulate the given when then formulation of requirements and has been given the name <a href="https://github.com/cucumber/cucumber/wiki/Gherkin">gherkin</a>. For the purposes of using this within a .net project a tool called <a href="http://www.specflow.org/">SpecFlow </a>is the one I currently choose to use, although in the past I have used <a href="https://cukes.info/">cucumber </a>and had to write Ruby code to access the actual code of the .Net system, the advantage of Specflow is that all the specifications, code to access the system and the system under test itself can exist in one place, and written in one development language.</div>
<div>
<br /></div>
<div>
I am not writing a post on how to perform BDD here, I am looking to highlight the advantages of a BDD tool like Specflow to the development process, and specifically the communication of detailed technical ideas between different groups and disciplines within the process without ambiguity of interpretation creeping in. That said, a simple test specification taken from the cucumber website provides a good place to start in terms of understanding how this fits into the dev process.</div>
<div>
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: blue; word-wrap: normal;"> Feature: <code style="color: green; word-wrap: normal;">CalculatorAddition </code>
In order <code style="color: green; word-wrap: normal;">to avoid silly mistakes </code>
As a <code style="color: green; word-wrap: normal;">math idiot </code>
I want <code style="color: green; word-wrap: normal;">to be told the sum of two numbers </code>
Scenario: <code style="color: green; word-wrap: normal;">Add two numbers </code>
Given <code style="color: black; word-wrap: normal;">I have entered 50 into the calculator </code>
And <code style="color: black; word-wrap: normal;">I have entered 70 into the calculator </code>
When <code style="color: black; word-wrap: normal;">I press add </code>
Then <code style="color: black; word-wrap: normal;">the result should be 120 on the screen </code>
</code></pre>
</div>
<div>
<br />
This specification is the example one that is provided when you add a new spec to a unit test project in visual studio using the Specflow add in, but provides a good point to explore the way gherkin solves the problem of interpretation of requirements.<br />
<br />
This specification is very easy to understand, a non-technical person e.g. the customer, could read this and sign-off that this details what they need the system to be able to do. The question is how does this satisfy the needs of the testers to validate that the system does what is described. Well that is the beauty of the cucumber/specflow system. This series of definitions constitute a test as well as a requirement specification. The specflow framework executes a piece of code for each of these lines, the code in question being hooked into by a regular expression match of the definition itself against an attribute on the code method. And the convention is that the 'Then' definition will validate the outcome of the action against the expectations (do an assert if you prefer). The code that needs to be written to hook this into the production system is very straightforward<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> [Binding]
public class CalculatorAdditionSteps
{
Calculator calc = new Calculator();
[Given(@"I have entered (.*) into the calculator")]
public void GivenIHaveEnteredIntoTheCalculator(int value)
{
calc.InputValue(value);
}
[When(@"I press add")]
public void WhenIPressAdd()
{
calc.DoAddition();
}
[Then(@"the result should be (.*) on the screen")]
public void ThenTheResultShouldBeOnTheScreen(int expectedResult)
{
Assert.AreEqual(expectedResult, calc.Result);
}
}
</code></pre>
and this form the basis of a simple unit test. As you can imagine detailing a fully functional production system will involve significantly more code to be written, but with the advantage that if the specifications drive the process, the tests come for free, and the architecture and code design is driven towards a clear and easily instantiated structure. Minimal coupling and dependencies make the production and maintenance of the 'hook' code here significantly easier.<br />
<br />
When performing this as a BDD process a simple Calculator class will satisfy the needs of the test as far as being able to build<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> class Calculator
{
internal void InputValue(int value)
{
throw new NotImplementedException();
}
internal void DoAddition()
{
throw new NotImplementedException();
}
public object Result { get; set; }
}
</code></pre>
And when run the test will fail. It is also possible to work with only the specification, before the 'hook' code is written, at which stage running the test will give an inconclusive result, highlighting that the specification has been detailed, but that no work has been performed to hook this into validate the system, potentially meaning the system has not had the new functionality added.<br />
<br />
There are shortcomings to this approach, but as a way of removing the element of Chinese whispers from the development process it goes a long way to solving the problem.<br />
<br />
I will showcase a situation where this approach proved problematic in a future blog, a situation where it did test the system successfully but where the overhead of creating and maintaining the specifications and the hook code outweighed the advantages provided.</div>
Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-59477011070891359952015-04-07T09:40:00.000-07:002015-04-09T01:29:18.604-07:00Visualisation of evolution<div dir="ltr">
Even though we know that the end result is what we are after, and speed is one of the most important factors, it would be nice when assessing different mutation and breeding operator combinations, and the affect of the applied fitness function to track the evolution of the population, or at least that of the fittest member(s) graphically to quickly see, and convey to intetrsted parties what is happening. </div>
<div dir="ltr">
To this end I will explore the possibility of hooking a visualisation interface into the algorithm with a minimum of code churn, and minimal speed impact.</div>
<div dir="ltr">
The approach I will take is to use a consumer object to handle the generation complete event. The responsibility of not blocking the processing thread will fall to this consumer, and all details of the rendering of the interim results will be totally hidden to the gentic algorithm itself. This approach will mean that if a web enabled front end, or simply a different desktop UI, were needed you merely need to construct this and inject it.</div>
<div dir="ltr">
<br /></div>
<div dir="ltr">
I have chosen to use a WPF Gui as I have experience of automatically updating graphs in this medium. Another technology may be better suited to your skill set. The WPFToolkit offers a very good charting control, which can plot data the is updated in real time very easily with data binding. I will not go into the details of the WPF application itself, or the structure of such an application, however the details of displaying the evolution are what we are interested in, so that is what I will focus on. But I will say that my chosen architecture employed an <a href="https://msdn.microsoft.com/en-us/magazine/dd419663.aspx">MVVM </a>pattern<br />
<br />
The code for each chart is very simple, with the UI layer being simply</div>
<div dir="ltr">
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> xmlns:chartingToolkit="clr-namespace:System.Windows.Controls.DataVisualization.Charting;assembly=System.Windows.Controls.DataVisualization.Toolkit"
<chartingToolkit:Chart Title="Fitness" >
<chartingToolkit:Chart.Axes>
<chartingToolkit:LinearAxis Orientation="Y" ShowGridLines="False" Minimum="{Binding MinFitness}" Maximum="{Binding MaxFitness}" />
</chartingToolkit:Chart.Axes>
<chartingToolkit:LineSeries DependentValuePath="Value" IndependentValuePath="Key" ItemsSource="{Binding GenerationFitness}" IsSelectionEnabled="True" />
</chartingToolkit:Chart>
</code></pre>
</div>
<div dir="ltr">
<br /></div>
<div dir="ltr">
Where the MinFitness and MaxFitness are values calculated as results are generated to give a sensible range for the graph, and the GenerationFitness property is a collection holding the points to plot in the graph. This is bound to a view model that exposes the data without exposing the detail of the GA, and this takes the form:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> class ViewModel: NotifyingObject
{
private Model theData;
private double _minFitness;
private double _maxFitness;
private double varf = 0.01d;
private string _results;
private int _delay=0;
public ViewModel()
{
theData = new Model();
GenerationFitness = new ObservableCollection<KeyValuePair<int, double>>();
GenerationF = new ObservableCollection<KeyValuePair<int, double>>();
GenerationR = new ObservableCollection<KeyValuePair<int, double>>();
theData.NewGeneration += GotNewGeneration;
theData.FinalResults += GotFinalResults;
ResetFitness();
}
public int PopulationSize { get { return theData.PopulationSize; } set { theData.PopulationSize = value; } }
public int MaxGenerations { get { return theData.MaxGenerations; } set { theData.MaxGenerations= value; } }
public double MinFitness { get { return _minFitness; } set { _minFitness = value; OnPropertyChanged(); } }
public double MaxFitness { get { return _maxFitness; } set { _maxFitness = value; OnPropertyChanged(); } }
public string Results { get { return _results; }set{_results = value; OnPropertyChanged();} }
public int Delay { get { return _delay; } set { _delay = value; OnPropertyChanged(); } }
public ObservableCollection<KeyValuePair<int, double>> GenerationFitness { get; set; }
public ObservableCollection<KeyValuePair<int, double>> GenerationR { get; set; }
public ObservableCollection<KeyValuePair<int, double>> GenerationF { get; set; }
public ICommand Stop { get { return new RelayUICommand("Stop", (p) => theData.Stop(), (p) => theData.IsRunning); } }
public ICommand Start
{
get
{
return new RelayUICommand("Start", (p) =>
{
ClearAll();
theData.Start();
}
, (p) => !theData.IsRunning);
}
}
public ICommand Clear
{
get
{
return new RelayUICommand("Clear", (p) =>
{
ClearAll();
}
, (p) => !theData.IsRunning);
}
}
private void ResetFitness()
{
_minFitness = 0d;
_maxFitness = 1d;
}
private void GotNewGeneration(object sender, GenerationEventArgs e)
{
Application.Current.Dispatcher.Invoke(() =>
{
GenerationFitness.Add(new KeyValuePair<int, double>(e.Generation, e.Fitness));
if (e.Generation ==1)
{
MaxFitness = e.Fitness * (1d + varf);
MinFitness = e.Fitness * (1d-varf);
}
MaxFitness = Math.Max(MaxFitness, e.Fitness *(1d + varf));
MinFitness = Math.Min(MinFitness, e.Fitness * (1d - varf));
GenerationF.Add(new KeyValuePair<int, double>(e.Generation, e.F));
GenerationR.Add(new KeyValuePair<int, double>(e.Generation, e.R));
Debug.WriteLine(String.Format("Generation: {0}, Fitness: {1},R: {2}, F: {3}", e.Generation, e.Fitness, e.R, e.F));
});
Thread.Sleep(Delay );
}
private void GotFinalResults(object sender, FinalResultsEventArgs e)
{
Results = String.Format("R: {0}{1}F: {2}{1}Fitness: {3}{1}{1}From Values:{1}{4}", e.R, Environment.NewLine, e.F, e.Fitness, String.Join(Environment.NewLine, e.GeneValues));
}
private void ClearAll()
{
ResetFitness();
GenerationFitness.Clear();
GenerationR.Clear();
GenerationF.Clear();
Results = "";
}
}
</code></pre>
<br /></div>
<div dir="ltr">
The model behind this does the work of instantiating the GA and it relays the results of each generation and the completion of the run in a meaningful manner to the view model:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> class Model: NotifyingObject
{
private double targetR = 0.95d;
private double targetF = 0.5d;
public double TargetR { get { return targetR; } set { targetR = value; OnPropertyChanged(); } }
public double TargetF { get { return targetF; } set { targetF = value; OnPropertyChanged(); } }
public EventHandler<DoubleEventArgs> NewFitnessValueArrived;
public EventHandler<GenerationEventArgs> NewGeneration;
public EventHandler<FinalResultsEventArgs> FinalResults;
private IGAProvider gaProvider;
private GeneticAlgorithm ga;
private int _maxGenerations;
private bool _isRunning;
public Model()
{
PopulationSize = 10000;
MaxGenerations = 100;
////initialise the GA and hook up events
const double crossoverProbability = 0.02;
const double mutationProbability = 0.8;
gaProvider = GetGaProvider();
var crossover = new Crossover(crossoverProbability, true)
{
CrossoverType = CrossoverType.SinglePoint
};
//var crossover = new AveragingBreeder() { Enabled = true };
//inject the mutation algorithm
var mutation = new SwapMutate(mutationProbability);
//var mutation = new PairGeneMutatorWithFixedValueSum2(mutationProbability){Enabled = true};
//var mutation = new SingleGeneVaryDoubleValueMaintainingSum3(mutationProbability, 1d, 0.2d) { Enabled = true };
gaProvider.AddMutator(mutation);
gaProvider.AddBreeder(crossover);
}
public void Start()
{
const int elitismPercentage = 10;
var dh = new DoubleHelpers();
ga = gaProvider.GetGA(elitismPercentage, dh, PopulationSize);
GAF.GeneticAlgorithm.GenerationCompleteHandler generationComplete = ga_OnGenerationComplete;
GAF.GeneticAlgorithm.RunCompleteHandler runComplete = ga_OnRunComplete;
gaProvider.HookUpEvents(generationComplete, runComplete);
Task.Factory.StartNew(() => ga.Run(gaProvider.Terminate));
IsRunning = true;
}
public void Stop()
{
ga.Halt();
IsRunning = false;
}
private void ga_OnGenerationComplete(object sender, GaEventArgs e)
{
(gaProvider as DoubleChromosomes).CurrentGeneration = e.Generation;
var fittest = e.Population.GetTop(1)[0];
var r = (gaProvider as DoubleChromosomes).GetR(fittest.Genes.Select(x => x.RealValue));
var f = (gaProvider as DoubleChromosomes).GetF(fittest.Genes.Select(x => x.RealValue));
FireGenerationArrived(e.Generation, fittest.Fitness, r, f);
}
void ga_OnRunComplete(object sender, GaEventArgs e)
{
IsRunning = false;
var fittest = e.Population.GetTop(1)[0];
var r = (gaProvider as DoubleChromosomes).GetR(fittest.Genes.Select(x => x.RealValue));
var f = (gaProvider as DoubleChromosomes).GetF(fittest.Genes.Select(x => x.RealValue));
FireFinalResults(fittest.Genes.Select(x => x.RealValue), r, f, fittest.Fitness);
}
public IGAProvider GetGaProvider()
{
var gaProvider = new DoubleChromosomes(TargetF, TargetR){MaxGenerations = _maxGenerations};
return gaProvider;
}
private void FireGenerationArrived(int generation, double fitness, double r, double f)
{
var h = NewGeneration;
if (h == null)
return;
h(this, new GenerationEventArgs(generation, fitness, r, f));
}
private void FireFinalResults(IEnumerable<double> geneValues, double r, double f, double fitness)
{
var h = FinalResults;
if (h == null)
return;
h(this, new FinalResultsEventArgs(geneValues, r, f, fitness));
}
public bool IsRunning { get { return _isRunning; } set { _isRunning = value; OnPropertyChanged(); } }
public int PopulationSize { get; set; }
public int MaxGenerations {
get { return _maxGenerations; }
set { if (gaProvider != null) { gaProvider.MaxGenerations = value; } _maxGenerations = value; OnPropertyChanged(); }
}
}
</code></pre>
<br />
The NotifyingObject that both view model and model derive from is a useful class for implementing the INotifyPropertyChanged interface:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public abstract class NotifyingObject : INotifyPropertyChanged
{
protected void OnPropertyChanged([CallerMemberName] string propertyName = "")
{
var eh = PropertyChanged;
if (eh != null)
eh(this, new PropertyChangedEventArgs(propertyName));
}
public event PropertyChangedEventHandler PropertyChanged;
}
</code></pre>
<br />
and the RelayUICoommand is an extension of the <a href="https://msdn.microsoft.com/en-us/magazine/dd419663.aspx#id0090030">RelayCommand</a> class which simply adds a Text property for the button caption<br />
<br /></div>
<div dir="ltr">
The app that I created allows the used to set the number of chromosomes in the population and the maximum number of generations to allow. As the algorithm progresses, the values of R and F are plotted along with the fitness, all for the best solution of the current generation. As a reminder the target values are R=0.95 and F=0.5, a fitness of 1 would be the ideal, and the out of the box SwapMutation and a single point Crossover operations are being employed. Equally the fitness evaluation has not changed.<br />
<br />
I have produced videos of the output (slowed down) using a population of just 4 chromosomes and 10000 chromosomes to compare the results. Due to the very small population size of the first the algorithm actually evaluates to a worse solution than one found earlier in the evolution, but this highlights the plotting of the results.<br />
<br />
<iframe width="420" height="315" src="https://www.youtube.com/embed/x5bM9zO6QrY" frameborder="0" allowfullscreen></iframe>
<br />
<br />
<iframe width="420" height="315" src="https://www.youtube.com/embed/gHNFhADSaTo" frameborder="0" allowfullscreen></iframe>
</div>Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-47052168580698569612015-03-31T01:01:00.001-07:002015-03-31T01:02:07.238-07:00Quantum ForkQuantum Fork?<br />
<br />
What? you may ask. Well, as I mentioned in the post yesterday the need to clearly and effectively communicate ideas, requirements and specifications is vital to a successful software project delivery, and vital to the sanity of the developers by removing the pressure from managers to quantify what is clear to a dev but not others. So this blog is taking a change, a fork you may say. The investigation of techniques for solving complex maths problems will continue, but alongside that will be a discussion of methods to communicate between members of an interdisciplinary team (and with those outside the team) in a clear way, preferably without the need to 'translate' the information for each audience member. Quantum Fork as you can follow both paths simultaneously, think of yourself as a superposition of readers.<br />
<br />
The decision to rename the blog and include posts on both subjects was made so that you will all reap the benefit of getting all the content I put out, and I will get the benefit of not losing you as readers due to a change or URL.<br />
<br />
I look forward to posting my first communication themed musings over the next week.Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-68885536466546081562015-03-30T05:31:00.000-07:002015-04-09T01:30:05.595-07:00Need for VisualisationThe idea of an evolving solution to a problem is a great one, but how do we sell it to non technical people and how do we show them the progress we are making and the avenues of investigation we are trying? Its difficult to explain that the mutation and breeding operators we tried in the last two posts have not had the desired effect without being able to explain the quantification of the fitness we are using. The best way to convey any complex idea is with pictures, so I am looking to create a graphical display of the evolution process, or at least the numeric results of each generation, so a side-by-side comparison of the process can be seen when an operator is changed, or the fitness evaluation is changed as we will look to do in a few posts time.<br />
<br />
Due to time constraints I have not managed to get a working system to display these results in a way that I would like, so I am not including the details in this post, but they will come soon.<br />
<br />
The problem of describing a problem like the one we are attempting to solve is always going to be a difficult one, and is just one example of the difficulty in communicating within a multidisciplinary environment. Not only do we have managers that need results, and want detailed explanations of timescales to get results with tangible progress indicators; we need to ensure that the needs of the system as outlined by business analysts are understood by devs and testers alike, and that end users can be provided with useful documentation, and, and, and. The channels are endless, and that brings its own problems. I will be looking at this set of problems in either this blog or a sister blog soon, and I will provide a link if necessary when it comes about.Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-48155835114558906562015-03-12T08:07:00.003-07:002015-03-17T08:52:44.810-07:00Implementing a Breeding AlgorithmIn the previous post we saw that introducing a mutation operation that naively you would think could improve the final solution found had little impact. The question is what about an operation that mixes solutions to create a new potential solution? For the sake of a biological analogy, a breeding operation where two (or more) parents (solutions from a given generation) 'mix' in some way to produce a child solution that becomes part of the population in the next generation. So I spent some time creating two such operators, very simplistic one, to see how they affect the outcome for the example problem I described in my first post.<br />
<br />
<b><span style="font-size: large;">Averaging Operator</span></b><br />
<b><span style="font-size: large;"><br /></span></b>
The first and simplest breeding operator I could think of is one where the value of each gene is an average of the corresponding values in the parents:<br />
<br />
<table border="2" bordercolor="#0033FF" cellpadding="3" cellspacing="3" style="background-color: #99ffff; width: 100%px;">
<tbody>
<tr>
<th>Chromosome</th>
<th>Gene 1</th>
<th>Gene 2</th>
</tr>
<tr>
<td>Parent 1</td>
<td>3</td>
<td>6.2</td>
</tr>
<tr>
<td>Parent 2</td>
<td>5</td>
<td>4.1</td>
</tr>
<tr>
<td>Child</td>
<td>4</td>
<td>5.15</td>
</tr>
</tbody></table>
<br />
One problem with this is that it will instantly violate the simple sum of gene values constraint we imposed. As such the actual code goes a step further and 'normalises' the genes of the child chromosome to have a sum that matches that of the parents.<br />
This operator is not a true breeding operator in its purest form as it does modify the values of the genes rather than simply create a new chromosome by resequencing genes selected from the parents. As such it is more of a breeding and mutation operator in one. The reason behind the decision to do this was that in a continuous variable problem, the best solution may possibly exist in a space 'between' the solutions that have already been explored, so a mutation and a resequencing combined may help to explore the problem space more rapidly. Remember that in the first post a single point crossover breeding operator was used, this takes part of each parent and concatenates to form a child, the table below showing how the same parents can produce 2 different children by changing the point of crossover<br />
<br />
<table border="2" bordercolor="#0033FF" cellpadding="3" cellspacing="3" style="background-color: #99ffff; width: 100%px;">
<tbody>
<tr>
<th>Chromosome</th>
<th>Gene 1</th>
<th>Gene 2</th>
<th>Gene 3</th>
<th>Gene 4</th>
</tr>
<tr>
<td>Parent 1</td>
<td>1</td>
<td>2</td>
<td>3</td>
<td>4</td>
</tr>
<tr>
<td>Parent 2</td>
<td>9</td>
<td>8</td>
<td>7</td>
<td>6</td>
</tr>
<tr>
<td>Child</td>
<td>1</td>
<td>2</td>
<td>7</td>
<td>6</td>
</tr>
<tr>
<td>Child</td>
<td>1</td>
<td>8</td>
<td>7</td>
<td>6</td>
</tr>
</tbody></table>
<br />
and we have the results of this non-mutating breeding operator to compare with those of a hybrid breeder-mutator operation.<br />
<br />
Results do show that this operator gives no noticeable advantage over the out of the box pure crossover operation, with R=0.941 and F=0.235 with the SwapMutate operator unmodified from the first post.<br />
<br />
the code for this operator is:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public void Invoke(Population currentPopulation, ref Population newPopulation, FitnessFunction fitnesFunctionDelegate)
{
Debug.WriteLine("Breeding from pop size of {0} into pop size of {1}", currentPopulation.PopulationSize, newPopulation.PopulationSize);
//first check sixe of new population, if its big enough already just return
if (newPopulation.PopulationSize >= currentPopulation.PopulationSize)
return;
//also cannot perform with less than 2 chromosomes
if (currentPopulation.PopulationSize < 2)
return;
while (newPopulation.PopulationSize < currentPopulation.PopulationSize)
{
//otherwise take 2 chromosomes at random from the current population (include elite ones)
var parents = currentPopulation.Solutions.SelectRandomMembers(2);
//craete the new chromosome
//assumes both parents have the same the sum of gene values
var sum = parents.First().Genes.Sum(g => g.RealValue);
var newGenes = new List<double>();
for (int i = 0; i < parents.First().Genes.Count; i++)
{
newGenes.Add((parents.First().Genes.ElementAt(i).RealValue + parents.Last().Genes.ElementAt(i).RealValue) / 2);
}
//now normalise to the same sum
var newsum = newGenes.Sum();
var m = sum / newsum;
var ch = new Chromosome(newGenes.Select(g => g * m));
newPopulation.Solutions.Add(ch);
}
}
</code></pre>
<br />
<br />
<span style="font-size: large;"><b>Normalised Sum Operator</b></span><br />
<span style="font-size: large;"><b><br /></b></span>
A slight variation on the averaging breeder is to instead simply sum the gene values of the parents, then 'normalise' the sum of gene values to meet the constraint. Again this does not offer any great advantage with R=0.978 and F=0.244<br />
<br />
from code<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public void Invoke(Population currentPopulation, ref Population newPopulation, FitnessFunction fitnesFunctionDelegate)
{
Debug.WriteLine("Breeding from pop size of {0} into pop size of {1}", currentPopulation.PopulationSize, newPopulation.PopulationSize);
//first check sixe of new population, if its big enough already just return
if (newPopulation.PopulationSize >= currentPopulation.PopulationSize)
return;
//also cannot perform with less than 2 chromosomes
if (currentPopulation.PopulationSize < 2)
return;
while (newPopulation.PopulationSize < currentPopulation.PopulationSize)
{
//otherwise take 2 chromosomes at random from the current population (include elite ones)
var parents = currentPopulation.Solutions.SelectRandomMembers(2);
//craete the new chromosome
//assumes both parents have the same the sum of gene values
var sum = parents.First().Genes.Sum(g => g.RealValue);
var newGenes = new List<double>();
for (int i = 0; i < parents.First().Genes.Count; i++)
{
newGenes.Add(parents.First().Genes.ElementAt(i).RealValue + parents.Last().Genes.ElementAt(i).RealValue);
}
//now normalise to the same sum
var newsum = newGenes.Sum();
var m = sum / newsum;
var ch = new Chromosome(newGenes.Select(g => g * m));
newPopulation.Solutions.Add(ch);
}
</code></pre>
<br />
<span style="font-size: large;"><b>Conclusion</b></span><br />
Although the operators selected for use in the algorithm obviously do have a factor in the obtaining of a good solution, it appears that they do not have a significant impact on just how good a solution can be found. It may be a leap of faith at this stage but it is my belief that the fitness calculation will have a far more significant impact, and that is something that I will explore in the next post.<br />
And on that theme, the next post may possibly be delayed as I am currently in hospital and have no access to my PC to do any coding, but hopefully I will get my hands dirty again fairly soon and be able to produce another post.Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-3322570847105807902015-03-10T08:20:00.001-07:002015-03-17T08:53:01.384-07:00Implementing a mutation algorithmIn the previous post we looked at solving a continuous variable problem with a genetic algorithm, and found that the restriction of the mutation algorithm meant we found a good, but not excellent solution as the value of each gene was set at the time the population is created. These could be sequenced into many different solutions (chromosomes), but did not actually change value. With a continuous variable problem we clearly need to tweak the values, so we need to try to create a new mutation algorithm to do this. There are a number of questions that spring to mind instantly when thinking about how to do this:<br />
<br />
<ol>
<li>How do we vary the genes?</li>
<li>How many genes in a chromosome should change in a given mutation?</li>
<li>What if a mutated chromosome is unviable?</li>
</ol>
<div>
The answer to these and all the other questions that are whizzing round your head vying for attention is simple IT DOESN'T MATTER.</div>
<div>
<br /></div>
<div>
the way we vary the genes is an implementation detail, we could do it in any number of ways, add a small value to it, subtract one, multiply by 1.0001, half it, anything. And how many we change is again an implementation detail, we change 1 or 2 or all it doesn't matter, any choice made here simply creates a different mutation algorithm, the evolutionary process will still continue. There may well be a sweet spot for the degree and detail of the mutations for finding a good solution quickly, but that is not the be all and end all for GAs, we just want to explore as much of the problem space as possible. Finally, if we mutate in such a way as to create an unviable chromosome, it will be culled, and will not have an effect on future generations. That said, for very specific mutation algorithms we may wish to mutate in such a way that some constraints are not violated. In the example problem from the last post, where the sum of the gene values must be 1, we may wish to chose to mutate in such a way that this is not violated. This constraint would always be violated if we simply change the value of a single gene within a chromosome, so a mutation that is aware of the problem may be required to allow some viable mutations to be created. </div>
<div>
In terms of software design this presents a horrible smell. We are coupling low level code to high level concepts, and specifically creating a single target mutation algorithm, it is not ideal, but the creation of an algorithm that will only create unviable chromosomes would be pointless, so some forethought must be used when formulating a mutation algorithm.<br />
<br />
<b><span style="font-size: large;">New Mutations</span></b><br />
<br />
I will look at 2<span style="font-size: large; font-weight: bold;"> </span>mutation operators in this post, and detail the results of using each. The first of these mutation operators simply takes a single gene of a chromosome, modifies its value and adjusts the other genes so that the constraint of the sum of gene values remains a constant. In practise this operator will select a proportion of the population of solutions not tagged as being elite solutions (elite solutions are earmarked to remain unchanged between generations) and mutate these. The code I implemented to test this mutation is:<br />
<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">1: public void Invoke(Population currentPopulation, ref Population newPopulation, FitnessFunction fitnesFunctionDelegate)
2: {
3: var eliteCount = newPopulation.Solutions.Count(ch => ch.IsElite);
4: var numberToMutate = (int)System.Math.Round((newPopulation.PopulationSize - eliteCount) * MutationProbability);
5: newPopulation.Solutions
6: .Where(ch => !ch.IsElite) //get the non elite
7: .Take(numberToMutate) // selct the right percentage of these
8: .ToList()
9: .ForEach(ch => DoMutate(ch));
10: }
11: private void DoMutate(Chromosome ch)
12: {
13: //need to select a value from the chromosome mutate then ensure the sum is maintained
14: var geneIndexToMutate = rand.Next(ch.Genes.Count);
15: var gene = ch.Genes.ElementAt(geneIndexToMutate);
16: //now change this value, it must have a new value of less than SumOfGenes, so
17: var newGene = new Gene(rand.NextDouble() * SumOfGenes);
18: //and now modify the others, so find new sum, what do we need to multiply this by to get 1??
19: var m = 1d / ch.Genes.Except(new []{gene}).Sum(g => g.RealValue) + newGene.RealValue;
20: newGene = new Gene(newGene.RealValue * m);
21: //now multiply every gene By this to get mutated chromosome with correct sum
22: var AllNewGenes = ch.Genes.Take(geneIndexToMutate)
23: .Select(g => new Gene(g.RealValue * m))
24: .Concat(new[] { newGene })
25: .Concat(ch.Genes.Skip(geneIndexToMutate + 1)
26: .Select(g => new Gene(g.RealValue * m)))
27: .ToList();
28: ch.Genes.Clear();
29: ch.Genes.AddRange(AllNewGenes);
30: //Debug.WriteLine("New Sum {0}", ch.Genes.Sum(g => g.RealValue));
31: }
</code></pre>
A guide to implementing genetic operators with a full example of a mutation operator can be found at <a href="http://aiframeworks.net/blog/gaf-part-5">http://aiframeworks.net/blog/gaf-part-5</a>.<br />
<br />
The mutation requires knowledge of the sum of gene values constraint, and this is set on the construction of the operator, this can be seen as a shortcoming of the operator for general use, but for the purposes of investigating the effects of value mutation on chromosomes made up of a set of continuous variables, I have chosen to overlook this.<br />
<br />
The results of using this operator are not as impressive as one would possibly expect. Using the out of the box SwapMutator values of R=0.988 and F=0.247 are found. Using the new mutation operator, R=0.993 and F=0.248 are found. A slight improvement in the objective mimisation, with a loss in the constraint. Now this is a very simplistic mutation of the gene values, with no memory of the previous value retained or used in the mutation. As such I thought I should try another variation on the operator<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">17: var mult = (minMultiplier + this.range * rand.NextDouble());
18: var newGene = new Gene( mult * geneValue);
</code></pre>
<br />
where minMultiplier +this.range will give a value between 0.5 and 1.5 based on a range value between 0 and 1. This produces results of R=0.995 and F=0.248 for a range of 0.9; and R=0.990, F=0.247 for a range of 0.2.<br />
So the retention of some of the parent's identity does not seem to help.<br />
<br />
<span style="font-size: large;"><b>Mutate by Pairs</b></span><br />
<span style="font-size: large;"><b><br /></b></span>
The second mutation I chose to try was the mutation of a chromosome by changing a pair of genes and leaving all others untouched. By this method the sum of the genes values could be maintained by careful selection of the new values. The code to perform this mutation is very similar:<br />
<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGWTiEUsNaWxeIB4-mtarEqwVQ6ih2B35Xr1lPDVaaKd2V0ddfi94_xxHx_GzKJbgL-LIApsfOJvAFqTaFURKujvw5zG0E2TB1qgmPfdzmcBgcWlpP7m0AlXWTPpZL2QpANMQ892OO29OO/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;">1: public void Invoke(Population currentPopulation, ref Population newPopulation, FitnessFunction fitnesFunctionDelegate)
2: {
3: var eliteCount = newPopulation.Solutions.Count(ch => ch.IsElite);
4: var numberToMutate = (int)System.Math.Round((newPopulation.PopulationSize - eliteCount) * MutationProbability);
5: newPopulation.Solutions
6: .Where(ch => !ch.IsElite) //get the non elite
7: .Take(numberToMutate) // selct the right percentage of these
8: .ToList()
9: .ForEach(ch => DoMutate(ch));
10: }
11: private void DoMutate(Chromosome ch)
12: {
13: if (ch.Genes.Count <=1) return;
14: //need to select a value from the chromosome mutate then ensure the sum is maintained
15: var geneIndexToMutate = rand.Next(ch.Genes.Count / 2);
16: var geneIndexToMutate2 = geneIndexToMutate;
17: while(geneIndexToMutate>=geneIndexToMutate2)
18: geneIndexToMutate2 = rand.Next(ch.Genes.Count / 2, ch.Genes.Count);
19: //now change this value, it must have a new value of less than SumOfGenes, so
20: var newVal1 = -1d;
21: var newVal2 = -1d;
22: while(newVal1 <=0 || newVal2 <=0)
23: {
24: newVal1 = rand.NextDouble();
25: newVal2 = (ch.Genes.ElementAt(geneIndexToMutate).RealValue+ ch.Genes.ElementAt(geneIndexToMutate2).RealValue) - newVal1;
26: }
27: //and now modify the others, so find new sum
28: var AllNewGenes = ch.Genes.Take(geneIndexToMutate)
29: .Concat(new[] { new Gene(newVal1) })
30: .Concat(ch.Genes.Skip(geneIndexToMutate + 1)
31: .Take(geneIndexToMutate2 - geneIndexToMutate - 1))
32: .Concat(new[]{ new Gene(newVal2) })
33: .Concat(ch.Genes.Skip(geneIndexToMutate2 + 1))
34: .ToList();
35: ch.Genes.Clear();
36: ch.Genes.AddRange(AllNewGenes);
37: //Debug.WriteLine("New Sum {0}", ch.Genes.Sum(g => g.RealValue));
38: }
</code></pre>
<br />
which gives results of R=0.989 F = 0.247.<br />
<br />
<b><span style="font-size: large;">Conclusions</span></b><br />
<br />
So it appears that the form of the mutation is not the limiting factor in finding a good solution to the problem. There may well be a better mutation possible, but the quality of the mutation id more likely to impact on the speed of convergence (not strictly a convergence as its a pseudo random exploration of the problem space) rather than the final result.<br />
<br />
In coming posts I will explore the use of different breeding operators and finally the effect of the fitness calculation on the final result.</div>
Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-56029581002997941152015-03-04T01:46:00.001-08:002015-03-09T10:21:30.924-07:00Interlude - Why a blog?<p dir=ltr>You may think 'interlude, but we just started, I don't need a break yet, give me more!' and you would be right.  This post comes now because I should have written it first, or put some of the contents in to the first post, but I was excited about getting some ideas down on paper to make a starting point for our learning journey in the potential use of GA that I couldn't hold back.  So I am writing it now as an aside to my ongoing work on GA.</p>
<p dir=ltr><b>So why write a blog?</b></p>
<p dir=ltr>I started writing this blog for a number of reasons, I know from past experience that the best way to get a good understanding of something new is to try to explain it to others.  Teaching is the best way to learn.  So that was a nice selfish reason.  My day job is a bit mundane at present, a lot of repetitive coding, so a more interesting distraction mentally is welcome.  And the reason why now, is that I joined <a href="http://simpleprogrammer.com/">John Somnez</a>'s <a href="http://devcareerboost.com/blog-course/">blogging course</a> to give myself the boot up the backside to do more.  He gives a load of reasons why writing a blog is a good idea, mostly based around self promotion, career progression and money.  All very important factors in the life of a software dev, and all have their place in my list of priorities, but above all else I love to learn and challenge myself.  I also recently started a non-software blog on my personal life (<a href="http://tribewardle-ms-journey.blogspot.com/">diagnosed with MS recently</a>) and I am enjoying writing  that immensely, so a software based blog I thought would be a great idea to keep my mind fresh on interesting dev topics, and importantly dev topic I care about.</p>
<p dir=ltr><a href="http://devcareerboost.com/blog-course/">John's blogging course </a>is a very useful education tool, very informative, easy to follow and eloquent in its encouragement for everyone to get blogging, go and give it a read, and if you start blogging let me know I love to read what others are doing (after all that how I got here by reading John's blog)</p>
<p dir=ltr>Next post will be back on topic</p>
Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0tag:blogger.com,1999:blog-2700301865750835750.post-36657618873047119592015-03-02T03:00:00.000-08:002015-03-17T08:53:31.055-07:00Can a Genetic Algorithm Approach Solve a Continuous Variable Minimisation Problem?Some time ago I was working on a project where the main blocker to make the whole thing a game changing, world beating, make millions and retire, product was the ability to solve a large constrained minimisation problem in a continuous variable problem space.<br />
<br />
Please don't be put off by the maths at this stage, I am simply looking to set the scene, I will not be delving into equations, rather the concepts of solving a class of problems. Essentially I had an equation something like:<br />
<br />
<span style="font-size: large;"></span><br />
<div class="separator" style="clear: both; text-align: center;">
<span style="font-size: large;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO5RC7TpwfLSPIkUkFnrU1IoEAsPc93fAr_esAUmFRc9p3NooB8sG4ah4Y1xkKd-wXzbUEE2Dxqkr3CqDfCzrpn1ryzGOFnN41LOmzBMwKp2pBDNRohJkrk2mpk3JZHNRAEpVBF__1d7sV/s1600/whole.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgO5RC7TpwfLSPIkUkFnrU1IoEAsPc93fAr_esAUmFRc9p3NooB8sG4ah4Y1xkKd-wXzbUEE2Dxqkr3CqDfCzrpn1ryzGOFnN41LOmzBMwKp2pBDNRohJkrk2mpk3JZHNRAEpVBF__1d7sV/s1600/whole.PNG" /></a></span></div>
<br />
<br />
with this example being a simplified version in 2 dimensions, the real problem I was tackling had 19 dimension, but the concept is the same.<br />
<br />
To give some real values to the problem for those of us who don't think in terms of pure mathematics, my simplest test scenario involved having a limit of N = 2 a matrix for k of<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_Fj3042NKrUu2P7Me67PM2cvZ_ZE3eW8Bu7zAQx02y3ZossXMTM8R7r0u9cglol7iQUiovcvXOotnexjtqaMgccd8ATDr1OA5PzOU6lPFnHNu-9dATRJsF7agdszim7kFSyZEx13O3lX0/s1600/matrix.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_Fj3042NKrUu2P7Me67PM2cvZ_ZE3eW8Bu7zAQx02y3ZossXMTM8R7r0u9cglol7iQUiovcvXOotnexjtqaMgccd8ATDr1OA5PzOU6lPFnHNu-9dATRJsF7agdszim7kFSyZEx13O3lX0/s1600/matrix.PNG" /></a></div>
<br />
<br />
setting the values of H and R to be 0.5 and 0.95 respectively, we have<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjU6DUdNqjp85sCtcr2qty1VdYFhu5kYSA1YveejmvBCwJD-efROvZs9RT5i1aEFqCejgSBdIKtHOFjMatLLnxQ7qa-MtjDsRagTmaWzL0ddFdK4DLPNuTlceI43AJ4TCmNJUXJpr1LLixg/s1600/expanded.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjU6DUdNqjp85sCtcr2qty1VdYFhu5kYSA1YveejmvBCwJD-efROvZs9RT5i1aEFqCejgSBdIKtHOFjMatLLnxQ7qa-MtjDsRagTmaWzL0ddFdK4DLPNuTlceI43AJ4TCmNJUXJpr1LLixg/s1600/expanded.PNG" /></a></div>
<br />
which on the face of it seems fairly straightforward. Brute force should make it fairly easy to solve, it can be reduced to a one dimensional problem, and very basic techniques could be used to solve. But without going to town on expanding the problem, imagine adding a third dimension, or simply increasing N to 3.<br />
<br />
There are many techniques out there for solving such constrained non-linear minimisation problems, such as the use of Lagrange multipliers or numerical approaches like interior point interpolation. However I was, and still am interested in whether using genetic algorithms could be a fruitful avenue of investigation for this category of problem.<br />
<br />
Genetic algorithms have a number of advantages over purely mathematical approaches. The problem of local minima being found being a key one, and from a computational point of view, the scalability of a GA approach drew me in.<br />
<br />
<b>What is a Genetic Algorithm</b><br />
<b><br /></b>
The idea behind a GS Approach is that given a problem that we have no idea of how to get to a good solution of, but where we know a good solution when we see it, we can explore the problem space finding progressively 'better' solutions until such a time as we are satisfied with the best solution tested.<br />
In practice this means that we need to have a way to evaluate how 'good' a solution to the problem is. For the problem described earlier, we need a way to determine if for a given set of x the solution is acceptable, and how good a solution it is. As we have a constrained problem, only solutions for which the 2 constraint equations are satisfied are permissible. A simple measure of how good each permissible solution is would be how far from 0 the minimisation part resolves to.<br />
Given this ability to evaluate how good a solution is (known as the fitness of the solution) the algorithm needs a way to generate new potential solutions in an educated manner to find ever better solutions, and then to iterate this loop.<br />
<br />
The term genetic algorithm takes its origins from Darwinian evolutionary theory. A population of potential solutions exists, from which the fittest are selected, a further generation of solutions is born (variation in the population created via cross breeding and mutation), the weakest being culled, the fittest surviving, reproducing and evolving through the generations.<br />
<br />
Given a broad enough initial population and sufficient generations, all corners of the problem space should be explored, with good, but not necessarily the best solutions not swamping out rarer but stronger strains.<br />
<br />
<b>Seems ideal?</b><br />
<b><br /></b>
Yes it does, calculating the fitness of a solution in this way is very fast and cheap. The calculation of the fitness of any solution is an atomic process, so a huge population is not a problem as we can easily scale out. So what is the problem and why aren't all mathematical problems approached this way?<br />
<br />
One problem comes that a solution may be investigated that is a poor fit and discarded, another that is a fair but not excellent fit is returned in preference. over time we get a healthy population of fair solutions, but nothing good enough. Through mutation an cross breeding we will always continue to explore new solutions until the algorithm exits, however the details of the breeding and mutation limit the ability to find an 'ideal' solution.<br />
<br />
The mere fact that we have discarded any possibility of a solution that violates the constraint equations out of hand means that we may be ruling out entire sections of the problem space that could evolve to yield a better solution.<br />
<br />
A post that explores the idea of constraints that tighten through the generations is <a href="http://scicomp.stackexchange.com/questions/5118/multi-objectives-fitness-function-with-hard-and-soft-constraints">here</a>, and suggests that ruling out solutions early in the evolutionary process is a bad idea, with solutions that violate the constraints becoming less favourable in later generations, but in early generations having the opportunity to influence the overall population.<br />
<br />
<b>How to implement a GA in .NET</b><br />
<b><br /></b>
A very good question, you could spend years writing the evolution, mutation, breeding and selection code. However, good news, there is a great framework that does the hard work for you, allowing you to concentrate on the problem you are really interested in. Using <a href="http://aiframeworks.net/gaf">GAF</a> allows you to concentrate on evaluating the fitness of solutions, defining how the mutations occur and how breeding is performed. With only the need to define 2 functions (evaluate the fitness of a solution and determine if we should stop evolving) you can solve a problem. A good introduction to the framework is given <a href="http://www.codeproject.com/Articles/873559/Implementing-Genetic-Algorithms-in-Csharp">here</a> with examples of problems you can solve.<br />
<br />
The specific functions I used for this problem are:<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> private static double CalculateFitness(Chromosome solution)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> var tol =1e-3;</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> var unfit = 0;</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> //need to calculate the proximity of the three equations to 0</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> var sum = solution.Genes.Sum(x => x.RealValue);</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> if (System.Math.Sqrt(System.Math.Pow(sum - 1, 2)) > tol) return unfit;</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> if (GetDeltaR(solution.Genes.Select(x => x.RealValue)) > tol)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> return unfit;</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> return 1d-(GetDeltaF(solution.Genes.Select(x => x.RealValue)));</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> }</span><br />
<br />
<span style="font-family: inherit;">Where GetDeltaR and GetDeltaF calculate the difference between the result of the calculation and 0, the returned fitness is best at 1 and a solution is totally unfit if this evaluates to 0.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">private static double GetDeltaF(IEnumerable<double> enumerable)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> return System.Math.Sqrt(System.Math.Pow(GetF(enumerable) - TargetF, 2));</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> private static double GetDeltaR(IEnumerable<double> enumerable)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> return System.Math.Sqrt(System.Math.Pow(GetR(enumerable)-TargetR, 2)); </span><br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> }</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">and</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">private static bool Terminate(Population population, int currentGeneration, long currentEvaluation)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> {</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> return currentGeneration > 400;</span><br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"> }</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: inherit;">The decision to terminate is simply based on the number of generations here, only because experience </span><span style="font-family: inherit;">has told me that this will not reach a solution where delta F is less than an acceptable level.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><b>So it doesn't solve the problem?</b></span><br />
<span style="font-family: inherit;"><b><br /></b></span>
<span style="font-family: inherit;">Not as such. With the fitness calculation above and the choice of breeding and mutation algorithms (I used the out of the box single point crossover breeding and swap mutation provided by GAF) a very good solution is not found. However it does find a solution, and with some minor tweaks to only the fitness evaluation (taking into account the ideas of relaxed constraints in the linked article) a solution with R = 0.943 and F = 0.235 can be found in seconds. Clearly this is not an ideal solution, but it is pretty good as a first stab.</span><br />
<span style="font-family: inherit;">For a problem such as this, the key to getting a good solution is choosing the correct mutation algorithm and breeding process. The swap mutation used simply swaps round the elements of the solution, they do not 'vary', so a solution of (1,2) could mutate to (2,1) but could not become (1.1,1.9). As we are dealing with continuous variables, the inability to mutate in this way will limit the proximity to the solution that we can get.</span><br />
<span style="font-family: inherit;">The crossover breeding takes 2 solutions, and uses part of each to produce a child solution (e.g. 1,2 and 3,4 could become 1,4) which is acceptable given a suitable mutation and sufficient generations to explore all evolutionary branches.</span><br />
<span style="font-family: inherit;"><br /></span>
I am interested in developing a new mutation algorithm that modifies the elements (termed genes in genetic algorithm circles) of the solution and will detail this process in a later blog.<br />
<br />
Suffice to say with little effort using GA to solve a complex mathematical problem is a good way to go. The effort to get a truly 'good' solution, may be a little greater, but to get a solution there is little effort. I have previously attempted to solve the class of problems described here using interior point interpolation with some success, however the starting point of these interpolations is key to finding a good solution as they are very prone to finding local minima. As such, a hybrid approach may be the answer, with GA selecting a range of potential starting points from which to interpolate rather than using either approach in isolation.<br />
<span style="font-family: inherit;"><b><br /></b></span>
<span style="font-family: inherit;"><b><br /></b></span>
<span style="font-family: inherit;"><br /></span>Anonymoushttp://www.blogger.com/profile/01577538907635765828noreply@blogger.com0