Real Coders Coding

Lambdas, Extension Methods, and Asynchronous Execution Example

Bogdan Varlamov (a.k.a. Phantom Stranger) — Tue, 08 Sep 2009 17:18:26 +0000

I've often found myself working on a class in some legacy application using objects hidden away in third-party libraries and thinking, "Wow... it would be really great if I could call this current existing method but have it process asynchronously (without blocking) and then notify me when it is complete."

The only way to do this would be to inherit the legacy object and add a new "Async" version of the method, right? And what about those pesky sealed classes? Not to mention the problems with having to refactor your whole application (and perhaps any dependent components used by other applications).

It can be quite a mess! But, luckily there is a simpler way...

What I've done is create an extension library that allows any object to do asynchronous processing.

Here is an example of how to use this:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
 
using _5DSolutions.Extensions.AsyncHandler; //here are the extensions
 
namespace Example
{
    public class Program
    {
        static long _asyncComplesCounter = 0;
        static long _disposedCounter = 0;
 
        static void Main(string[] args)
        {
            GoCrazy();
        }
 
        private static void GoCrazy()
        {
            //here we have some "legacy" object that we cant update/add new methods to
            TestObject tmp = new TestObject();
 
            int counter;
 
            Console.WriteLine("How many times would you like to asynchronously print?");
 
            while (!int.TryParse(Console.ReadLine(), out counter))
            {
                Console.WriteLine("Enter an integer number only, less than {0}", int.MaxValue);
            }
 
            while (counter > 0)
            {
                //here we are using the extension method on some "TestObject" class that we can't modify
                tmp.HandleAsynchronously((sender, e) =>
                    {
                        Console.WriteLine("Asynchronously calling a synchronous method");
                        tmp.DoSomething();//this blocks
                        Console.WriteLine("Woohoo! We are done blocking the synchronous method call, notifying through the completed event handler...");
                    },
                    (sender, e) =>
                    {
 
                        System.Threading.Interlocked.Increment(ref _asyncComplesCounter);
                        Console.WriteLine("---An Asynch write has finished! -- {0}", System.Threading.Interlocked.Read(ref _asyncComplesCounter));
 
                    },
                    null,
                    (sender, e) =>
                    {
                        System.Threading.Interlocked.Increment(ref _disposedCounter);
                        Console.WriteLine(">>>>>>>>> disposed");
                    },
                    null);
                counter--;
            }
 
            Console.ReadLine();
            tmp = null;
            GC.Collect();
            Console.WriteLine("Total completed events: {0}", _asyncComplesCounter);
            Console.WriteLine("Total disposed bg-workers: {0}", _disposedCounter);
 
            Console.WriteLine("Go again?");
 
            if (Console.ReadLine().Equals("Yes", StringComparison.InvariantCultureIgnoreCase))
            {
                GoCrazy();
            }
            else
            {
                //just exit
            }
        }
    }
}

There is nothing tricky going on with the TestObject class. In fact, you should be able to use your own class. Or you can use this test object:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
 
namespace Example
{
    public class TestObject
    {
        public TestObject()
        {
 
        }
 
        public void DoSomething()
        {
            Console.WriteLine("Synchronously writing this then waiting for 0.5 seconds...");
            System.Threading.Thread.Sleep(500);
            Console.WriteLine("--Synchronous write Done.");
        }
    }
}

The real magic actually lives in the Extensions project:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ComponentModel;
 
namespace _5DSolutions.Extensions.AsyncHandler
{
    public static class ObjectAsyncExtension
    {
 
        public static void HandleAsynchronously(this Object caller, DoWorkEventHandler onDoWork, RunWorkerCompletedEventHandler onCompleted)
        {
            HandleAsynchronously(caller, onDoWork, onCompleted, null, null, null);
        }
 
        public static void HandleAsynchronously(this Object caller, DoWorkEventHandler onDoWork, RunWorkerCompletedEventHandler onCompleted, ProgressChangedEventHandler onProgressChanged, EventHandler onDisposed, object argument)
        {
            //check required params
            if (null == onDoWork)
                throw new ArgumentNullException("onDoWork");
 
            if (null == onCompleted)
                throw new ArgumentNullException("onCompleted");
 
            BackgroundWorker tmp = new BackgroundWorker();
 
            tmp.DoWork += onDoWork;
            tmp.RunWorkerCompleted += onCompleted;
            tmp.RunWorkerCompleted += (sender, e) =>
            {//note, we subscribe twice to "dispose" of the object
                //and raise the Disposed event
                tmp.DoWork -= onDoWork;
                tmp.RunWorkerCompleted -= onCompleted;
                tmp.ProgressChanged -= onProgressChanged;
                tmp.Dispose();
                tmp = null;
            };
 
            if (null != onProgressChanged)
            {
                tmp.WorkerReportsProgress = true;
                tmp.ProgressChanged += onProgressChanged;
            }
 
            if (null != onDisposed)
            {
                tmp.Disposed += onDisposed;
                tmp.Disposed += (sender, e) =>
                    {//i dont think this is really necessary...
                        //but leaving it in for reference
                        tmp.Disposed -= onDisposed;
                    };
            }
 
            if (null != argument)
                tmp.RunWorkerAsync(argument);
            else
                tmp.RunWorkerAsync();
        }
 
    }
}

Of course you can substitute your own named delegates for the lambdas in the Program class in this example, or your own lambdas to do actual work.

The main concern I had with this implementation was whether or not this code would be subject to thread leaks.

I tested this code with 30k calls to HandleAsynchronously and watched the threads/memory/handles fluctuate, but always drop back down as garbage collection was performed. (I am explicitly calling it in the code samples above to better observe potential issues, but it should happen naturally without the need for an explicit call).

Filter Logs by Changing log4net Configuration

Bogdan Varlamov (a.k.a. Phantom Stranger) — Wed, 20 May 2009 17:37:09 +0000

We've all been there before--trying to track down some insipid bug which is impossible to reproduce on a developer's machine. What is the response? To add more logging of course!

That's all well and good, but eventually your log files are going to start getting huge. Don't get me wrong, I LOVE applications with good logging. (In my opinion it's much better to "overlog" than "underlog" [or not log at all!]). But, sometimes it makes sense to cut back on the amount of logging to save disk space, or to make the log files easier to digest...

How can this be done? Certainly not by removing logging from code!

As you may or may not know, log4net provides various "logging levels" which range from "INFO" to "FATAL" with several in between. However, in some cases I've found that these levels are not enough.

For instance, there are some applications where it makes sense to log every single method entry and exit (yes, I DO realize this is usually discouraged as it gives an architectural layout of your application to competitors, etc.). But, there is no appropriate logging level to correspond to this. Ideally, one might have something that looked like this:

 
 
private void DoSomething()
{
        log.Enter(System.Reflection.MethodBase.GetCurrentMethod().ToString());
 
//do more code here
 
        log.Exit(System.Reflection.MethodBase.GetCurrentMethod().ToString());
}

But alas, there is no such thing. We have to use the "Debug" logging level if we are to log enters/exits, like so:

 
 
constant string LOG_ENTER = "Enter - ";
constant string LOG_EXIT = "Exit - ";
 
private void DoSomething()
{
      if(log.IsDebugEnabled)
      {
           log.Debug(LOG_ENTER + System.Reflection.MethodBase.GetCurrentMethod().ToString());
      }
 
//do more code here
 
      if(log.IsDebugEnabled)
      {
           log.Debug(LOG_EXIT + System.Reflection.MethodBase.GetCurrentMethod().ToString());
      }
}

This works (although it's not the prettiest code in the world). We can keep track of when a method is entered, when it is exited, calculate how long it takes, look for inefficiencies, troubleshoot much easier, etc.

But when all of that is done, we don't want to ALWAYS log the method entries and exits. We could change our logging level, but since we are using "DEBUG" all of our other debug log statements will also disappear. What can we do?

Luckily there is an easy way to filter out all of our method enters/exits just by updating the configuration file with a string filter setup to filter out "Enter" and another one for "Exit" like so:

Just add this to the "appender" element in the log4net section in your configuration file and uncomment/comment out as necessary.

Not So RESTful Days

Flash — Tue, 19 May 2009 18:13:44 +0000

One of my clients asked me to write a very simple RESTful web service to retrieve user data (name, email, address, etc.) based on login credentials. Having recently finished working on a PHP-driven RESTful interface for another client (based on the Zend Framework), I thought this would be a simple and straightforward task. As comedian Bill Engvall says: "Here's your sign!". I should have known better.

My first effort was to simply refactor my previous work. It took me a few hours but it worked great... on my development machine. However, once I ported it to the host for testing, it crapped out because they host their site on a Windows server under IIS7. So, having very little control, there goes the possibility for url rewrites (on the server level).

Next, I thought back to an recent WNC .NET Developer's Guild presentation given by Joel Reyes (of Microsoft) entitled ADO.NET Data Services. His code examples demonstrated a simple RESTful service in action. So, another few hours and I had a great working example. A few hours?!? Well, I had to download a few things to get my Windows XP up to snuff (such as .NET 3.5 SP1 among other things). There was my next sign... it didn't occur to me then that the host (Web.com) wasn't up to date either. Too late! After I deployed a quick test app I realized they were behind the times when I got this error:

Configuration Error

Description: An error occurred during the processing of a configuration file required to service this request. Please review the specific error details below and modify your configuration file appropriately.

Parser Error Message: Could not load file or assembly 'System.Data.Entity, Version=3.5.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies. The system cannot find the file specified.

Source Error:

Line 33:

Source File: \\HOSTING\DFS\20\3\3\4\2030628433\user\sites\dampp-chaserdb.com\www\test\web.config Line: 33

As a sidebar... I tried to ask their front-line technical support staff for some information on their current version of the .NET framework and if they had a workaround for URL rewrites on the box the site was hosted on. Yeah right!. That was another wasted 30 minutes.

Alack! And Alas! I decided it was time to force feed myself some ASP.NET MVC and quick. ASP.NET MVC has the ability to return XML and JSON via a RESTful interface. So, to keep this note short, I'm heading back to tuck into some tutorials. I purchased Effective REST Services via .NET by Kenn Scribner and Scott Seely so I've got to read through it quick. I'm also using blog posts from Piers Lawson as a guide. Here's a link to Part 1. Just filter by the REST tag to get a list of the other posts in the series. There's a bunch there.

One helpful note: If you find yourself using Piers' posts like I did, I had to fix an error that was preventing me from getting the same results as he did. In Part 1 in the Guid.aspx file use this code snippet instead of his example:


    
        <% foreach (var guid in (IEnumerable)ViewData.Model)
              {
         %>
              <%= guid.ToString().ToUpper()%>
        <% } %>

This may be because of the version of ASP.NET he used as compared to the latest bits.

I really dig MVC having been first exposed to it when writing custom components in Joomla. I'm psyched now that I have the chance to mix some MVC peanut butter with some ASP.NET chocolate. Now I'm off to get some REST.

(That was lame, sorry)

Reinstalling SQL Server 2005 (or 8 Hours of My Life Wasted)

Flash — Thu, 15 Jan 2009 15:03:41 +0000

The main purpose of RealCodersCoding is to post about problems we, the authors, have overcome in our daily coding adventures. Sometimes, the problems we are faced with are complex and require some interesting coding yoga to to work out. Other times the solutions are drop-dead simple, but we refuse to recognize them. Here's a little post about how I wasted an entire freakin' day to one of the latter.

I've been coding in the PHP/Joomla/MySQL space for the last 6 months and haven't cracked open my local MS SQL Server 2005 instance lately. When I tried to do so yesterday I was met with this gem--the app wouldn't connect to my local instance. Instead I got some bizarre error message about remote connections. Diving into my Event Viewer to see if there's any clues there, I'm told that the master database is corrupt. The error message helpfully guides me to the Books Online for information on how to rebuild the master database. The easiest solution would be to use a valid backup and just restore it. Ahem! Backup?!? This is my dev machine and I don't do much backing up of anything. I guess I'm stupid that way.

So, I do some research and tried all of the various methods the blog posts and KB articles tell me to try, as in:

All to no avail! So, I'm in a time crunch here. This maintenance work I'm doing should only take a few hours (that's all I'm getting paid for, at least). I decided, screw it, I'll do a reinstall. That'll fix it. That always works, right?

Right off the bat, I know I'm in trouble. I can't completely uninstall SQL Server 2005 via the traditional Add/Remove Programs (ARP). Artifacts are left over including all of the hot fixes and patches that have been applied over time. I follow other blog posts and msdn articles on how to manually uninstall SQL Server 2005, as in:

There's a ton out there, but they all pretty much say the same thing. The MSDN article did reference a Windows Install Clean Up utility which eventually helped me get rid of the hangers on in ARP. Finally, and we are talking about 2 hours into it, I start installing the new instance. I'm doing this from one of my MSDN DVD's. I navigate the DVD menu and manually run setup.exe from the proper directory (I'm installing the Developer version). The initial install was a snap!

However, I realize (a little later when nothing worked) that I only installed the client tools. No server! No services! No management tools! Arrg! So, I uninstall (again). Clean up after myself and start again. Now, I go a little slower to make sure I make the correct selections in the installer. However, now I'm not given any options for the server components. I go right from the registration screen to the feature selections. I skip the component selections completely. WTF?

My assumption is that there must be some latent artifacts from the previous installation (registry settings, installed services, files, etc.) that make the installer think I still have an instance set up. More blog posts and articles (and hours) and I still can't get the server components installed!

I finally give up and install SQL Server 2000. But that doesn't solve my problem since the DB I need for my maintenance work has been SQL Server 2005-ied and won't open in 2000 any more. :( I swear, cry, go eat dinner and resolve to fight this to the death (of one of us).

Later, I uninstall Visual Studio 2008 (thinking that maybe it's got to be installed after SQL Server 2005). That didn't work (and now I'm faced with the long reinstall [and I'm doing that as I type]). I'm about to tear my hair out (except I don't have any) when I start poking around the DVD directories and stumble across something very interesting: There's a setup.exe in both the Tools directory:

H:\ENGLISH\SQL2005\DEVELOPER\SQL Server x86\Tools\setup.exe

and the Servers directory:

H:\ENGLISH\SQL2005\DEVELOPER\SQL Server x86\Servers\setup.exe

I never once looked in the Servers directory. I just assumed (yeah, yeah, I know) that there's only one setup.exe that I need to deal with.

So the moral to this blog post, if moral there needs to be, is to make sure you know that you are installing the right thing. Otherwise, you'll end up like me - 8 hours into a 3 hour job. And I still have to finish installing VS.NET 2008. If all goes well, I should be done with this work sometime tomorrow.

System.ExecutionEngineException when opening a WPF Window from Windows Form

Bogdan Varlamov (a.k.a. Phantom Stranger) — Mon, 22 Dec 2008 14:50:06 +0000

One of the great things about .NET 3.5 is that it is fully inter-operable with .NET 2.0. You just have to reference the right assemblies and you can use WPF and Windows Forms from the same application! This is great for slowly migrating a .NET 2.0 application over to .NET 3.5; all you have to do is add the new functionality while keeping your old .NET 2.0 logic. Once you are ready, you can slowly migrate your old code over into .NET 3.5 as needed.

Well...at least it's all that easy in theory; there's just one pesky little hiccup--a System.ExecutionEngineException...

What I was trying to accomplish was to create a .NET 3.0 WPF application and run a Windows Form with all of my old .NET 2.0 logic. This was easy and worked great. Then, I tried adding a WPF window which would open when a button was clicked in the Windows Form.

I compiled my application, ran it, clicked the button and the WPF window opened up just as I was hoping. Excellent! I continued working in debug mode and made some code changes. I clicked the button again to see the code changes in action and...System.ExecutionEngineException.

It took me forever to figure out what the problem was. I undid all of my code changes, I even created a blank testing application and still got the same exception.

Apparently the problem is that a .NET 3.5 WPF application has a ShutdownMode property which by default is set to "OnLastWindowClose." This means that when the last WPF Window in the application is closed, the application itself shuts down.

Oddly enough the application doesn't close the Windows Form when it "shuts down"--it just disposes of internal objects.

So this is what was happening:

The application starts and shows a Windows Form.
Then the Windows Form shows a WPF Window.
Then the WPF Window closes and the application "shuts down" while leaving the Windows Form window open.
When I try opening another WPF Window I get the exception because the application I'm trying to open a window for has already been shut down.

There are two ways to get around this problem. The first is the "right way" to do it as described in the MSDN article; it is by setting the ShutdownMode property to OnExplicitShutdown. Then to shutdown the application you could create an event handler method to be called when the Windows Form's "FormClosed" event fires. This event handler would call the applications "Shutdown()" method.

Here is an example:

App.xaml.cs

using System;
using System.Collections.Generic;
using System.Configuration;
using System.Data;
using System.Linq;
using System.Windows;
 
namespace MultiWindowTest
{
    ///
    /// Interaction logic for App.xaml
    ///
    public partial class App : Application
    {
 
        private void Application_Startup(object sender, StartupEventArgs e)
        {
            Form1 f1 = new Form1();
            f1.FormClosed += new System.Windows.Forms.FormClosedEventHandler(f1_FormClosed);
            f1.ShowDialog();
        }
 
        void f1_FormClosed(object sender, System.Windows.Forms.FormClosedEventArgs e)
        {
            this.Shutdown();
        }
    }
}

An alternative way to avoid this exception is a bit of a hack; but still works. You can leave the ShutdownMode property alone and instead create a WPF window that is hidden. If you had an application that shows a splash-screen or a logon window it would make more sense to do this. The application loads, you show the logon window--once they logon this window is hidden and the Windows Form is shown. Once that form is closed you would once again show the logon WPF Window. Hiding a WPF Window does not trigger the Shutdown of the application, so you will be able to open and close as many WPF Windows as you like without worrying about getting the exception.

Detecting a physical network disconnection with .NET Sockets

Bogdan Varlamov (a.k.a. Phantom Stranger) — Fri, 31 Oct 2008 13:56:46 +0000

I recently ran into an issue with an application that was not detecting a physical network problem and was consequently losing data while attempting to write it to the socket.

I know what you are thinking. Sounds like poor programming; surely the .NET Socket class has a clean and Object Oriented way of handling network failures.

That is what I thought at first. The .NET Framework is filled with tons of goodies--namespaces with classes that make coding life easier. I am usually fairly satisfied with what is available in .NET (especially in .NET 3.5), but when it comes to the System.Net.Sockets namespace, there is just more to be desired.

At first glance, it seems like the Socket class provides exactly what you need to get the job done. There is even a "Connected" property that is exposed, which seems like a natural thing to check before trying to send data over a connection.

The way that this application was designed, however, makes it a moot point as to what the Connected property returns (If you check the MSDN, you will see that the Connected property actually returns whether or not the last IO operation on the socket was successful--not whether or not there is a connection currently).

This application was coded to handle failures to write to the Socket. The logic flow for this process looked something like this:

Add all outgoing messages into an outgoing Queue for storage in case of problems
Peek at the Queue to get the next message (not Dequeueing/Requeueing since the order of the messages matters in this application)
Attempt to write the bytes of the message to the socket
If all bytes were written successfully, Dequeue the message as we no longer need to store it for sending
If the entire message was not transmitted, reset the connection and retry sending upon reconnection

I looked at the code, and could not tell where the failure was occurring. This application was logging successful socket writes and the application on the other side never got the bytes! How was this nightmare possible?

Well... it was all thanks to the magic of TCP/IP. The "magic of TCP/IP" is a topic for another article (or book even) so I won't go into great detail here. Basically, once a TCP/IP Socket connection is "established" it is considered valid even if the physical connection is broken. In fact, if you unplug the OTHER computer, your computer won't know the connection is gone until you try sending data and the TCP/IP stack fails to deliver it after a certain timeout. TCP/IP was meant to be a robust protocol, being able to recover from intermittent physical connectivity failures. And it does a great job at what it was designed to do...

However, in my scenario I needed to know instantly (or almost instantly) whether or not the message was delivered to the other machine--not 30 seconds, 2 minutes, or 2 hours after we tried sending another message.

How could I know BEFORE a TCP/IP retry timeout whether or not the connection was still there? The answer: TCP keep-alives!

These are short messages that are basically just empty packets that need to be ACKed. When this message is sent from one computer to another, the TCP/IP stack replies to it with an ACK. The best thing about keep-alives is that any system that implements TCP/IP automatically supports keep-alives. Once they are turned on if the connection between the two TCP/IP stacks is physically aborted and a keep-alive is not ACKed within a certain time the sending stack knows that the connection is broken.

I searched how to set TCP Keep Alives from the managed .NET sockets and finally came across Larry Cleeton's blog entry which shares this method:

 
static void SetKeepAlive(Socket s, bool on, uint time, uint interval) {
/* the native structure
struct tcp_keepalive {
ULONG onoff;
ULONG keepalivetime;
 
ULONG keepaliveinterval;
};
*/ 
 
// marshal the equivalent of the native structure into a byte array
uint dummy = 0;
byte[] inOptionValues = new byte[Marshal.SizeOf(dummy) * 3];
BitConverter.GetBytes((uint)(on ? 1 : 0)).CopyTo(inOptionValues, 0);
BitConverter.GetBytes((uint)time).CopyTo(inOptionValues, Marshal.SizeOf(dummy));
BitConverter.GetBytes((uint)interval).CopyTo(inOptionValues, Marshal.SizeOf(dummy) * 2);
// of course there are other ways to marshal up this byte array, this is just one way
 
// call WSAIoctl via IOControl
int ignore = s.IOControl(IOControlCode.KeepAliveValues, inOptionValues, null); 
 
}

The "time" variable represents the time to wait between each TCP Keep Alive that is sent out. The "interval" variable represents the amount of time that the TCP/IP stack should wait to receive a response from the other machine.

By turning on the keep alives you will be able to know when the physical network connectivity is broken.

Minimize Asynchronous Socket Callbacks With Dynamic Buffer Sizes

Bogdan Varlamov (a.k.a. Phantom Stranger) — Wed, 08 Oct 2008 14:06:30 +0000

I've been doing a lot of work with .NET Sockets lately, and I will be posting more articles discussing what I've been working on in greater detail. In this article, I want to talk about the concept of reading a Socket asynchronously, raising events to notify your application when bytes are received, and minimizing the number of times the callback method is processed.

If you are familiar with .NET Sockets you might already know that there are asynchronous methods to receive bytes from Sockets (Microsoft has an Asynchronous Server Socket Example available if you are interested in seeing code for doing this).

In the above code sample, a StateObject class that contains a byte buffer with a static size is passed around. When the asynchronous socket read happens, incoming bytes are stored into this buffer and then passed back to the receive callback method. This example works fine, but I think there are some optimizations that can be applied.

To see where these optimizations can be applied let's think about what happens in the typical Socket reading application, but with real world examples. Let's pretend that our task is to pick up items that arrive at a train station and take them to our home.

In the Microsoft example this would look something like this:

You wait at the train station with a bag that can hold 1024 items. 3 items arrive, you stick them into your bag, then run home to drop them off. Then, you come back to the train station and wait for more items.

If 8000 items arrive, you put 1024 into your bag and take them home, then come back and put the next 1024 into the bag, take them home, and repeat until all of the items are home.

An alternative to this would be to just grab all of the items that arrive, stick them into a bag that holds exactly that amount of items, and bring them back once.

I haven't found a way to do exactly that in .NET Sockets, so the next best thing I have come up with would look like this:

You wait at the train station with a bag that can hold 0 items. 3 items arrive, you run home, get a bag that can hold 3 items, run back to the train station and fill your bag. Then, you bring the items home, get a bag that can hold 0 items and return to the train station to wait again.

At first this sounds inefficient...you have to switch out bags and run back and forth twice! Seems wasteful, but when you think about it you will notice that no matter how many items arrive you can always grab all items in just two trips.

So, what? It costs more to allocate memory for the new buffers every time than to keep reading into the same one!

While that may be true, when buffering into a fixed size buffer you will inevitably have some "left over space" in the buffer. So, you may be receiving into a 1024 byte buffer, but you only receive 1000 bytes. Now you have 24 empty bytes that are being wasted...And you will undoubtedly have to "trim up" your buffer before using the received bytes in the application.

In most cases you will probably do an Array.Copy to move over the bytes with real data into a separate byte[] of the appropriate size--allocating space for the new array, and using twice the memory during the copy operation!

In theory it might be less costly to reuse the same buffer, but in practice it might actually be more efficient to create the properly sized array in the beginning. Your application is going to do something with the received data, so in most cases the bytes need to be in a usable state...

I'm not advocating that you use my model for all of your socket communications logic, but there are some places where it might especially make sense to do so...

When your application:

receives greatly varying amounts of data (e.g. an app that receives a periodic and small keep-alive message, while also receiving large image files)
performs intensive processing each time data is received (e.g. received bytes are being buffered and scanned for application specific protocols)
receives large chunks of data intermittently

I've got some sample code to illustrate what my train station example would look like translated into C#:

//will be used to listen for incoming channel connections
TcpListener _tcpListener;
 
//socket will be used to determine connectivity/etc.
Socket _incomingSocket; 
 
//will control the socket buffer
//NOTE: if you implement some real-time network traffic analysis
//         or are using an application level protocol and know the min.
//         size of a message, adjust this value as appropriate!
int _incomingSocketByteBufferSize = 0;
 
public bool StartIncomingCommunications()
{
     bool returnValue = false;//assume problems
 
    try
    {
        //start listening
        _tcpListener.Start();
 
         //let us accept the connection when we get it
         _tcpListener.BeginAcceptSocket(
          new AsyncCallback(DoAcceptIncomingSocketConnection), null);
    }
    catch (Exception caught)
    {
          HandleLogging(caught);
    }
 
    return returnValue;
}
 
        //the callback to handle the incoming socket connection
        private void DoAcceptIncomingSocketConnection(IAsyncResult iaResult)
        {
            try
            {
                //this application only allows one socket connection at a time
               //so we need to check that our socket variable isn't already being used
                if (_incomingSocket != null)
                {
                    try
                    {
                        //we already connected by some socket
                        //we will close the current connection and establish a new one
                        //an alternative would be to reject the incoming socket
                        _incomingSocket.Close();
                        _incomingSocket = null;
                    }
                    catch (Exception caught)
                    {
                        HandleLogging(caught);
                    }
                }
 
                    // End the operation and get the socket
                    _incomingSocket = _tcpListener.EndAcceptSocket(iaResult);
 
                    //create a byte buffer to store received information
                    //this buffer is of size 0 by default and serves as a way to notify when bytes are received
                    byte[] readingBuffer = new byte[_incomingSocketByteBufferSize];
 
                     //the last parameter in this method should be a custom object
                     //i'm just passing along my readingBuffer since i only have one socket
                     //and don't care about anything other than the buffer
                    _incomingSocket.BeginReceive(readingBuffer, 0, readingBuffer.Length, SocketFlags.None, new AsyncCallback(DoReceiveCallback), readingBuffer);
 
                    NotifyOfConnectivityStatus();
 
                    //start accepting again
                    //this will allow us to process new connections
                    //an alternative would be to only do this once the current connection is closed
                    _tcpListener.BeginAcceptSocket(
                        new AsyncCallback(DoAcceptIncomingSocketConnection), null);
 
            }
            catch (Exception caught)
            {
                HandleLogging(caught);
            }
        }
 
        /// this method processes the receive callback
        private void DoReceiveCallback(IAsyncResult iaResult)
        {
            try
            {
                byte[] buffer = (byte[])iaResult.AsyncState;//extract our original buffer
 
                if (buffer.Length == 0)//this buffer was used as a notification of bytes received
                { //we will replace our "zero bag" with the appropriately sized one
                    buffer = new byte[_incomingSocket.Available];
                }
                else//read real data (got our appropriately sized bag)
                {
                    //some method that does something with the bytes we just got...
                    OnBytesReceivedOverIncomingChannel(buffer);
 
                    //maybe more bytes arrived while we've been sitting here?
                    if (_incomingSocket.Available > 0)//still bytes available for reading
                    {
                        buffer = new byte[_incomingSocket.Available];//get the bag for the bytes just arrived
                    }
                    else//done, go to notification cycle (get the zero-sized bag)
                    {
                        buffer = new byte[_incomingSocketByteBufferSize];
                    }
 
                }
 
                try //MSDN recomends trying to read from a socket and handling exceptions as a way
                {   //of detecting when the connection is broken..
                     //go back and wait for more data (go back to the train station)
                    _incomingSocket.BeginReceive(buffer, 0, buffer.Length, SocketFlags.None, new AsyncCallback(DoReceiveCallback), buffer);
                }
                catch (SocketException sckEx)
                {
                    if (sckEx.SocketErrorCode == SocketError.ConnectionReset)//socket conn got closed...
                    {
                        NotifyOfConnectivityStatus();
                    }
                    else //some other weird error
                    {
                        throw sckEx; //pass it up to the higher try/catch
                    }
                }
            }
            catch (Exception caught)
            {
                HandleLogging(caught);
            }
        }

To see that code work you have to paste it into your own class, and add your own methods for HandleLogging, and NotifyOfConnectionStatus, etc. I'm working on something for which I will post the complete code later, so keep checking RealCodersCoding.com! Hopefully you get the general idea from the code in this article...if not, just leave me a comment : )

Tips on Creating Excel Spreadsheets via PHP

Flash — Mon, 06 Oct 2008 16:16:52 +0000

One of my current clients has an old FoxPro (old as in created in '97 and upgraded in '99) database application that the users can't stand. It frequently crashes and it's all too easy to lose valuable data after a lengthy data entry session (just by clicking the sadly misnamed Save button). We are porting them over to a LAMP-based web application. One of the early tasks in this project is to analyze their existing database structure. This application involves hundreds of FoxPro DBF files which are tables (data and schema) that reside in various child directories.

My initial approach (following the KISS principle) was to inspect some of these files and see what's in 'em and how to get the schema out of 'em. I downloaded a trial version of DBF Viewer 2000 and it's a fine tool. I was able to view the data and the schema and, using SnagIt, I captured the text and pasted the values to an Excel spreadsheet. But purely manual approach would take way too long for me. Time to whip out a quick script. The problem: .NET or PHP or other? I'm a recovering .NET dev working on PHP apps. I surprised myself by deciding to go for the PHP option. And I'm glad I did. Here's how I was able to create an Excel spreadsheet with hundreds of worksheets in PHP.

I assumed I'd be able to plug into some existing PHP-based Excel spreadsheet API and knock this out. Boy was I surprised. There's a few classes related to Excel over at PHPClasses.org but they were either too simple (not enough features I needed) or unsupported or just plain useless. The Spreadsheet Excel Writer PEAR package has way too many bugs in it and I couldn't get it to work on my setup (probably my fault). It then hit me that moden versions of Excel (since 2003?) can read and write XML (given a proper schema). After some quick testing (involving creating a simple spreadsheet and exporting to XML and viewing the output in a text editor) I decided the best solution would be to create an Excel-flavored XML file.

Here's the source data in Excel:

And here's the 92 line XML output for this data:




 
  Brian
  Brian
  2008-10-06T15:14:58Z
  2008-10-06T15:17:02Z
  One Who Serves
  12.00
 
 
  10170
  20655
  600
  360
  False
  False
 
 
  
 
 
  
    This is A1
    This is B1
   
    This is A2
    This is B2
   
   
   
  
  
   
    
    
    
   
   
   
    
     3
     17
     11
    
   
   False
   False
  
 
 
  
  
  
   
    
    
    
   
   False
   False
  
 
 
  
  
  
   
    
    
    
   
   False
   False

However, most of the elements are either static or unimportant (for my simple needs). So, here's the part that's most important:

 
  
    This is A1
    This is B1
   
    This is A2
    This is B2

Everything before the first Worksheet tag became a static heredoc header and the footer is simply the closing tag. I didn't require setting any so I removed that element completely.

Now that I figured out how to layout an XML fragment that represents a worksheet, all I needed to next was the following:

loop through the directories where I stored the DBF files in question (using PHP's RecursiveDirectoryIterator class)
parse each DBF file for the data schema (here's how I learned how to do this)
loop through the parsed data (using foreach loops on arrays)
generate the appropriate and tags

After much trial and error, I finally succeeded in creating an well-formed XML document. I probably should have used some existing PHP XML API but, for the sake of time, I went with the brute force method of hacking my way to a well-formed XML document (via heredoc strings and concatenating variables and new line characters, etc.). However, this XML file did not open in Excel for many reasons and Excel was not being reasonable with me and giving me information I can use for debugging my text. I kept getting (nearly) useless errors like this:

I couldn't find the log mentioned either. After much more experimenting, I finally learned the following lessons:

Worksheet names need to be unique (duh but not so obvious in the heat of the moment).
- Set the ss:Name attribute for the tag.
Worksheet names are limited to 31 characters.
- This sucked since I was using a version of the file path to create unique worksheet names.
- I solved this one by using sequential numbers for the worksheet names and creating a sort of table of content worksheet that maps the files to their matching worksheet (cumbersome but effective).
The ss:ExpandedRowCount attribute for the tag needs to match the number of actual rows to be displayed in each worksheet.
- This one took me a long time to debug because of the aforementioned useless Exel error message.
In the end, I have my Excel spreadsheet with 183 worksheets in it (one for every DBF file and the master TOC). I can regenerate it at will. And, most importantly, I can refactor it later and use it again to extract the data from the DBF files (probably to export the data to MySQL for later use). While I was fighting the ExpandedRowCount issue, I did find another method of creating Excel Spreadsheets in PHP that I may use in the future - PHPExcel. Once I got it up and running on my dev machine, I was amazed at what I could do with it. My script would be much cleaner without all of those heredoc strings in it. Plus, PHPExcel is well-documented and has a growing base of users which can only bode well for the future of this set of classes.

System.Threading.Timer Stops Firing in Windows Server 2003 SP1

Bogdan Varlamov (a.k.a. Phantom Stranger) — Wed, 24 Sep 2008 13:16:35 +0000

Do you have an application that starts misbehaving randomly? You aren't quite sure what is going on, but it seems like for some reason your Timer just stops firing it's event handler--and once it stops it never starts back up.

This drove me nuts! But, it is definitely a problem in the Windows Server 2003 SP1. This bug is particularly devious since even if your application uses extensive logging it maybe be nearly impossible to find proof of a timer not firing in your logs.

Short of implementing a Homer Simpson type of "everything is okay alarm" is there anything you can do to determine whether or not your application might be suffering from dying timers?

When I realized that my application had timers that randomly stopped firing, I set out on an Internet quest to figure out if anyone has ever had this problem before. And, yes, I found many many forum threads and blog posts already discussing this issue (just do a quick Google Search if you don't believe me). The problem was that everything I found was from over a year ago and only mentioned .NET 1.1. Well... I was having what seemed like the same issue, but with .NET 2.0.

It was nearly impossible to find anything from Microsoft about the issue. The only thing I came across was this article which had a misleading title, but contained the following (plausible) explanation:

The Timer classes are implemented as a linked list of timer objects. When the first System.Threading.Timer object is created, the thread pool manager starts a thread to process the linked list. Every timer object is added to the linked list. The thread that process the linked list cycles through the linked list and determines when the timer event is expected to be signaled against the current clock count.

If the timer object has expired, the thread asynchronously queues a callback function before the thread updates the time that the timer event is expected to be signaled. After the thread has processed all timer objects in the linked list, the thread updates the time that the linked list was last processed. Then, the thread calculates the shortest time that the thread should sleep before the thread reprocesses the linked list for the next elapsed timer object.

Sometimes, when the system is under stress or when the linked list includes many timer objects, the processing thread may be pre-empted by a higher priority thread before the whole linked list has been processed. When this behavior occurs, the time that a timer event is next expected to be signaled is calculated to be earlier than the timestamp when the linked list was last processed. Therefore, the time that the timer event is expected to be signaled is in the past and never expires.

Because the time that the timer event is expected to be signaled has already passed, the thread may calculate a negative period to wait before the timer event must be signaled. When the thread has a negative period to wait, the thread enters a sleep state for a long time.

I decided that the only way to be sure that this bug is caused by the Server 2003 SP1 (and not the .NET 1.1 Framework SP1 as the Microsoft article claims) would be to test using the .NET 2.0 Framework.

After some more grueling searching I came across a Google Groups Topic that contained source code to test System.Threading.Timer bug. Basically, all it does is create 30,000 Timers and tells them all to go off in 5 seconds. Then it prints out the updated count each time one fires--so if all of them fire the application will count up to 30000.

I built the testing application from the source, and ran it on the 2003 SP1 machine. The first time I ran the process all 30k timers fired (this got me worried), but after running it a few more times I was able to see that sometimes hardly ANY fired at all (the lowest I saw was around 11,000).

This proved that this System.Threading.Timer problem is well alive in .NET 2.0 Framework and that it is in fact caused by Server 2003 SP1 (not only the .NET 1.1 Framework SP1). I have tested the same process on SP2 and SP3 and each time all 30000 timers fire.

I have not tried to install the hotfix which Microsoft says will also fix the problem, but there is one available here: KB900822. Please feel free to respond to this post if the hotfix has worked for you.

Spreading the Word w/ Information Radiators

Flash — Thu, 18 Sep 2008 20:55:28 +0000

I was introduced to the world of agile software methodologies and, in particular, Scrum at my last company. In fact, on my first day new whiteboards were delivered to almost every office and a book called Agile and Iterative Development: A Manager's Guide by Craig Larman was handed out to at least one member of each dev team. We were told to choose an agile approach that works best for our team and go for it. However, I later learned that we were to choose any approach we wanted as long as it was called Scrum. In that effort, I helped to wrangle our usual cat herding approach to software development into something closely resembling Scrum processes. It was fun and challenging and a very eye-opening experience.

Along the way, we struggled with various ways to maintain transparency so that we could satisfy our higher-ups (or the chickens in Scrum parlance) desire to know the status of our sprints and when we would be ready to deploy, deploy, deploy. Mostly I'm talking about burndown charts and product/sprint backlogs. If you have no clue as to what a burndown chart or backlog is... I implore you to follow the links and come back for the rest of the story. The chickens wanted an easy way to know where we were and the dev teams wanted an easy way to keep them up to date (and out of their hair). We started w/ Excel backlogs and burndowns, toyed with 3rd party commercial software and ended up with a custom wiki/issue tracker implementation that works but is not very lightweight. It really involves our internal TWiki, our custom bug tracker and several tools to integrate the two so they communicate properly (involving java, groovy, .NET, XML, etc.). What we really needed and should have started with was some sort of task board (otherwise known as an information radiator) to make sure we were following Scrum principles as opposed to just displaying data for the sake of technology.

Now, at my current job, we are implementing Scrum but only in spurts (one project at a time) and keeping technology out of the mix as much as possible (at first). I really wanted to stay away from Excel and wikis and what not and use a task board. The model I followed came from Henrik Kniberg's excellent paper called Scrum and XP from the Trenches:

My information radiator consists of two big, sticky pieces of wall size Post-It paper, index cards (w/ tape) to indicate major tasks and tons of yellow Post-It notes representing individual tasks. I generate the grid for the burndown chart online (set to a grid spacing of 2 lines per inch), print it out and tape it the task board.

This works for us so far. The chickens can walk by and find out how we are doing in the current iteration and the dev team can easily see what specific tasks are left to do and who's doing what. Eventually, we'll move to a more permanent surface (although the sticky paper and tape is working fine and hasn't failed me yet). I doubt we'll go so far as actually using electrons to do the representin'. I mean, c'mon, I'm only a software developer. Geez, who's got time to write internal tools these days?

I came across a few other examples from various developers across the 'net and thought you'd be interested to see what your peers are working on:

AgileJoe at LostTechies.com

Many examples at mountaingoatsoftware.com

Not so low tech version from Serge Beaumont

From AgileCoaching.dk

Who says you need a wall when a window works fine