Removing Duplicate Items from an Abstract Generic List

Ryan Baxter — Thu, 31 Jul 2008 07:40:00 -0700

I’ve got some explaining to do. I was hesitant in posting this code for fear that it might be too niche to benefit anyone. It may be, but the underlying problem affects many programmers working in the IT industry. What do you do when the company legacy system’s data model doesn’t work with your fancy ORM (Object-relational mapping)?

“Any problem in computer science can be solved with another layer of indirection. But that usually will create another problem.” - David Wheeler

At my last job, the ERP (Enterprise resource planning) system was something of mystery and voodoo. Only a few had sufficient knowledge to work with it and because of its arcane nature it was deemed untouchable. Within my first months as an employee I wrote a rudimentary ORM to serve as a layer between our client applications and the ERP system. Its performance was terrible. I wrote it off as “experience” and the department ultimately decided to ignore interfacing directly with the ERP system.

I’m using the SubSonic ORM on a few ASP.NET projects with my current employer. SubSonic has worked great, but a few of its database requirements have left me in the dust with yet another ERP system. The ERP’s tables have no primary keys, constraints, or relationships, but rather than write a complete ORM I decided to roll my own data layer.

All of the strongly-typed collections in the ERP’s data layer implement the AbstractList type. AbstractList implements List where ItemType implements IUniqueIdentifier. That is a mouthful, but the key (pun intended) to removing duplicate items is to make sure they’re unique.

The IUniqueIdentifier interface contains only one property, UniqueIdentifier.

public interface IUniqueIdentifier
{
    string UniqueIdentifier
    {
        get;
        set;
    }
}

If the ERP system contained a table called Customers a Customer type implementing IUniqueIdentifier would be created.

public class Customer : IUniqueIdentifier
{
    private string uniqueIdentifier;

    // etc...

    public UniqueIdentifier
    {
        get { return this.uniqueIdentifier; }
        set { this.uniqueIdentifier = value; }
    }
}

I’d also have a CustomerCollection class:

public class CustomerCollection : AbstractList<Customer, CustomerCollection>
{
    // etc...
}

After such a long-winded introduction I can feel better about dumping the following code on anyone that has happened to read this far (kudos to you).

public abstract class AbstractList<ItemType, ListType> :
List<ItemType> where ItemType:IUniqueIdentifier where
ListType:AbstractList<ItemType, ListType>, new()
{
    public ListType RemoveDuplicates()
    {
        Dictionary<string, int> uniqueStore = new Dictionary<string, int>();
        ListType list = new ListType();

        foreach (ItemType item in this)
        {
            if (!uniqueStore.ContainsKey(item.UniqueIdentifier))
            {
                uniqueStore.Add(item.UniqueIdentifier, 0);
                list.Add(item);
            }
        }

        return list;
    }
}

I’d love to hear how others have worked around legacy systems and still kept their code clean.

crunchlife: Removing Duplicate Items from an Abstract Generic List

Removing Duplicate Items from an Abstract Generic List