Label Cloud

Wednesday, March 18, 2009

Linq Provider for Oracle Coherence – Pt 2

In previous post I published about a Linq for Oracle Coherence. Linq makes it very convenient to add functionality to a provider, however creating a functionally reach and complete provider is a very complicated process.

Latest functionality I’ve added was ability to use coherence extractors in the query. Linq does not allow you to change its keywords and syntax, however, all functions are converted to linq expressions and are passed to the ExpressionVisitor for evaluation. First thing is to create an extended function to provide the functionality

public static T Extractor<T>(this object obj, string Name)
{
    throw new NotSupportedException();
}
public static T ChainedExtractor<T>(this object obj, string Name)
{
    throw new NotSupportedException();
}

The functions themselves do not have any functionality and are used solely to be converted to a linq expression. I’ve created them as Generic functions to be able to strongly type against extracted data

In the QueryTranslator class VisitMethodCall() function, we add functionality to evaluate the expression

if (m.Method.DeclaringType == typeof(LinqFunctions))
{
    if (m.Method.Name == "Extractor")
    {
        this.Visit(m.Arguments[1]);
        string right = (string)globalFilter;
        globalFilter = new Tangosol.Util.Extractor.ReflectionExtractor(right);
        return m;
    }
    else if (m.Method.Name == "ChainedExtractor")
    {
        this.Visit(m.Arguments[1]);
        string right = (string)globalFilter;
        globalFilter = new Tangosol.Util.Extractor.ChainedExtractor(right);
        return m;
    }
}

m.Arguments[0] has the reference to the parent object. m.Arguments[0] contains the argument. This allows us to create a Coherence Extractor that is used in the filters higher in the expression tree.


A Note: I hope that the project gets some visibility and comments. Oracle Coherence is an excellent product with great .NET potential. Check out the project on google code: http://code.google.com/p/linqtocoherence


Share/Save/Bookmark

Tuesday, March 17, 2009

Linq provider for Oracle Coherence (Linq to Coherence)

I am very impressed with Coherence from Oracle. Coherence provides a distributed in-memory cache and processing fabric. However it is a lot more then just a cache. It can be used for everything from messaging to cross platform communication medium. There is too much to talk say about it, so read more information at Oracle: http://www.oracle.com/technology/products/coherence/index.html

Coherence works very nicely with .Net however, in the days of Linq, I wanted to write a Linq provider for it. My code is based largely on the Linq provider documentation on MSDN (http://msdn.microsoft.com/en-us/library/bb546158.aspx) and excellent series on creating a linq provider by Matt Warren (http://blogs.msdn.com/mattwar/pages/linq-links.aspx)

I am using Google Code to host the project under Artistic License. Please check out the full source code at http://code.google.com/p/linqtocoherence/.

Below is a rundown on two main classes. The main part of the code that deals with Coherence is in two classes CoherenceQueryProvider and CoherenceQueryTranslator.

CoherenceQueryProvider accepts a connection to the INamedCache – a reference to coherence cache that will be queried.

public class CoherenceQueryProvider  : IQueryProvider
{
   public INamedCache Cache { get; set; }
   public CoherenceQueryProvider ()
    {
    }

   public CoherenceQueryProvider(INamedCache cache)
   {
       Cache = cache;
   }

In the Execute method, CoherenceQueryProvider translates the Where clause to a Coherence Filter and executes the filter against the Cache objects to return array of values.

public object Execute(Expression expression)
{
  if (Cache == null)
      throw new InvalidOperationException("Cache is not properly set");

  // Find the call to Where() and get the lambda expression predicate.
  InnermostWhereFinder whereFinder = new InnermostWhereFinder();
  MethodCallExpression whereExpression = whereFinder.GetInnermostWhere(expression);
  LambdaExpression lambdaExpression = (LambdaExpression)((UnaryExpression)(whereExpression.Arguments[1])).Operand;

  // Send the lambda expression through the partial evaluator.
  lambdaExpression = (LambdaExpression)Evaluator.PartialEval(lambdaExpression);

  IFilter filter = new CoherenceQueryTranslator().Translate(lambdaExpression);

  object[] data = Cache.GetValues(filter);
  Type elementType = TypeSystem.GetElementType(expression.Type);
  return data;
}


CoherenceQueryTranslater uses the visitor pattern to convert the Linq Expression from the where clause to Coherence Filter. Coherence filters are nested to converting one to the other is relatively simple

protected override Expression VisitBinary(BinaryExpression b)
{
  this.Visit(b.Left);
  object lastGlobal1 = globalFilter;
  this.Visit(b.Right);
  object lastGlobal2 = globalFilter;
  switch (b.NodeType)
  {
      case ExpressionType.AndAlso:
          globalFilter = new AndFilter((IFilter) lastGlobal1, (IFilter)lastGlobal2);
          break;
      case ExpressionType.OrElse:
          globalFilter = new OrFilter((IFilter) lastGlobal1, (IFilter)lastGlobal2);
          break;

There is a lot more code in the classes to handle other filters, but a lot of it is pretty repetitive. The work on the linq provider is not done and I still have to implement some of the coherence functionality. Full code and usage sample is available on google code http://code.google.com/p/linqtocoherence/

Check it out and post your comments / suggestions.


Share/Save/Bookmark

Sunday, March 15, 2009

Embrace Change

We as developers spend our careers creating tools, innovating, making things better through technology. However, as a developer we also constantly deal with users who are refusing to change. Examples are everywhere

  • Doctors refusing to replace their filing cabinets with record keeping software. I’ve heard of this just this weekend from my wife.
  • People “Hate” new look and feel of software because it looks different (Office 2007 is a great example. I’ve also heard “hate it” comments about new Facebook look)
  • Users refusing to use new procedures and software because “old way works just fine”

The world is changing all the time. Technology accelerates this change and we can either fight it, or embrace it and make it work for us. I say – Embrace Change
ECbaseball_800x600


Share/Save/Bookmark

Wednesday, March 04, 2009

noop.nl - Top 50 New Software Development Books and other lists

I generally don’t link to other blog entries since that doesn’t add that much value to people. However, this post is not regarding the specific blog entry. Jurgen has an excellent blog dedicated to software development and management of development teams.

He also created excellent lists of TOP *EVERYTHING*. The last one published is Top 50 New Software Development Books. Other lists are at http://www.noop.nl/top-lists/

This is one of the blogs I would definitely recommend subscribing to.

Technorati Tags: ,,


Share/Save/Bookmark

Wednesday, February 11, 2009

Production Debugging a Memory Leak

I wrote before about not believing in regular system reboots. One of the services we wrote had a serious memory leak and process size grew over 1GB within 2 days requiring us to perform regular service restarts. This is not something that we were able to replicate in development or QA environment so I’ve decided to do some production debugging.

I love reading the blog of Tess Ferrandez on low level .NET Debugging. http://blogs.msdn.com/tess. The has a series walk trough sessions one of them is on Memory Leaks http://blogs.msdn.com/tess/archive/2008/03/25/net-debugging-demos-lab-7-memory-leak.aspx

I can’t really provide the original code for our service, but I was able to replicate the basic leak in a sample app, and below are steps to find out what it is.

Sample (on skydrive.live.com)

LeakyCache.zip-download


Sample Setup: Open LeakyCache.zip  Compile it if you want, or just run the included executable. Click “Leak” to leak memory.

LeakyApp

  1. Download Debugging Tools for Windows form Microsoft and install it on the server that is running the problem application.
  2. Copy SOS.DLL from “C:\Windows\Microsoft.NET\Framework\v2.0.50727” to “c:\Program Files\Debugging Tools for Windows (x86)” to get access to debugging library for .NET 2.0
  3. Execute ADScript to take a memory dump of the LeakyCache application
    "c:\Program Files\Debugging Tools for Windows (x86)\adplus.vbs" -hang -pn LeakyCache.exe -o c:\temp\LeakDump

LeakyAppDump

  1. Start WinDbg
    "c:\Program Files\Debugging Tools for Windows (x86)"\windbg
  2. From the File Menu, select “Open Dump File” and open the created dump file from C:\temp\LeakDump\
  3. Load SOS debugging using command
    .load SOS

Now the fun begins :)

  1. Run !dumpheap –stat
     DumpHeap
    What you’ll see is that the most memory is used by data type is System.String (53MB) and Dictionary+Entry (22MB). Also notice that there are more then 1 million string entries. Most of them are very small (<55 bytes average).
  2. To see the entries. Use command  (Press CTRL+BREAK to stop the flow) to see the list of addresses.
    !dumpheap -type System.String -max 100
    !do 022b8978 
    DO
    Substitute the address of one of the items instead of the 022b8978
    I underlines a Text String that you can see. In my experience of debugging my apps, based on the data, I can tell what is stored, and probably have some ideas about where that data is generated or should it be cleaned.
  3. Run !gcroot [Reference] to see exactly what class is holding a reference to the object
    GCRoot

A walkthrough like this will not necessary solve a problem in the application, but it can point out to a possible issue in the application that can be solved. To me, a memory leak is not a problem that should be ignored, but is a bug that can be fixed.

Note: Huge thanks to Tess for the wonderful blog http://blogs.msdn.com/tess

Technorati Tags: ,,


Share/Save/Bookmark

Monday, February 09, 2009

Keep release PDBs to help with debugging production problems

Not many developers know that PDB files are generated during release builds are just as helpful as they are in debug builds.

For some background information, PDB Files contain debugging symbols that are used by .NET Debuggers (including Visual Studio) to lookup source code information. If symbols are available, debugger will be able to show not just the function where exception happened, but also the line number in the source file where exception occurred.

Currently, our current build process copies results of every build into a separate output folder, away from the source code. A new step was just added to make a copy of PDBs into a subfolder as well. Here’s a snippet of XML that I’ve added to the .csproj target

    <CreateItem Include="$(TargetDir)\*.pdb">
      <Output TaskParameter="Include" ItemName="PDBFiles" />
    </CreateItem>
    <Copy SourceFiles="@(PDBFiles)" 
            DestinationFolder="$(OutputPath)\PDB" />

One way to use PDB files is to provide them with your application. If PDB file is available at the time of exception, Exception information will include line numbers and source code file name in the exception.

You can also debug release versions of the executable using Visual Studio. From the Tools menu, select “Attach to Process”, Select your executable. After debugging session starts, In Debug menu, Window->Modules, Right click on the module for your executable, and select “Load Symbols From”. Point to your PDB file, and you are done. It will be important to have source code available if you want to step through. That however is a completely different issue.

Technorati Tags: ,,


Share/Save/Bookmark

Tuesday, February 03, 2009

The Social Processional Network (Twitter, Friendfeed, Facebook, Linkedin, etc..)

I guess the original idea behind a social network is to socialize – communicate to others. The idea of group communication is not new at all. Some people might remember Bulletin Boards from 15-20 years ago, Compuserve / AOL (Still alive somewhat) / NNTP News groups (still available but are very different). Social networks got popular with MySpace and FaceBook. They were originally discarded as “play time” activities and are even now often restricted from access through corporate networks. However their professional use is increasing rapidly.

Stack Overflow

Even though I’ve been using social networking in my professional day-to-day for a wile, I’ve recently started to get a lot more engaged. Some of the dynamics of the site are astounding. I wrote a few comments on my www.StackOverflow.com experience. I’ve been using the site as a public forum for development related questions. It is amazing how fast other developers respond to the questions asked. if a question in well phrased, it can get multiple answers within minutes (sometimes even seconds)

LinkedIn

Another tool is my network arsenal is my LinkedIn. The network grows exponentially as you start adding contacts. At this point, I have 185 direct connections. However, my total network is more then 2.2 million people. Considering that I only add people that I’ve been communicating with into my contacts list, that’s 2.2 million people that I can be pretty comfortable in asking for an introduction to. Until recently, I’ve used LinkedIn to do some background research on potential candidates. However, recently, I’ve started to get involved in the professional groups and ask questions. Unlike StackOverflow, LinkedIn has a very broad range of people that use it to connect to each other. This allows me to tap into the pool of resources and information that is wider then any one area of expertise. At the same time, since this is not a pure “social” network, all communication is professional.

Blog

StackOverflow and LinkedIn are great places to ask for information. Blog on the other hand is a great way to share it. There are blog networks that will provide you not only a place to host your blog, but as a community of readers that are interested in the contents. That will get you feedback on the topics you write.

Twitter

Micro Blogging with Twitter is relatively new (from October 2006). However it is hard to understate the extend of the idea, the importance (and simplicity) of the technology and the reach of the network. There is a lot of talk about twitter on the web

Twitter can be used for anything from saying “I am bored” or “I am having lunch” to a more professional “Listening to so-and-so at the conference” or “Found this great article at http://www…” the short statement you make is broadcasted to all subscribers. You can include a subscribers' @name and the message (though still public) will be flagged on his screen so it is noticed. The whole interaction feels like room full of people with multiple conversation going on. You focus on a conversation with one or two people, but you can still overhear others. If you hear something interesting, you join in.

FriendFeed

FriendFeed is an interesting service that takes information push to the next level. It creates one channel for sharing all your information. It connects to over 30 networks including Twitter, Blogs, Facebook, LinkedIn, Flickr, Del.ici.os, and others. It allows to setup friends and will follow their feeds as well. You can even setup an “imaginary” friend to organize someone’s information if they are not participating in FriendFeed.

Importance of Information Push

One thing to notice, is that I am putting a lot of emphasis on pushing the information to others. That is the important aspect of having a social network. Google does excellent job in providing an ability to search and access public information. However, it is the ability to interact by asking questions, by publicizing your ideas is what makes social networks really useful in professional atmosphere.


Share/Save/Bookmark

Saturday, January 31, 2009

Kids on the Net (KidZui review)

The internet age now starts much earlier. My got my first computer when I was 10 and it was Radio-86RK built by my father.

My kid’s computer experience is much different: desktop and two laptop computers are basically always available for them to us.

We got a few computer games as presents, but none of them were as good as the kids websites that we found. Here’s some of  the more favorite ones

Check out my full list of kids related sites at http://delicious.com/guessman/kids

One of my latest finds was . KidZui is a browser specifically designed for kids. Instead of working as a filer and preventing kids from going to restricted sites, it is designed to be the one and only application that the kids will interact through. The User Interface is great, very kids friends. Every page is hand picked by KidZui and verified to be Child Safe. This is really a Closed System browser.

Downloading the application from http://www.kidzui.com/downloading is straight forward. I had to install the application, and register myself as a parent. My KidZui account allows full monitoring of what my son views through the program. It will also send me a daily activity report with screenshots of the sites that he visited.

My son was able to pick and character, not only give him the name, but also select what the character would look like. The browser looks excellent. Very interactive, colorful, easy to navigate. KidZui folks did a truly remarkable job. I’ve put some screenshots of the UI below.

My son loves the browser. He can get around to all his favorite sites by himself. The search bar on top allows to quickly look through all available content.

We still try to limit the amount of time my kids spend at the TV and the computer. But while he’s on it, he is totally loving it.

Kidzui

Kidzui2

Technorati Tags: ,,


Share/Save/Bookmark

Tuesday, January 27, 2009

Batch converting Unicode files to ASCII in Subversion (utilizing Powershell)

Recently we’ve worked on importing our database structure into subversion. After scripting out the database schema into thousands .SQL files, and checking them in, we’ve realized that the files were creating in Unicode. Subversion used binary encoding when sending the files into the repository. Even though client side tools were able to work on files without any problems, all server-side tools (like FishEye, Bamboo, etc…) considered them binary, and were not able to properly process them. Batch converting them to ASCII took a little research, but overall was relatively straightforward.

After checking the structure out to the local drive, I’ve used a 1 line Powershell script convert the files to binary (note: this is one lone line)

dir --recurs -include *.sql | foreach {$FileName = $_; $fileData = get-content -path $_; out-file -filePath $FileName.FullName -inputObject $fileData -encoding ascii}

After having all files changed to ASCII, we had to remove the svn:mime-type property. To do that, run the following:

snv propdel svn:mime-type –R *.sql

Finally, we checked in the changes back into the repository.

Technorati Tags: ,


Share/Save/Bookmark

AddToAny.com

Just made a small addition to my blog. I am using buttons from www.addtoany.com to provide a better interface to subscribe to the blog, as well as share links to the individual entries.

Hopefully readers will find them usefull

Technorati Tags: ,


Share/Save/Bookmark

Thursday, January 22, 2009

Remotely shutdown Outlook (Another IPhone hack)

I wrote about creating an Outlook rule to help out with IPhone / Exchange push issue. One thing I realized, is that sometimes, I leave the Outlook running after I leave the office. That breaks the Exchange push to the IPhone. I wanted to be able to shutdown the Outlook remotely.

  • Download PSExec from the Microsoft SysInternals suite (http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx)
  • Create a batchfile “ShutdownOutlook.bat with one line “c:\SysInternals\pskill Outlook.exe” (Replace c:\SysInternals\ with the path to pskill)
  • Create a rule
    • Process messages when they arrive
    • from “Your User Name” and Subject “Shutdown Outlook”
    • Start a Program “ShutdownOutlook.bat”

Now, all you have to do is send yourself and email with a subject “Shutdown Outlook” to shut down your outlook. I guess this can be used for other situations as well.

Update: Just corrected the command in the batch file. Originally had PSExec not PSKill (think one type the other)

Technorati Tags: ,,


Share/Save/Bookmark

IPhone Exchange push peeves and some workarounds

I am really impressed with my IPhone. I’ve switched to it from the old Blackberry Curve and now every time I use a smart phone without touch screen, it is VERY, VERY annoying.

There are however a few peeves that I have about its functionality. One of the main ones is Exchange Push connectivity. It works great when you only have an inbox, however, with multiple mail folders and server side filtering, IPhone does not receive notifications of the email in the other folders. This is definitely an IPhone issue, since Exchange push protocol fully supports monitoring of multiple folders. A minor Outlook rule made the Synchronization work the way I want it it.

  • Setup a client side rule that is applied to every email before it is processed. Mine is to display a notification box (Another Outlook issue is that if messages are moved to a folder in a server, popup notification will not be display!)

When I am in the office, my Outlook is opened, and messages get sorted into their appropriate folders. When I am out, the Outlook is generally closed. Since the client rule can not be applied, Emails will stay in the Inbox, and will be properly pushed to the IPhone. Once outlook is opened, emails will get sorted into appropriate folders.

Technorati Tags: ,


Share/Save/Bookmark

Monday, January 19, 2009

CAPICOM.dll Removed from Windows SDK for Windows 7

Its not that often that I hear that of a system component of Windows SDK being removed from a future version of windows. As the matter of fact, this is the only time that I know off (I am sure it happened before)

Karin Meier from Windows SDK Team announced on his blog that CAPICOM is now considered to be depreciated and is providing alternatives at http://msdn.microsoft.com/en-us/library/cc778518(VS.85).aspx

Technorati Tags: ,


Share/Save/Bookmark

Saturday, January 17, 2009

OpenID – Using it and Liking it

You might have noticed a little icon on some of the sites you are visiting. The icon is a log of the OpenID network. From the openid.net, “OpenID is a free and easy way to use a single digital identity across the internet.”

OpenID was developed in 2005 and is now accepted by many mainstream websites. Some of the big players are Google, Yahoo!, IBM, Microsoft, AOL, MySpace. More history on Wikipedia. OpenID is a very open API and a number of Open Source free libraries are available for a variety of languages. A list more then 30 is available at the http://wiki.openid.net/Libraries

For me the benefit of OpenID is tremendous. I have a large number of sites that I register on, and having to remember my User ID and Password is a hassle. There is the option of using the same one for all the websites, but then having them in sync is even worse. OpenID provides a perfect solution. One secure website to manage remember and use.

I was following a discussion on StackOverflow on why NOT to use OpenID. Main points are

  • If the UserID / Password is compromised then attacker will get access to all sites that are registered with OpenID
  • Overall complexity, an less then technical user would have a hard time registering and using OpenID

My suggestions to OpenID use,

  • Do not use it on the sites where account security it critical (For example anywhere money is involved)
  • Use a trusted OpenID provider (Google, Yahoo!)
  • Create a strong password on the OpenID provider

Here some more Pro/Con links with very good points to read.

http://www.shoemoney.com/2007/02/20/11-reasons-why-openid-rockssucks/
http://radar.oreilly.com/archives/2007/02/pros-and-cons-o.html
http://lifehacker.com/software/technophilia/one-openid-to-rule-them-allor-not-302156.php
http://idcorner.org/2007/08/22/the-problems-with-openid/


Share/Save/Bookmark

Wednesday, January 07, 2009

Benefits and Hindrances of Regular Server Reboots

First of all, stackoverflow.com is very, very cool. I’ve talked about it before, and would like to reiterate the point. The site gets tremendous amount of traffic and is great for asking any development questions or starting technology related discussions.

Now to the main point.

Over the years that I’ve doing software development and architecture, I had a chance to work directly on server and data center architecture. One of the most important points of the software and hardware design was stability, which is generally measured in amount of uptime. We’ve spent a lot of time looking for memory and other resource leaks. Servers needed to be designed with the same resiliency in mind.

However, I’ve also worked with IT managers with extensive experience, who followed a different paradigm: Weekly, controlled reboots of all servers. I looked around and the policy is not at all uncommon:

I brought this question to Stack overflow for more comments, please check them out there.

http://stackoverflow.com/questions/410413/benefits-and-hindrances-of-regular-server-reboots

Some consider this a Foolish Policy, For Others, this is a weekly test that is good if you can afford it. Even though I definitely see benefits of testing startup scripts and resource cleanups, I would have to stand behind my original view: Reboots for the sake of rebooting are an overkill, adds downtime and waists resources personnel resources. Scheduled maintenance windows for server maintenance (hardware and software) and are a completely different story.


Share/Save/Bookmark
Directory of Computers/Tech Blogs