Label Cloud

Showing posts with label Open Source. Show all posts
Showing posts with label Open Source. Show all posts

Tuesday, April 07, 2009

Linq to Coherence + Attributes + MetaData = Cool

First, we got the Linq Provider for Oracle Coherence

Second, we got attribute based serialization

Third, Metadata in the serialization stream

Once we put all that together, what we get is a set of very clean way of storing and querying data in Oracle Coherence.

A Coherence Linq provider now supports passing a CoherenceQueryTranslator as a parameter. I am providing a MetadataCoherenceQueryTranslator that uses getProperty method to access property originally serialized by the Generic Serializer. Here’s almost all relative .Net Code:

Person Class

[POFSerializableObject(StoreMetadata=true)]
    public class Person// : IPortableObject
    {
        [POFSerializableMember(Order=0,WriteAsType=POFWriteAsTypeEnum.Int16)]
        public int ID { get; set; }
        [POFSerializableMember(Order=1)]
        public string FirstName { get; set; }
        [POFSerializableMember(Order = 2)]
        public string LastName { get; set; }
        [POFSerializableMember(Order = 3)]
        public string Address { get; set; }
        [POFSerializableMember(Order = 4)]
        public string Title { get; set; }
        public Person()
        {
        }
    }
Add object function
INamedCache cache = CacheFactory.GetCache("dist-Person");
for (int i = 0; i < 1000; i++)
{
cache.Add(i, new Person()
{
   ID = i,
   FirstName = string.Format("First Name {0}", i),
   LastName = string.Format("LastName {0}", i),
   Address = string.Format("Address {0}" , Guid.NewGuid()) ,
   Title = i % 2  == 1 ? "Mr" : "Mrs"
   });
}

Query using Linq Query:

CoherenceQuery<Person> coherenceData =
 new CoherenceQuery<Person>(
	 new CoherenceQueryProvider(CacheFactory.GetCache("dist-Person"), 
		 new MetadataCoherenceQueryTranslator()));
string likeClause = "%8";
var people = from person in coherenceData
			 where
				(person.FirstName.Like("Test")
				 || person.LastName.Like(likeClause))
				 && person.Title == "Mrs"
			 select new { person.Title, person.ID, person.LastName };
IFilter filter = ((ICoherenceQueryable)people).Filter;
dataGridView1.DataSource = people.ToArray();

Internally, MetadataCoherenceQueryTranslator, will convert the linq query into a filter and execute the query against the Java POFGenericObject


Share/Save/Bookmark

Monday, April 06, 2009

99.9% pure .NET Coherence (100 % No Java Code Required)

Its not 100% .NET Coherence, since a developer is required to modify configuration files on the java service to specify the generic POF Serializer.

So to start: The .NET Class:

 

    [POFSerializableObject(StoreMetadata=true)]
    public class Person
    {
        [POFSerializableMember(Order=0,WriteAsType=POFWriteAsTypeEnum.Int16)]
        public int ID { get; set; }
        [POFSerializableMember(Order=1)]
        public string FirstName { get; set; }
        [POFSerializableMember(Order = 2)]
        public string LastName { get; set; }
        [POFSerializableMember(Order = 3)]
        public string Address { get; set; }
        [POFSerializableMember(Order = 4)]
        public string Title { get; set; }
        public Person()
        {
        }
        
    }

Note the StoreMetadata=true argument. When StoreMetadata is specified, Generic serializer will first first write a string array of property names. On server side, we must specify a Generic Java serializer. This is needed to be able to store objects for filtering. Here’s an interesting note. Unless filters are invoked, the object WILL NOT be deserialized on the java side. That means, Put, Get, GetAll calls without a filter, do not require Metadata to be written into the cache. Now. To specify Java Generic Serializer. Distributed Cache Configuration:

      <distributed-scheme>
      <scheme-name>dist-default</scheme-name>
      <serializer>
		<class-name>com.tangosol.io.pof.ConfigurablePofContext</class-name>
		<init-params>
		  <init-param>
			<param-type>string</param-type>
			<param-value>custom-types-pof-config.xml</param-value>
		  </init-param>
		</init-params>
	  </serializer>
      <backing-map-scheme>
        <local-scheme/>
      </backing-map-scheme>
      <autostart>true</autostart>
    </distributed-scheme>

custom-types-pof-config.xml

<pof-config>
  <user-type-list>
    <!-- include all "standard" Coherence POF user types -->
    <include>coherence-pof-config.xml</include>
    <!-- include all application POF user types -->
    <user-type>
      <type-id>1001</type-id>
      <class-name>com.Coherence.Contrib.POF.POFGenericObject</class-name>
      <serializer>
        <class-name>com.Coherence.Contrib.POF.POFGenericSerializer</class-name>
        <init-params>
           <init-param>
             <param-type>int</param-type>
             <param-value>{type-id}</param-value>
           </init-param>
           <init-param>
            <param-type>boolean</param-type>
             <param-name>LoadMetadata</param-name>
             <param-value>true</param-value>
           </init-param>
         </init-params>        
	  </serializer>
    </user-type>
  </user-type-list>
</pof-config>

For now, you must specify a user-type for each .Net object. On the java side, the server will be using the POFGeneicSerializer, and all values in the object array indexed by the property names. A generic getProperty method is implemented to allow filtering on any property that was used in the serialization. Property evaluation is happening on the server, so only filtered data is returned.

Here’s a simple loop to add an object into the cache

INamedCache cache = CacheFactory.GetCache("dist-Person");
for (int i = 0; i < 1000; i++)
{
    cache.Add(i, new Person()
    {
       ID = i,
       FirstName = string.Format("First Name {0}", i),
       LastName = string.Format("LastName {0}", i),

       Address = string.Format("Address {0}" , Guid.NewGuid()) ,
       Title = i % 2  == 1 ? "Mr" : "Mrs"
    });
}

A cool side effect, data can be accessed from .Net and from Java code in the same fashion.


Share/Save/Bookmark

Attributes based Coherence POF Serializer

As much as I love Oracle Coherence, it is a very much a Java product. .Net has lots of cool tricks that can be used during programming, however, they are not implemented in Coherence .Net

Many .Net developers are used to using attributes to define object serialization. A Coherence Generic serializer allows a developer to specify POF Serialization using object attributes as well. Here’s a simple example

    [POFSerializableObject()]
    public class Person// : IPortableObject
    {
        [POFSerializableMember(Order=0,WriteAsType=POFWriteAsTypeEnum.Int16)]
        public int ID { get; set; }
        [POFSerializableMember(Order=1)]
        public string FirstName { get; set; }
        [POFSerializableMember(Order = 2)]
        public string LastName { get; set; }
        [POFSerializableMember(Order = 3)]
        public string Address { get; set; }
        [POFSerializableMember(Order = 4)]
        public string Title { get; set; }
        public Person()
        {
        }
        //#region IPortableObject Members
        //public void ReadExternal(IPofReader reader)
        //{
        //    ID = reader.ReadInt32(0);
        //    FirstName = reader.ReadString(1);
        //    LastName = reader.ReadString(2);
        //    Address = reader.ReadString(3);
        //    Title = reader.ReadString(4);
        //}
        //public void WriteExternal(IPofWriter writer)
        //{
        //    writer.WriteInt32(0, ID);
        //    writer.WriteString(1, FirstName);
        //    writer.WriteString(2, LastName);
        //    writer.WriteString(3, Address);
        //    writer.WriteString(4, Title);
        //}
        //#endregion
    }

To force Coherence to use the serialization attribute, use the the Coherence Generic Serializer in the POF Configuration File

    <user-type>
      <type-id>1001</type-id>
      <class-name>CoherenceSample.Person, CoherenceSample</class-name>
      <serializer>
        <class-name>Coherence.Contrib.POFGenericSerializer, Coherence.Contrib</class-name>
      </serializer>
    </user-type>

Some important points to mention:

  • Object must include POFSerializableObject attribute
  • Generic Serializer will serialize Properties and Members
  • Objects MUST NOT implement IPortableObject. Doing so will force Coherence to use ReadExternal and WriteExternal functions instead of a custom serializer
  • Order parameter of the attribute is optional. The order of serialization is Order, Alphabetical Ascending. Meaning. Multiple attributes can have the same order argument, and will be serialized in alphabetical order
  • WriteAsType parameter is optional and is usually derived based on the source type.
  • Serializer uses Converter.Convert() to convert between object types.
  • Most (but not all) main types are implemented. Check out the source code for specifics
  • Hardcoding a serializer still provides better performance due to extra boxing and object conversion performed by the serializer.

Please check out the latest code on Google Code: http://code.google.com/p/linqtocoherence

Technorati Tags: ,,,


Share/Save/Bookmark

Wednesday, March 18, 2009

Linq Provider for Oracle Coherence – Pt 2

In previous post I published about a Linq for Oracle Coherence. Linq makes it very convenient to add functionality to a provider, however creating a functionally reach and complete provider is a very complicated process.

Latest functionality I’ve added was ability to use coherence extractors in the query. Linq does not allow you to change its keywords and syntax, however, all functions are converted to linq expressions and are passed to the ExpressionVisitor for evaluation. First thing is to create an extended function to provide the functionality

public static T Extractor<T>(this object obj, string Name)
{
    throw new NotSupportedException();
}
public static T ChainedExtractor<T>(this object obj, string Name)
{
    throw new NotSupportedException();
}

The functions themselves do not have any functionality and are used solely to be converted to a linq expression. I’ve created them as Generic functions to be able to strongly type against extracted data

In the QueryTranslator class VisitMethodCall() function, we add functionality to evaluate the expression

if (m.Method.DeclaringType == typeof(LinqFunctions))
{
    if (m.Method.Name == "Extractor")
    {
        this.Visit(m.Arguments[1]);
        string right = (string)globalFilter;
        globalFilter = new Tangosol.Util.Extractor.ReflectionExtractor(right);
        return m;
    }
    else if (m.Method.Name == "ChainedExtractor")
    {
        this.Visit(m.Arguments[1]);
        string right = (string)globalFilter;
        globalFilter = new Tangosol.Util.Extractor.ChainedExtractor(right);
        return m;
    }
}

m.Arguments[0] has the reference to the parent object. m.Arguments[0] contains the argument. This allows us to create a Coherence Extractor that is used in the filters higher in the expression tree.


A Note: I hope that the project gets some visibility and comments. Oracle Coherence is an excellent product with great .NET potential. Check out the project on google code: http://code.google.com/p/linqtocoherence


Share/Save/Bookmark

Tuesday, March 17, 2009

Linq provider for Oracle Coherence (Linq to Coherence)

I am very impressed with Coherence from Oracle. Coherence provides a distributed in-memory cache and processing fabric. However it is a lot more then just a cache. It can be used for everything from messaging to cross platform communication medium. There is too much to talk say about it, so read more information at Oracle: http://www.oracle.com/technology/products/coherence/index.html

Coherence works very nicely with .Net however, in the days of Linq, I wanted to write a Linq provider for it. My code is based largely on the Linq provider documentation on MSDN (http://msdn.microsoft.com/en-us/library/bb546158.aspx) and excellent series on creating a linq provider by Matt Warren (http://blogs.msdn.com/mattwar/pages/linq-links.aspx)

I am using Google Code to host the project under Artistic License. Please check out the full source code at http://code.google.com/p/linqtocoherence/.

Below is a rundown on two main classes. The main part of the code that deals with Coherence is in two classes CoherenceQueryProvider and CoherenceQueryTranslator.

CoherenceQueryProvider accepts a connection to the INamedCache – a reference to coherence cache that will be queried.

public class CoherenceQueryProvider  : IQueryProvider
{
   public INamedCache Cache { get; set; }
   public CoherenceQueryProvider ()
    {
    }

   public CoherenceQueryProvider(INamedCache cache)
   {
       Cache = cache;
   }

In the Execute method, CoherenceQueryProvider translates the Where clause to a Coherence Filter and executes the filter against the Cache objects to return array of values.

public object Execute(Expression expression)
{
  if (Cache == null)
      throw new InvalidOperationException("Cache is not properly set");

  // Find the call to Where() and get the lambda expression predicate.
  InnermostWhereFinder whereFinder = new InnermostWhereFinder();
  MethodCallExpression whereExpression = whereFinder.GetInnermostWhere(expression);
  LambdaExpression lambdaExpression = (LambdaExpression)((UnaryExpression)(whereExpression.Arguments[1])).Operand;

  // Send the lambda expression through the partial evaluator.
  lambdaExpression = (LambdaExpression)Evaluator.PartialEval(lambdaExpression);

  IFilter filter = new CoherenceQueryTranslator().Translate(lambdaExpression);

  object[] data = Cache.GetValues(filter);
  Type elementType = TypeSystem.GetElementType(expression.Type);
  return data;
}


CoherenceQueryTranslater uses the visitor pattern to convert the Linq Expression from the where clause to Coherence Filter. Coherence filters are nested to converting one to the other is relatively simple

protected override Expression VisitBinary(BinaryExpression b)
{
  this.Visit(b.Left);
  object lastGlobal1 = globalFilter;
  this.Visit(b.Right);
  object lastGlobal2 = globalFilter;
  switch (b.NodeType)
  {
      case ExpressionType.AndAlso:
          globalFilter = new AndFilter((IFilter) lastGlobal1, (IFilter)lastGlobal2);
          break;
      case ExpressionType.OrElse:
          globalFilter = new OrFilter((IFilter) lastGlobal1, (IFilter)lastGlobal2);
          break;

There is a lot more code in the classes to handle other filters, but a lot of it is pretty repetitive. The work on the linq provider is not done and I still have to implement some of the coherence functionality. Full code and usage sample is available on google code http://code.google.com/p/linqtocoherence/

Check it out and post your comments / suggestions.


Share/Save/Bookmark

Saturday, January 12, 2008

Query ExPlus 2.0.2.8 is released

I finally got a chance to release a binary of the latest Query ExPlus v.2.0.2.8

I didn't put time into the project for a while, however, some developers in the community did, and I'd like to extend my appreciation to those guys.

The new functionality since last version is:

  • Handle result sets with NULL primary keys
  • Select All for the grid
  • MRU implemented using Genghis library
  • OleDB Support
  • Fixing Order by in Oracle Browser

The project is still a one executable 172KB in size. The only prerequisite is the .NET 2.0 framework

You can access the project homepage on Source Forge

http://sourceforge.net/projects/queryexplus

The direct download link for the release is

http://sourceforge.net/project/showfiles.php?group_id=176790&package_id=206668&release_id=567994

The download is available as either compressed executable or compressed source

 

Technorati Tags: , ,


Share/Save/Bookmark

Tuesday, May 22, 2007

ScriptDB

I was looking for a free utility to script database objects from command line. There aren't that many of them out. However, I did find one that was very good at getting the job done and came with full source code. ScriptDB from ElsaSoft.

I made a few changes to the code to make it easier to use the ScriptDB in the build scripts and accept more options from the command line. I was spoke with the original developer Jesse Hersch and he agreed to publish the new application to on an open source repository.

So now: Please check out the openly available ScriptDB on CodePlex at http://www.codeplex.com/ScriptDB. The application does almost everything I need from a command line tool, and now that it is easily available to the community, hopefully it will be improved farther.


Share/Save/Bookmark

Monday, March 05, 2007

Update to Smart Client Software Factory

Blaine Wastell is writing about a soon to come out April Update to Smart Client Software Factory. The new functionality is extremely interesting:

  • WPF Interoperability
  • Offline Application Block (this was released a while back, but I didn't see support for it recently)
  • WCF Support
  • Possible support for WWF (Windows Workflow). This is 2nd priority, but that means it might be released in the future updates, if not in the April update.


Share/Save/Bookmark

Tuesday, January 30, 2007

Free Linux Driver Development!!!

This is truly amazing and something that Microsoft will never be able to do. Linux kernel development com unity agreed to support EVERY hardware device available as long as manufacturer is able to provide specifications or answer questions. All NDA requirements will be met as well. All development will be done free of charge so their developers "will have more time to work on drivers for all of the other operating systems out there" This really shows the power of the open source development efforts.


Share/Save/Bookmark

Tuesday, October 10, 2006

Query ExPlus 2.01 beta is out

  • Serious bug with saving list of connection strings is fixed.
  • Settings are now saved using .NET 2.0 settings API
  • help for command line arguments (-help)

Check it out at http://www.sourceforge.net/projects/queryexplus


Share/Save/Bookmark

Query ExPlus 2.0 beta is now on SourceForge

I've now officially merged code for the Query ExPlus 2.0 into the sourceforge repository. Look for the new release under www.sourceforge.net/projects/queryexplus

Here are some main highlights of the release:

  • Writen in .NET 2.0 - You will need Framework 2.0 installed as it is not backwards compatible
  • Code refactored to make it easier to extend
  • Support for command line parameters to automatically connect to the server
  • Copy data as CSV from the grid output is implemented

All new development is done on the 2.0 branch. I am not planning to backport new functionality unless I feel that it is very critical to the application use.

Please check out the source and the compiled appplication. As always, any comments are welcome.


Share/Save/Bookmark

Tuesday, September 26, 2006

Trip to Microsoft

I got a chance to visit Microsoft in its Seattle headquarters. so a few notes to follow.

But first thing first: Airline travel sucks, especially domestic airline travel. Just when I thought its impossible to cramp more seats onboard an airplain, Continental Airlines proved me wrong. I trully felt like a sardine for a bit.

Well enought complaining. I got a chance to finish some functionality in the Querty - Query Analyzer Alternative. New functionality includes: OleDB connectivity option, Copy from grid, more UI enhancements, and some bug fixes. I trying to use Querty as much as I can internally and future set is prioritised based on my need. The next to come is probably Find/Replace functionality.

It really looks close to be able to merge the code with the Query ExPlus as a version 2 of the trunk. Check out the download, any feedback is appreciated.


Share/Save/Bookmark

Saturday, September 09, 2006

Query Express is now Query ExPlus on SourceForge

After downloading Query Express, I got in touch with the developer, Joseph Albahari, and asked him to put the project on SourseForge. A few more emails later regarding the Licensing (Source Forge does not support permissive free license), and a few days later, the Query ExPlus was born. Check it out at http://sourceforge.net/projects/queryexplus/


Share/Save/Bookmark
Directory of Computers/Tech Blogs