Thursday, June 17, 2010

Tools For Administration

ViSit Anywhere provides a number of specialized tools for performing administrative tasks.  Our previous experience with ViSit and ViSit Web have shown that geospatial projects often evolve over time.  To manage this evolution a robust administration framework must be present.  ViSit Anywhere's change management provides the foundation for ensuring that the project configuration can be easily and transparently applied over the entire user base.  The post discusses some of the tools available to administrators for managing project change.  It is important to point out that most ViSit Anywhere users (even those making complex edits) are unlikely to need to use these tools.

The Project Tree

As stated previously on this site, a ViSit Anywhere project is managed by a hierarchy of project objects, including the object types used, the project schema and the project instance data.  Administrators can view the project hierarchy by activating the ribbon's Project tab and selecting the Project button from the Manage chunk.  This displays the main project tree control in a dockable panel.  The project tree control is based on a Outlook 2007 style navigation bar.  This type of control has a main display area (typically showing the full project tree and a set of navigation bars or buttons on the bottom that allow the view to be switched.  In our case, you can create sub-trees simply by selecting a node in the visible tree and dragging it onto the bar on the bottom.  Selecting a button will switch between tree views.

The docked panel is only the visible face of the ViSit Anywhere tree controller.  This object acts a server of project tree branches for other ViSit Anywhere tools and applications.  These sub-trees have the same functionality as the tree in the main view, but often add other features.  For example, the Place navigation tool collects all the available places, and will navigate when a Place is double-clicked.  The exact same functionality is available in the main tree (except that the Places may be dispersed across several branches there).

Right-clicking on a tree now presents the object's contextual menu that provides several tools for project configuration.  These will be discussed later in this document.  The items displayed in the contextual menu are controlled either by the selected node, its parent or one of the top level controllers (such as the type controller or the change manager).

Check Mode

It is very important for administrators to know that a configuration is valid.  ViSit Anywhere provides a special Check mode on the project tree to assist in this task.  To activate check mode either click on the Check button from the context menu of a node in the project tree, or select the Check button from Manage chunk in the Project ribbon page.  When this is done the selected node and all its child nodes are checked (errors bubble-up to the top level node to facilitate their location).  While in check mode, the image icons associated with the tree nodes take on an addition overlay that indicates the state of the object (based on its internal checking algorithm).  The state might be OK, Warning or Error.  Nodes in a warning or error state have an additional text message associated with the tree node text.  To disable check mode, just un-check the Check button.

Discovery is a foundation technology in ViSit Anywhere.  The idea is that certain project types, in under well-defined conditions do know how to "discover" (basically create configured objects) themselves without the need for an administrator to perform the configuration.  For organizations with data already in a relation database, Discover can be used to import the database schema (and eventually the data) into the new project.
  This is an important topic, so I will be spending more time on discovery and integration in an upcoming post.  For now, lets say that to use the discovery tools, simply select a tree node and then select the Discover Mode button, either in the project tree's contextual menu or the  Manage chunk on the Project ribbon page.  This puts the select item into Discovery state, at which time it tries to discover children objects.  Discovered children are displayed in the project tree with a question mark overlay on their node icon.  Discovered children can be added to a project simply by integrating them using the Integrate item in the discovered child's contextual menu.  Integrated objects become fully functional project objects.

As a simply example, consider the integration of a new snap-in application into ViSit Anywhere.  In order to add a ViSit Anywhere application to a project, we simply copy the .NET assemblies to the ViSit Anywhere program directory.  Next, we open the project tree and navigate to the Types object.  The object represented by this node is the type controller and it implicitly know how to discover ViSit Anywhere types.  When we activate Discover mode on the Types object (left side of the following screenshot) it scans the .NET assemblies in its domain path and shows the ViSit Anywhere types that are available for integration with the project.  The discovered types are shown in the project tree, decorated with the Discovered overlay (right side of the following screenshot).  The object can be integrated into the project (as fully validate project types) by selecting the Integrate action in the contextual menu associated with the discovered object.  We will be discussing the implementation of ViSit Anywhere snap-in applications in an upcoming post.

ViSit Anywhere also provides standard project configuration tools.  These are best used when starting without existing data.  In these cases, discovery is of little help (there is nothing to discover).  Fortunately ViSit Anywhere provide the tool to create new project objects from the project tree.  In this case, the parent object knows what type of children that it can have, so and which types can be created from nothing, so the list of objects to create is filtered.  For example, the screenshot below shows all the types that might be created as direct children of a Module object.  Note, object creation in this way make most sense for schema objects.  Type objects are best discovered and instance object typically created editing tools or discovered from existing data.  Once the user has created an object, it can be configure easily using the object properties controls.  This type of object creation and configuration is based on long standard Windows usage idioms and this allows users to quickly learn how to do basic configuration.

The configuration of the project objects is generally performed with the project tree.  This configuration often means that one object depends on the configuration of another, for example, a Relation object associates one Table to another.  In cases like this, it is often important to inspect and configure these object at the same time.  This can be difficult in very large projects, as such objects are often implemented in a disjoint, modular fashion.  To aid administrator deal with these types of complex associations, ViSit Anywhere provides Project Tree navigators in an objects context menu.  In the example below, when we right click on a table, we automatically see navigations to its parent tables.  Selecting one of these items will move the selected node in the Project Tree to the related object.

Drag and drop is another important technique used in ViSit Anywhere configuration.  By select a node in the Project Tree we can move, copy and depending on the drop site, even create new objects.  For example, by dropping one table column on another, in another table we can create a relation between the two tables.  Drag and drop often provides a simple, unambiguous way to achieve complex configurations.
Configuration Wizards
In some cases, configuration is very complex and may require a valid sequence of operations to effectively preform the task.  In these cases, ViSit Anywhere provides configuration wizards that allows the administrator to proceed step-by-step through the complex configuration task.

The Properties Control

The Project Tree allows us to configure the overall structure of the ViSit Anywhere project and it allows us to navigate and invoke different aspects of the project objects.  However, to configure the details of each object another tool is required.  This is the job of the Properties Control.  Each object in a ViSit Anywhere project implements a unique set of controls to facilitate its configuration.  This uses standard Windows user interface idioms which helps administrators in dealing with the diverse set of configurations possible.  For example, the screenshot below shows how we can customize the color of a line graphic by configuring the associated context.  Note, that context set is a hierarchy of objects that allows properties to be inherited and overridden.

The Properties Control is closely associated with the Project Tree control.  In order to facilitate the administration of the project, when the selection in the Project Tree changes, the read-only properties control is automatically updated so that the administrator can also check the specific configuration of any object quickly and easily.  Also it is important to point out that all the configurations discussed in post are change managed.  This means that when an administrator has finished a configuration task, he simply has to commit his changes to the ViSit Anywhere project server and the new configuration will automatically be installed on all other client machine when they synchronize to the server.  There is no need to make provisions for special configuration distributions.



This post has given a brief introduction to many of the tools available to ViSit Anywhere administrators.  We believe that they provide a unique environment for configuring geo-spatial data projects.  In upcoming posts we will look closer at many of the features we have introduced here.

Friday, June 4, 2010

ViSit Anywhere Update

At the risk of sounding trite, we would like to say that the Spatial Networking blog is back. We understand that it has been a while since a new post appeared here and we suppose we have no excuses. Of course we have been very busy and yes, the product has advanced to a state where we are ready for production use. The important point is that this blog is back and we will try to provide a bit of information about the new features on a regular basis.

ViSit Anywhere now sports a large number of features and functions. So many in fact that we have a difficult time quantifying them all. For this reason, we would like to present a tour of some of the latest release. The features that were previously discussed, integrated change management, CAD engine independence and superior user interface are still there (and better), but now we would like to discuss some of the less well-known features that we rarely have time to demonstrate. The extent and quality of these feature, demonstrates our commitment to building the best possible application for managing Geo-spatial information.

We propose to post 2 or 3 short blog entries each week, describing new and important features of ViSit Anywhere. These entries will include:
  • Project Management tools - ViSit Anywhere uses advanced property controls, drag-and-drop and object discovery to make the work of administrators and integrators easier.
  • Non-Change Managed Information - In ViSit Anywhere all information can be change managed, but sometimes it is good to have a store of private information. This new feature allows such information to easily integrated into your project.
  • Graphic layers and Managed Contexts - Most graphic editors manage information in a set of layers and ViSit Anywhere is no exception. This entry describes how we define layers for managed graphic data using hierarchical contexts. This provides the foundation for a powerful framework for managing static symbology.
  • Graphic Layer tools - We provide a set of tools for managing the display of graphic layers that is at once robust and simple to use. The challenge is to have complete control over the displayed layers while avoiding the need to adjust their display.
  • The Relational Model - This entry discusses recent additions to our relational data model which allows separation and simplification in the presentation of attribute data.
  • More on the Relational Model - In this entry I would like to discuss the notions behind our spatial query tool and how it integrates with the way we centralize the display and interaction with tabular data.
  • Functions and Expressions - One of the most important run-time configuration tools is the table expression and function framework. This allows users to manipulate project information using C# scripts and special purpose expressions. These are compiled on-the-fly at run-time and used to transform or agglomerate information and present them to the user.
  • Application Settings and State - ViSit Anywhere now includes a new framework for saving the application state (without resorting to writing information in the Windows registry). This framework means that the function and layout of ViSit Anywhere is now persisted between editing sessions resulting in a much better user experience
  • Project Archives - ViSit Anywhere now has the ability to archive projects for re-use and re-distribution quickly and easily. These archives (and ViSit Anywhere change management) mean that simulations and restructuring can be performed without fear of breaking project integrity. In addition to project data, it is also possible to archive and distribute application environments. The result is that a user can be fully operational, for a cold start, in under 10 minutes.
  • Basic Network Editing - ViSit Anywhere has been designed to edit and maintain topological network data. This topic will describe the network model and the tools used to maintain it. We will also discuss some upcoming network features.
  • Cartographic Plots - ViSit Anywhere has number of tools for managing complex, multi-view cartographic plots. This topic will survey the tools available and discuss some of the future directions.
  • Ribbon Layout - The Office 2007 Ribbon user interface presents an opportunity to increase the usability of an application. Unfortunately the layout of the ribbon can often be optimized for a particular task. We have recently re-worked the layout of the ViSit Anywhere ribbon and in doing so, have laid the foundation for run-time, user driven ribbon layout.
  • User Roles - Another new technique for customizing the usability of ViSit Anywhere is through user roles. In this entry we will discuss how editing roles can be enforced at the user level and some future applications of these roles.
  • Data Integration - ViSit Anywhere is a great tool for integrating data from a variety of sources. Its "discovery" framework can be used to integrate data from Shape, Mid/Mif and other sources.
  • Auto-Synchronization - One of the benefits of robust change management and strict relational rules is the ability to have a production quality database that represents the stable state of the system. ViSit Anywhere provide auto-synchronization for this - in particular auto-synchronization to Oracle (with SDO geometry).
  • Navigation - Navigation in large geo-spatial datasets is hard problem. ViSit Anywhere provides a number of tools to address the difficulty, including gazetteers, external tables and favorite places.
  • GPS Integration - Today, GPS data is everywhere. To exploit this Géotech is working to integrate GPS with ViSit Anywhere. A side effect of GPS integration is the ability to transform from one system of coordinates to another. Given the state of web GIS tools, like Google (TM) or Bing (TM) Maps, the ability to manage coordinate systems provides new opportunities for desktop GIS.
  • Work Management - Today GIS must integrate with the enterprise information fabric if it is no remain relevant and deliver its full benefit. We believe believe that the integration point that will enable this is the work management system. In this way every edit in ViSit Anywhere are associated directly with a task that is important for the enterprise. This entry will discuss the foundation of the work management system integration in ViSit Anywhere.
  • Work Management 2 - This is a continuation of the previous entry. In this entry however, we will focus on the implemented tools for work management.
That it! The list is long, but we have been working hard and we really want to share information about these important features. After reading these entries we hope that you will agree with us when we say that we believe ViSit Anywhere is a geo-spatial tool unlike any other.

Wednesday, June 4, 2008

Store Yourself in Context

The ViSit Anywhere product was designed and implemented over several years. When we started the project we did the right thing. We locked ourselves in a quite room for one week to examine what worked and what didn't work in ViSit v3 and ViSit/Web. We noted all the things we always wanted and all the thing we could never do. We concentrated on the what and not the how. After a week we had what we considered to be the foundations of the new product. I don't think this initial vision has changed in any significant way during the product development. The first vision of the product was that we would have a rooted hierarchy of objects that would describe the project schema, the applications that could be run and the data that was operated on. Today this vision manifests itself in the ViSit explorer tree, as discussed in the data model.

When we started implementing this vision we came very quickly to a tool with a reasonable level of classic ViSit behavior, continuous mapping, well defined layers and tables associated with graphic elements. We continued in this way until the China release (our third iteration). At this time, we found that the underlying project architecture was becoming more and more brittle. We started have to re-migrate all our projects after every new release. This was taking more and more of our time, until we finally felt that using .NET serialization was just not a good way to store project configuration and data. We decided that if we were going to progress, our objects would have to know how to store themselves.

At the same time, we decided that if we were going to have change management, we would have to implemented ourselves, and one way to implement change management is to save everything before a change and restore it if we had to rollback. So to us, having our change-managed objects store themselves was the corner store of building the product. With this new insight into how we would have to save projects and how saving objects is directly related to change management (with a good dose of design philosophy from Holub) we took a long hard look at our product architecture.

What followed in the Denmark release of ViSit Anywhere was what I like to refer to as the C-D extinction (in honor of the K-T extinction that wiped out the dinosaurs). That is, objects that were important in the ViSit v3 API suddenly lost their utility and were abandoned as the ViSit Anywhere API emerged. An example of this is the use of the primary ViSit objects, sites, themes and tables. In the ViSit v3 API there was a controller object which could be queried to access the desired ViSit object, which could then be used to perform an application task. The ViSit Anywhere model uses a cooperating set of objects, with fewer controller objects. For example, in pre-Denmark ViSit Anywhere, the SiteTreeFactory object could be used to create a sub-tree of sites that could then be used by an application. The controller object would be responsible to all site interaction, including keeping track of who was attached and who was not. The post-Denmark API simply has a hierarchy of sites that can be queried by path expression. Much of the functionality that was implemented by the SiteTreeFactory was now distributed directly over the hierarchy of SiteLinkages. For example, a parent site knows that it and some of its children might intersect a geographic region. It can then ask its child sites if they intersect the region.

This type of implementation is an expression of the object-oriented concept stating that you should not ask an object for information about itself so that you can perform an operation, but rather, you should ask the object to perform the task itself. And the task I want to talk about now, is object storage.

This all sounds very well and fine, but there is one more twist that really allows this implementation to prove its worth. Our objects are maintained in a strict hierarchy. Every object has a place in the hierarchy and there is only one object that is special (the root of the hierarchy). Some objects have children, some don't. When we save an object, it is stored relatively to other objects in the hierarchy.

Now, while objects have to know how to store themselves, they need to have some information on how they should be stored. If fact, it is better if a on object can describe its storage in a generic format, such as XML, then delegate the actual storage of the information to another object. To accomplish this we introduce a serialization context. The serialization context exposes a container object that the storable object uses to store its content. The serialization context is responsible for writing the container to some persistent store, for example the file system or (as we shall soon see), a relational database.

The serialization context object has been implemented as an interface, allowing us to re-implement the object when we wish to change the persistent store. In addition, only two objects are capable of creating a root serialization context, the root collection (i.e. the root of the object tree), which defines how a local project is stored, and a proxy object that represents the server store. For the iterations from Denmark to Malawi, we used a single serialization context, the XmlSerialzationContext. This object managed the storage of object content as XML files, with the hierarchical structure of the object tree represented as a hierarchical set of files.

Note, the idea of using a generic member to allow objects to perform some task, and then providing details on exactly how to represent the results of the operation with a context object is a recurring architectural pattern in ViSit Anywhere. A large part of the ViSit Anywhere API involves asking object to describe themselves using some context object. This subject will be dealt with in a future blog, but it is important to understand the use this concept.

So back to serialization (or storage) of objects. Basically the process goes as follows, the root object creates the root serialization context because it knows the type of serialization that is being used. Root objects are either the local project or the server project. Both these objects have to know how to locate their stores, so this is not a problem. Once this is done, it is possible to create a child serialization context for each child of the root . In this way, child objects are stored (in some way) relative to their parent object, through the use of a child serialization context. Serialization contexts must provide a way to create child contexts, so again this is not very difficult. By recursively applying these rules, all objects can be serialized.

On a side note to this, we can see how the change manager uses serialization to orchestrate change. When an object is added to a change scope (before it has been modified) the change manager can create a child backup serialization context using local revision information. When the object is changed it can then serialize the new state. If we have to return to the original state, we just have to load the backed version and store it as the current version. Thus change management of storable objects (typically schema objects) can be accomplished by applying simple rules of serialization.

While this may all seem good and change management is an important thing, you might be asking why I want to talk about serialization and object storage now? In fact, the last two iterations, Nepal and now Oman, have shown uses for the serialization context that were not previously planned.

The big feature of the Nepal release was to abandon on simple file system server for one that could be used to store objects over an HTTP connection. Previously, we were using a server project on a shared drive, locking the file hierarchy so that only one person could access it at a time, and using this as a server store. This server store was based on an XmlSerializationContext that pointed to the server store area rather than the local store area.

When we started thinking about how to implement an HTTP server, we came to the conclusion that by implementing a new type of serialization context we could push the store over the wire to store objects on a remote server. To do this, we simply implemented a serialization context that could communicate with a couple of WCF (Window Communication Framework) services. This provides a framework where we can have a server be hosted on a web server, via a TCP/IP port or simply using an in-process file system connection (or all three flavors), without having to change any program code. That's because WCF abstracts the networking protocols, making them look like the same thing to the application program. WCF is a technology that, for us, arrived just in time. If we would have started implementing this before WCF arrived, we probably would have spent a lot of time doing it, and then we would have had to re-implemented to conform to WCF.

But the really interesting thing is that rather than taking hundreds of hours to implement a new server technology, with client-side and server-side components, we implemented one new serialization context (about 400 lines) and a simple server-side object that accepted files and copied them to the correct directories (again, about 400 lines of code). So rather than spending weeks to implement our new server store, we spent 1 week (and most of that was discussing the fine details).

By this time we were starting to appreciate the ideas of strict object encapsulation, but history would repeat itself. In the current, Oman iteration, one of the big features is to be a special purpose client that will automatically synchronize a ViSit Anywhere project to an Oracle database (with the graphic objects represented as Oracle SDO geometry objects). Given the structure and complexity of the ViSit Anywhere synchronization sequence this task promised to be difficult. But, after a weekend of thought and about 1 day of design discussions, it became clear that we could solve this problem by implementing a new Oracle serialization context. In this case, the oracle context would know how to connect to the database and push the ViSit Anywhere geometry (managed CAD elements) into Oracle Locator. Again, the heart of the problem was solved in about a week, probably in less than 1000 lines of code.

The Oracle auto-synchronization client also provides a pattern for pushing ViSit Anywhere projects into any relational database, simply by implementing a new serialization context.

These two examples underline the advantages of the extensible ViSit Anywhere API and how object-oriented concepts like type polymorphisms and strict encapsulation can lead to stunning productivity improvements. To top it all off, this is exactly what Holub predicted!

Thursday, May 29, 2008

The ViSit Anywhere User Interface

With the first ViSit Anywhere beta release now shipping, I wanted to give non-beta testers an idea of what the first release is about and what to expect. Before we start, I want to point out that there are two flavors of ViSit Anywhere, standalone and hosted in Microstation. We have made every effort to ensure that the two tools have he same functionality and the same user interaction logic. Unfortunately, the two tools will never be exactly the same. The standalone client allows us a bit more latitude of what we can do with the dialogs and windows and Microstation has additional editing tools and it processes vector and raster graphic more rapidly. In this entry I am going to be showing the standalone tool and I will talk about the Microstation version in the future, but keep in mind, the things I will talk about here apply to both versions.

Another point I want to make is that ViSit Anywhere is a culmination of our experience with ViSit v3 in Microstation and our ViSit/Web product. We appreciated the simpler user interaction in ViSit/Web, but we felt that it lacked the complex editing functionality of Microstation. We have tried to get the best of both world with ViSit Anywhere by providing all the detailed functionality of the Microstation/ViSit product with some of the simply usage idioms of ViSit/Web.

An Office 2007 Ribbon Tool Bar

Lets start with the basic screen shot shown to the left. I have annotated the image with some numeric callouts to highlight some of the features. You might want to open the image in another browser window to follow along (i.e. click right and open in a new window).

As you can see, the application uses the an Office 2007 ribbon tool bar user interface. Under this type of application, a ribbon (callout 1) presents tools as buttons and others UI items across the top of the application, rather than using tool bars. This allows us to group tools according to function. The larger area means that we can use larger buttons which makes them easier to access (and, we hope, understand). If you are worried that the ribbon takes too much space it can be minimized and then accessed as a ribbon popup. This allows maximum space for the graphic and other windows.

In addition, the interface has a user-customizable quick access tool bar (callout 2). Users can quickly and easily add commonly used tools to this tool bar. This makes them instantly available when the ribbon is minimized. It is displayed in the window frame by default, so the tool bar takes zero client space (as shown in the image), but the user can optionally display the toolbar at the bottom of the ribbon to reduce the distance need to reach the a button when performing click-intensive work.

Note also that the ribbon consists of several ribbon pages, similar to a tool bar with tabs. This allows us to group functionality by task. Indeed, when we were designing the application, we made sure to have ribbon pages with all the tools required to perform a given task, even if that meant that some tools are present on several pages (see callout 3).

Finally, note that the graphic space (callout 4) contains a full-featured CAD drawing tool. We can attach and detach maps and data layers as required, including raster images. ViSit Anywhere features an easy to use auto-select interaction mode. With this interaction idiom, the user hovers the mouse over a graphic element to begin the selection, if the user clicks the left mouse button, the element is selected. If the user clicks the right mouse button, the next closest element is selected. Selection is shown by drawing the graphic element in a highlight color. If the mouse pointer is moved, the selection will change. If the user leaves the mouse hover a little longer over the element a tool-tip is shown with information about the element under selection, as shown in the screen shot. Note that the tool-tip is entirely configurable by the ViSit Anywhere administrator, and information in the tool-tip can come from a number sources. For example, the tool-tip in the screen shows the diameter of the pipe under selection and the name of the graphic context used to edit the item. It also shows how many equipment elements are on the pipe. This is not a attribute of the pipe, but is created by counting the number of elements associated with the pipe. We also have information from the active tool, the containing site and the current graphic composition. To a ViSit v3 administrator, this may seem complex, but to the user, the information is natural and simple to understand.

Simplified Spatial Queries

The next screen shot (left) shows another of the features that emerged during development that we have found intuitively interesting and easy to use. That is, spatial requests. In ViSit Anywhere to do a simple fast spatial query all the user must do is to select a spatial table from a drop down list. A data grid opens at the bottom of the main window to hold the results of the query. Each time the user navigates in the graphic (zoom in, zoom-out, pan etc.), the rows of the grid are updated using the extents of the view as the request zone. Once the user has the zone that interests him, it is possible to freeze the query, so that view updates do not change the request results. The user can then highlight and localize the rows in the grid to perform their task. Note the highlight used to show a queried element is not same color as the edit highlight, so this is one way a user can delimit the extents of an editing task and then process the elements one by one. Note also that the data grid has all the features that you would expect from a modern application, such as sorting and filtering of columns and grouping rows hierarchically (as shown in the screen shot). We also provide tools to print or export the (filtered) query results.

Complex Network Editing

I mentioned previously that ViSit Anywhere provides a full-featured CAD tool to perform, for example, complex network edits. There are a lot of interesting things on this screen shot. First, note that the spatial query data grid is still open, but it has been minimized to the tab in the lower left corner of the application. ViSit Anywhere features simple, unambiguous docking windows that really allow users to optimize their workspace.

Now look at the editing operation that is taking that is taking place. This is a network node move. All the connected network segments must move with the node being edited. The network segments are polyline objects, so that when the network segment is stretched or compressed the last line segment in the polyline must change size and orientation. Note also, that the editor shows the vertex points of the line segments making up a network segment. This is very important when performing network segment edits, because the user can move a vertex by clicking on (or nearly on) a segment vertex or insert a new vertex by clicking on a zone with no segment vertex.

Note also, that the network equipment on the segments being moved must also be stretched with the segment. In this case, we maintain the same relative linear position of the segment equipment as the length of the network segment changes.

Modify Attribute Data

I have one final screen shot to show. This one shows how a user can edit the attributes of a network object using the sophisticated data grid controls in ViSit Anywhere.

In this case we are using the Modify Attributes tool and a network valve has been selected. When editing attributes in ViSit Anywhere, we show a modal dialog, which makes the process of submitting new change unambiguous. That is, while the dialog is open the change has not been submitted and by clicking the Cancel button, the change can be abandoned. Oh yeah, don't forget, in ViSit Anywhere you can always undo changes (even changes from a previous session) and the undo will be applied to both the graphic and the attribute data. Changes are final only when they are committed to the server as a new revision.

If you examine the attribute grid, you will note couple of things. First, some columns have been grouped into a section called eau détails régulations. This section is actually a secondary table associated with the equipment table. It is also possible for users to group, hide and rearrange the attributes in the form to suit their needs. The configuration can also be saved and re-loaded to perform specific tasks. The idea here is to allow the user to optimize the work environment to suit the task at hand. This is especially useful when there are a lot of repetitive changes to be made.

Note also that that the attribute grid supports a large number of field editors to help users to enter their data more easily and efficiently.

I think these four screen shots give you the flavor of what ViSit Anywhere is all about (or at least the first beta release). I will write a more detailed description some of the tools I have shown here in a future entry. Maybe I can get a screen cast that demonstrates how easy it is to perform some of these tasks.

Friday, May 16, 2008

ViSit Anywhere Change Management

A couple of years ago when Bentley first released Microstation v8, we were considering how to move ViSit forward onto the new platform. At the time it was clear that the new design file format was going to to result in a lot of re-work in ViSit just to stay where we were with in Microstation/J. This was very disconcerting since we felt that if we were to make this move, we would have to provide something extra to encourage our customers to move the new platform. We understood that new products, at least at the out set, are generally less stable and have fewer features. As we looked at migrating 10 or 15 years of MDL code from MS/J to MS/v8, we felt that the task would certainly be a difficult and time consuming.

At the time of the MS/v8 release Bentley had promised to fix what we considered to be one of the most difficult GIS problems - concurrent editing using optimistic change management. MS/J had been released with ProjectBank (a change management platform) with much fanfare, but it seems the system never caught on. Even though we did deliver a ProjectBank component for ViSit v3, the system never had the stability or the performance that we were looking for. So with every new MS/v8 release we patiently waited for news of the promised change management system and when it did appear (late in the Microstation XM product stream release), it was not what we were hoping for.

Meanwhile, we decided that if we were to have a change management system we would have to implement it ourselves. Our experience with ProjectBank had given us some insight into what was required, but ProjectBank only managed graphical CAD elements. We really needed something that would manage changes in both the graphics and the attribute data.

Many GIS systems (including Bentley) were starting to use Oracle Workspaces for change management, but since most of our customers where not using Oracle we did not feel that this could be made to work for us. In addition, the demonstrations we had seen seemed to be overly complex and the Oracle licensing requirements were (for us) ambiguous.

At the same time, being software developers, we were being exposed more and more to the same type of change management in our source code version management tools. For example, we are currently using P4 (from Perforce). When we program, different developers are often working on the same source files, however P4's automatic change merging and conflict management means that we only have to manually resolve conflicts on very rare occasions. Our experience with GIS systems was similar. Typically we can have a large number of people editing and working with the GIS system at the same time, but since everyone is doing their own work, their changes rarely overlap. We feel that information systems of this type are ideal candidates for change management with optimistic locking.

At the time we were struggling with the issue of change management I was reading the book Holub on Patterns, which I then passed on to Dominique. One of Holub's pet peeves was application user interfaces that would show popup dialog asking users if they were sure they wanted to continue because the changes that they were about make could not be undone. This notion also struck a chord with us, as we wanted users to be able to edit data in ViSit without fear that they would somehow corrupt the project data. Holub's solution to this problem was to have objects that could save themselves. So as changes were being made, the objects being changed could silently and transparently save their own state and thus allow any change to be undone.

This notion inspire much discussion at Géotech and the result of these discussions is the ViSit Anywhere system of change management. One of the cornerstones of our implementation is objects that can store themselves. For this, we developed the IStorable interface. That is, any object that implements this interface could be change managed. At the same time, we were discussing where and in what format objects should store themselves. We had already had requests that configuration information be stored in an RDBMS and while we did not think that this was appropriate at the time, we decided to punt on the matter and develop an ISerializationContext object that would be passed to IStorable objects to provide infrastructure and hints on how we wish objects to store themselves. At a later date, if we need to store the objects in some other format it could be accomplished using the framework of the ISerializationContext. The idea of a context object that provides hints for object behavior became another cornerstone of the ViSit Anywhere implementation.

Currently objects store their state in XML files and the XmlSerializationContext provides the necessary infrastructure for performing the save. However, we have already exploited the existence of the ISerializationContext by implementing a RemoteXmlSerializationContext to store objects remotely using a WCF proxy object. Thus when implementing the remote server, we simply implemented the local server proxy to provide a remote serialization context. The RemoteXmlSerializationContext provides the infrastructure to store XML documents remotes, so no IStorable code needed to be changed.

The final major participant in the change management system is the ChangeManager. This object is responsible for coordinating all change management tasks. That is, when objects know they are in the process of being changed, the ChangeManager will create the appropriate context objects and request that the IStorable's save themselves. If there is an error in the process and the change can not be completed, the ChangeManager must coordinate the rollback.

At the time we were developing the change management system Microsoft was releasing the .NET framework v2. The release featured an object called a TransactionScope which was used to delimit the effects of an atomic change. Since at the time we were still using the .NET framework v1.1 (for compatibility with Microstation) we developed a similar framework, where the ChangeManager would deliver and control IChangeScope objects. The basic usage is that a method that wants to perform a manged change would request an IChangeScope from the ChangeManager. The IStorable objects participating in the change would then add themselves to the IChangeScope. The change would then be performed. Once all the work was complete, a call to the Complete member on the IChangeScope would signal that the change should be committed. IChangeScope's can be nested to allow complex, multi-participant transactions. The ChangeManager provides all the calls to save and rollback the objects. As the IChangeScope is an IDisposable object, transaction commit/rollback is performed when the IChangeScope disposes. If the IChangeScope Complete method has been called, the transaction is committed, otherwise it is rolled back. The ChangeManager handles all thread synchronization (since only one thread can hold the current change scope at any one time). The result is a system where objects that can store themselves can be easily transaction managed. The ChangeManager also must store the committed IChangeScope in case the user later decided to undo his local changes. The stack of local changes is called the long transaction. The long transaction is what has to be committed to the server once the user decides that the changes are valid.

The above process can be rather complex, but it gets a little bit worse still. IStorable object are typically schema objects that allow project features to be configured. Most user changes are changes to graphic objects with their associated attribute data, that is instance data. The two major instance objects in question here are site linkages and table instances. SiteLinkage's are collections of graphic data, roughly analogous to a Microstation design file. For the purposes of change management, SiteLinkage's are collections of managed CAD elements (MCE) . An MCE is a primitive graphic element that can be change managed. An MCE can be a point (represented by a symbol), a line or a polygon. Each MCE can be associated with zero or more text labels. All the managed graphics in ViSit Anywhere must be represented by these 3 primitives. But note, that SiteLinkage (which implements IStorable) may contain any number of MCE's. The overhead of creating an IStorable object for each MCE is just too high. For this reason we have implemented the notion of an IChangeController. An IChangeController is simply an IStorable objects that helps in change managing a collection of elements. The elements managed by the IChangeController defines the granularity of the changes that can be managed by the system.

TableInstance provides the same service for attribute data as SiteLinkage provides for graphic data. The TableInstance change manages attribute data at row granularity. That is, when you change the attributes for a single row, only that row is marked as having been changed. If two users edit data in the same row it is seen as a conflict, even if the two users edit different columns in the row. This is the meaning of the change granularity.

Now, the ChangeManager knows how to save and restore IStorable objects, but rows and graphic elements are not IStorable's. When these items are added to the IChangeScope, the ChangeManager must resolve the appropriate IChangeController and request that that object perform the necessary change management operation. In this way we have resolved most of the problems with local change management. The ChangeManager coordinates change operations, but in the end only objects that know how to store their own state can be change managed. Other objects can change manage collections of elements but they are responsible for doing the right thing when the ChangeManager makes a request.

Now, consider that we have a stack of changes on the local client and the user wants to commit this information to the server. The ChangeManager is again responsible for coordinating this task. The first thing that must be done is to create a Revision. A Revision is an collection of changes that can be committed to the server. Unlike in the local long transaction, an IStorable object may appear only once in the Revision. A RevisionEntry for any object gives the new state of the object at the time of the commit. Thus, if I had an object that I added and then modified twice in my long transaction, the RevisionEntry created would be the final configuration with an ADD marker. Thus, there is a fixed logic to merging changes from the long transaction to the Revision. Users synchronizing to the revision will not see the intermediate states of the objects, only the revision state (which is also the state of the object in the committer's local project).

To commit data to the server, we simply merge the long transaction changes into a Revision object and send the new states of the changed objects to the server. The server maintains a stack of revisions, that contains entries for each object in the project, serializing the Revision on the server increments the tip of the revision stack and provides the new state of the changed objects. Objects that are not part of the revision can be accessed by drilling down into the revision stack.

Before a user can commit his changes, he must be sure that his local project is synchronized to the tip of the revision stack. If not, applying changes can result in a corrupted server. Synchronizing a revision simply means applying the IStorable objects that are entries in the revision in question to the local project. Again, this is performed simply by object serialization, in this case, from the server to the client. Note also that this is how conflicts between changes are detected. If the user must recover an IStorable object from the server, because it has a new version in the revision and that user has changed that IStorable object (i.e. it has an entry in the local long transaction) this is a conflict. The user must resolve the conflict in order to synchronize the incoming revision. This can be done by accepting the server version and rolling back the local change, by ignoring the server change and keeping the local change or by a combination of the two. The point is that the user must have, what he considers to be the correct version on his local machine before he can commit his changes.

So in essence we see that ViSit Anywhere implements a change management system based on object's ability to save themselves. Simple objects, like graphic elements and attribute rows, may use controller objects to perform change management operations for them. Changes are made in nested change scopes that provide a transaction environment for developers. The ChangeManager object performs most of the heavy lifting, when it comes to the details of organizing changes and getting them to the project server, but the objects being changed always have to participate in the process.

If this infrastructure seems complex, that's because it is. Managing nested change scopes during commit and rollback is extremely difficult, even with the ChangeManager. Fortunately, a new method for programming change managed operations has emerged during the implementation of the network editing tools that greatly simplifies change managing instance data. This technique is built on top of the basic change management plumbing, but uses an in memory ADO.NET dataset to contain all the information that could possible change during the operation. Once the operation is complete, we only have to change manage the dataset. The DataSetManager takes care of all the details of loading the data and change managing it, including ensuring that attribute data and graphic elements are always changed in the same IChangeScope (since they are managed by separate IChangeController's). Change management with the DataSetManager will be the subject of a future blog entry.

Thursday, March 20, 2008

ViSit Anywhere Architecture

This article provides some insights into the ViSit Anywhere architecture, and shows exactly why we think that it can be used to as a foundation for work-centered GIS and geo-engineering applications. Physically, the architecture of ViSit Anywhere is straight forward. The following diagram summarizes the basic components.

I have added several components here to represent the different roles that can be taken by the various applications and users. Not all components need to reside on physically distinct machines.

The first thing to note is that this is a client server architecture. The consolidation point for ViSit Anywhere is the Project Server. This is a component that maintains the stack of changes that have been applied to a the projects used in ViSit Anywhere. An installation can consist of any number of projects. When creating a new project, the project server will simply deliver a project consisting of the basic ViSit Anywhere types, but without any schema or instance data.

Internally, the project server is implemented as a .NET WCF server and is delivered either as a HTTP web service or a Windows service that listens to clients on a TCP/IP port or both. In fact, the two configurations of the project service can be active at the same time, as some users may find it more convenient to connect to the project data through ubiquitous HTTP, while others may prefer the performance of a dedicated TCP/IP port. Switching between the two network schemes requires only a reconfiguration of the client.

The project server acts as a consolidation point for ViSit Anywhere licensing as well. When users connect to the project server their user credentials are sent to the license server where they are tested against the available block of licenses. In this way licensing has been localized on the server and the GIS administrator does not have to micro-manage installing licenses on individual machines. The licensing model has been design specifically for disconnected editing, where the synchronization and commit processes act data integration portals for the users.

I will come back to the role of the database server later in this blog. For now, I want to talk about the ViSit Anywhere clients. I have represented the clients with three separate icons. They represent an integration client, a number of standard workstations and a number of mobile workstations. Each of these roles is important for ViSit Anywhere usage.

The first client role depicted is the integration workstation. In this case, data from various sources and in various formats are integrated into the ViSit Anywhere project. ViSit Anywhere uses a simply abstract data format, based on XML to represent all underlying data items. Each component renders the underlying data in a format that is appropriate for its implementation. For example, a Microstation-based editing client maintains data in the generic format, but transforms it into a Microstation design file when it needs to edit or display it. The format used by the standalone editor does not maintain data in Microstation design files and these editors do not have to know how to display a Microstation design file.

The integration workstation will typically use Microstation as the primary CAD engine, as Microstation is well adapted for reading and converting various graphic formats. Also the general CAD editing tools found in Microstation are useful for creating symbols and other graphic items that will later be used by the other CAD engines.

The intergration workstation may also need to extract data from a ViSit v3 project using an RDBMS during project initialization. Project migration from ViSit v3 is performed by running a program that creates an XML description of the ViSit project, then building the project schema and instance data by "discovering" the described information in ViSit Anywhere. The migration process requires a connection to the RDBMS which may require a client database license. Once the project has been migrated and integrated into the ViSit Anywhere project, the database connection infrastructure on the client is no longer necessary. For this reason, we believe it necessary to have only a limited number of integration workstations.

The next client role is that of general workstation. This represents a typical in-office user. These users could be outfitted with either the ViSit Anywhere Microstation or ViSit Anywhere standalone. Microstation is typically only necessary when the structure of the data will require the higher performance offered by this tool. For example, Microstation is able to better optimize the display of large raster files. Even if Microstation is available (for example, because it is used for non-ViSit Anywhere work flows), users may prefer using the standalone version because of its simplified user interface. ViSit Anywhere licensing allows users to have both the standalone or Microstation version.

Standard workstation users will have the ability to modify the GIS data like all other user. They do not require a connection to a database or any other 3rd party software (including database clients and CAD editors). When these users are ready to commit their changes they can simply connect to the project server (via TCP/IP if the project server is on the LAN) and share their data. One interesting possibility that is presented by the ViSit Anywhere notion of "local project data", is to have users perform simulations that will not necessarily ever be committed. This is done by synchronizing to the tip of the project and making proposed changes. The user can then analyze the result of the changes and repeat the process if necessary, without fear of ever corrupting the project data. In addition, once users find configurations that are viable, they can be merged in to the project data simply by synchronizing to the tip of the project data, and committing their changes.

The final client role is the mobile worker. These users are typically equiped with a rugged laptop or tablet computer. They will typically synchronize the project on their mobile machine when they have the opporatunity. Their connection to the project server may only be sporadic and thus it is important that their data merges correctly and easily with the ever changing project data. ViSit Anywhere provides support for this through its change management system. Under this system the graphic and attribute data of managed elements are handled atomically. Since GIS data typically represents large sets of managed data, it is rare when two users modify the same element at the same time. That is, it is rare when there is a true conflict. Other, structural conflicts (like the key conflicts that occurs when two users both add a new element and the new elements both have the same key) can be resolved automatically by the system and do not require any user intervention. In ViSit Anywhere conflicts are resolved on the client's machine. There is not reason to upload revisions to the server or to use special tools to merge revisions. When users resolve conflicts the server provides the current server version and the last shared ancestor version to compare with the local version. The user simply modifies the local project so the data is correct then accepts the conflict as being resolved. The correct version is then accepted as the tip of the project when the changes are committed. I will talk more about conflicts and change management in an upcoming blog entry.

Note that on the above diagram all the connections between the client applications and the project server are shown in green. This is meant to represent the fact that these users can work productively without any connection to any other machine. The users only have to connect to share there changes with other users (by committing the changes) or to synchronize their local project to see the data other people have changed. In addition, no other 3rd party software is required to work productively while disconnected. This is equally true for disconnected mobile workers as it is for office workers.

The final component on this diagram is a server-side client that listens for project data changes and automatically updates an external RDBMS. The connection between the project server is shown in gray because this component is only scheduled for release in the second ViSit Anywhere beta. We are currently planning to implement auto-synch database clients for Oracle locator and Microsoft SQL Server 2008 (which also supports spatial data). We feel that These types of clients are very important as they will allow GIS project data to be easily accessible in other enterprise applications.

This concludes my basic look at the ViSit Anywhere architecture. The thing to understand is that ViSit Anywhere uses a centralized project server to consolidate and distribute data to all users, but once a user has synchronized to the server, that user can work autonomously, without further interaction with any external application until such as time as they want to share their changes or recover a newer version of the project data.

Wednesday, March 12, 2008

The ViSit Anywhere Data Model

Before we can really understand the novel features of ViSit Anywhere, it is important to understand its data model. When we were designing ViSit Anywhere, we started with the ViSit site model. From our point of view, the ViSit site model had proved its utility, as it had long provided a flexible, high performance foundation for our Microstation applications. ViSit Anywhere exploits the fundamental ideas that were present in ViSit, but it generalizes them and applies them over the various information layers used in the application.

Before I start with this discussion we should understand the three fundamental layers of the ViSit Anywhere data model, concrete data, schema data and type data. These layers exist in most application, but are rarely treated as explicitly as they are in ViSit Anywhere. At the highest, most concrete level we have the engineering project data. This is a model of an engineering asset, like a water or electricity network. The elements in this layer, the pipes, electric lines, equipment and serviced customers - all have concrete representations in the real world. This is the level of data that makes the system pay. The more closely this layer of data represents real world objects, the more utility you can extract from your system.

At the next layer in the data model we have what we like to call the schema data. This level can be thought of as an organized description of the concrete model. A well developed schema layer will make accessing and updating the (large) concrete data layer much easier. Many GIS applications have weak schema models. Users are allowed to add and removed data layers with ease and without constraint. The problem is that ensuring the integrity of such a model is very difficult. The schema layer is there to provide business rules and logic to ensure that he concrete data layer remains valid. We must also remember that the concrete data layer is there to help users do their work. Superfluous data items, unnecessary details and cumbersome graphic layers can impede users in getting their work done. A well designed data schema can transform unrelated data items into a work-centered application.

We call the lowest layer in the data model the types layer. By types we mean hard computer program objects, in our case .NET assemblies. ViSit Anywhere is designed to be an easily extensible application. We wanted to be able to add extension modules simply by dropping an assembly in the program directory and then allowing the new functionality to be discovered. Since new programs invariably need new schema and concrete data, it became apparent that we would have to close the circle and make the basic executable an element of the data model also.

The ViSit Site Model

The ViSit site model was developed over the past 15 years at Géotech as its primary spatial filter. The model uses parallel hierarchies of themes and sites. Theme are logical objects representing the business rules associated with some geographic asset. Sites are collections of related geographic information. Sites contain the geometric objects displayed in the GIS graphic view, while themes provide logic, as well as an extension point for hanging associated alphanumeric data and editing tools.

The diagram to the left shows how the hierarchies are arranged. Each site is associated with a geometric boundary that contains the site's geographic information as well as all the geographic information contained in the site's children. Thus, spatial filtering is a simply tree pruning operation (in the site hierarchy), because when a parent site's boundary is not contained in the area of interest, the child sites will not be contained either.

Each site is associated with a single theme. The themes are also grouped into a hierarchy. The theme hierarchy is arranged such that information that is applicable to a larger scale appears higher in the hierarchy than those applicable to a lower scale. For example, we might have state and city information. The scale of the state would be larger than that of the city. The state theme would be associated with one or more state sites (containing for example, geographic information of state-level features). The city theme would be associated with city sites. The city sites are children of a state site. The theme provides scale-dependant spatial filtering. For example, when we are at a scale where state data is not to be displayed, all sites associated with the state theme can be ignored.

This simple example shows how parallel hierarchies of theme and site can be used to easily filter very large data sets.

The ViSit Anywhere Data Model

The ViSit site model provides much of the insight that underlies the ViSit Anywhere data model. ViSit Anywhere manages project data in three hierarchies as shown is the project explorer dialog (left) used during project configuration. The top level hierarchy is the project data, which contains a model of the assets that are being managed by the application. The second hierarchy is the schema data that describes the project data.

The ViSit site model is implemented by placing themes in the schema hierarchy and sites in the project data hierarchy. A special site linkage object in the project data hierarchy is used to make the connection between themes and sites.

Finally, there is a types hierarchy that describes the assemblies and classes that are used to implement theme and project data.

Typical ViSit Anywhere users might not notice the structure of the data model, as it is transparent to the user, however, it is important for application administrators to understand the basic principals, so that they can optimize the organization of the project. Developers looking to extend the application should also understand the data model, as the object model is a simple implementation of the data model.

An important thing to understand is that the data model was designed with change management in mind. ViSit Anywhere provides a change managed environment for multiple users performing concurrent editing out-of-the-box, that is, not additional modules or configuration are required. A robust data model means that all data items, types, schema object and asset model objects are all change managed. Thus when any change is applied to the project that change is automatically distributed to all users, simply by synchronizing with the project server. The change might be a change in types (by the addition of a new application), a change in the project structure by the administrator (a schema change), or a change in the actual project data (by an edit from a standard user) - changes at all levels are changed managed using the same basic infrastructure.

In a future post I will explain how the structure of the data model enables the implementation of the change management system and how change management allows ViSit Anywhere users to more easily maintain their important geographic data and to derive more benefit from them.