SharePoint is a great platform for managing email and attachments and has several advantages in the right scenarios. In addition to providing the capability to store, organize, and search for content, SharePoint enables email to become part of the content that is shared throughout the organization. This improves collaboration and content re-use. There are a number of alternatives for moving emails to SharePoint, including out-of-the box methods such as email-enabled lists, managed folders, and third party applications such as Colligo Contributor Add-In for Outlook. The key to success is building an architecture that is scalable, while making it easy for information workers to use.

PST Files – While some of the capabilities of Exchange and SharePoint overlap, there are distinct differences and advantages of one platform over the other for different scenarios. When important files are stored on individual PST files throughout the organization, the information becomes inaccessible islands. Worse still, PST files pose security and data loss concerns, and often lead to the loss of enterprise content. In reality, PSTs are a thing of the past. Storage and sharing of email and attachments in SharePoint offers distinct advantages for collaboration, document management, and discoverability through a rich search interface.

Exchange Public Folders – While I don’t recommend deploying Public Folders for new applications, you needn’t worry if you already have them. They will continue to be supported by Microsoft in the next major version of Exchange server after Exchange 2007, and for at least 10 years thereafter. For new Exchange deployments, SharePoint is “strongly encouraged” for application development.

If deploying both Exchange 2007 and SharePoint, it doesn’t make sense to build up a huge Public Folder deployment. It’s confusing to users and additional overhead for the team managing Exchange. Microsoft is making major investments in SharePoint and has indicated that it’s the preferred application platform for collaboration and document management going forward.

SharePoint Advantages – As discussed, SharePoint offers a number of advantages over PST files and Public Exchange folders for email management:

• It supports the sharing of email across the enterprise
• User-based email and attachment selection ensures that “important” content is stored
• SharePoint centralizes critical enterprise content on secure company servers
• Email becomes “structured” enterprise content when custom metadata is applied
• The email body and attachments become findable and reusable
• Information workers can take advantages of the version control and history features
• Content types support retention policies and regulatory compliance requirements
• SharePoint improves collaboration and enterprise content management

“Out of the box” there are two ways (highlighted in this series of posts) that Microsoft products support the integration of Exchange and SharePoint for email management: (1) “send to” email-enabled SharePoint lists and (2) “auto copy” Managed Folders to SharePoint.

Email-Enabled Lists – Email-enabled self service lists can easily get out of control and become a significant drain on IT resources. Microsoft IT decided against using email-enabled lists internally. Most organizations don’t need the email-enabled functionality and the oversight it requires, and users prefer to access archived emails in .MSG format (rather than the .EML format created in SharePoint lists). With proper planning and design, email-enabled lists can be deployed successfully. However, Managed Folders are a better alternative.

Managed Folders – Introduced in Exchange 2007, Managed Folders provide administrators with an easy way for users to archive email. Managed Folders can be configured to “auto copy” emails added to them to SharePoint libraries. This incredibly insightful feature can significantly reduce mailbox sizes, while capturing the intended emails (in .MSG format) and attachments. Still, abuse must be avoided to ensure that SharePoint does not become a dumping ground. Managed Folders require a solid information architecture design, and trained administrators who understand scalability and how to manage it.

SharePoint Scalability – Storing emails and attachments in SharePoint can have great advantages for project and knowledge management in corporations, if they are secured in the proper context. However, SharePoint lists and libraries can quickly be bogged down with too much content if not carefully managed. The scalability of SharePoint lists is a major consideration and requires oversight, especially when using email-enabled lists. The default “All items” view can be particularly problematic.

Training and guidance may be required to change corporate culture so users understand where and how content should be stored. Training users to tag content with metadata will increase the usefulness and “findability” of documents. Folders, custom views, indexed columns, and a query-based design are best for large lists to ensure fast performance.

Third-Party Solutions – When deploying SharePoint for email management, it’s important to consider what you get out of the box with Microsoft, and what you get with a partner. Most gaps you find in SharePoint are filled by a very solid partner ecosystem. For example, Gold Certified Microsoft Partner Colligo Networks offers an Add-In for Outlook that makes it easy for users to drag-and-drop emails into SharePoint, while reducing the burden on IT. I would encourage you to try their .Net client-only solution to see if it can meet your Outlook and SharePoint integration requirements.

Well, that’s it for this series of posts on the OfflinesharePoint blog. I hope you found them interesting and informative. If you’d like to keep up with my latest posts, please visit my blog “SharePoint Joel’s SharePoint Land”.

- Joel Oleson.

Colligo Contributor will reflect server performance optimizations to the client since it supports folders, views, and filters. In addition, the performance of the Contributor interface is usually significantly better than the browser for content cached locally, for example when accessing content for a specific view. A unique feature in Contributor, which is not available out-of-the-box in SharePoint, is default folder-level metadata. This feature can provide advantages for tagging content since users can set default metadata values that get applied automatically when content is dragged into a SharePoint folder in Outlook. Colligo Contributor 3.2 supports large SharePoint folder hierarchies rendered through the Outlook interface.

Storing all emails and attachments in a single document library is a common practice and popular method for personal storage, however this is not a recommended best practice for knowledge repositories. In SharePoint, document libraries require special information architecture because of performance degradation associated with lists that contain a large number of items.

To gain a better understanding of the issues, there are a few documents that I would recommend developers and architects review, as outlined below.

• The whitepaper called “Working with large lists in Office SharePoint® Server 2007” has charts and metrics based on query time with performance considerations being a key indicator. The recommended limit of 2000 items per folder popularized in WSS 2.0 is not a bad one for those that don’t have the time to invest in designing a query based interface, though there is no performance “cliff” at 2000 items in a list. You may want to recommend to your users that they keep their lists under 1000 or use 3000 items as an upper limit at which point IT gets involved in assisting teams and groups to more efficiently scale their lists.
• Microsoft IT recently published a whitepaper titled “SharePoint Performance Optimization” that discusses ways of optimizing render time of sites and reducing database blocking. At Microsoft, the IT team has set-up a scan to detect large lists over 3000 items to ensure proper usage and divide up content where appropriate.
• Another useful article is the one on capacity boundaries on TechNet. It’s a good resource for seeing guidance on “limits” within the product along with charts on list performance.

If you currently have lists with a large number of items, or cannot avoid them in your application, below are a few quick ways to improve their performance.

1. Views can have a big impact on performance. The default “All Items” view should be avoided because it can take tens of seconds to render as a list grows into the tens of thousands. Views with too many items can can cause content database locking during query time. Changing the default view to something with a smaller number of items can often be a quick fix.
2. Filters are another way of limiting the number of items displayed in SharePoint. Note, however, these queries can be inefficient in large lists.
3. For document management deployments, use folders and sub-folders for better scale and retrieval. Keeping each folder with less than 2000 will support better performance. Even better, limit folders to one hundred or so to prevent unnecessary scrolling.
4. Using Indexed columns is another way to manage list scalability and increase index query optimization.
5. The most efficient method of retrieval is search queries.

Truth: SharePoint lists exhibit performance problems as list size grows, so it’s best to limit the number of items in lists and use views, folders, and filters to improve query performance.

Colligo Contributor reduces the IT administration required to manage the routing of email content to SharePoint since it enables users to set-up their own links between Outlook folders and document libraries in SharePoint, based on existing user permissions set SharePoint. Like Managed Folders, Colligo Contributor automatically extracts email properties and moves emails and attachments to SharePoint when users drag-and-drop them into a folder in their mailboxes. In addition, Contributor enables the users to tag content with custom metadata and choose content type at the time emails and attachments are moved to SharePoint – a feature that is not available through Managed Folders. This can reduce the workload for legal site administrators and ensure that the proper retention policies are applied to content at the time it is saved by the user.

The subject of this post is Managed Folders. Managed Folders were introduced in Exchange 2007 to provide administrators with an easy way for users to archive email. Any Managed Folder can be configured such that all emails sent to it are routed to SharePoint. It’s an incredibly insightful feature and when implemented properly can reduce mailbox sizes, while capturing the intended emails and attachments. When not implemented properly, Managed Folders can be abused, causing SharePoint to become a dumping ground.

One poor example of managed folder design could be a managed folder called “Keep.” If everyone is told to put their email in that one folder and it’s archived to SharePoint, it sounds like a perfect solution for email archiving. However, there are some serious drawbacks. First, scale is an issue. Putting all that junk into one list can overwhelm SharePoint since it doesn’t scale well to support millions of items in one list, especially if there is a single view. Second, what about the security of that list? Managed Folders require extensive administrative set-up. It’s an IT option in Exchange, not a feature that is end user configurable.

So what does a good Managed Folder design look like? An example might be a folder titled “Legal Hold,” which is used to archive items under legal hold because of an investigation or other circumstance. On the SharePoint side, a specific document library is set-up and secured, then a legal site administrator is responsible for any tagging and for managing the views for the LCA team. A special search view might be set-up with specific indexed columns to support a quick and easy search. To avoid performance problems, avoid the “All items” view. I’ll discuss performance issues related to SharePoint list scalability next.

Truth: Managed Folders can work well, but require a solid information architecture design & trained administrators who understand how to manage scalability.

Meanwhile… we are continuing the series of posts on the “Myths and Truths of Email Management with SharePoint”. In his last post, Joel Oleson discussed some of the pifalls of using email enabled lists in SharePoint. I wanted to expand on that a little further.

While very powerful, email-enabled lists have some additional drawbacks that should be well understood before deploying them. These are described below:

1. When an email is sent to an email-enabled list, attachments are stripped off and stored separately.
2. The email body is stored as a .EML file, which cannot be opened in Outlook.
3. Users cannot specify different content types.
4. Users cannot specify custom metadata.

Colligo Contributor is an easy-to-deploy and manage .NET client Add-In for Outlook that enables users to move content to SharePoint through a simple drag-and-drop interface. It stores emails and attachments together in a single .MSG file, which can be opened in Outlook. Both content types and custom metadata can be set at the time of drag-and-drop. In addition, emails can be automatically moved to SharePoint using Outlook rules.

Emailing a post to a blog … very cool or archiving an Exchange Discussion List to a SharePoint list … super cool … but be careful. Email-enabled self service lists can easily get out of control. Microsoft IT, which loves to use nearly every feature of SharePoint, decided against using email-enabled lists.

Email-enabled lists can be a significant IT resource drain. Without the proper planning and management, AD objects will be created with archiving and no lifecycle. Contact account naming standards are another reason. IT doesn’t want to see random contacts in AD.

Everyone wants to have the document library called “docs” and everyone wants to have the discussion list called “discussion”. If you have a process or even a workflow to get requests and manage these requests, you can better manage who needs them, when they are needed and for how long. So, my recommendation here is to know what you’re doing. Otherwise, it’s very easy to end up with a mess.

Like Public Folders, email-enabled lists can also pose security risks if not managed properly. Fortunately, they are more often secured to the context of the team so they are not as much of an exposure.

List scalability can pose problems with email-enabled lists. You don’t want to send all data from all users to one list. Put content in context in different site collections, sites, folders, as it relates to the context of the group, team, or project.

Most SharePoint environments don’t need the email-enabled functionality and the oversight it requires. Those that decide to use it should plan to set-up specific content objects and point the lists at them. Steve Smith’s document called “How to configure Incoming Email Enabled Libraries in MOSS2007 RTM using Exchange 2003 in an Active Directory Domain” explains in detail how to set-up email enabled lists. Don’t be surprised if it’s more complex to set-up than initially thought. I recommend setting it up in a preproduction environment first and learning how it works, then exercising administrative tasks and troubleshooting tasks around maintenance of the list, inbound SMTP, and AD contact objects.

In summary, I’d suggest you get answers to the following questions before implementing email-enabled SharePoint lists:

• Are those contact objects in a separate OU?
• How do you know if the inbound SMTP stops working?
• Can anyone send to the list?
• Does the item show up the way you expect it to, or are just attachments showing up?
• Does the metadata look like you expect it to?

The answers will help you better understand the nature of these special lists and help you better take advantage of this functionality, if you decide to use it.

Truth: Use with caution: To be successful, email-enabled lists require management & oversight to scale past a few thousand items.

Joel mentioned that SharePoint is a better option if you are migrating Lotus Notes applications. It is important to consider some gaps that need to be filled when doing these migrations:

1. mail and document management system integration,
2. a rich client experience on the desktop, and
3. offline caching for mobile and remote workers.

Lotus Notes users will be accustomed to a rich desktop client that integrates with the email application and works online and offline. Out-of-the-box SharePoint only supports an online browser-based experience. Outlook 2007 syncs with SharePoint document libraries, discussions, and a few standard lists, but it doesn’t support elements such as custom lists, content types, metadata, or views that users are used to in Notes applications. It also doesn’t support drag-and-drop from Outlook to SharePoint. For an overview of the offline SharePoint capabilities of Outlook, you may find this post from 2006 informative.

If you are looking to provide an equivalent to the Notes client, the Colligo Contributor Add-In for Outlook supports SharePoint’s advanced elements that are used to migrate Notes applications, and it integrates SharePoint with email. In addition, it can cache SharePoint lists and libraries (including views and metadata) so the applications and content can be used offline.

In early 2006, the Exchange Team at Microsoft outlined their thoughts about the future of Public Folders in a blog post titled “Exchange 12 and Public Folders.” It was intended to let customers know that Microsoft was de-emphasizing Public Folders for certain applications, but many misunderstood this to mean that Public Folders were dead.

Microsoft’s position was clarified in March 2008 in a post titled “Updated Exchange Public Folder Guidance,” which encourages use of SharePoint for new deployments and migration where overlap exists. This blog confirmed that Microsoft will continue to support Public Folders in the next release of Exchange and for an additional 10 years from its release.

This post also provides the first scenario-based examples where Public Folders and SharePoint are compared. I know both the SharePoint team and the Exchange team were involved since I was on the SharePoint team when it was designed and played a role in getting it cross posted on the SharePoint team blog.

Click to Enlarge

Above is my version of the table in the blog post. I’ve changed some of the wording to make it easier to understand, with a bit of commentary emphasis on really investigating SharePoint functionality when both Public Folders and SharePoint have overlap across a scenario. SharePoint is “strongly encouraged” for most application development, though in some instances there still may be some advantages to Public Folders. For example, the capabilities of team calendars in Exchange vs SharePoint should be compared against the specific requirements of the application. Also, if you are looking at email-enabled lists in SharePoint, you may find that public folders are a better match for your requirements.

While de-emphasizing Public Folders, Microsoft didn’t stop evolving them altogether. For example, in Exchange 2007 there were new ways of managing the Public Folders with Powershell cmdlets. Please note that the functionality and features require investments on the part of the customer. However, the current guidance is that Public Folders are now in maintenance mode.

It’s illustrative to look at what Microsoft IT is doing internally with Public Folders. They decided to impose constraints both on time and size to limit the growth of the stores. Microsoft IT also turned off replication to reduce the unnecessary growth of storage. New folders were locked down to exception-based requests, with new provisioning requests pointed to SharePoint (with the common exception of distribution list archiving).

So why is Microsoft IT moving more content to SharePoint? In contrast to Public Folders, SharePoint was designed as an application platform and it’s important to note that this is the continuing direction of Microsoft. Both Windows SharePoint Services and Office SharePoint Server are major investments for them and they will continue to augment their functionality to support the common scenarios of customers.

While Exchange Public Folders were previously the endorsed solution by Microsoft for sharing email, SharePoint is now the recommended solution. If you are still considering Public Folders, it’s important to understand it has several well known limitations and challenges, described below.

1. Public Folders and custom applications built on them are the bane of the life of an exchange administrator. Control, performance, scale, and replicating chaos are a number of common concerns. Public Folders require heavy upfront and ongoing management.
2. Security is also often a concern due to the public nature of email-enabled Public Folders. Organizations are exposed to large risks if the random content flying around in emails is not controlled, so Public Folders need to be carefully managed. Security and liability are definitely something to be considered in all cases where email content is stored. It’s important to understand who has rights to store email on the server.
3. Public Folders can also be an HR nightmare. These “public” social dumping grounds often become a place where pictures and music are exchanged. As well, DLs (Distribution Lists) can get buried in group and team email-enabled security groups.

Bottom line: if you are deploying both Exchange 2007 and SharePoint technologies (WSS or SharePoint Server), it doesn’t make sense to build up a huge Public Folder deployment. It’s confusing to users and additional overhead for the team managing Exchange.

But should you use Public Folders when you are doing a Groupwise, or Lotus Notes migration? For nearly all your Notes applications you should seriously look at SharePoint. For email it’s a no brainer… it’s Exchange. For discussion groups and archiving, you’ve got a decision to make. When migrating to SharePoint, you should do it with an understanding of what you get out of the box, and what you get with a partner. Most gaps you find in SharePoint are filled by a very solid partner ecosystem.

One last point. If you plan to customize SharePoint sites and site templates, have a look at this blog post on Features and Solutions. Features allow reusable pieces of functionality to be created and deployed to other sites, without modifying site templates. Solutions allow you to package Features in a cabinet (.cab) file and define important metadata about those Features.

Truth: The Exchange Team will include public folders in the next version and at a minimum support them for 10 years after the release. However, they encourage you to seriously consider SharePoint for application development.

Colligo Contributor Add-In for Outlook enables users to drag-and-drop emails and attachments right into Outlook folders that are synced to SharePoint document libraries. It’s easy to use because it supports the familiar Outlook operations, so information workers readily adopt it to move selected content to SharePoint.

Contributor Add-In also supports the manual application of metadata or automatic application of folder-level metadata. It is ideal for both compliance and collaboration applications since it extracts all the properties from emails (To, From, Subject, Date, etc.) and saves them as metadata. Users can select the content type so that hold periods can be specified. All emails moved through Colligo Contributor are saved in the .MSG format, so they can be easily opened again using Outlook.