Executive Summary
The real information in a website or an online market lies in its content. More informational, useful and interesting content is, the website will be more successful. More people will want to visit the site frequently. It is particularly true if a website is constantly adding content on a regular basis - be it articles, tutorials, news and opinion or retail information.
Content is what we communicate, and effective communication is about as fundamental a business requirement as one can get. Though there are millions of products available to buy on the web only the odd surfer or customer, who really needs the product or information, goes online to specifically look for that. So if the site simply sells the product and does nothing else to make the product or the information easily available or searchable by the surfer or customer then it is least likely that the web site will be successful. The dynamic nature of the content when effectively managed will be the most powerful engine that drives the success of a web site.
This paper discusses a structured approach to manage the life cycle of content from Content Analysis to the final delivery stage along with the importance of Content. It also explains the challenges involved in Content Aggregation and highlights the key factors which need to be considered while building an effective Content Aggregation System to power the web site’s sales, marketing and business strategy.
Introduction
Content creation and management is currently undergoing very profound changes. In brief, what's happening is a very fundamental step towards the separation of form from content, towards structuring data in a way that it can be reused in a variety of ways.
The surprising thing in this movement is that, unlike most digital trends in recent years, it is driven by the high end of the market: while page layout, image processing and HTML-based Web publishing are handled by shrink-wrapped tools, content creation and management is very clearly in the realm of high-end systems. Costing anywhere between hundreds of thousand to millions of dollars, we are talking here about tightly integrated custom-made client server applications that require heavy-duty databases and lots of specific programming. We are definitely not in the realm of end-user tools.
There is not much doubt, however, that this is where the next big market opportunity lies. In a funny way, we are back to where publishing was before DTP came around: content creation and management is once again the playground of larger players, and requires heavy investment.
With that in mind it is a good practice to have a clear understanding about Good Content, Content Life Cycle and the right implementation of the same. This paper focuses more on how the content is getting moved from different cycles till it reaches the end user.
Current Trend
The rapidly maturing market of on-line content has spearheaded this evolution, but that’s not where it stops. Increasingly, magazine and newspaper publishers are getting involved in this movement and, with the emergence of e-books, book publishers are beginning to be concerned by it, too.
According to one of the surveys, the contribution of the Content production activities increased from around $3.5 billion in 1996-97 to $3.9 billion in 2001-02, with a higher peak observed in 1999-00. Despite the contraction in later years, the average annual growth rate for industries in this component was recorded at 6.3 per cent over the period.
What is Content?
Content can be defined in number of ways. One way is to define that Content is all forms of information, and without limitation includes: text, formatted text such as HTML pages, interactive and/or dynamic Web pages (such as those generated from data files and databases), images, and animation, video and sound files. It may also include software, Web-based output from software applications and Web-based input (ie. transactions).
Content is more than information on a Web page. The way it is getting delivered, organized, and shown together form content. Imagine content as the fundamental block which holds the Web page together; build from different sources and make it easy to understand.
Content and E-business evolution
Content may be the once and future king, but it hasn’t always been recognized as such. Just as row-and-column database systems defined the nature of business automation in the ‘70s and ‘80s, it can be said that the automation of the creation, storage and delivery of a more random mix of data types has only penetrated business asset management activities over the past decade or less.
On the Internet content is king and always will be. This is because the Internet is the information superhighway and most people use it for information of some sort.
Content in the context of internet represents something of value on a web site. Content can be free information or links to other resources (may not be of competitors). Content of this nature generally will cost nothing to build. For example a product frequently asked questions (FAQ) page (furniture site), free maps (tourist site) or updated weather forecasts (travel site). The above free content information on any site may bring more customers.
With the evolution of E-Business, the perspective of looking at content also shifted. Content essentially breathes life into a website. The importance of good content can be effectively summarized by the following three points:
1. It increases conversion rates by keeping prospects on the site.
When prospective customers arrive at the site, they like to ‘dig deep’. They look for all the information they can find before they feel confident enough to commit to that purchase. Whether the site sells electronics, banking, subscriptions, or accessories, people want to know what they are buying.
If the site does not give them all the information they need, they will find someone else who does. Perhaps they will try epinions.com, consumerguide.com or other sites. Or maybe they will just do a search on Google.
And that’s very bad news for the conversion rates on any web site.
2. It differentiates the web site from its competitors
Most businesses have a number of direct competitors and all too often there is very little to truly differentiate the products or services from theirs. The web site may have some marginal distinctions, new offers or price deals.
Content can be used to differentiate one’s website with their competitors and can make the site an absolute magnet for anyone. It has become different because it becomes known for being the number one site.
3. Content is great for search engines
Creating great content for the web site is still a very smart thing to do. It’s good for customers, good for conversion rates, perfect for search engines and does a great job of separating the site from its competition.
Successful creation and maintenance of a good content requires a good understanding of the content lifecycle. The following sections of this document discuss in detail the content lifecycle as applied in the context of E-Business.
Content Engineering
From its creation to publication, content goes through many stages of refinement. It is both important and rewarding to engineer the Content – the lifeline of any website - with lifecycle management approach
The different stages of content lifecycle are:
Content Analysis
Content Creation
Content Extract
Content Load
Content Transfer
Content Aggregation
Content Management
Content Publish
The following paragraphs discusses each of these lifecycle phases in detail
Content Analysis
A content aggregation or integration begins with an analysis that defines what content an organization needs and how this content is sourced (created or purchased), stored, maintained and delivered.
The needs analysis is used to create a model of content that shows
• Who uses content, how they use it, what they use it for?
• How content flows through enterprise systems?
• How content changes over time?
A complete plan will include metrics to determine how effectively content is being sourced and used to impact performance.
Content Creation
There are content providers who will create or gather content from different resources and provide or sell this content to the online retailers or web. Content creation is one of the major businesses in the industry. It is very difficult for an online retailer who sells music and movies to create and manage content within there content aggregation system. In most cases, they have to rely on external content providers who will provide the content on a day to day basis.
A professional content writer can add more value to the site by writing about organization or any article which has to be posted on the web site.
Content Creation can also be a part of Content Management System. Content Management team can create, modify or edit the content from the provider. This has discussed later in this article.
Most of the Content Provider provides the content in the form of data file or xml file. This varies from provider to provider.
Content Extract
Extract the content from different content providers or other sources to the content landing server. The sources can be either a database system, or from a content provider FTP sever. The data file or input content file can be in any format. It can be a flat file delimited by any special character or can be an XML file.
Depending on the business requirement or data volume, one can design Delta calculation during the Content Extract process Stage. This delta calculation can be performed by comparing today’s load with yesterdays load. This will help in avoiding huge amount of data load, if there is no real change in the content.
Another option is to go for an ETL tool to extract the data. A customized program can be designed using different tools like Perl, SQL Loader etc. to load the data. There are so many articles on the net which will tell how to select an ETL tool or when to go for customized program.
Content Load
Once all the content lands on the Content Landing Server, It has to load it into the database. This database can be the content aggregation database system or it can be a staging database. Final choice depends on the application architecture and the requirements. There are many advantages if the data gets loaded into staging database. This will be discussed later in this document.
The other activities which need to perform before the content gets loaded into the database are filtering, formatting etc. This will have a huge performance gain in terms of time, space etc.
Pre filtering/ formatting process
Most of the content arrives from different sources may not have filtered or formatted depending on the requirement needs. In that case this is the best stage where all the data filtering and formatting gets applied. For e.g., if the site does not sell cassettes and if you are receiving content for cassettes along with the CD from the content provider, then it is not necessary to load the content into the database. By eliminating all the cassette related content, one can save real amount of effort, time and money.
The filtering processes can be summarized as below:
File Data transformations - string manipulation based on pattern matching
File Data Filtering - rejection of records based on predefined criteria
RegEx based Rules specification - perform pattern matching and string manipulation.
ETL tool may not be the real choice in this case if any project handles about millions of content records. The best way is to go for any customized program code using Perl or other utilities. Perl is one of the best tools that can be used to write any complex regular expressions to do different types of filtering and formatting.
Load Process
After formatting and filtering you can load the data into the staging database.
Staging database can be designed in a number of ways depending on the requirements. One approach is to design the database with transactional data. This will help in debugging data which is one week old or months old depending on any application. If it is required to keep transactional data from the provider in the database, then it should not have any referential integrity constraints on the tables. All the entities will be stand alone. For e.g., if movie related content is getting loaded, then movie, artist, award, review will stand as individual entities. There will not be any content relation between these entities. This will help the process to load the data faster into the database.
The second option is to have the tables with RI constraints and keep the back up in the flat file itself. Any util tools can be used to grep the data present in the flat files for any analysis if required.
Numbers of options are available to load the data from the landing server to the database. If the data is in the form of flat files, then it’s better to go for Oracle SQL loader. If it is from external database system, then the best approach is to use Informatica mapping.
Content Transfer
This is an optional staging process. If the system is designed with the initial staging area without any referential integrity, then this will help to make the content relational. Some of the filtering/ formatting rules can be executed during Transfer process. If the filtering requirement needs to have a look up table and based on the value, it has to execute some of the filtering, then this is the right place to implement those filters.
ETL tool like informatica or Oracle PL/SQL can be used for transferring content from initial staging area to the secondary staging area.
Content Aggregation
At its simplest level, content aggregation is the packaging and description of content so that it can fit into a distribution and delivery system. Content aggregation makes content easier to use by making it part of the systems that we use in our day-to-day life.
Content aggregation is of growing importance today for three reasons:
The vast increase in the amount of information available
The growing use of enterprise software systems to integrate all aspects of managing and operating an organization
The popularity of information appliances such as web sites, PDAs and advanced cell phones (but also cars, clothes washers and rice cookers with sophisticated digital displays) that require new ways to package content
Content aggregation is the integration of content into systems so that it can be used when, where and how it is needed. This is the stage where content from different sources marries together.
Content can be anything from a text document, to a video clip, to a sophisticated simulation. Real-time information feeds are another form of content that can be integrated into many types of systems.
So here is the final stage where the content get aggregated in the form in which it can delivered to any systems or any type of applications.
Content Management
Content management is the organizing, categorizing, and structuring of information resources (text, images, documents etc.) so that they can be stored, published, and edited with ease and flexibility. A content management system (CMS) is used to collect, manage, and publish content, storing the content either as components or whole documents, while maintaining dynamic links between components.
Content Management team will be responsible for managing the content and publishing it to the external world. They will make use of the Aggregated content and responsible for generating meta data and packaging the content in the right way.
A 'good' Content Management application will
Describe the content accurately making it easy to find
Tell you what the content can be used for (this includes rights management)
Tell you what the content has been used for and what other pieces of content it has been used with
Tell you what other pieces of content might be relevant
Provide your systems with the information they need to track and present the content properly
Packaging has two closely related functions:
To organize pieces of content into larger aggregations
To format the content so that it can be used on the target delivery system
In many ways packaging is the most complex part of content integration. It requires an intimate knowledge of information design, to put the content together into meaningful packages (in the case of learning it is instructional design skills that are needed).
It also requires deep technical knowledge of the delivery systems, whether these learning management systems, enterprise portals, special devices such as PDAs or the electronic support systems built into applications and devices. Packaging of content for use with the flight information systems of an airliner is very different than what is needed for a Point-of-Sales terminal at a retail outlet.
Content Publishing
Content publishing is the final stage of content aggregation. It makes use of content description and content packaging to make sure that the right information gets to the right person at the right time. The information must be available in the format required by the delivery system and the network must be managed to ensure that latencies are kept to a minimum and loads are reasonable.
Content publishing technology is in its infancy and is waiting for description and packaging technologies to mature before it can come to the forefront. Most enterprise systems have some content routing capability, as do network management systems, but an integrated solution is still some years away.
The following diagram depicts content flow in a typical application of content engineering for an e-commerce website.
Factors to be considered while building Content Aggregation Project
1)System
Restartability
Traceability
Flexibility to Add a new Provider
Reusability
2)Productivity
Effort involved in Adding a New Source of Content into the System
3)Hardware/Software Cost
Software Cost
Storage Space
Data Storage- Better Storage of Data(Both Transactional and Relational)
4)Data
Provider Specific Storage of Content
Precedence Rules – A set of rule to decide which provider content has precedence over the other provider.
Data Debugging
Relational Data Storage of Provider Data
Storage of Provider Specific Keys
Supports Multithreading capability to load files
Internationalization
Generic Database
Delta Identification
Supports Data Staging
Supports different types of data files(XML, Flat files)
Audit and operational reports for each data load
Challenges involved in a Content Aggregation Project
Understand your workflow
An in depth or thorough knowledge reveals the inter-dependencies of a large organization operation. It is very important to understand the business operation within an organization.
Understand your users, or who they are
Who is the final end user and how they were going to use your content
Understand your content
Understand your content structure and the format in which it is available. Then apply filtering and formatting and transformations that are required for your current business needs.
Look at the big picture
Understand the overall picture about entire architecture of your application and track how you will manage and where your content fit into the big picture
Implementation Challenges
. Deployment of a new IT architecture
. Training of staff in the use of the CMS
. Content Load from different Content Providers
. Migrating content into the CMS or Setting up the Initial Content
. Determining suitable workflows
. Changing work practices and processes
. Sustainability. The goal of all these activities is to ensure that the Content Aggregation solution is sustainable into the future. Much work will be required after the initial implementation project to achieve this goal.
Conclusion
The conclusion that could be drawn from these discussions is that, although the failure rate of Content implementations is high, an engineering approach to Content Lifecycle Management maximizes the chances of website’s success. If done correctly, Content can yield large business benefits, both in making the Web an effective medium for execution of sales, marketing and business strategies and in expanding the website to the rest of the world.
Acknowledgments
The author thanks and appreciates the help provided by Mr. Raghu Vishwanath for constructive comments and encouragement.
References
http://www.webassociates.com/
http://http://www.kmworld.com/
http://www.contentworld.com/
About the Author
Daison Jose is a Certified PMP having 8 years of expereince and 5 years of industry experience with a primary focus on e-Commerce application development. He specializes in managing,design and development of Web Applications,Content Management applications and Object Oriented applications. He has worked extensively in Retail domain with expertise in solution definition and packaging. Daison has successfully delivered many e-commerce projects for global clients. He holds a Masters degree in Computer Application and has published many articles. Daison can be reached at daisonj@gmail.com
