The writing team wishes to acknowledge the many ideas and efforts that our colleague Doug Nebert contributed to the concepts and development of GEOSS and the GCI and his constructive comments on early versions of this paper.
Table of Contents
Purpose and Scope of this Document 4
Global Earth Observing System of Systems (GEOSS) 5
GEOSS Common Infrastructure (GCI) 6
Community Objectives 6
GEOSS Information System: What community portal developers should know 7
GEOSS Architecture 7
GEOSS Use Cases 9
Data Sharing Principles – GEOSS Data-CORE 11
Interoperability Arrangements 11
Roles for Metadata and Documentation 11
Accessing GEOSS Data and Services 11
Community Portals in the GEOSS Architecture 11
Community Portal Interactions with GEOSS 12
Community Portal/GCI Functional Interfaces and Recommendations 12
Experience of the Architecture Implementation Pilots 17
Challenges in Developing Community Portals 17
Resources for building a community portal 18
GEOSS Registration Process 22
Accessing the Discovery/Access Broker (DAB) 22
Purpose and Scope of this Document
Over the last ten years the Group on Earth Observations (GEO) has been implementing the Global Earth Observation System of Systems (GEOSS) to provide decision support information and tools to users in a wide variety of communities. This ‘system of systems’ is linking together observing systems around the world, supporting development of new systems where gaps exist, and promoting common technical standards to enable development of coherent data sets. And to further facilitate data access, sharing, and coordination a GEOSS Common Infrastructure (GCI) has been developed that enables a user to locate and access a great variety of observations, tools, and other resources. The GEOSS Portal (http://www.geoportal.org/web/guest/geo_home_stp) is a component of the GCI that provides a broad view of all of the resources available from GEOSS.
The GEOSS Portal allows a user to narrow their search by the nine GEO Societal Benefit Areas (SBAs). However, most communities are also interested in developing their own portals that tailor searches of GEOSS resources according to their own needs and that integrate both GEOSS and non-GEOSS resources. They also wish to develop their own specialized applications and components that provide value-added services utilizing GEOSS and non-GEOSS resources. To do this they will need to interact with various GCI components directly, an approach consistent with the GEO System of Systems concept and which GCI supports. Also, servicing a broader set of needs of the GEO communities will strengthen their connection to GEOSS, promote the best practices of GEOSS, and enhance its sustainability.
The purpose of this document is to help the developers of community portals utilize the GCI and maximize the benefits they can get from it. We define a Community Portal as “a community-focused website that provides a human interface to content that may come from distributed resources.” The document was prepared based on recommendations from several events in 2013: The GEOSS Future Product Workshop; a working meeting for the Vision and Architecture for the GEOSS Information System; the Sprint to Plenary for the GEO-X Plenary; and the strong emphasis on community portals within the GEOSS Architecture Implementation Pilot, Phase 6 (AIP-6). Community Portals were initially identified as essential components for GEOSS based on responses to a Request for Information from the GEO Architecture and Data Committee in 2007. This document collects and builds on subsequent work both in GEO Tasks and external to GEOSS.
The authors of this document include some the key architects and engineers who have developed the components of the GEOSS Common Infrastructure (GCI) as well as representative from a number of communities who have or intend to develop Community Portals. The intent has been to ensure that the recommendations are responding to the actual needs of the various communities. The document reflects the reality that the collection of communities will have both common and specialized needs and that the recommendations should accommodate this diversity to the greatest extent possible.
This document focuses on Community Portals but the recommendations also address user clients and applications as well as the infrastructure elements that the communities have developed to support their own discipline-specific needs. It aims to increase the content available to a Community Portal by accessing the GEOSS Information System and, in particular, the data collections that are registered in GEOSS. The recommendations will promote the best practices and conventions that have been established during the development of the GCI and provide guidance and assistance to the communities on their use. The document will also target access to GEOSS services but with the recognition that communities will likely need to access non-GEOSS resources in support of their research and applications.
This is not a requirements document, rather, it is a collection of recommendations, best practices and available resources aimed to spur further development and use of Community Portals to benefit users of Earth Observation information.. Further, as additional communities actively engage with GEOSS it is envisioned that they will enrich the accessible components and services and guide the continued development of the GCI components as they evolve to better serve the needs of the communities.
Global Earth Observing System of Systems (GEOSS)
The GEO Vision is to realize a future wherein decisions and actions, for the benefit of humankind, are informed by coordinated, comprehensive and sustained Earth observations and information.
Key GEO Objectives include:
Improve and Coordinate Observation Systems
Advance Broad Open Data Policies/Practices
Foster Increased Use of EO Data and Information
A major barrier to meeting the GEO objectives is the difficulty in discovering and accessing the rich set of EO data and information resources that exist today or are planned in the future. The various agencies of individual GEO members have developed data and information systems that support their collections of EO data but these are targeted to their own specific programs and user communities. A major GEO initiative is to develop the capability to link these systems together such that the full set of EO resources are available to the entire set of users through a Global Earth Observing System of Systems (GEOSS). By coordinating the development of a GEOSS Common Infrastructure (GCI) and the policies and procedures for its use, GEO is enabling users across the nine Societal Benefit Areas access to this diverse but integrated set of EO resources.
Figure 1. GEOSS Overview
GEOSS Common Infrastructure (GCI)
GEOSS consists of a global and flexible network of content providers and aims at providing decision-support tools to a wide variety of users. A set of core components and functions, known as the GEOSS Common Infrastructure or GCI, were designed and deployed to:
Allow GEOSS resources (e.g. systems, data, services) to be easily discovered and accessed
Improve interoperability for existing and future observation systems
Document and share a set of interoperability arrangements and best practices that can be used as a reference for future projects
Build an Open Infrastructure in accordance with the GEOSS Data Sharing Principles
A detailed description of GEOSS and the GCI is provided in Section 3.
As the recommendations contained in this document have been developed a deliberate effort has been devoted to understanding and responding to the general needs the various GEO communities. This has included discussions with community representatives on the writing team that have been involved in developing or at least planning the development of portals for their respective communities (overviews of a number of these portals are presented in a later section). While it is impossible to completely or precisely represent the objectives of the diverse set of GEO communities what follows is our current understanding of the common goals associated with the development of community portals.
Most portal developers will want to provide their stakeholders with free, easy and open access to data tools, and services. Data resources may be raw data and data products, but also metadata about observational programs, projects, and observational platforms. Decision makers at all levels are key stakeholders for GEOSS, and these are often governmental and non-governmental organisations and communities. All communities and stakeholders are welcome, and encouraged, to use and contribute to GEOSS.
The providers of observational assets may or may not have their own platform for registering and storing their observational assets. In case they have their own platform, they are willing to provide access to this platform so that externals can have access to the assets. Externals can be humans or machines, and in both cases, the information providers are willing to organize the information according to a certain standard. Providers wish to keep control over their legacy data and favor the hosting, maintenance and update of such data for the benefit of the community.
Communities will need to be in dialogue with the individual information providers and will be interested in following the progress of the information providers’ activities and in understanding if they meet certain targets. The communities may also like to define certain Indicators which will document and quantify changes in the topic area that is subject to observation. The data resources are meant to support the production of these Indicators.
The communities are faced with the challenge of supporting the discovery and access of data from multiple sources and generally are adopting a system of systems architecture (SoS) in their own environments with these characteristics:
1) A three-level tier model: access, mediation and client:
a. The access level is where the observational assets is stored, and will be operated by the information providers.
b. The mediation level will hold a registry of information providers and harvest information from these. This level will also make sure that the harvested information is in agreement with agreed protocols.
c. The client level will be the place where the stakeholders and the governance body will have access to the harvested information.
2) The community platform may be linked up to a more general purpose platform with a global scope that includes other information providers, Indicators/SBAs, stakeholders, and topic areas. Ideally, the local and the global platform should share protocols. If not, a mapping mechanism between the protocols should exist. It should also be possible to associate assets at the global platform to the community platform and the opposite.
3) The protocol for the harvesting of information should be compliant with international recognized standards like ISO.
GEOSS Information System: What community portal developers should know
The GEOSS Information System is being developed to support distributed deployment, access and management of services on Earth Observation data. It is comprised of the GEOSS Common Infrastructure and all of the relevant components and services provided by the participating GEO communities. An overview of the GCI architecture is shown in Fig. 2 below. The initial set of GCI components included a registry that contained metadata required for the discovery and access of these resources, a Clearinghouse that would harvest registered metadata catalogues to support search of the resources and a GEO Web Portal to present the GCI services to the user. The GCI also included additional registries of standards and interfaces used by the GEO community and best practices based on their on-going implementation experience.
Figure 2. The components of the GEOSS Common Infrastructure (GCI)
Since the initial implementation of the GCI was made operational in 2007, considerable evolution and expansion in portal services and distributed search capability has taken place. A new component, the Discovery and Access Broker (DAB) has been implemented that enables a broader search of EO data resources by harvesting the contents of the Clearinghouse and brokering them with the extensive data collections supported in major EO repository catalogues. For a number of commonly used client protocols, the DAB can support search by a set of standard criteria and access to data and information resources that are registered with GEOSS and provided by a diverse set of EO data providers. The DAB also supports discovery and access to data collections that visible via distributed search of catalogs and brokers of the weather and Earth observation satellite communities. The original role of the Clearinghouse as the primary harvester of GEOSS-registered catalogs has been overtaken to the extent that the Clearinghouse and CSR are now just one among multiple resource registries (all supported by the DAB). Today, the GEO Web Portal (now called the GEOSS Portal) has evolved to interface with DAB to provide access to the wider set of EO resources and has made advances in providing community-focused services.
Although to date much of the focus of GEO has been on the development of the GCI and the registration of relevant Earth observation data and services the GEOSS vision includes all of the community-developed components that contribute to GEO objectives. This broader picture of GEOSS is shown in Figure 3 where the components are characterized as part of a service layer in a 3-tier model.
The top tier (Client Tier) is the one with which people deal directly. It provides the interfaces to describe and use the services offered. It includes the GEOSS Web Portal, Community Portals and Client Applications;
The middle tier (Mediation Tier) embodies all the business processes required to respond to requests issued by clients. The services in general embody everything from authentication to complex geoprocessing on sets of data from various repositories and from generation of map views to statistical charts that the client gets back at the end of the process;
The lower tier (Access Tier) provides read and/or write access to data, whether it is geospatial data, accounting records, or catalogue entries stored in any of a dozen different types of registries.
Figure 3. GEOSS Components
In the figure, “ACCESS” implies the services in the Access Tier, i.e., ftp, WMS, WCS, WFS, OPeNDAP, Order, SOS, SAS, SPS. Some of the ACCESS services are also provided in the Mediation Tier and maybe access by the Clients. For more detail on the figure including descriptions of each component and service see the GEOSS AIP Architecture http://www.earthobservations.org/documents/cfp/201302_geoss_cfp_aip6_architecture.pdf
GEOSS Use Cases
As with the Internet on which it is largely based, GEOSS will be a global and flexible network of content providers allowing decision makers access to an extraordinary range of information. Use Cases are reusable functions deemedmost essential to using GEOSS. The GEOSS use cases provide easy, familiar processesfor providers to publish GEOSS components, as well as for users to discover and access them across many communities of practice. A summary of GEOSS Use Cases was developed within the Architecture Implementation Pilot (AIP) activity and provides a summary of the functionality to be provided by GEOSS (Figure 4 below).
Figure 4. Use Cases: the essential functions of GEOSS
The uses cases identify two types of “actors” that are involved in GEOSS but the full complement and their responsibilities are:
Operates GCI components and approves registrations
As evident from their description, the SBA Integrators have the responsibility to integrate all relevant components of the GCI and those of the community to meet the requirements of the particular SBA community. Because of this role the SBA Integrators are the primary audience for this paper.
Data Sharing Principles – GEOSS Data-CORE
The GEOSS 10 Year Implementation Plan defines the GEOSS Data Sharing Principles:
"There will be full and open exchange of data, metadata, and products shared within GEOSS, while recognizing relevant international instruments and national policies and legislation. All shared data, metadata, and products will be made available with minimum time delay and at minimum cost. All shared data, metadata, and products for use in education and research will be encouraged to be made available free of charge or at no more than the cost of reproduction.”
GEOSS has committed to: 1) maximize the number of documented datasets made available on the basis of full and open access; 2) create the GEOSS Data Collection of Open Resources for Everyone (GEOSS Data-CORE), a distributed pool of documented datasets with full, open and unrestricted access at no more than the cost of reproduction and distribution; and 3) develop flexible national and international policy frameworks to ensure that a more open data environment is implemented, thus putting into practice actions for the implementation of the GEOSS Data Sharing Principles;
GEOSS depends on resource providers accepting and implementing interoperability arrangements, including technical specifications for collecting, processing, storing, and disseminating shared data, metadata, and products. GEOSS interoperability is based on non-proprietary standards, with preference to formal international standards. Interoperability is focused on component interfaces thereby minimizing impact on systems other than those interfaces to the shared architecture.
Roles for Metadata and Documentation
Accessing GEOSS Data and Services
Community Portals in the GEOSS Architecture
Figure 5 below illustrates how Community Portals and their associated infrastructure elements may interact with the GCI. Note that Community Portals are themselves a resource that can be discovered and accessed through the GEO Web Portal—giving them more visibility. Also, note that the portal can utilize the GCI to provide a custom view of GEOSS resources, so that only those resources of interest to the community are visible. The diagram also shows the interactions where the GCI can access the Community Infrastructure components.
Figure 5. Interactions between GEOSS Common Infrastructure and Community components.
Community Portal Interactions with GEOSS
As described in Sec. 2.3, community portals provide access to data and information resources useful to its stakeholders and are typically developed in support of a particular program, science discipline and/or application. These resources may be registered and accessible from the GCI or from other sources. The community portals can access both GEOSS and non-GEOSS data, provide a view of them customized for their users and support value-added services on the data resources (processing, analyses, visualization, etc.). Figure 5 in Section 3.7 introduced the interfaces between the GCI and the community components and demonstrates the two-way interactions between the GCI and community components.. The following sections will more fully describe these interfaces and present recommendations to guide the development of the community components. The goals of these recommendations are to enable the community to fully utilize the capabilities of the GCI and to facilitate the use of the communities’ capabilities by the full GEO community.
Community Portal/GCI Functional Interfaces and Recommendations
The highlighted interface shown below is the link from the GCI to a community-developed portal and represents the most basic interactions between the GCI and the community components. The community portal should be registered in the GCI if it provides access to data and/or services, either directly or indirectly through a brokering service, which would be useful to the GEO community at large. This link could be the only interaction between the GCI and the Community Portal but the extent to which it supports additional interactions with the GCI should be noted in the registration process.
Figure 6. Link from GEOSS Portal to Community Portal
Table 1. Link to Community Portal - Recommendations
If it provides open access to EO data and services and/or related information the portal should be registered in the CSR or DAB.
The next interface noted in Fig. 7 below defines what might be called a Community GEOSS Portal that accesses the GCI components but provides a particular thematic view of GEOSS resources. The community portal or brokering service could access the GCI DAB for each query synchronously or a community catalog could be constructed by harvesting relevant information from the GCI. Community clients may interact with the DAB using its GOESS API or via one of the many discovery interfaces published by the broker (e.g. OGC CSW, OpenSearch, OAI-PMH etc.). A complete description of the use of these interfaces is provided in Section 5.2 Tutorials. Whereas the GEO Portal has to serve the entire GEO community, these types of portals can assist their users by highlighting those GEOSS resources that are most relevant to its target set of users and also control how those resources are presented. These portals also can support integration of GEOSS and non-GEOSS resources. As a result of the direct interactions with the GCI the developers of these portals are in a position to utilize and promote the GCI standards and best practices within their respective communities.
A Community GEOSS Portal should use the OpenSearch interface to the DAB.
Needs to be discussed, but where there are options (e.g. CSW, DAB APIs) we should provide a recommendation and rationale.
A Community GEOSS Portal should be clearly identified as such and registered in the CSR or DAB.
These portals would be candidates for a direct link on the main page of the GEOSS Portal.
The next interface highlighted in Fig.8 refers to cases where the GCI accesses components of the Community Infrastructure to provide its clients (GEOSS Portal or others) with access to the target community resources. The current distributed search function that allows the DAB to access community catalogs (CWIC, FedEO, WIS, etc.) is a case of utilizing this type of interface.
Figure 8. GCI Access to Community Infrastructure
Table 3. GCI Access to Community Infrastructure - Recommendations
In building its own catalogue of EO data and services the GEOSS profile of ISO 19115 should be used.
To better support integration/search of community and GEOSS collections.
The next interface that is shown in Fig. 9 below is not actually between components but instead represents the interactions between the implementers of access services among and across the different GEO communities. The integration and inter-use of data and services from all GEO communities and disciplines are greatly facilitated if sharing and collaboration are envisioned during the implementation of such services.
Figure 9. Integration of GEOSS and non-GEOSS Resources
Table 4. Integration of GEOSS and non-GEOSS Resources - Recommendations
Use OGC WFS for a feature layer of (water) data sites.
These are a couple of recommendations from David’s Water SBA presentation
Use OGC SOS 2 as the web data service for (WaterML 2.)
The final interface between the GCI and community components highlighted in Figure 10 shows the registration of appropriate community resources in the GCI. As communities become more closely engaged and interdependent with GEO and the GCI they will be more likely to promote and enable the use of GEOSS standards and best practices for the access of data and services and also the registration of Earth observation data and services in the GEOSS registries.
Figure 10. Registration of Community Resources
Table 5. Registration of Community Resources - Recommendations
Experience of the Architecture Implementation Pilots
Challenges in Developing Community Portals
(rough ideas in need of extensive editing)
Focus vs. General Utility – Developers of community portals are focused on the immediate needs of their respective communities whereas the GEOSS/GCI is attempting to assure that all resources are generally available to address the full range of Societal Benefit Areas. While it may take some additional resources to assess and utilize the Best Practices of GEOSS the benefit to the communities is the access to broad set of GEOSS data and information as well as the shared experience of other GEO community development teams.
Evolution of GCI – The development of the GCI is a long-term process and its components have and will evolve to meet evolving requirements of the GEO community and to incorporate advances in information technology. This can certainly impact those communities that are using the GCI but their close engagement with this evolutionary process and input into the design decisions can mitigate many of the communities’ concerns.
Missing Functionality – In some cases the GCI has not addressed some of the technical challenges that are important to the various communities. An example is user authentication and authorization.
Duplication of Products/Services – In a large enterprise such as GEOSS that integrates many catalog and broker capabilities it is inevitable that individual products and services will show up multiple times in result sets.
Resources for building a community portal
Name of the Portal: Energy Community Portal
In the field of renewable energy, the most known and likely oldest international initiative is that of the World Meteorological Organisation (WMO) which consistently organized for many decades the collection and exchange of meteorological data. Measurements of solar radiation made in each country by the national weather bureau are stored in the World Radiation Data Centre in Saint-Petersburg, Russia.
Focusing on the African continent and solar radiation, many efforts have been made to produce maps of solar resources (Solar Radiation Atlas of Africa, 1991) or data sets of radiation estimated from satellite images, e.g. HelioClim (Blanc et al. 2011; Lefèvre et al., 2013), or NASA SSE (Stackhouse et al, 2004; Zhang et al, 2009). Experience shows that potential users of such data do not necessarily know all available data sets depicting the resource in a given region. Should the user know the existence of the HelioClim database, it may be difficult for him to identify the appropriate data set since the HelioClim database comprises several data sets, having the same geographical coverage but different spatial and temporal resolutions, and different temporal coverage (Blanc et al. 2011).
A tool permitting to search and discover data sets on renewable energy resources and the properties of these data would be highly beneficial to the users of those data. It should allow simple keywords based queries as well as advanced spatial and temporal based queries on selected area of interest, regions and/or themes. Data users should be able to view the main properties of the answers, thus helping them selecting the most appropriate data set(s).
Conversely, providers of data have an interest in having their data exploited by users. Most often these data are available in servers on the Web, thus of easy access. However, putting data into a Web server does not ensure their visibility. Therefore, they will benefit from a discovery tool since the existence of their data sets will be more widely known.
The advantage of having a single point of access to search for renewable energy related information is an asset for all actors. It tackles a large panel of data sources in renewable energy to search for (solar, wind, bioenergy, geothermal…), provides a global coverage and enables a uniform presentation of each available data set or application.
The webservice-energy Community Portal
This webservice-energy community portal is an initiative of MINES ParisTech launched in 2008 to support the renewable energies community. Since 2011 it evolves as a Spatial Data Infrastrcuture (SDI) with the addition of a a key component such as a geospatial catalog. The webservice-energy catalog is build-upon the Catalog Service for the Web (CSW) standard promoted by the Open Geospatial Consortium (OGC).
The catalog gathers the description of geo-referenced resources that a data provider wish to made available as a formal reference about his/her resource. This description includes mandatory and facultative fields as a form of a metadata record. ISO standards (ISO 19115, 19119, and 19139) support this formal metadata record description.
The catalog provides two mains features:
Search and discovery of resources
The catalog provides human friendly and machine-to-machine interaction to search and discover its internal resources. Search can be made ranging from simple free text keywords based approach to advanced spatial and temporal based queries on selected area of interest, regions and/or themes.
The catalog gives access to detailed metadata information’s as human friendly web-based pages as well as machine-to machine oriented presentation such as XML files in conformance with ISO 19115/19119 standard.
The catalog content can be harvest and synchronized with other CSW based catalog such as the one from UNEP-GRID or broker such as the GEOSS Discovery and Access Broker (DAB). This enables the content of the webservice-energy catalog to be accessible through the GEO Portal as well as on the UNEP catalog. Either the full catalog content of only part of it can be harvested. Fine-tuning the content to be shared is made possible with existence of virtual CSW server entry points. Well-defined criteria can be applied (e.g. Only deliver metadata according to a given theme such as “solar energy”).
The catalog enables Create, Read, Update Delete (CRUD) operations on all or part of its metadata records either by human actions under the administration section of the web-based platform and/or by machine-to-machine operation.
The catalog is capable of harvesting remote resources. This powerful feature enables the catalog to interact with remote web-based geo-referenced resources. Theses resources can be deployed on geospatial web-based servers such as Mapserver or GeoServer as well as with other CSW catalog. This last option is made possible thanks to the respect of CSW standards and allows to create networks of catalogs sharing seamlessly their resources for the benefit of users and resources providers.
The webservice-energy community portal also offers access to:
Application schema developed by the international community on solar resource knowledge. These schema helps to develop web services that are interoperable, i.e., that can be invoked by various portals.
A gallery of energy-related visualizations tools, apps and WebGIS client that make an extensive use of content available in the geosatial catalog.
Figure 1: Webservice-energy Community Portal
Figure 2: Webservice-energy Catalog Name of the Portal: EnviroGRIDS portal
Description: The resources of the enviroGRIDS system are accessible to the large community of users through the BSC- OS Portal that provides Web applications for data management, hydrological model calibration and execution, satellite image processing, report generation and visualization, citizens oriented applications, and virtual training center. The portal publishes through Internet both the geospatial functionality provided by Web technologies, and the high power computation resources supported by the Grid technologies.
GeoNetwork: search, discover, and access EnviroGRIDS core data sets.
Geoportal: allows users to search, discover, and access data sets in the Black Sea catchment.
Greenland: generates and executes on the GRID workows processing satellite images.
gSWAT: allows users to calibrate SWAT hydrological models on the GRID.
eGLE: implements both a user interface, and the tools for the development, the execution and the management of teaching materials.
BASHYT: is a Collaborative Working Environment (CWE) on the web that relies on complex "physically based" hydrological, land cover and ocean models to support decision makers through a user-friendly Web interface.
Types of services: All componentes are able to publish and consume OGC services.
Types of involved infrastructures: SDI and Distributed computing platform (Grid)
Relation to GEOSS/GCI:
all datasets produced by enviroGRIDS are registered in the GCI.
EnviroGRIDS discovery client is interacting the the GEOSS CSR (using the CSW interface).
Further reading: EnviroGRIDS Special Issue on “Building a Regional Observation System in the Black Sea Catchment" freely available at: http://thesai.org/Publications/ViewIssue?volume=3&issue=3&code=SpecialIssue
GEOSS Registration Process
See the workshop “Bringing GEOSS services into practice” (http://www.geossintopractice.org).
Chapter 9 explains how to register resources in GEOSS.