CDMS Evaluation Page 1 of 11 ANNEX 8 WMO Climate Database Management System Evaluation Criteria System Name: Version: Contributing Country: Contact Information Contact Person: Telephone: FAX: Email address: Postal address: Date: General Criteria ! Usability ∀ Does the system include on-line documentation? Yes No ∀ Do forms and menus provide context sensitive help? Yes No ∀ Do forms and menus prevent illegal user input in entry fields? Yes No ∀ Are error messages generated for illegal inputs? Yes No ! Language Support ∀ List the language(s) supported by this system: ∀ Will the developer provide additional language support if requested by a Member Nation?Yes No - Explain any conditions associated with these requests: ! Manuals ∀ Installation Guide Yes No ∀ User=s Guide Yes No ∀ System Administrator=s Guide Yes No ∀ Programmer=s Guide Yes No ∀ Database Model and System Design Concepts Yes NoCDMS Evaluation Page 2 of 11 ! Implementation and Maintenance Costs In a supporting document, provide realistic estimates of initial and recurring annual costs of implementing and operating this system. Provide estimates for a typical installation that includes a breakdown of equipment, personnel, and software licensing fees. These estimates should include details of equipment types (or model numbers), the number and level of expertise of required personnel, and purchase and/or licensing fees associated with commercial software required by the system. Provide a summary of these costs in the space provided, below. Cost Summary Currency Type: ∀ First-year costs - Equipment purchases - Software purchase/licensing fees - Personnel ∀ Annual recurring costs - Software licensing fees - Personnel Note: Please provide additional cost information regarding >typical= and >minimal= configurations that may apply to the installation of this system. ! What method(s) will be provided for users to obtain help and assistance when they encounter problems with using or administering the system? (Check all that apply) The user will be expected to solve their problems using the provided documentation. A user group email facility will be established that will allow users to exchange questions and comments with each other to obtain help. The developers of this system will monitor this facility and provide answers and advice as needed by the users. An archive of Frequently Asked Questions (FAQs) will be maintained as part of this user group facility. A help desk facility will be established and maintained to answer questions that a user may encounter. The help desk will provide help on system usage (data entry, data extraction, etc.) The help desk will provide system administration assistance. On-site service and assistance will be provided to users when requested.CDMS Evaluation Page 3 of 11 Key Entry of Paper Forms ! Is the on-screen layout of the data input form similar to the layout of the paper form containing the data? Yes No ! Does the system allow the user to design customized data input forms? Yes No ∀ Is language support provided in this capability? Yes No ! Form entry data commit to database ∀ Can the user commit data to the database from the data input form? Yes No ∀ Can the user produce a data file from the data input form and enter the data to the database as a separate process? Yes No ∀ Does the system allow for both of the above data input capabilities? Yes No ! Validation checks during key-entry ∀ Are checks made to test the validity of data types for numeric, text, and >tick mark= entry fields? Yes No ∀ Are checks made for valid data ranges? Yes No - Are these validity checks >station dependent= Yes No ∀ Do the validity checks test for impossible combinations of data values? (ex: maximum temperature < minimum temperature) Yes No ∀ Does the system allow key entry personnel to ignore validity checking and continue entering data? Yes No ! Double key entry (form data must be entered twice to ensure accuracy of data key input) ∀ Does the system allow for double key entry? Yes No ∀ Does the system allow for single key entry? Yes No ∀ Does the system allow for both types? Yes No ! Quality Control ∀ Are quality control procedures applied during key entry as the form data are entered by the data entry personnel? Yes No ∀ Are quality control procedure applied during the process of committing data to the database? Yes No ∀ Is the quality control process performed as a process independent of the key entry process? Yes No ! Are default value fields masked during the key entry process? Yes No ! Does the system allow key entry personnel to skip entry fields? Yes No ! Is the key entry process documented? Yes No ∀ Is on-line documentation available? Yes No - Is the on-line documentation available to the operator during the key entry process?Yes No ∀ Does the documentation include key entry examples Yes NoCDMS Evaluation Page 4 of 11 Import of Digital Data ! Indicate the standard GTS message format this system can ingest into the database. ∀ TEMP Yes No ∀ PILOT Yes No ∀ SYNOP Yes No ∀ METAR Yes No ∀ CLIMAT Yes No ! Does this system have the ability to import key entry data files containing multiple stations or multiple forms for a single station? Yes No ! Does this system have the ability to import data from other systems such as Automated Weather Stations (AWS)? Yes No ∀ Does this system allow the user to define the input format of data files that a user wants to import to the database? Yes No ! Can this system import data from CLICOM? Yes No ∀ Indicate the type of CLICOM data formats that this system can import. - 15-minute observations Yes No - Hourly observations Yes No - Daily observations Yes No - Monthly observations Yes No - Upper air observations Yes No - Station Normals Yes No - Station Metadata Yes No ! Can this system automatically schedule when a data file is imported into the database?(for example, off-line import of data during non-peak hours) Yes NoCDMS Evaluation Page 5 of 11 Validation and Quality Control ! Does this system provide quality control flags to indicate the original data capturing methods for each observation? (For instance, was the observation entry derived from a real-time data feed, a key entry process, or an electronic data import?) Yes No ! Does this system store the most recent QC results at the entity level? Yes No ! Does this system offer replacement estimates for observations that >fail= QC? Yes No ∀ Can a user override the value suggested by the QC process? Yes No ! Does this system record the date of the QC change? Yes No ∀ Is the date recorded at the record level? Yes No ∀ Is the date recorded at the entity level? Yes No ! Does this system maintain a copy of the original observations that are replaced during the QC process?Yes No ∀ Is the original observation, replaced by the QC process, stored as an entity in the relational database? Yes No ! Does this system provide QC status flags to indicate the level of QC applied? Yes No ∀ Are the QC status flags stored at the record level? Yes No ∀ Are the QC status flags stored at the entity level? Yes No ! Does this system provide QC flags that indicate the reason for replacing an observation? Yes No ! QC and validation processes details ∀ Are observed values compared to the valid range of values for the particular observation type? Yes No - Are the range of values obtained from defined or calculated range limits determined for each station Yes No ∀ Is the consistency of a set of observations validated by comparing an observation to other observations in the observation record? Yes No ∀ Is temporal coherence validated by comparing observations to other observations (of similar type) in the time series? Yes No ∀ Is spatial coherence validated by comparing an observation to similar observations from adjacent stations? Yes No - Are graphical displays of QC information provided? Yes No ∀ Are observations tested using statistical methods? Yes No ∀ Are the QC routines compliant with WMO guidelines? Yes NoCDMS Evaluation Page 6 of 11 Database Model Used for Climate Data Storage ! Is documentation available which describes the definition of each table in the database and the relationship between tables? Yes No ∀ Is the data model type described in the documentation? Yes No ! Does the system use a consistent naming convention for tables, entities, indexes, and views? Yes No ! Does the database support internal Unicode that provides a unique number for every character, no matter what the platform, no matter what the program, no matter what the language?Yes No ! Does the database include a programming interface that allows applications to pass queries to the Relational Database Management System (RDBMS) and receive the results back to the application?Yes NoCDMS Evaluation Page 7 of 11 Data Extraction ! Does the system have a Graphical User Interface that allows users to enter data extraction parameters into defined fields? Yes No ! Does the system allow users to enter data extraction queries (such as SQL) at a command line prompt?Yes No ! Select the statement that best describes the initial amount of training required for a user to begin extracting data from the database. ∀ Data extraction training requires < 1 hour training (intuitive) Yes No ∀ Data extraction training requires < 1 day of training Yes No ∀ Data extraction training usually requires > 1 day training Yes No - Typical number of days of training: ! Is the data extraction process documented? Yes No ∀ Is on-line documentation available? Yes No - Is the documentation available to the operator during the data extraction process?Yes No ∀ Does this documentation include examples? Yes No - Does the documentation include examples of simple and more complex data queries? Yes No ∀ Does the system include documentation for queries entered from a command prompt? Yes No - Does the system provide references to external documentation sources, such as SQL language manuals? Yes NoCDMS Evaluation Page 8 of 11 Metadata ! Are station metadata elements required to extract climate data? Yes No ∀ Select the station metadata items that are required to be in the metadata system in order to select climate data from the database. Station name station number/identifier latitude/longitude elevation country climate element (temperature, etc) other (specify elements): ! Are the number and type of metadata elements managed by this system extensible to include other types of metadata information required by the user? Yes No ! Does the system support graphical data types such as pictures of the climate station or scanned information such as instrument manuals? Yes No ! Does the system allow for the maintenance of historical metadata information that provides a history of changes to station behavior? Yes No ! Does the system store information describing individual variables? Yes No ∀ Does this metadata include information describing the instrument used to perform observation?(height, type, exposure, change dates, maintenance,etc.) Yes No ∀ Are beginning and end dates recorded for each variable? Yes No - Are these dates updated whenever an observation is inserted into the climate data archive? Yes No ∀ Does the metadata contain observation schedules describing when climate observations are supposed to be recorded by the observer? Yes No ! Describe aspects of your metadata system that provide additional capabilities that are not included in the questions presented, above:CDMS Evaluation Page 9 of 11 Output Products ! Climate database management systems normally provide a collection of standard output products that use the stored data. Using the following list of standard types of products, indicate the products that this system can produce. Data listings and/or tabulated data Hourly data Daily data 10-day data Other (specify): Tabular summary of statistical analysis Daily summary 10-day summary Monthly summary Annual Summary Other (specify): Typical types of statistical analyses Means, totals, and standard deviations Normals Extreme values Frequency analyses Count of missing values Counts based on thresholds (ex: # days <> threshold value) Graphical products Time series plots of single data variables Time series plots of multiple variables Wind roses Upper air sounding Station model plots of multiple stations Contour analyses Other analyses (attach documentation for additional output and analysis products) ! Indicate the type of output formats supported for output data and/or products ASCII text HTML XML PDF Comma separated values (CSV) Export to CLICOM (DataEase) WMO message formats CLIMAT CLIMAT TEMP Other (specify): Gridded data output (specify):CDMS Evaluation Page 10 of 11 GIS formats (specify): Spreadsheet formats (specify types): Other (specify): ! Do the statistical analysis and output routines conform to WMO Guides? Yes No ! Does the system provide the user with the ability to add additional products to the system by providing programming guidelines or a documented Application Programming Interface (API)? Yes NoCDMS Evaluation Page 11 of 11 Data and System Administration ! Are built-in facilities provided for routine backup and restore of the database? Yes No ∀ Can the database be backed up while the system is operational? Yes No ∀ Do the backup and restore facilities support a variety of hardware devices that can be added and configured by the user? Yes No ! Does the system provide facilities to recover the database and database information in the event of a system crash? Yes No ! Does the system provide logging of individual transactions? Yes No ! Does the system provide transaction processing that allows the user to >roll-back= transactions to previous conditions following an update or insertion of data? Yes No ! Are built-in facilities provided for security management and configuration? Yes No ∀ Do facilities exist to provide different database access privileges at the user level? That is, can individual users be granted select, update, insert, and administrative privileges? Yes No ! Can the system performance be optimized at each installed location to account for different hardware, software, and data usage configurations? Yes No ! Are system monitoring tools provided to allow the administrator to monitor the system performance on items such as memory usage, number of transactions performed, status of system logs, and data storage space available? Yes No ! Does the system provide the capability of remote (off-site) administration? Yes No Describe the process that will be developed for fixing user-discovered software >bugs': Describe the process that will be developed to accommodate system software upgrades: