BNHM Collection Management System Evaluation, Spring–Summer, 2009
Combined Initial Scoring, second draft 10/6/2009SpecifyArctosCollectionSpace
Specify (1.0=fully)Arctos (1.0=fully)CollectionSpace (1.0=fully)Would this system be able to handle your institution's needs with respect to the following criteria?OVERALL WEIGHT 1-Low 5-HighI think this is out of scopeOur institution doesn't care about thisYes, certainlyI think so, but I'd want confirmationI think so, but I'd want it demonstratedI don't think so, but have the developers confirmNot in the way(s) we need/wantNoExplain if you answered "no"EssigMVZPAHMAUCMPHerbariaBot Garden#NotesTotal PointsPossible PointsOverall57Total PointsPossible PointsOverall60Total PointsPossible PointsOverall70
40%40%40%FUNCTIONAL CRITERIA234.0533270252.634673200.835057
System currently used in …
110Entomology collection5555505
110Vertebrate Zoology (mammals, fish, birds, herps)5555505
000.75Anthropology and Archaeology505053.755
000Botanical Gardens5050505
Core collections management processes00
110.75Object Entry (the management and documentation of the receipt of objects/specimens lots that are not currently part of the collection)51C: Extend existing schema for domain55553.755
110.75Acquisition (documenting and managing the addition of objects/specimen lots to the permanent collections of the institution; includes accession)52C: Extend existing schema for domain55553.755
110.75Inventory Control (the maintenance of up-to-date information identifying all objects/specimen lots for which the institution has legal responsibility, including objects/specimen lots on loan, unaccessioned, or previously undocumented items and enquiries)5355553.755
110.75Location & Movement Control (the documentation and management of information concerning the current and past locations of all objects/specimen lots in the institution's care to ensure that the museum can locate any object at any time)5455553.755
110.75Location: Museum storage5555553.755
0.750.750.25Barcoding & RFID tagging46343414
110.75Cataloguing (the compilation and maintenance of primary information describing, formally identifying, or otherwise relating to objects/specimen lots in the collection; includes object history/provenance)5755553.755
000Event-based cataloging support48040404
110.5Bulk entry capability59S: Workbench.55552.55 Management (the documentation and management of information about the conservation of objects from a curatorial and collections management perspective)45103.243.2424
000Propagation and care of live species4511C: Modeling service soon.040404
0.750.75Rights and Reproductions (documenting and managing the preparation of images, casts, and models)412A: Casts and models treated as parts03434
0.75Risk Management (the management and documentation of information relating to potential threats to an institution's collections and the objects for which it is temporarily responsible. Includespreventative measures, disaster planning)4130034
0.500.75Insurance Management & Valuation Control (documenting and managing the insurance needs of objects and valuation control -- the management of information relating to the valuations placed on individual objects, or groups of objects)414S: Appraisals included240434 Management (the management and documentation of temporary exhibitions and permanent displays, including the processes of developing, co-ordinating, and implementing an exhibition and display program)415A: Exhibit treated as internal loan0.842424
110.75Dispatch (the management and documentation of objects/specimen lots leaving the institutions premises) distinct from loans and deaccession51655553.755
110.75Loans (managing and documenting the borrowing of objects for which the institution is responsible for a specific period of time and for a specified purpose, such as display, research, education, or photography)51755553.755
110.75Loans: Client/Partner Requests518A: collection-specific messaging to curator.55553.755 Balance of trade tracking419A: Could do ad hoc report0.840.840.84
110.75Deaccession & Disposal (transfer, sale, exchange, or destruction of objects and of deaccession (documenting the disposal))52055553.755
110.75Collecting events521C: Extend existing schema for domain55553.755
110.75Collection location management523C: Extend existing schema for domain55553.755
110.75Geographic coordinates52455553.755
110.75Stratigraphic45525C: Extend existing schema for domain444434
000Sensor data and environmental monitoring426040404
110.75Lab measurements52855553.755
110.75ID, annotation history52955553.755
10.50.25Genetics455531S: in schema. A: integrates with Genbank. C: Link to Genbank or file442414
110.75GUID support532S: UUID. C: Flexible IDs55553.755
000Integration with lab systems (LIMS) and/or field systems (FIMS)455533040404
Taxonomy and Semantics000
000.25Ontology maintenance, semantic indexing4536040414
Data Sharing000
110.75Data export in specific formats537C: Flexible template-driven export55553.755
110.75Data import from necessary formats538A: Templates, headers. C: Flexible template-driven import55553.755
00.750.75Data accessible via APIs and web services539A: DiGIR, ColdFusion API's053.7553.755
0.750.750.75Media files of different type can be associated with objects and sets (image, audio, video)540A: various formats; zoom via browser only.3.7553.7553.755
000.5Advanced media file functionality (e.g., image zoom, time-indexed video/audio)441040424, copyright management442A: Copyright message on media.242434
0.750.50.75Document management including field notebooks and correspondence5433.7552.553.755
Temporal Data Support000 support for the temporal placement of specimens (geological eras and years; administrative boundary changes)444A: Geological eras242424
Administration, Other46000
110.75Audit Reports (audit reports on data, user access profiles, deletion of records, etc.)54755553.755
000Membership & donations (money & objects)349030303
000Outreach, public services, tours350030303
000Calendar and scheduling351030303
000Business resumption/disaster recovery552050505
000Resource scheduling (include facilities rentals)354030303
User Interface000
0.750.750.75Help Features (menu, command and icon driven, on-line help, tutorial, etc.)5553.7553.7553.755
0.750.750.75Date Formats (date formats for entry, searching, display and output)5563.7553.7553.755
0.750.50.75User Customization (customization of screens)457S: Significant user prefs342434
0.7500.5Multilingualism (system function in an alternate language)458S: Built for other languages, but none added yet. C: Architecture supports but not planned.340424
0.90.9Character Codes/Languages (Latin 1, UNICODE, etc.)559A: Unicode. C: Unicode04.554.55
0.2510.75Public Access (public access via the Internet, searching, displaying, printing, etc.)560S: ITP in process. A: UI is expert driven1.255553.755
0.50.75Multimedia Files (audio/video, 3-D imaging, zoom, image files, management of digital files, etc.)561A: display images, links to other files02.553.755
110.75Metadata (standards for metadata (NISO, MPEG-7, DIG35), import and exporting (EXIF), etc.)562C: Standards yes, but 55553.755
110.75General Requirements (query language, boolean searching)56355553.755
110.75Range Searches (range searching on numeric fields, attribution dates and date fields)56455553.755
110.75Wildcard Searching (character substitution for wildcard searching)56555553.755
110.75Query Results (query results browsing, sorting results, etc.)56655553.755 (phonetic and proximity searching, narrowing the search scope, etc.)567A: Search within search2.552.553.755
110.75Pre-defined Reports (report layout formats, reports that come with a collections management system)56855553.755
110.75User Defined Reports (allows the user to define reports, labels, generate forms, sorting, calculating, printing and saving reports)569S: iReport3. A: ColdFusion55553.755
Data Management000
0.510.5Batch Update and Data Correction570A: Significant capabilities in user-driven batch. S: via Workbench only?2.55552.55
Collections Publishing and Public Access000
010.75Collections Publishing Module. The system should allow the user to publish collections information contained in the system on the web, either for public or password-protected use.57105553.755
00.40.5Community Cataloging and Curating. (a) a user registration and accounts; (b) tagging (keyword-based); (c) custom collections/playlists; (d) annotate at the item or list level; (e) assignment of rights to add to, alter, or view the contents of user-created lists/collections.4572A: User can message curator about ID, etc.041.6424
30%30%30%BUSINESS CRITERIA19.801253852.1085526320.687916673854.4418859624.496607143472.04884454
0.460.630.69Financial support and sustainability52.2812553.1562553.468755 has a multi-year funding model that is comprehensive and sustainable, that considers a diverse community of partners, and strategy for ongoing funding.73 on that plan, what resources/financial commitments are actually in place for the next 2 years to cover • Operations of the community, e.g. governance, engagement, communications • Support of the product • Development of the product • Further deployments of the solution77
0.40.750.7Hardware, software, and staffing requirements and costs (TCO) for deployments are known, can be modeled and are acceptable.74
0.40.750.7Hardware, software, and staffing requirements and costs (TCO) for ongoing maintenance are known, can be modeled and are acceptable.75
00.50.7Does the supplier/community have a clear sustainability plan?76
0.750.750.75Are there other funded projects that cooperate or are complimentary to this one?76
0.750.750.75Is there a plan for funding solutions built on this platform to enhance research?77
0.750.750.75How fundable is future development of the application, or deployments? • Also speaks to how widely attractive the solution is across Museums supporting different disciplines78
0.430.740.90Governance and Leadership5S: Need to look closer2.16666666753.67554.55
0.50.750.9Project has clear sponsorship and a working governance model. 79
0.50.750.9Project has clear process and criteria for decision-making and priority-setting.80 UCB have significant influence in the direction of the project?81
0.50.750.9Several institutions have proven that they will provide leadership in the ongoing development of the software.82
0.40.660.9Is there a clear process by which the community can influence the direction of the project, rather than the vendor model?83 project has a roadmap for ongoing development and priorities. (including function and sustainability)84
0.570.710.79Solution, Provider and Community Maturity42.27333333342.82666666743.1428571434
0.7510.5How long has the project/community been in existence85
0.7510.25System has been in use in production environments and has a good track record. (the more, the better)86
0.811Solution is being actively developed and/or updated, and bugs are being fixed.87
0.20.331Development is occurring at multiple sites, and there is sufficient open and transparent communication and coordination to enable reuse and deployment planning.88
0.660.661Project has a stated process/infrastructure for source code control, using a professional development/design methodogy.89
1Project has a stated QA process and uses tools for automated testing.90 supporting the project is robust; with no single point of failure.91
Have there been quality, reliability or availability issues or security concerns with the project over the past year???92
0.500.560.63Provider, Service and Support 4242.2542.5254
0.80.750.75Institution or other entity has stepped up to provide services such as training, support, and consulting.93S: help desk
00.250.25Consulting and support services are available from a 3rd party? • Is there a plan to do so? (speaks to scalability, future adoption levels/market share/sustainability)94A: AlasConnect? C: Planned / resources for deployment, extension is available from the community or a 3rd party? • Is there a plan to do so? (speaks to scalability, future adoption levels/market share/sustainability)95 established transparent process with supporting infrastructure is available for registering, prioritizing and tracking support requests. 96 release cycles with adequate communication.97
0.750.50.5Documentation for users and programmers is thorough and current.98
0.750.750.75Quality of relationship with and between supplier and the community99
0.20.750.8Responsiveness of supplier and community to UCB needs99
??Metric: • How long to get and provision a deployment (e.g. a new collection) from initial point of interest?100 Hosted Solutions5151.552.55
For any SaaS (externally hosted offerings), service levels must be transparent clearly understood throughout all levels of the service chain in terms of • Availability/performance • data ownership and end-of-term transfer • intellectual property • disaster recovery • security • pricing assurances • penalties (e.g. what incentives are in place to enforce a service level agreement) • etc., etc.101 packaging42.840.843.24
Installation and configuration of software is documented and straightforward. 102
0.650.490.75Supported platforms42.641.98434
0.80.660.75Software can run on required platforms • Server-side • Client103
0.50.330.75Platforms are aligned with established architecture/ infrastructure roadmap 104Not HTTP
0.750.750.9Licensing & Copyright 43434
111Open source license allows for flexible development, but community has committed to avoiding forking of the source code.105GNU GPL is clear where rights are held, and rights are held in an entity that the community trusts. 106
???Is there some form of legal recourse that needs to be available / considered regarding the quality/function of the solution?107
0.560.50.72Skills and staffing31.6831.532.163 for the solution set (including application and underlying technologies) are readily available in the market 108 rampup is easily achievable for core underlying technologies (languages, modeling tools, middleware, databases, web servers, unique infrastructure, etc.)109 of effort / investment necessary to train core support staff110Client install of effort / investment necessary to train core museum staff110Expert database is what they expect of effort / investment necessary to train developer/researcher staff111Expert database is what they expect
30%30%30%TECHNICAL CRITERIA35.787945.29113924438848.8636363674.058884.14772727 scale horizontally and vertically5151545
System must scale easily, and horizontally versus just vertically. This means it should not require a bigger computer to handle a large load, but that components can be separated onto different machines. It also means that the application must be written in a manner that can reasonably accommodate changes in the underlying technology stack (e.g., changes to database, content repository, web server, etc.).112S: Client-server model limitations Multi-tenancy5252.5545
Allows the addition of new collections and customers without the overhead of separate duplicated instances of the core application and underlying platform. Dimensions of multi-tenancy would include UI, security, data model, vocabulary, etc. Refer to customizablity issues below. 113S: Useful organization model but not true multi-tenancy easily customizable and extensible5y3.551545
For each deployed collection - the data model, associated system software and user interface (UI) for the system must easily accommodate local customization, including but not limited to: • Addition of fields/properties on core entities • Deletion or hiding of existing fields/properties on core entities • Custom validation for all fields/properties • Setting local, custom labels for all fields/properties • Customizing the general look and feel of any web-based elements. • Customizing the layout and pageflow/workflow for common operations • Vocabularies, with multiple variations per tenant/collection114S: Schema Configuration allows config at the discipline level (e.g., hide tables, fields; make required fields; attach fields to user-defined pick lists) reusability30.630.632.43
Core parts of the platform used (e.g. repositories for digital assets, workflow engines, etc.) can be leveraged for other uses and customers. (Thereby providing greater/wider opportunity for cost recovery).115 for SaaS (Software as a Service)40.840.843.24
System must support lightweight deployments, ideally with multiple clients on a given installation, without compromising the security policies or deployment-specific customizations of any of the individual clients. This is often known as support for Software as a Service, or SaaS.116
0.60.450.9Uses common technology stack5352.2554.55 must use common and broadly deployed technology stack, libraries, languages, and architecture & design principles. This means: no obscure databases, operating systems, programming languages, etc. 117 also means adhering to accepted practices of data modeling (e.g., normalized, relational forms), and of application design (e.g., the Model-View-Controller paradigm, and the separation of content and presentation). This is important both for managing deployment costs, but as well to ensure reasonable software maintenance costs.118
0.850.40.925Independent of Proprietary Systems (and Open Source)43.441.643.74
10.21System should not depend upon costly, proprietary, licenses. This does not mean that it *must* be open source, so long as the required components do not have high-cost licenses. It does mean that the architecture should avoid dependence on specific proprietary features of a given system (like platform specific libraries, stored-procedure mechanisms, etc.).119
111Ideally, the application itself should be open source, although this may not be an absolute requirement. If the system is not open-source, additional consideration must be given to mitigate the risks of the vendor going out of business, or changing license terms to make the deployment much less cost-effective. Allow for these tradeoffs with a SaaS provider119
10.21Open source principles should be applied also to the underlying technology stack, as much as is practical and possible.120 for vendor or technology lock-in?121 agnostic52.552.554.55
System must run well (and equivalently) across a range of widely deployed client platforms. At a minimum, it must run well on MSFT Windows XP and later, and on Mac OSX and later. 122
00.80.9Web-based clients505454.55
Ideally, client should access the applciation with a web-client and adhere to general web portability and accessibility standards. The web-client should operate on broadly accepted web browsers.123
0.670.50.8Is well documented4y2.684243.24
0.750.50.8Any application must be well documented, from the architecture, to system requirements, to end-user documentation and training, to the actual software.124 of end-user documentation 125 of developer documentation 126 of operations documentation 127 of architectural/system documentation 128
Supplier vs. community contributions - both need to be evident129 permissions-based access controls54.554.554.55
The system must support permissions and role-based access controls on data and operations in the system. A role/permission model is preferred that is coupled with flexible authentication and authorization tools. Associated admin tools must allow for definition of local groups/roles, and assignment of privileges/ permissions of different areas of the system (down to the field level).130 CAS & external user authentication40.44143.24
The system should integrate with the existing UCB Single-Sign-On authentication system (CAS) deployed here on campus for CalNet authentication, but should support access from external users as well.131
00.750.8Has flexible, well documented APIs/ SPIs5053.75545
The system should expose flexible Application- and/or Service-Programming Interfaces to facilitate extension, customization and integration with complementary systems. Ideally exposed with light-weight API's such as REST. These must be well documented. 132 enforcement3S: "Interactions". A: e.g., student data entry has to be approved.0.931.532.253
?The system should support the grouping of multiple procedures into workflows, fully customizable by the user, that can accommodate restrictions, limits, and ticklers. These can be used, for instance, to require that an accession number be assigned before an object name is given, or that out-loans cannot be finalized without the approval of a curator or qualified staff member.133
111Complete data revision history5555555
Audit trails exist that document data changes, times and by whom. E.g. Security assignments and changes.134
0.250.251Usability is core to design principles.135
0.250.251Accessibility is core to design principles. E.g. meets ADA guidelines.136
Can features be turned on or off based on configuration? • Ideally at runtime rather than compile time137
How well architected is the solution to interoperate with other systems and tools?138
Supports well known and adopted standards as part of key design principles. E.g. OAI-PMH.139
0.30.5Security5A: reviewed by team.01.552.55
Have any security audits been applied to the solution (application with underlying infrastructure). Can this be demonstrated?140