Guidelines for Evaluating Digital Scholarship
MESA Guidelines for Evaluating Digital Scholarship for the purpose of Hiring, Tenure & Promotion
Digital scholarship continues to develop rapidly in the many fields of the humanities and social sciences that govern the study of the MENA region and the Islamic world and their multiplicity of cultures, politics, societies, economies and languages, historically and today. All research is a venture into the unknown, and the greater the unknown, the higher the risk and potential return for scholars, including in evaluations of their work at every phase. Digital scholarship, despite the most careful planning, still has more uncertainty factors than more familiar research methods and products in humanities and social sciences, ranging from the reliance on new and developing software and technical consultations, to collaborations, the need for financial support, and more time-consuming iterative modelling.
Every aspect of academic work is digital today to some degree, including library catalogs, word processing, basic data management, scanned and OCR’ed materials, publication, indexing, and archiving. However, a gap has emerged between scholars for whom these kinds of operations describe the extent of their digital scholarly operating sphere, enormous in itself, and scholars who rely on and experiment with less familiar, more computationally demanding possibilities, for example: text encoding, Geographical Information Systems, relational databases, linked data, and network analysis.
Digital scholarship poses unique challenges for scholars, specifically in their professional relationships to their institutions and to their colleagues. Today, whether or not individuals are engaged personally in digital scholarship, they are almost universally obliged to engage with this scholarship through the work of their colleagues. Where this engagement includes an obligation to evaluate and make recommendations for the purposes of hiring, review, tenure and promotion, difficulties and tensions regularly emerge.
This document offers guidelines for the evaluation of digital scholarship related to the study of the Middle East, and more broadly, the MENA region and/or Islamic world. While individual institutions will craft guidelines for their local evaluation procedures, it is our goal that the MESA guidelines serve as a fundamental point of reference in the evaluation of digital scholarship about the MENA region and/or Islamic world. In the processes of hiring, promotion, and tenure, it is crucial that everyone involved understand the official requirements and institutional expectations that will govern the process. We recommend that students and scholars for whom these guidelines may be immediately relevant read and discuss them with department chairs, deans and any other evaluating authority. At the same time, we believe this document is relevant for our entire community, and recommend it to everyone.
In order to ensure that digital scholarship is fairly and adequately evaluated, these proposed MESA guidelines point to best practices that are being widely adopted. In addition, we propose several considerations that are particularly relevant in scholarship relating to the MENA region and/or Islamic world. Although this document focuses exclusively on issues of digital scholarship, these guidelines also recognize that digital and computational research are methodology choices, not a moral good or the sign of a better scholar or more cutting-edge scholarship. Evaluators at every level must be sensitive to this and not penalize or reward scholars because they are or aren’t digital.
Real concerns exist among practitioners of digital and computational humanities and social sciences about the structures and processes for evaluating their research, publication, teaching, and service records. Meanwhile, other scholars are curious and enthusiastic to explore digital potentials, but are wary of the risks connected with venturing into these lesser- or un-known realms, unsure of how their efforts and products will be perceived and assessed, not to mention whether they will be successful. New forms of publications exist as well as entirely new scholarly products: websites; blogs; interactive texts, maps, or images; podcasts; online courses; specific software; and more. People on both sides of the evaluation equation recognize that there is still a dearth in most institutions of people with sufficient understanding and digital experience who can serve as effective mentors or evaluators in all relevant fields of humanities or social sciences. Real concerns also exist among evaluators about their abilities to distinguish sound digital scholarship, measurable outcomes, and significant contributions to research, teaching and service when compared with more familiar fields and forms of scholarly activity.
As with all research, individuals being assessed have a responsibility to articulate and demonstrate clearly the scientific contribution of their work and its significance. The work should then be evaluated on its scholarly merits and its contribution. Scholars have the right to assume that their work will be assessed by senior colleagues who can validate any scholar’s claims, based on personal experience and fluency with the topic and the methods use in its study. When reviewing digital materials, evaluators must be able to view the material in the full technical environment in which it was intended to be viewed, as specified by the scholar being evaluated. Evaluators – whether institutional colleagues, superiors or external referees – must be qualified to understand digital scholarship, even if at varying levels, and have a degree of familiarity with the nature of digital work. If/when necessary, evaluators should be recruited from outside the specific sub-field of the scholar being evaluated. Digital expertise can itself be considered a sub-field for the purposes of assessment, in conjunction with the relevant expert(s) from history, literature, anthropology, etc.
From the outset of a career, and then at regular intervals, administrators and researchers alike can usefully clarify and calibrate in writing their mutual expectations regarding fields of performance; quantity of activity; time frames for production and modes of evaluation. Expectations should be aligned as to research, publication, teaching and service achievements, and address how digital aspects of each realm of activity may require specific or additional resources, affect timetables, and the character of the resulting products. Perhaps more than has been customary in the past, researchers may find it useful to document the progress of work, including missing or insufficient resources, setbacks and dead-ends. This record serves as evidence of labor, results and costs.
Most digital projects require external funding on a continuing basis, making the digital humanist more like the physical or life scientists with labs elsewhere on campus. External grants funds bring money into a university, sometimes creating additional pressure on more scholars to “go digital.” Many funders also expect a web or social media presence to publicize progress and results. Applications for and management of funds for digital research projects may consume significant amounts of time, adding further challenges to a research agenda.
Digital scholarship is frequently a collaborative effort, which poses additional challenges when being evaluated as academic product. The leading role and contributions of the Principal Investigator can mask the equally important and integral contributions of affiliated scholars, without whose expertise and participation the project would not progress. The multi-faceted aspects of digital scholarship may create demands for short-term skilled project labor from individuals, all of whom must be acknowledged and rewarded. Crowd-sourcing, drawing on the expertise of many people to volunteer small contributions, has yielded very mixed results to date. The disparity may be the result of scale or specificity, but in an academic context it is clearly connected to the question of how to quantify and reward such endeavors. Once the issue of recognition is resolved, it is likely that more scholars and students will be willing to invest their research time in furthering these projects.
Thus, to the extent possible, the roles, expectations, time commitments and benchmarks from each person on the team and the anticipated frameworks for evaluation should be designed before the project begins and at regular intervals thereafter, to ensure effective evaluation processes. At the same time, evaluators in traditionally solo-author disciplines are invited to recognize that collaborations across disciplinary boundaries – which are typical of most digital work – often demand more work of all parties rather than less, as all team members must understand to a reasonable degree the contributions of each to the collective result.
III. Specific MESA Considerations & Needs
We enthusiastically recommend the guidelines proposed by Todd Presner on “How to Evaluate Digital Scholarship” and append his text as an integral part of this proposal. Presner offers a notably concise and lucid statement of principle and practice for academic review committees, chairs, deans and provosts in the Humanities, Social Sciences, Arts and related disciplines. His guidelines address: fundamentals for initial review; crediting; intellectual rigor; crossing research, teaching and service; peer review; impact; approximating equivalencies; development cycles, sustainability, and ethics; and experimentation and risk-taking. In addition to Presner’s article, we append the recommendations composed by the Department of History of the University of Toronto as an example of how Presner’s approach can be usefully adapted to a specific university context.
To the extent that MESA members are affiliated to disciplinary departments in their universities, the associations’ guidelines can be useful. Where this is not the case, the lacuna may create uncertainty (stressful and unhelpful), misunderstanding or even skepticism (harmful) about achievements in digital scholarship. Many MESA members have experienced first-hand the extent to which disciplinary guidelines may be based on norms grounded in scholarship that focuses on Europe, the U.S. or other cultures or parts of the globe which have traditionally attracted greater attention that the MENA region or the Islamic world. Huge compendia of documentary sources and manuscripts for the study of Western history, society and culture were catalogued by hand in the nineteenth and early twentieth centuries. These same resources have been easily converted to digital formats as well, including digitized copies of literary and archival materials openly accessible online. For the MENA region and/or Islamic world, some fields of study can already rely on similar compendia. These include documentary and manuscript collections, reliable text editions, online repositories, scholarly communications and publishing, and podcasts. Yet many other fields have relatively few digitized resources. Around the world, extensive collections of manuscripts in Arabic, Persian, Ottoman Turkish, Urdu, Armenian and other languages, as well as archival documents, material artifacts and images, still remain uncatalogued, that is, unlisted and undescribed. Essentially, they are invisible and so unusable to scholars until reliable catalogs are created with sufficient metadata to make them discoverable by their specific potential audiences.
Such catalogs, however, can only be created by experts who are trained as scholars, possessing the requisite language fluency and critical evaluation skills. The work of identifying and describing these manuscripts requires training similar to that underpinning most humanistic and social science research with the addition of advanced training in paleography, textual analysis, codicology and manuscript production, as well as familiarity with a corpus of existing scholarship. It is crucial to the field that scholarly activities like the creation of catalogs and critical editions be recognized as of comparable scientific importance with the production of scholarly monographs and articles.
Creating sound critical editions in a digital age also expands the potential for analysis with additional tools and formats in environments that present and act on texts in new and powerful ways. At their most basic, these can make long-familiar scholarly modes of inquiry and communication more efficient; at the experimental end of the scale, they make possible innovative manipulations and analyses that expand the dimensions of scholarly endeavor.
Advances in machine-reading technologies offer one example of the specific realms of ongoing digital development. OCR capabilities for Arabic, Persian, Ottoman Turkish, and languages using other alphabet systems are advancing rapidly so that paleography and transliteration will enjoy advanced machine-supported research possibilities. Computational scientists continue to develop technologies for reading these languages, in print and handwritten form. However, the ongoing work of fine-tuning these same technologies, to perfect them for many MENA region and/or Islamic world languages, can only be done in partnerships with the scholars who know the original materials and the languages. Computational scholars from computing sciences focus on solving problems defined as such in their own fields; their research does not include the application of their solutions and their refinement to the level of accuracy required by the end-user humanists or social scientists. This creates gaps which ultimately have to be filled by computationally trained humanists or social scientists, without whose input the computational success remains functionally inaccessible.
The development of digital scholarship and its capacities may have benefits especially valuable in the many conflict-ridden areas of the MESA region and/or the Islamic world. Digital solutions can overcome some of the many barriers to scholarship erected by political, security and economic realities, and so sustain the establishment of scholarly communication, cooperation and preservation. They allow for the sharing of ideas and data, while enabling scholarly exchanges, teaching, and outreach otherwise unimaginable due to distance, lack of funds, and other barriers. Further, digital solutions are crucial for heritage rescue and management, a very real concern in the early 21st century. Not only is it possible to alert a global community to threats to people, buildings, archives, libraries and material goods, but their rescue can sometimes be effected through the creative use of digital resources. Cultural artifacts, monuments, images, sound recordings, and scans of all kinds of texts can be collected and curated digitally to preserve and make available a sustainable record of human existence and achievement, to be remembered, studied and celebrated in response to attempts to destroy them. One telling example of how crucial such a project can be is the digitization of the Timbuktu manuscripts in their entirety by the Hill Museum and Manuscript Library of Saint John’s University (MN), which also provided open access to the material online.
 This geographical and topical description of the span of our field seems to reflect more closely the real scope of scholarship represented in IJMES and at MESA meetings at present.
 Todd Presner, “How to Evaluate Digital Scholarship,” Journal of Digital Humanities 1/4 (Fall 2012). This article appeared in a special issue of the Journal of Digital Humanities entitled “Closing the Evaluation Gap,” all of which is relevant to the subject of this report.
 For example: American Historical Association, Modern Language Association, College Art Association. Numerous links to guidelines from individual institutions may be found in the very useful article: Kristine M. Bartanen, "Digital Scholarship and the Tenure and Promotion Process," http://www.academiccommons.org/2014/07/24/digital-scholarship-and-the-tenure-and-promotion-process/. (all links last accessed 14 August 2017)
http://www.hmml.org/the-timbuktu-manuscripts.html (accessed 13 Aug. 2017)