Peer Review Record

Manuscript ID: JOSTA-202610-DE78

Article Title: Methodology for Evaluating Artificial Intelligence Chatbots: A Review with Applications to Agricultural Advisory Systems

Double-anonymous review Accepted

Review Summary

ItemDetails
Submission Date11 October 2026
Editorial ScreeningCompleted
External Reviewers2
Review Rounds2
Revision TypeMajor Revision
Final DecisionAccepted
Acceptance Date09 November 2026

Editorial Assessment

The manuscript is a narrative review of methodological approaches for evaluating AI-based chatbots, with particular reference to agricultural advisory systems. Reviewers considered it timely and relevant, noting that it brings together several established evaluation instruments in one place, offers a useful methodological perspective on reliability and knowledge testing, recognises that usability and satisfaction alone cannot establish chatbot reliability, and proposes a five-layer evaluation protocol that could serve as a useful conceptual framework. The manuscript was found to be generally well written and concise. Reviewers nonetheless requested revisions to strengthen its theoretical rigour and presentation: more detail on how the literature was identified, screened, and selected; justification for prioritising particular instruments over other chatbot evaluation frameworks; clearer distinction between reliability and validity when discussing Cronbach's alpha; attention to cultural adaptation, language validation, and psychometric evaluation when applying tools developed outside agriculture to smallholder farmers; fuller treatment of controls, effect sizes, and testing effects in pre-test and post-test designs; a distinction between independent and self-reported evaluations of agricultural advisory chatbots; and clearer guidance on how the five-layer framework would be implemented in the field. Reviewers also asked for corrections to notation, terminology, and formatting in the tables, and for minor language refinements. The authors satisfactorily addressed all comments, after which the manuscript was accepted for publication.

Review Timeline

Submission Received

11 October 2026

Manuscript submitted through the JOSTA submission portal.

Editorial Screening Completed

15 October 2026

Manuscript passed preliminary editorial checks.

Peer Review Initiated

17 October 2026

Two independent reviewers accepted the review assignment.

Reviewer Reports Received

30 October 2026

Reviewers recommended publication following major revision.

Revised Manuscript Received

06 November 2026

Authors submitted a revised version addressing reviewer concerns.

Final Decision

09 November 2026

Accepted for publication.

Published Online

16 November 2026

Article published in Volume 2, Issue 4.

Key Issues Addressed During Revision

Reviewer concerns are summarised by category. Specific comments and author responses are confidential.

CategoryNature of ConcernStatus
Review Methodology Fuller description of how the literature was identified and how potentially relevant studies were screened and selected, beyond the databases and general search terms already given ✓ Addressed
Instrument Selection Rationale Brief explanation added of why the five standardised instruments and the knowledge-testing approach were prioritised over other existing chatbot evaluation frameworks ✓ Addressed
Reliability versus Validity Clarified that a high Cronbach's alpha indicates a degree of internal consistency but does not establish construct validity or unidimensionality, with a short explanation of the difference between reliability and validity ✓ Addressed
Applicability to Smallholder Contexts Discussion added on the need for cultural adaptation, language validation, and psychometric evaluation before using instruments developed outside agriculture with smallholder farmers, particularly those with limited digital skills ✓ Addressed
Knowledge Testing Design Discussion of pre-test and post-test assessment strengthened with the importance of appropriate controls or comparison groups, estimation of effect size, and consideration of testing effects ✓ Addressed
Independent versus Self-Reported Evidence Distinction made between findings from independent evaluations and results from developers' own evaluations when discussing agricultural advisory chatbots such as Farmer.Chat and AgroLLM ✓ Addressed
Five-Layer Evaluation Framework More concise explanation of how the five layers would support real field evaluation, the reasoning behind the choice of layers, and whether they are meant to be applied sequentially or concurrently ✓ Addressed
Limitations & Practical Application Fuller argument on the practical application of the approach, building on the limitations of the review already identified by the authors ✓ Addressed
Tables, Notation & Terminology Mathematical symbols, reliability coefficients, and abbreviations checked for correct use, with inconsistencies in terminology in Tables 1 and 2 resolved ✓ Addressed
Language & Formatting Minor language refinements made, including subject–verb agreement and more formal academic wording, together with correction of formatting problems and typographical errors ✓ Addressed

Peer Review Statement

Journal of Sustainable Technology in Agriculture (JOSTA) employs a double-anonymous peer review process. The identities of authors and reviewers remain mutually confidential throughout the review and publication process. Reviewer reports and correspondence are not published.

Voluntary reviewer disclosure. JOSTA respects reviewers' autonomy regarding attribution. A reviewer who wishes to be identified in the published article may notify the editorial office at any point before final acceptance. Upon written consent, the reviewer's name and affiliation will be acknowledged in the Publication & Reviewer Details section of the article. Reviewers who do not provide consent remain anonymous and are listed as Anonymous. This policy is strictly voluntary; no reviewer is under any obligation to disclose their identity.

This page provides a transparent summary of the editorial and peer-review process in accordance with JOSTA's open-science commitments. It does not constitute a full disclosure of reviewer comments or author responses.