Manuscript ID: JOSTA-202610-DE78
Article Title: Methodology for Evaluating Artificial Intelligence Chatbots: A Review with Applications to Agricultural Advisory Systems
Double-anonymous review Accepted| Item | Details |
|---|---|
| Submission Date | 11 October 2026 |
| Editorial Screening | Completed |
| External Reviewers | 2 |
| Review Rounds | 2 |
| Revision Type | Major Revision |
| Final Decision | Accepted |
| Acceptance Date | 09 November 2026 |
11 October 2026
Manuscript submitted through the JOSTA submission portal.
15 October 2026
Manuscript passed preliminary editorial checks.
17 October 2026
Two independent reviewers accepted the review assignment.
30 October 2026
Reviewers recommended publication following major revision.
06 November 2026
Authors submitted a revised version addressing reviewer concerns.
09 November 2026
Accepted for publication.
16 November 2026
Article published in Volume 2, Issue 4.
Reviewer concerns are summarised by category. Specific comments and author responses are confidential.
| Category | Nature of Concern | Status |
|---|---|---|
| Review Methodology | Fuller description of how the literature was identified and how potentially relevant studies were screened and selected, beyond the databases and general search terms already given | ✓ Addressed |
| Instrument Selection Rationale | Brief explanation added of why the five standardised instruments and the knowledge-testing approach were prioritised over other existing chatbot evaluation frameworks | ✓ Addressed |
| Reliability versus Validity | Clarified that a high Cronbach's alpha indicates a degree of internal consistency but does not establish construct validity or unidimensionality, with a short explanation of the difference between reliability and validity | ✓ Addressed |
| Applicability to Smallholder Contexts | Discussion added on the need for cultural adaptation, language validation, and psychometric evaluation before using instruments developed outside agriculture with smallholder farmers, particularly those with limited digital skills | ✓ Addressed |
| Knowledge Testing Design | Discussion of pre-test and post-test assessment strengthened with the importance of appropriate controls or comparison groups, estimation of effect size, and consideration of testing effects | ✓ Addressed |
| Independent versus Self-Reported Evidence | Distinction made between findings from independent evaluations and results from developers' own evaluations when discussing agricultural advisory chatbots such as Farmer.Chat and AgroLLM | ✓ Addressed |
| Five-Layer Evaluation Framework | More concise explanation of how the five layers would support real field evaluation, the reasoning behind the choice of layers, and whether they are meant to be applied sequentially or concurrently | ✓ Addressed |
| Limitations & Practical Application | Fuller argument on the practical application of the approach, building on the limitations of the review already identified by the authors | ✓ Addressed |
| Tables, Notation & Terminology | Mathematical symbols, reliability coefficients, and abbreviations checked for correct use, with inconsistencies in terminology in Tables 1 and 2 resolved | ✓ Addressed |
| Language & Formatting | Minor language refinements made, including subject–verb agreement and more formal academic wording, together with correction of formatting problems and typographical errors | ✓ Addressed |
Journal of Sustainable Technology in Agriculture (JOSTA) employs a double-anonymous peer review process. The identities of authors and reviewers remain mutually confidential throughout the review and publication process. Reviewer reports and correspondence are not published.
Voluntary reviewer disclosure. JOSTA respects reviewers' autonomy regarding attribution. A reviewer who wishes to be identified in the published article may notify the editorial office at any point before final acceptance. Upon written consent, the reviewer's name and affiliation will be acknowledged in the Publication & Reviewer Details section of the article. Reviewers who do not provide consent remain anonymous and are listed as Anonymous. This policy is strictly voluntary; no reviewer is under any obligation to disclose their identity.
This page provides a transparent summary of the editorial and peer-review process in accordance with JOSTA's open-science commitments. It does not constitute a full disclosure of reviewer comments or author responses.