Background

The number of people with dementia continues to rise worldwide and has become a major social problem, imposing a burden on society in terms of medical and care costs. Therefore, the development of preventive intervention systems for dementia is urgently needed. For an effective preventive intervention, it is considered effective to start the intervention at the preclinical stage of dementia (PCSD), which is the first stage of the disease. However, existing methods for detecting PCSD (PET, MRI, cerebrospinal fluid tests, blood tests, etc.) are all highly invasive and specialized, making it difficult to screen subjects at the PCSD stage during health examinations of older people living in the community.

Therefore, Shimura et al. developed the Color Words Pick-Out Test (CWPT) [1], a neuropsychological test using a questionnaire, as a simple, non-invasive, and universally applicable screening test for PCSD. The CWPT is a neuropsychological test based on the Stroop effect. In the CWPT, subjects are first presented with a narrative text containing colored words, as shown in Figure 1 A. Subjects are asked to read the text while remembering the episodes of the story, and at the same time to write symbols on the colored words (circle the colored letters if they match, cross them if they do not), while identifying the match between the meaning of the word and the color printed on the page. After a certain amount of time, the subject stops identifying the colored letters of the story and answers a question (Figure 1 B) about the episode of the story he/she has memorized, without looking at the text. During scoring, the total score (Index_1) is calculated by multiplying the number of correct responses to the color word discrimination task by the percentage of correct responses to the story episode questions, and is used to determine the subject's level of cognitive functioning.

Fig 1 Sample CWPT stories and questions

A: Sample of story of CWPT

B: Sample of questions

Subsequently, Shimura et al. provided evidence using the Color Kanji Pick-out Test (CKPT) [2,3], a Japanese version of the CWPT, and established diagnostic criteria using large-scale data [4]. Existing neuropsychological tests (e.g., MMSE and CDR) cannot detect minor cognitive decline before the onset of dementia due to ceiling effects. In contrast, the CKPT shows a normal distribution of scores even in subjects at this level and may reflect mild cognitive decline, making it a promising screening test for subjects in early dementia prevention interventions [5] The CKPT also has the advantage that a small number of examiners (one narrator and several observers) can examine many subjects simultaneously. The CKPT also has the advantage that a small number of examiners (one narrator and several observers) can examine a large number of subjects simultaneously. However, it also has the disadvantage of requiring a great deal of time and effort for scoring and data entry after the test.

To overcome the disadvantages of CKPT, we developed Electronic CKPT (eCKPT), an application of CKPT that can be performed on a tablet device. eCKPT can automatically perform everything from test orientation, measurement time control, scoring, and result feedback. The eCKPT can automatically perform test orientation, measurement time control, scoring, and result feedback. Therefore, subjects can take the test individually at their own time and place, and the eCKPT is expected to be widely used throughout Japan. In addition, it is expected that by knowing the evolution of their cognitive status (change in eCKPT score over time), subjects will become interested in rehabilitation for dementia prevention, and this will lead them to take action to prevent dementia. In the future, it is expected to be translated into different languages and disseminated around the world.

Objective

It is not clear whether the newly developed eCKPT measure has the same clinical utility as the paper version of the CKPT. In general, when a new measurement index is developed, it is necessary to demonstrate the reliability and validity of its measures [6]. Therefore, the purpose of this study was to examine the reliability and validity of the newly developed eCKPT measures in healthy adult subjects.

Methods

Forty-three employees working in a nursing home in Japan who did not have any physical or mental disorder that interfered with their work performance were included in the study. Of the subjects, 40 were included in the analysis after excluding one who was color blind and two who had measurement deficits. Subjects were informed in advance, both verbally and in writing, of the nature of the study, and informed consent was obtained. This study was approved by the Institutional Research Ethics Committee (Approval No. 19-57).

Measurements were taken twice. At the first measurement, in addition to the eCKPT, the Mini Mental State Examination (MMSE), an index of general cognitive function, and the Wisconsin Card Sorting Test (WCST), an index of frontal lobe function, were measured. In addition, 24 of the participants were given a second measurement of the eCKPT 6 months after the first measurement.

The eCKPT was measured by launching the application on a tablet device (iPad Pro 12.9 inch, Apple) placed on a stand and listening to an audio narration through headphones (Fig 2 A). Responses were made by touching the screen with a stylus (Fig 2 B).

The subject listens to the voice narration explaining the measurement procedure while looking at the application screen displayed on the tablet device, and then proceeds with the measurement as in (1) through (6) by touching the screen with a stylus in his hand.

Fig 2 Measurement scenery of eCKPT application

A: Tablet screen display

B: Measurement view

(1) Explain that the test consists of two tasks:

"Finding Kanji characters (traditional Japanese characters) that mean colors in the text and judging the correctness of the meaning of the characters and the colors of the description" and "Understanding the content of the story" while reading the text of the story.

(2) Explain how to answer the questions on the test.

Touching a character on the screen with the stylus once will put an "〇" on the character, touching it a second time will put an "×" on the character, and touching it a third time will make the character disappear.

(3) The following explains the method for judging the accuracy of color Kanji characters.

Kanji characters that mean colors are displayed in various colored fonts. If the meaning of the kanji character and the font color match (e.g., the kanji character for "red" is displayed in red font), the answer is "〇”. If there is a mismatch (e.g., the kanji character for "red" is displayed in blue font), the answer is "×".

(4) Take a short practice test to confirm the flow of the test and how to answer the questions.

(5) Read the actual question passage in 2 minutes, and write "〇" and "×" for each colored Kanji character in the passage, and judge whether it is correct or incorrect.

(6) Answer the questions (3 choices) about the content of the actual story sentences.

As the measurement results of the eCKPT, in addition to the number of correct answers for judging the accuracy of colored kanji characters, the percentage of correct answers to questions about the content of the story sentences, and the time required for inspection, an total score was calculated using the following formula as in the CKPT. A higher total score indicates better cognitive function.

Total score = Number of correct answers for judging the accuracy of colored kanji characters × Percent correct on questions about the content of the story sentences

The MMSE is a measure of global cognitive function. It is widely used internationally for efficient assessment of cognitive function and consists of 11 domains, including disorientation and short-term memory [7]. Results are scored on a 30-point scale, with higher scores indicating better cognitive function.

The WCST is a test of concept and sentence conversion deficits and is positioned as a frontal lobe function test [8]. In this study, we used the Keio version of the WCST, which corrects for problems in scoring the original method, using software modified to be administered on a personal computer [9]. Participants were asked to look at patterns on cards displayed on the computer screen and classify them into either color, shape, or number concepts. From the measurement data, the number of correct responses (number), the number of nonretentive errors (number), and the number of retentive errors (number) were tabulated.

Statistical Analysis

In the statistical analysis, the χ-square test, unpaired t-test, and Mann-Whitney's U test were used to compare the two groups based on the MMSE scores; Spearman's rank correlation coefficient was used to analyze the correlation between the eCKPT total score and the WCST (validity study). Intraclass correlation coefficients (ICC) were also obtained from the two eCKPT measures (reliability study). SPSS Statistic Version 24 (IBM Japan, Inc.) was used for all analyses, and the significance level of the analyses was 5%.

Results

The MMSE scores of the subjects ranged from 29 to 30 points, and group comparisons between the 29- and 30-point groups based on MMSE scores revealed significant group differences (p < 0.05) in age, total score on the eCKPT application, number of color Kanji_ correct answers, number of correct answers on the WCST, and perseverative errors (Table 1).

Table 1 Results of Comparison of Basic Attributes and Measures by MMSE Scores

Mean ± Standard Deviation
MMSE: Mini Mental State Examination, eCKPT: Electronic Color Kanji Pick-out Test, WCST: Wisconsin Card Sorting Test

The ICC was calculated from the two eCKPT measurements and ICC = 0.50 (p<0.01) to verify the reliability of the eCKPT (Table 2).

To validate the eCKPT, a correlation analysis was performed with the WCST. The analysis showed that the number of correct answers [ρ=-0.377 (p=0.02)] and the number of perseverative errors [ρ=-0.375 (p=0.02)] on the WCST were significantly correlated with the total score on the eCKPT. On the other hand, there was no significant correlation between the number of non-perseverative errors on the WCST and the total score on the eCKPT [ρ=-0.290 (p=0.07)].

Table 2 Reliability testing of two CKPT application measurements

Mean ± Standard Deviation

eCKPT: Electronic CKPT, ICC: Intraclass Correlation Coefficients

Considerations

Based on the MMSE results, the subjects in this study maintained a high level of overall cognitive function. This may be due to the fact that the subjects were paid employees working in nursing homes, and personnel with appropriate cognitive function required for their duties were selected during the employment screening process. On the other hand, a comparison of MMSE scores between groups (30 vs. 29 points) showed that the total eCKPT score was significantly lower in the 29-point group, despite the slight difference in MMSE scores. These results suggest that eCKPT scores may reflect small cognitive declines that are not detectable on the MMSE due to the ceiling effect [10].

The ICC of the two eCKPT measurements taken over a 6-month period was 0.5. Based on the criteria of the previous studies [11], the reliability of the eCKPT measures is rated as moderate (ICC = 0.41-0.60). The increase in total score from the first to the second session suggests a practice effect. In the future, it will be necessary to provide sufficient explanation and practice prior to scoring to promote understanding of the test method and proficiency in using the device.

A significant correlation was found between the total eCKPT score and the WCST (number of correct answers, number of perseverative errors). It has been reported that poor performance on the WCST is associated with the risk of developing preclinical Alzheimer's disease [12]. Perseverative errors on the WCST are also known to be an indicator of preclinical cognitive decline [13]. Therefore, the significant relationship of the eCKPT with the WCST, which reflects early cognitive impairment, suggests that the eCKPT total score is a valid assessment index for detecting early cognitive decline.

As a future prospect, we are considering clarifying the sensitivity and specificity of PCSD determination by eCKPT by comparing it with the results of diagnosis by a dementia specialist.

Acknowledgment

We would like to thank the staff of Sousei Day Care Centers (Tomitsuka Club, Sanarudai Club, Irino Club, and Yamanote Club), Hamamatsu Institute for Human Sciences, Inc. and Home Care Support Office Megumi, and Mr. Junichi Takase of SoftUp J for their cooperation in this research. This research was supported by Mitsubishi Foundation.