Oct 22, 2024

September 18, 2023 . Outcomes of Providing Children Aged 7-12 Years With Access to Evidence-Based Anxiety Treatment Via a Standalone Digital Intervention Using Immersive Gaming Technology: Real-World Evaluation Outcomes of Providing Children Aged 7-12 Years With Access to Evidence-Based Anxiety Treatment Via a Standalone Digital Intervention Using Immersive Gaming Technology: Real-World Evaluation Authors of this article: 2Norfolk and Suffolk NHS Foundation Trust, , Norwich, , United Kingdom 3Eastern Academic Health Sciences Network, , Cambridge, , United Kingdom Corresponding Author: Abstract Background: Anxiety disorders are among the most common mental health conditions in childhood, but most children with anxiety disorders do not access evidence-based interventions. The delivery of therapeutic interventions via digital technologies has been proposed to significantly increase timely access to evidence-based treatment. Lumi Nova (BfB Labs Limited) is a digital therapeutic intervention designed to deliver evidence-based anxiety treatment for those aged 7‐12 years through a mobile app incorporating immersive gaming technology. Objective: We aimed to evaluate the real-world impact of providing access to Lumi Nova through UK National Health Service–funded mental health services. Methods: We analyzed precollected anonymized data routinely captured through the implementation of Lumi Nova from children aged 7‐12 years, who lived in the United Kingdom and had the opportunity to use the intervention for at least 1 week over an 18-month period. Engagement indices included whether the game key was activated, number of unique sessions, time spent engaging, and number of “challenges” completed. Clinical outcomes were assessed using the Goal-Based Outcomes measure and Child Outcome Rating Scale. Demographic data were analyzed to assess the health equality implications of Lumi Nova. Results: Of 1029 eligible families invited to use Lumi Nova, 644 (62.5%) activated their game key, of whom 374 (58.1%) completed at least one in-game graded exposure challenge. The median number of unique sessions was 6 (IQR 3‐12) and the median time spent engaging with the intervention was 42 (IQR 15‐79) minutes. For the subset of young people with paired outcomes, there were statistically significant small to medium improvements in goal-based outcome scores (n=224; t223=5.78, P<.001; d=0.37, 95% CI 0.25‐0.52) and Child Outcome Rating Scale scores (n=123; t122=5.10, P<.001; d=0.46, 95% CI 0.27‐0.65) between the first and last data points. Two in 5 young people’s scores reflected a change that would be considered reliable. Analysis of demographic characteristics tentatively suggested that children from ethnic minority backgrounds and those living in the most deprived neighbourhoods may be less likely to access Lumi Nova, but children from socioeconomically deprived areas were more likely to successfully complete a challenge once they accessed the intervention (P=.02). However, the level of missing data and small number of children in some demographic groups limited meaningful statistical comparisons. Conclusions: This study provides initial evidence that Lumi Nova may be associated with improved outcomes for those aged 7‐12 years seeking anxiety treatment in real-world settings. However, the lack of a control comparator group and information about concurrent treatments accessed by the young people, in addition to substantial attrition, limited the analysis that could be conducted and confidence in the conclusions drawn. JMIR Ment Health 2024;11:e52866 Outcomes Paired GBO data was available for 224 cases and paired CORS data for 123 cases. There was a statistically significant increase in GBO scores between the first and last available data point (t223=5.78, P<.001). The size of the effect was in the small to medium range (d=0.37, 95% CI 0.25‐0.52), indicating that, on average, young people who engaged with the game made progress toward their selected goal. At the individual level, 39.3% (n=88) of cases saw a reliable improvement in GBO scores, 14.3% (n=32) a reliable deterioration, and 46.4% (n=104) had no reliable change. There was also a statistically significant increase in CORS scores between the first and last available data point (t122=5.10, P<.001). The size of the effect was in the small to medium range (d=0.46, 95% CI 0.27‐0.65), indicating that parents or caregivers of children who engaged with the game reported improvements in functioning. At the individual level, 37.34% (n=46) of cases saw a reliable improvement in CORS score, 9.8% (n=12) a reliable deterioration, and 52.8% (n=65) had no reliable change. Paired anxiety ratings pre and post at least one challenge were available for 363 cases. The average anxiety rating prechallenge was statistically significantly higher than the average postchallenge rating (t361=8.35, P<.001). The size of the effect was again in the small to medium range (d=0.44, 95% CI 0.33‐0.55). There was no association between the size of the average discrepancy between pre and post challenge anxiety ratings and change in GBO or CORS scores. No associations were evident between the number of challenges completed or time spent engaging with the intervention and change score for either the GBO, CORS, or average anxiety ratings pre and post challenges. Equality of Access, Engagement, and Outcomes Of those young people whose parent or caregiver activated their game key, 51.79% (492/1029) identified as female and 48.21% (458/1029) as male. Females were significantly more likely to complete at least one challenge than males (n=200, 61.9% vs n=158, 53.5%, P=.02). There were no other significant differences between males and females in markers of engagement or outcomes. More young people whose parent or caregiver activated their game key identified as White (480/644, 91.42%) than the percentage of those aged 7‐12 years who identify as White in the general population of England and Wales (n =3,174,513, 73.5%). Conversely, children of Asian and Black ethnicities were underrepresented among Lumi Nova users compared to the population of those aged 7‐12 years in England and Wales (n=12, Asian 1.7% vs n=486,124, 11.2% and Black n=17, 1.3% vs n=237,872, 5.5% respectively). It was not possible to conduct meaningful statistical analyses of any differences in engagement or outcomes between ethnic groups due to the small number of Lumi Nova users in the sample who identified as belonging to each group. Further, it should be noted that, due to commissioning arrangements, Lumi Nova was only available in certain areas which may not have reflected the demographics of the country as a whole, and often only to those already under the care of mental health services. Compared to the general population of those aged 7‐12 years, Lumi Nova users were more likely to live in the least deprived areas of the United Kingdom and less likely to live in the most deprived areas. However, young people living in the most deprived areas were significantly more likely than those living in more affluent areas to complete at least one challenge (n=87, 66.4% vs n=125, 53.8%; OR 1.68, 95% CI 1.08-2.64, P=.02). There were no significant differences in the outcomes of young people living in the most deprived areas compared to the most affluent areas. Of those young people whose parent or caregiver activated their game key, 11.8% (n=121) reported a disability, a higher percentage with a disability than in the general population of those aged 7‐12 years (n=190,287-445,469, 4.4%‐10.3%). However, young people with disabilities were less likely to successfully complete at least one challenge than nondisabled users (n=32, 42.7% vs n=326, 59.6%, OR 0.5, 95% CI 0.3-0.81, P=.005). The number of young people with a disability for whom paired outcome data was available was small (GBOs, n=20; CORS, n=11). However, within this small cohort of users, improvements in GBO and CORS scores were smaller than for nondisabled young people. The nature of reported disabilities (ie, physical, learning, or neurodevelopmental) was not recorded. Discussion Principal Findings There is a pressing need to improve young people’s access to evidence-based mental health treatments. To meet this need in the context of growing demand for mental health support and a limited pool of qualified practitioners, innovative approaches are required, including improved use of digital technologies. Lumi Nova, a digital therapeutic intervention which uses immersive gaming technology to deliver an evidence-informed anxiety intervention for those aged 7‐12 years, is an example of an innovation with the potential to facilitate increased access to timely treatment. We conducted a real-world evaluation of Lumi Nova, to better understand user engagement and outcomes, and to explore whether these might differ across demographic groups. We found that the majority (n=644, 62.58%) of families invited to access Lumi Nova activated their game key, of whom 58.1% (n=358) completed at least one challenge. However, only a relatively small proportion of users went on to complete multiple challenges, resulting in a median number of challenges completed of 1 (IQR 0‐3). The median number of unique sessions was 6 (IQR 3‐12), but these sessions were typically short relative to the usual duration of face-to-face therapy sessions, with the median total time spent engaging with the intervention being 42 (IQR 15‐79) minutes. For the subset of young people for whom paired outcome measures were available, there were statistically significant small to medium improvements in perceived progress toward goals and parent-reported functioning between the first and last data points. Two in 5 young people reported an improvement that would be considered reliable (ie, the magnitude of the change is such that it is unlikely to be accounted for by measurement error) for both the GBOs and CORS. Available paired anxiety ratings suggest that young people’s anticipated anxiety prechallenge was, on average, higher than the anxiety they reported post challenge, providing some support that the mechanisms of graded exposure were operating as would be anticipated. However, there was no evidence of an association between either the extent of mismatch in anxiety expectation or the child’s level of engagement and the outcomes observed. Interpretation of these findings is complicated by the small proportion of the overall sample for whom paired data was available, the uncontrolled design and limited available comparison data from studies of interventions delivered in real-world treatment settings. However, analysis of routinely collected data from the Child Outcomes Research Consortium [ 30 ] suggests that the outcomes observed for young people accessing Lumi Nova are broadly in line with those observed in young people receiving routine treatment for child and adolescent mental health services. Given the relatively brief duration of the intervention received, that we observed outcomes of a comparable magnitude to those seen in routine practice could be interpreted as preliminary evidence that Lumi Nova may offer a particularly efficient treatment option. However, we are unable to rule out the possibility that the improvements observed may simply reflect natural recovery or the effect of other support received alongside Lumi Nova. Further, as paired outcome data was only available for a subset of those who activated the intervention, we cannot be sure that these findings would generalize to all Lumi Nova users, let alone the wider population of children seeking treatment for anxiety. Comparing the demographic characteristics of Lumi Nova users to the general population of those aged 7‐12 years in England and Wales suggested that children from ethnic minority backgrounds and those living in the most deprived neighbourhoods may be less likely to access Lumi Nova than other young people. However, once users from more deprived areas did access Lumi Nova, they were more likely than affluent users to complete at least one challenge. Given that ethnic minority and economically disadvantaged young people are more likely to experience mental health difficulties than their peers [ 12 , 13 ], this is an important indicative finding that requires further investigation. However, it should be borne in mind that, due to commissioning arrangements, Lumi Nova was only available in certain mental health services around the United Kingdom during the implementation period studied. These services were located in diverse areas of the United Kingdom including the East of England, London, Wales, Essex, Yorkshire, Bristol, Gloucestershire, and Kent, but only within certain organisations and specific NHS mental health teams. As such, the demographic comparative work relating to deprivation and ethnicity undertaken in this paper has used UK-wide data to reflect this diverse geography, in lieu of the absence of population data in these specific areas for those aged 7‐12 years. Furthermore, since the option for families to access the intervention via self-sign-up was implemented toward the end of the period studied, and only in certain regions, in most cases, only those already in contact with mental health services would have been offered Lumi Nova, yet no quality mental health service access data on a specific age basis is publicly available which met the needs of this study analysis. It is possible that further mitigations are needed to overcome barriers to help-seeking among marginalized groups. For instance, expansion and further promotion of the self-sign-up option may help to overcome structural and attitudinal barriers to accessing traditional mental health services [ 11 ]. When examining differences in outcomes based on user characteristics, we found some evidence that females and those from less affluent areas may be more likely than other users to complete at least one challenge, but, overall, we identified few differences between the engagement and outcomes of Lumi Nova users across demographic groups. However, the small numbers of young people in the sample who identified as belonging to some minority groups and the high volume of missing data limited the statistical comparisons that could meaningfully be made. There were preliminary indications that young people with a disability may be less likely to successfully engage with the intervention than children without a disability. The small size of those subgroups involved, alongside the limited available information about the nature of disabilities reported, precludes any firm conclusions from being drawn from this finding. However, further research investigating the accessibility of digital interventions such as Lumi Nova to young people with disabilities is warranted. Strengths and Limitations One of the key strengths of this study was that it involved a real-world evaluation, reflecting pragmatic usage of the intervention as part of routine practice in NHS settings and families outside of experimental research contexts, conferring external validity to the findings [ 31 ]. This approach also allowed for analysis on a larger dataset of 1029 people, even given the extent of attrition, than would have been possible to collect in a controlled study carried out with similar resources and over a similar timeframe. Moreover, the scope of data collected and analysed in this study was more extensive than is typically made available for commercial digital interventions and services, incorporating demographic, engagement, or access and outcome data, not just data regarding initial access. Collectively, this allowed for an in depth exploration of not only whether Lumi Nova appears to be effective to treat young people’s anxiety, but an insight into the platform’s users, acceptance and real-life usage in the home, and early indicators about who it is more or less likely to benefit. These insights are important for practitioners and commissioners to consider if wishing to implement Lumi Nova into future services. However, there were also several limitations to the approach taken in our study. Relying on retrospective routinely captured data completed by young people themselves without support by professionals meant the dataset had more missing data than would be anticipated in a prospective study design where missing data could be identified and addressed during the research process, posing some challenges during the data cleaning process. We were also limited by data relating to the population census statistics as a comparator, as no meaningful mental health service data is available for this specific age range. The UK-wide implementation of Lumi Nova, albeit across selected services only, meant that we were limited to comparisons of the user cohort with UK-level population data rather than using mental health service data or local small area census data. We were also limited by the choice of measures and other information available within the dataset. For instance, no standardized self-report measure of anxiety symptom severity was available and data on disabilities was captured as a binary variable, with no information available on the nature of the disability, which may have been relevant to understanding accessibility. Importantly, the real-life approach also meant that this study had no control comparator group and there was limited information on what (if any) treatment young people received in addition to Lumi Nova from the NHS or other care providers, compounding the limitations imposed by the uncontrolled study design when evaluating the cohort’s clinical outcomes. This means that conclusions of effectiveness need to be treated with caution as improvements relating to natural recovery or the impact of other treatment interventions delivered in the same period as Lumi Nova usage cannot be ruled out. Further, interpretation of the findings was made more challenging by the high level of missing data. This challenge was heightened by the impossibility of distinguishing between missing data due to a child disengaging from the intervention and missing data due to a child playing the game offline following the initial download and their data not being transferred to the central data hub as a result. Implications and Recommendations Providers of digital therapeutics should be encouraged to collect, monitor, and make available for analysis, data on how interventions are being used, by whom, and the outcomes observed. It should be noted that the quality and completeness of routine data collection by traditional child and adolescent mental health services is often poor [ 25 ]. As such, we must be cautious of holding digital interventions to standards of monitoring and evaluation that far exceed those of nondigital interventions. However, we should work toward thorough and systematic collection and analysis of routine data across all forms of child and adolescent mental health treatment. Learning from this evaluation has already been used by BfB Labs to refine their processes to improve the quality and completeness of the data they are able to capture. For instance, extra data save points have been implemented to minimize the loss of data in the event the child does not continue to the next save point or connect to the internet for their data to be uploaded. This improved data will be used by BfB Labs to continue to evaluate and refine their product. Further research is needed to explore how any barriers to access or engagement can be reduced among those with disabilities, young people from marginalized ethnic backgrounds, and those living in the most deprived areas to ensure all young people can benefit equally from interventions such as Lumi Nova. We are aware of an ongoing study (ISRCTN Registry, ISRCTN11131689) specifically aiming to understand the barriers to uptake and usage of Lumi Nova among children growing up in circumstances of economic disadvantage. This study aims to understand how the intervention can be implemented to maximize usage and engagement among those living in postcodes in the 2 most deprived deciles of England. Future research should also evaluate the benefits and costs of offering Lumi Nova and other digital therapeutic interventions via self-sign-up routes versus requiring families to be on-boarded by mental health clinicians. On a broader level, this study also raises methodological questions regarding the best way to provide evidence for digital therapeutic interventions. Digital interventions must continually and rapidly evolve if they are to keep pace with technological advancements, compatibility issues, and user expectations. As such, there is a need to ensure that approaches to evaluation balance scientific rigour with the ability to provide evidence that is timely and has sufficient ecological validity to retain relevance to real-world settings. Researchers, commissioners, providers, guidance authors, and innovators must work together to agree on what constitutes a sufficiently robust approach to evaluation in this rapidly developing field. Conclusions Providing the growing number of children seeking mental health support with timely access to care poses a significant challenge. This study provides indicative evidence that use of Lumi Nova may be associated with improvements in functioning and goal-based outcomes among those aged 7‐12 years seeking treatment for anxiety difficulties. However, the strength of this study’s findings is limited by a lack of a control comparator group and missing information about any concurrent treatments being received by young people using the intervention. As such, Lumi Nova has the potential to play an important role in improving access to mental health treatment for children experiencing anxiety. However, further research is needed to confirm these findings as natural recovery or the impact of other interventions cannot be ruled out. Future research should consider how engagement with the intervention can be maximized and sustained where beneficial for that child’s needs, particularly in children from less advantaged groups. Acknowledgments This study was funded by the National Institute for Health Research Invention for Innovation programme (NIHR202504). The views expressed are those of the authors and not necessarily those of the National Institute for Health Research or the Department of Health and Social Care. The authors wish to acknowledge the assistance of Manjul Rathee, CEO, Nabeena Mali, BFA, and Kelly Tham, BSc, from BfB Labs in providing and interpreting the dataset for analysis. We are grateful to Dr Sarah Robinson, ClinPsyD, and Sophie Knight, MPhil (Eastern Academic Health Science Network), for their contribution to the conception of this study and advisory input to the wider project. Thanks also to Aoife Dunne, MSc (research assistant psychologist), for her input to the wider project and contribution to co-ordination of patient and public involvement activities. We are also extremely grateful to the young people who gave up their time to contribute to this study as young advisors. Conflicts of Interest