Skip to main content
Download PDF

In-vivo design feedback and perceived utility of a genetically-informed smoking risk tool among current smokers in the community

BMC Medical Genomics volume 14, Article number: 139 (2021) Cite this article

Abstract

Background

The use of genetically-informed personalized risk information for behavioral disorders, namely smoking and smoking-related behaviors, is a promising yet understudied area. The Genetics and Smoking Risk Profile, or RiskProfile, leverages genetic and environmental information to communicate one’s risk for smoking-related diseases. Although prior studies have examined attitudes toward genetic results, little research has investigated these perceptions through a lens of in-vivo testing; that is, user-centered design feedback in response to personalized genetic results being returned contemporaneously. This qualitative study engaged current smokers in usability testing of the RiskProfile within the context of concurrently receiving this personalized, genetically-informed smoking cessation intervention.

Methods

Eighty-nine participants who were current smokers responded to open-ended interview questions on perceptions of smoking-related genetic information and the content and format of the RiskProfile intervention that they had received moments before. Data were analyzed via the conventional content analysis approach in which themes were allowed to emerge throughout the analysis.

Results

Participants were able to reference and offer design input on specific elements of the RiskProfile. Overall, current smokers perceived the RiskProfile to have high potential utility. Constructive feedback that current smokers offered about the tool centered around suggested improvements to optimize its usability and technical content.

Conclusions

The detailed and constructive feedback from participants highlights that in-vivo feedback offers a useful design approach that addresses concerns of rigor and relevance when returning genetic results. This unique method demonstrated perceived utility and constructive design feedback for the RiskProfile among current smokers and can play an important role in optimizing the design and implementation of personalized genetic risk interventions moving forward.

Peer Review reports

Background

Despite numerous strides in the precision medicine initiative [1], there remains a lack of widespread use of genetic information in clinical and community settings among individuals with behavioral health disorders such as smoking [2]. Pharmacogenomic advances continue to enhance treatment for breast cancer [3], psychiatric conditions [4], and dosing for specific drugs such as the anticoagulant warfarin [5, 6]. Yet, relatively less attention has been paid to the prospect of utilizing genetic information to communicate personalized risk and promote health behavior changes, such as use of evidence-based pharmacotherapy and quitting or reducing smoking [2] [e.g., 7–9, 10–12]. Genetic information has the potential to guide prevention and treatment efforts across disorders [9, 13], especially when considered in tandem with environmental or lifestyle information [14].

There is noted public interest in genetic testing for psychiatric disorders, including disorders of addiction. That interest varies based on context (e.g., having symptoms of a disorder, having a family member with a diagnosis, age, entertainment value, skepticism) and perceived potential impact and value of the genetic results [15, 16–18]. To meet this demand, numerous studies have assessed stakeholders’ perceptions about the hypothetical use of genetic information in healthcare [19–26, 27] as well as the consequences of returning actual genetic information to patients and consumers [10, 12–17, 27–31]. This work spans multiple settings, designs, types of genetic information (e.g., direct-to-consumer testing, medical genomic sequencing panel results), and ways of presenting genetic information (e.g., active versus passive). Collectively, this work has noted many barriers and facilitators toward implementing genetic information into healthcare. Pressing barriers include disagreement over who is responsible for discussing genomic information with patients and consumers and, relatedly, the insufficient supply of genetic counselors to meet the growing demand for personal access to genomic information [8, 22, 24, 32–36, 37]. There are also concerns over inadequate opportunities for education and training about genetics as a whole [22, 25, 26, 34] as well as fears of insurance discrimination [19, 15]. Implementation facilitators include the relative lack of evidence for adverse patient and consumer reactions to receiving genetic information [7, 16, 30]. Gains in knowledge, self-efficacy, and overall engagement with their results have also been noted among patients and consumers [28–30] in addition to positive behavior changes such as disclosing results with another person or lifestyle changes [12, 17, 31, 38].

To facilitate optimal implementation of genomics in behavioral health, a working group was recently established to continue bridging this gap between genetic information and behavioral health [39, 40]. There is now a growing body of work on the impact of genetic information about smoking and smoking-related disorders (e.g., nicotine addiction, lung cancer). These studies have informed current smokers’ perceptions about hypothetical and actual genetic results [41, 42, 43] and have assessed behavior change, decision making, motivation, and other related outcomes subsequent to return of personalized genetic results [7, 44–46, 11, 12, 17, 18, 47]. Despite these studies providing a wealth of important findings, no known studies have solicited user-centered design feedback of a genetic risk communication tool concurrent with the return of smoking-related genetic results. That is, studies have not examined the “in-vivo” perceptions of current smokers when they are, in the moment, engaging with personalized genetically-informed interventions. As such, there remains a scientific gap involving user-centered design feedback on personalized genetic risk tools using methods that maximize personal and contextual relevance and minimize recall bias. To maximize the rigor and relevance of this evidence, user-centered research methods may help researchers and clinicians gain a better understanding of an innovation as individuals concurrently engage with the innovation.

A key scientific gap is on perspectives from current smokers in the context of receiving personalized genetic information. Methods from the fields of implementation science and design thinking can aid in establishing the consistent use of genetic information for behavioral disorders in real-world, contextually-relevant scenarios by uncovering innovation-, individual-, organization-, and system-level barriers to integration and use of this evidence [48, 49]. An important next step that will build upon past research is to ensure that the tools used to communicate genetic information for behavioral health are designed in a way that respond to known implementation barriers and facilitate the use of genetic information to positively impact patient empowerment, self-efficacy, and behavior change [50, 51].

Study aims

The aim of the current study was to qualitatively analyze participants’ “in vivo” perceptions and feedback on a genetically-informed smoking risk tool within the context of concurrently receiving this personalized tool as part of a proof-of-concept intervention study [52]. Following genetic testing via 23andMe, current smokers received a personalized, genetically-informed smoking cessation intervention. The intervention included the smoking risk tool, hereafter referred to as the Genetics and Smoking Risk Profile, or RiskProfile. The RiskProfile was previously developed and initially validated [53] and then subsequently demonstrated utility in motivating progress toward smoking cessation, including significant pre-post reductions in cigarette smoking [52]. The current study applied in-vivo methods to qualitatively investigate and further inform the design of the RiskProfile, an approach that builds upon prior return of smoking-related genetic results studies and may be easily incorporated into future studies.

Methods

Genetics and smoking risk profile (RiskProfile)

The RiskProfile was created in an iterative manner whereby its design was optimized using separate samples of participants (see 53 for more details). It consisted of a tri-fold brochure that first presented a visually-appealing introductory outer panel that primed participants to expect information about their genetics and smoking-related outcomes (see Additional file 1: Fig. 1: How genes and smoking impact my risk; reproducced with permission from 35). The inner flap provided an overview of the 23 human chromosomes with chromosome 15 highlighted as it was specifically examined in the study. The larger inner panel provided personalized results that revealed the genetic variants for specific single nucleotide polymorphisms. These genetic results were combined with phenotypic information (i.e., number of cigarettes per day at baseline) into an algorithm that determined the individual’s risk for developing each of the three smoking-related disorders (lung cancer, lung diseases such as chronic obstructive pulmonary disease (COPD), and difficulty quitting smoking) (see https://osf.io/tmwyn/ for an overview of the algorithm). The large, inner panel also provided actionable information about the benefits of healthy behavior changes. The back panel referred the individual to resources to help with smoking cessation and included Quitline-, text-, and app-based tools, as well as potential medications to discuss with their physician. See Additional file 1: Overview of the RiskProfile's contents.

Table 1 Overview of feedback on the RiskProfile
Full size table

Discussion of the RiskProfile with participants was scripted and (1) gave an overview of genetics in lay terms, (2) emphasized the importance of genes and environment in smoking-related behaviors and diseases, (3) encouraged cessation, or reduction of smoking if full cessation was not possible, and (4) acknowledged the early stage in which this research was being conducted, noting that new genetic markers for smoking-related diseases are still being discovered and that feedback on the RiskProfile would be valuable for improving future research and practice.

Study protocol

Study context

As part of the genetically-informed smoking cessation intervention, participants had been recruited into a three-visit feasibility study to demonstrate proof of concept for the RiskProfile [52]. Participants were recruited into the study using a variety of both active and passive methods, including existing institutional registries, posted flyers, and online ads (see 53 for more details). Participants aged 21 or older who were current smokers of tobacco from the [greater St. Louis, MO] region were eligible for the study. In Visit 1, participants provided informed consent, answered baseline questions about their current smoking and perceptions of using genetic results, and provided DNA via saliva sample to be sent to 23andMe. Participants consented to being interviewed and audio recorded during visits, giving their saliva as part of the 23andMe process, and allowing study staff to analyze their raw genetic data to generate the RiskProfile. The consent form also included information about how participants’ biological samples would be treated, as well as privacy, risks, and benefits of the study (full informed consent form is available at https://osf.io/tmwyn/). The research team utilized the raw data from the 23andMe genetic reports and self-reported ancestry information to create the RiskProfile (see above). Specifically, we selected clinically valid genetic markers in the CHRNA5 gene region that have been identified in multiple genome-wide association study as having robust associations with nicotine dependence, smoking-related lung cancer and other lung diseases, and difficulty quitting smoking. Participants were then invited to Visit 2 to assess their current smoking and to receive and discuss the results of their RiskProfile. Participants also provided feedback about their “in-vivo” reactions to the tool and its potential utility for promoting smoking cessation; this served as the basis for the current study. Importantly for Visit 2, because all participants were current smokers, the RiskProfile by design never communicated a message of “low risk”, despite the participants’ genetic risk results. Further, all participants were given a consistent, strong recommendation to quit smoking and referral to freely accessible smoking cessation resources. Visit 3 was a phone-based, brief follow-up assessment to assess potential change in current smoking one month following the intervention. Analyses for that visit are outside the scope of the current qualitative study (see 52).

Current study

The present study was conducted as part of Visit 2 in the three-visit intervention study. Using a standardized verbal script (see https://osf.io/tmwyn/), a trained research team member presented each component of the RiskProfile (e.g., the concept of genetics, their risk) to participants, provided guidance on how to interpret the results, and offered multiple opportunities to ask questions. After the tool was delivered, the presenter left the room and a trained interviewer asked semi-structured, open-ended interview questions about the RiskProfile tool: (1) When you first signed up for this study, what did you most want to learn in regard to your smoking behaviors? (2) How might a tool like this genetics and smoking risk profile be helpful in guiding smoking cessation attempts? (3) What are some of your concerns with using a tool like this profile to guide smoking cessation attempts? (4) How could this genetics and smoking risk profile be improved to help motivate smoking cessation attempts? Participants were also asked to share any final thoughts about the content or format of the RiskProfile, and interviewers were trained to prompt participants as necessary throughout the interview. Participants were monetarily compensated $25 for this visit.

Analyses

Transcription of in-vivo responses followed a verbatim approach whereby responses were written exactly as they were said by participants except that filler words (e.g., “um,” “like,”) were filtered out to aid in analysis [54]. A co-author (AD) on the paper conducted the transcriptions. Qualitative analysis followed a conventional content analysis approach where themes were allowed to emerge from the data [55]. The order for analysis included creating open codes, collapsing codes into categories, and then conceptually ordering the categories into themes in an inductive manner (see Additional file 1: Table 2: Breakdown of initial codes, categories, and themes across questions). Microsoft Excel was used to analyze and count the codes, categories, and themes [56, 57], and multiple codes per response were allowed (see Additional file 1: Text 1: Details about coding with Microsoft Excel). Researchers engaged in constant comparison whereby codes and categories were checked throughout data analysis to ensure their continued relevance and appropriateness. This is necessary because later responses can inform earlier codes and vice-versa [54]. Relevant quotes were identified as necessary.

Each question was analyzed by two raters (JLB and AD) who regularly met to discuss similarities and differences in their analyses. Analyses often aligned except for specific codes used (e.g., semantic differences) or the structuring of themes (e.g., one person may have summarized positive and negative feedback as one theme and the other may have divided it into two themes). All coding discrepancies were discussed and resolved collaboratively by both raters to achieve consensus for each question before continuing the analysis.

Results

Results are presented as five domains; relevant themes accompany each one and all are summarized in Table 1. The domains were structured around the original questions asked to participants, although responses from multiple questions comprise each domain.

Sample demographics

The sample consisted of 89 participants (mean age = 47 years; female = 51%; White/Caucasian = 56%; Black/African American = 36%; another ancestry = 8%; Hispanic or Latinx = 2%). All participants consented to being audio recorded for transcription except for two. Summaries of those participants’ responses were recorded via written record.

Domain 1: motivations for receiving the RiskProfile

Motivations for receiving the RiskProfile centered upon two different themes of smoking—strong interest in the genetics of smoking and information-seeking for broader health reasons.

Strong interest in the genetics of smoking

Most participants were motivated by their interest in the genetic component of the study, and some were engaged on a personal level in which they deeply reflected and drew substantial meaning from the information being presented to them. For example, one participant wanted to know “if I had any genetic factors involved [in smoking-related outcomes]” while another simply wanted to know if there was “a genetic variant that caused you to be more addicted to the tobacco or … made it harder to quit.”

Interestingly, even participants who were not well-versed in the genetics of smoking were often familiar with the influential role that genetics plays in smoking and related behaviors. For example, one participant noted “I wanted to […] just see more about how the addictive gene affects our family and me personally.” Others discussed genetics in a deterministic manner, implying that they either thought or hoped that it explained their smoking behaviors more than their environment (e.g., lifestyle or behavioral factors). As one participant stated, “So I guess I was interested in the genetic aspect of it to see if there were factors working against me that I may not have a lot of control over.”

Information-seeking for broader health

There was a subset of participants who participated in the study to gain very specific types of information about their health. For example, some participants were interested in their overall risks for disease and smoking-related illnesses and how this could impact their health. Some participants even hoped for a formal diagnosis or prognosis from this study. Others wanted information to support their efforts to quit smoking (e.g., cessation medications). As one participant noted, “I wanted to learn how hard it would be to quit and what would be the best way to quit.”

Domain 2: perceived utility of the RiskProfile

Participants focused on two key themes of the RiskProfile’s utility—how it can inform and educate or potentially motivate behavior change.

Informing and educating

Participants commented on how useful it is for individuals who smoke to know their genetic risk because that can inform future health and behavior decisions. As one participant said, “I wasn’t sure how high of a risk I was [for smoking-related outcomes], but to see I was in the higher risk category—that was pretty eye opening. And if I ever have children too, it would be good to have some kind of knowledge to help make an environment that they wouldn’t want to smoke [in].” The RiskProfile can also help inform individuals about why they smoke and/or struggle with craving nicotine. Although most participants were already aware of the harms of smoking, several were of the mindset that more information and education, particularly when presented in a unique and more personalized way, was useful. One participant stated, “I think it’s just one more bullet in the chamber, one more weapon in your belt, however you want to put it.”

Motivating behavior change

Several participants commented on the various ways that the RiskProfile can help participants focus on their health and the long-term goal of quitting because of its motivational impact. For example, one participant stated that the tool was “Just more motivation. It tells you how critical it is to […] attempt to quit.” As one participant noted, “I think […] it will help [others] change their lifestyle and I would like to think that it’s going to help me change my lifestyle.”

Domain 3: potential concerns about using the RiskProfile

Participants expressed a range of potential concerns about using the RiskProfile. These included having no concerns, concerns about under-utilizing the tool, and privacy concerns.

No concerns

It is worth noting that the majority of participants did not have any concerns about the genetic smoking risk tool. Participants generally remarked on how they did not find anything concerning with the smoking risk profile tool and mentioned that it could be helpful to use. For example, one participant commented “No, no concerns. I love the idea.”

Lack of utilization

Many participants were concerned about the RiskProfile being under-utilized by themselves and others moving forward. Some of these concerns involved personal agency following receipt of the tool, for example, including acknowledgment of now facing the difficulty of cigarette reduction or cessation and fear of such efforts failing. A couple of participants also noted the concern that the tool might not be used after the visit, possibly because it is simply not useful enough for individuals who already know the risks of smoking. For example, one participated noted, “I know the hazards [of smoking] and […] at a certain point […] we just don’t care because a lot of [us] figure, ‘I’ve lived my time.’” Additional concerns were about potential unanticipated consequences of the RiskProfile and whether the tool could be enabling for individuals with a low genetic risk for smoking-related outcomes. As one participant said, “I don’t so much have any concerns besides [someone] being at genetically lower risk and thinking that they have a free pass to smoke now because they’re at low risk.”

Privacy

Many participants expressed concerns about the privacy of their information, but not about the information in the RiskProfile itself. These concerns were mitigated by participants’ trust that the research team associated with this project were upholding the highest possible privacy standards, but such standards were not assumed to be upheld by third parties. Participants expressed broad concerns about the safety of their genetic information, third party tracking, distribution, and use of their genetic information (e.g., insurance companies).

Domain 4: suggestions for improving the effectiveness of the RiskProfile

Participants provided a range of feedback about the RiskProfile, including leaving it as-is, improving the tool’s content, and improving the tool’s impact.

Leave it as-is

Although several participants noted areas of improvement for the RiskProfile, most had no suggestions and/or voiced approval. As one participant said, “I think that it did what it was designed to do. I don’t know how it could be improved.”

Improve jargon, technical details, and layout

Some participants commented on the formatting, layout, information, and content of the RiskProfile. Although these comments were quite diverse, they did provide concrete suggestions to improve the tool in order to better motivate cessation attempts. Feedback on key areas of improvement included simplifying the layout, adding more information about the genetic information that went into making the profile risk score, and improving the language because the tool was not intuitive and required explanation. Some participants also suggested to include risk for family members. Finally, participants recommended periodic updates to the tool as new information (e.g., relevant genetic markers, risk percentages, cessation aids) are made available.

Improve impact and meaningfulness

There were several responses that provided insight beyond the format or content of the tool itself. Many participants offered improvements that would help make the RiskProfile more impactful and meaningful, thus potentially improving motivation to attempt smoking cessation. Some suggestions included expanding what went into the risk profile score, such as asking more information about behaviors, environment, markers, and family history. Other suggestions included ways to make the experience more personalized, such as offering information about personalized smoking cessation. Other suggestions of how to maximize the impact and meaning included increasing the perceived severity of disease risk such as more sensationally displaying the potential consequences of continued smoking (e.g., that individuals with COPD often use oxygen tanks) and arranging follow-up professional outreach with a healthcare professional. A few participants also discussed ways to improve the reach of the study, such as more advertisement and targeting younger adults. This feedback highlights the importance of reaching and engaging the right audience with highly personalized and actionable content, all of which is perceived to be essential to more widespread implementation of this tool.

Domain 5: general positive feedback

There was much generalized positive feedback offered by participants that cut across all of the other four domains. Participants seemed happy about the experience as a whole, from the in-person aspect of the study to the personally engaging information that the RiskProfile provided. Many noted that the tool was helpful and motivating and that the information presented was unavoidable and eye-opening. Additional positive feedback about the RiskProfile centered around the fact that the information was relevant and credible, the tool itself would be useful for young people (e.g., prevention), and the layout was appropriate. Notably, one participant specifically liked that even though it was a personalized tool, there was no identifying information on the tool itself. As they said, “I mean you’re handing me a paper that doesn’t have my name on it so even if I dropped it, nobody’s going to know it was me.”

Discussion

The current study qualitatively analyzed in-vivo responses about a personalized, genetically-informed Genetics and Smoking Risk Profile (RiskProfile). As a whole, participants expressed favorable views of the RiskProfile, providing detailed information about specific components of the tool that would not have been available without the in-vivo design. Results across five inter-connected domains provide a richer understanding of current smokers’ motivations to participate, the specific perceived benefits and potential utility of the RiskProfile, and the concerns and suggested opportunities to improve this tool. To date, few studies have studied how to translate responses to personalized genetic susceptibility information into evidence-based interventions designed to communicate disease risk and motivate behavior change [15, 58, 59]. Thus, the current study expands past research in at least two ways: the use an in-vivo study design to concurrently study the genetically-informed tool as it is being delivered to participants; and the ability to apply this rich, highly-specified in-vivo design feedback to improve the content and format of the RiskProfile moving forward.

Relative advantage of the concurrent in-vivo design

The in-vivo approach used in the current study extends prior research that examined participants’ attitudes about receiving and using both hypothetical [19, 23, 41, 42, 60, 27, 47] and personalized [7, 10–18, 28–38, 47, 2] genetic information for various purposes by analyzing participants’ feedback about the RiskProfile concurrently with its delivery. This study capitalized on the opportunity to engage participants using qualitative and user-centered design methods to gain a deep understanding of their motivations, perceived utility, concerns, recommended improvements, and positive aspects of the RiskProfile. This was done from the context in which participants received the personalized genetic risk tool. This approach yielded detailed, contextually-relevant, and timely feedback. Thus, such deep, in-vivo analyses appear advantageous in gaining an understanding of the potential utility of personalized genetically-informed tools. Accordingly, the information provided by participants will influence future iterations of this work in ways that would not have been possible without the current design. Future studies that utilize personalized genetic information may be able to optimize the design of genetically-informed tools by implementing a similar methodology.

Although current smokers will likely benefit from behavioral interventions that are more closely aligned with their participatory design input, this in-vivo method carries important advantages for researchers as well. For instance, this type of in-depth, contextually-relevant qualitative feedback design may be particularly sensitive to detecting plausible mechanisms of behavior change that warrant additional investigation in future research. Potential behavior change mechanisms that were discussed by participants across domains included health-related cognitions (e.g., perceived risk of disease, perceived benefits of smoking cessation, perceived value of cessation aids) and engagement factors (e.g., personal relevance, comprehension, sharing of results) [61, 62]. To date, most studies that communicated genetic susceptibility information to participants have had little impact on behavior change, including smoking cessation [45, 12, 17, 28, 63]. However, there are notable exceptions [46, 52] as well as a more recent meta-analysis to suggest that genetic risk feedback may motivate behavior change among at-risk individuals [31]. These findings highlight the need for additional research focused on the potential mechanistic effects of genetically-informed behavioral interventions and the thoughtful design of these interventions to activate identified behavioral targets.

Ability to quickly apply in-vivo results

Due to the current study’s design, there are three key take-aways that were only made apparent due to the in-vivo design of the study and which can immediately be applied to next steps in this research trajectory (see Challenges and Future Studies below). While these take-aways are specific to the RiskProfile, they offer insights for other teams who are currently working on returning genetically-informed information to participants. First, the overarching positive feedback demonstrated broad support among current smokers for the RiskProfile, a novel genetically- and environmentally-informed risk score that cannot be obtained through any direct to consumer or other third-party genetic testing services. However, the information provided by the RiskProfile may not be sufficiently motivating for all levels of smokers. Reflecting upon some participants’ responses about prevention opportunities and “targeting youth,” it is possible that this risk profile would have value for prevention programs or early smoking cessation efforts among new smokers. For the tool to be maximally beneficial among adult (and presumably longer-term) smokers, concerns raised in the current study need to be addressed. These include expanding the parameters of the risk profile (i.e., genetic and environmental information), addressing privacy concerns, consulting with a smoking cessation expert and genetic counselor, and receiving feedback about participants’ experiences with the RiskProfile after more time has passed and they can reflect more deeply about the tool.

Second, the genetic and environmental components that comprised the risk score in the RiskProfile need to be expanded. This addresses feedback from across all five domains. Many participants wanted specific information about risk to family members, education about risk to self and family, enhanced post-visit utility, and improved meaningfulness of the tool. In the future, the risk score algorithm and tailored intervention components could be supplemented with information about family history of smoking, smoking triggers, household and partner smoking, and other related factors. Including a more well-rounded risk score may make the tool more motivating and comprehensible to participants. It may also naturally create more dialogue between the researchers and participants.

Finally, there is a tension between the intensity and amount of information in the RiskProfile, and this relates to the modality in which the information is delivered. For instance, participants who thought that the tool did have enough information may be assuming that there will always be a researcher to explain the report in real-time. Conversely, those who think it has enough or too much information may be assuming the report will ultimately be a “stand alone” tool that needs to be very concise and easy to grasp without additional support or explanation from health professionals. Balancing this tension will be critical as the RiskProfile is expanded and updated in the future.

Challenges and future studies

There will be challenges moving this research forward. A key one will be overcoming concerns with genetic testing, including ownership and use of the data [19, 15, 64]. These concerns were echoed via privacy concerns in the current study, although they were tempered by the trust in the research team, which may generalize more broadly to healthcare professionals. Also, as documented in the larger proof of concept study [52], it is encouraging that participants ultimately expressed no decision regret in association with their participation by the end of the study. There is also the barrier of genetic literacy and expectations about what the RiskProfile, or future iterations of it, can offer to participants. Despite careful wording and the use of scripts during the visit, some participants still expected to learn information that went beyond the scope of this study. Setting clear and reasonable expectations of participant benefit, while also acknowledging the range of perceptions of potential benefit, reflect another important tension to balance in future in-vivo research [64]. Relatedly, reinforcing the educational component of the tool will clarify participants’ expectations, as education is an oft-cited barrier to the translation (e.g., implementation, uptake) of genetic information among multiple stakeholder groups [21, 26, 34, 64, 65].

There are three main layers of examination and validation that will drive future directions of this research. First, immediate improvements can be made to the tool based on the current in-vivo feedback, and acceptability and feasibility should be iteratively assessed via in-vivo feedback with qualitative and quantitative metrics in future versions of the RiskProfile. Second, efficacy and effectiveness testing are needed to demonstrate clinical utility of the RiskProfile as measured by the tool’s impact on smoking cessation and other risk-reducing behaviors. Results from this study also suggest the need for dedicated testing of mechanisms, including cognitive (e.g., perceived risks and benefits, perceived value of cessation aids) and engagement-related (e.g., personal relevance) factors, that may uncover essential intervention targets for future research. Finally, as the focus of this study was on individuals in the community who smoke, it is unknown to what extent this type of tool would or should impact the smoking cessation care that healthcare professionals provide. It is plausible that patient-specific genetic and clinical risk information could facilitate patient-centered discussions and highlight the urgency and potential benefits of smoking cessation for healthcare professionals; however, to change their clinical practice, healthcare professionals must be convinced that this tool provides unique and useful insight. If found to be clinically useful, a future step would be to integrate the RiskProfile into community health agencies or clinical settings with healthcare practitioners delivering the tool to participants. Alternatively, an online version of the RiskProfile could be implemented in the future, such that individuals could access their personalized results and interpretation of those results, without the need to attend in-person counseling.

Limitations

This study is not without limitations. First, as a qualitative study, we prioritized saliency of expressed responses to construct thematic content rather than quantitative metrics that often provide more summative evaluations. Relatedly, although we encouraged honest critiques of the usability and usefulness of the tool, it is possible that participants were reluctant to share more harshly negative feedback. Second, the potential confounding effects of the study staff were not examined (i.e., effects of who explained the genetic information, who administered the questionnaires during the visit, who coded the data). Third, it should be noted that organizational and clinical guidelines do not yet support the return of genomic information to participants for clinical reasons. Therefore, the current study was not designed to inform clinical decisions and participants were encouraged to speak with their physician about smoking cessation efforts. For instance, given the large consumer interest in reduced risk nicotine options, such as nicotine vaping products, some participants may have desired clinical recommendations about these alternatives with respect to their specific genetically-informed risks. Although we were unable to include such recommendations in this study, examining the utility of lower risk nicotine products in relation to genetic risk, particularly among individuals with high genetic risk, is an important area of scientific inquiry moving forward. Finally, 23andMe uses genotyping technology that comprises mostly European populations. This is a limitation in the field of genetics for genotyping and sequencing and has been discussed elsewhere (23andme.com/ancestry-composition-guide; 66).

Conclusions

The current study presents an in-vivo approach to assessing participants’ attitudes and perceived utility about receiving personalized genetic information from a novel Genetics and Smoking Risk Profile (RiskProfile). Through detailed in-vivo, user-centered design feedback, we identified five inter-connected domains—the motivations, perceived utility, potential concerns, suggestions for improvement, and positive aspects of the RiskProfile. The RiskProfile was largely well received by participating current smokers and, with modifications that align with end-user feedback described herein, may hold promise as a useful smoking cessation tool in the future. This study highlights the advantages of using an in-vivo design to optimize the design of genetically-informed intervention tools and to maximize the rigor and relevance of end-user feedback when returning genetic information to participants.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

COPD:

Chronic Obstructive Pulmonary Disease

RiskProfile:

Genetics and Smoking Risk Profile

References

  1. Collins FS, Varmus H. A new initiative on precision medicine. New England J Med. 2015;372:793–5.

    Article  CAS  Google Scholar 

  2. Peterson EB, Chou W-yS, Gaysynsky A, Krakow M, Elrick A, Khoury MJ, Kaphingst KA. Communication of cancer-related genetic and genomic information: A landscape analysis of reviews.

  3. Jeibouei S, Akbari ME, Kalbasi A, Aref AR, Ajoudanian M, Rezvani A, Zali H. Personalized medicine in breast cancer: pharmacogenomics approaches. Pharmacogen Per Med. 2019;12:59–73.

    CAS  Google Scholar 

  4. Hamilton SP. The promise of psychiatric pharmacogenomics. Bio Psych. 2015;77(1):29–35.

    Article  Google Scholar 

  5. Millican EA, Lenzini PA, Millian PE, Groxxo L, Eby C, Deych E, et al. Genetic-based dosing in orthopedic patients beginning warfarin therapy. Blood. 2007;110:1511–5.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Shahin MHA, Johnson JA. Clopidogrel and warfarin pharmacogenetic tests: what is the evidence for use in clinical practice? Current Opin Cardio. 2013;28:305–14.

    Article  Google Scholar 

  7. Hartz SM, Olfson E, Culverhouse R, Cavazos-Rehg P, Chen L-S, DuBois R, et al. Return of individual genetic results in a high-risk sample: Enthusiasm and positive behavioral change. Gen in Med. 2015;17:374–9.

    Google Scholar 

  8. Moldovan R, Pintea S, Austin J. The efficacy of genetic counseling for psychiatric disorders: a meta-analysis. J Gen Counsel. 2017;26:1341–7.

    Article  Google Scholar 

  9. Norman P, Brain K. An application of an extended health belief model to the prediction of breast self-examination among women with a family history of breast cancer. Brit J Health Psych. 2005;10(1):1–16.

    Article  Google Scholar 

  10. Chen L-S, Kaphingst KA, Tseng T-S, Zhao S. How are lung cancer risk perceptions and cigarette smoking related? – testing an accuracy hypothesis. Transl Cancer Res. 2016;5(Suppl5):S964–71.

    Article  PubMed  Google Scholar 

  11. Roberts JS, Gornick MC, Carere DA, Wuhlmann WR, Ruffin MT, Green RC. Direct-to-consumer genetic testing: user motivations, decision making, and perceived utility of results. Pub Health Genomics. 2017;20:36–45.

    Article  Google Scholar 

  12. Olfson E, Hartz S, Carere, DA, Green RC, Roberts JS, Bierut, LJ. Implications of personal genomic testing for health behaviors: the case of smoking. Nic Tobacco Res. 2016;2273–2277.

  13. Khoury MJ, Iademarco MF, Riley WT. Precision public health for the era of prediction medicine. Am J Prev Med. 2016;50:398–401.

    Article  PubMed  Google Scholar 

  14. Belsky DW, Moffitt TE, Caspi A. Genetics in population health science: Strategies and opportunities. Am J Pub Health. 2012;103:S73–83.

    Article  Google Scholar 

  15. Meiser B, Guo XY, Putt S, Fulleton JM, Schofield PR, Mitchell PB, Yanes T. Psychosocial implications of living with familial risk of a psychiatric disorder and attitudes of psychiatric genetic testing: A systematic review of the literature. Am J Med Gen Part B. 2020;1–20. doi:https://doi.org/10.1002/ajmg.b.32786

  16. Waltz M, Meagher KM, Henderson GE, Goddard KAB, Muessig K, Berg JS, Weck KE, Cadigan RJ. Assessing the implications of positive genomic screening results. Per Med. 2020;17(2):101–9.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Stewart KFJ, Wesselius A, Schreurs MAC, Schols AMWJ, Zeegers MP. Behavioral changes, sharing behavior and psychological responses after receiving direct-to-consumer genetic test results: a systematic review and meta-analysis. J Community Genet. 2018;9:1–18.

    Article  CAS  PubMed  Google Scholar 

  18. O’Neill S, Lipkus IM, Sanderson SC, Shepperd J, Docherty S, McBride CM. Motivations for genetic testing for lung cancer risk among young smokers. Tob Control. 2013;22(6):406–11.

    Article  PubMed  Google Scholar 

  19. Kimball BC, Nowakowski KE, Maschke KJ, McCormick JB. Genomic data in the electronic medical record: Perspectives from a biobank community advisory board. J Emp Res Hum Res Ethic. 2014;9:16–24.

    Article  Google Scholar 

  20. Prom-Wormley EC, Clifford J, Bourdon JL, Barr P, Blondino C, Ball J, et al. Developing community-based strategies with family health history: Assessing the association between community resident family history and interest in health education. Soc Sci Med. 2019. https://doi.org/10.1016/j.socscimed.2019.02.011.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Salm M, Abbate K, Applebaum P, Ottmamn R, Chung W. Use of genetic tests among neurologists and psychiatrists: Knowledge, attitudes, behaviors, and needs for training. J Gen Counsel. 2014;23:156–63.

    Article  Google Scholar 

  22. Zhou YZ, Wilde A, Meiser B, Mitchell PB, Barlow-Stewart K, Schofield PR. Attitudes of medical genetics practitioners and psychiatrists toward communicating with patients about genetic risk for psychiatric disorders. Psych Gen. 2014;24(3):94–101.

    Article  Google Scholar 

  23. Fernandez CV, Strahlendorf C, Avard D, Knoppers BM, O’Connell C, Bouffet E, et al. Attitudes of Canadian researchers toward the return to participants of incidental and targeted genomic findings obtained in a pediatric research setting. Gen Med. 2013;25:558–64.

    Google Scholar 

  24. Klitzman R, Appelbaum PS, Fyer A, Martinez J, Buquez B, Wynn J, et al. Researchers’ views on return of incidental genomic research results: qualitative and quantitative findings. Gen Med. 2013;15:888–95.

    Google Scholar 

  25. Nurnberger JI, Austin J, Berrettini WH, Besterman AD, DeLisi LE, Grice DE, et al. What should a psychiatrist know about genetics? Review and recommendations from the residency education committee of the International Society of Psychiatric Genetics. J Clin Psych. 2018;80(1):e1–6.

    Google Scholar 

  26. Wilson BJ, Islam R, Francis JJ, Grimshaw JM, Permaul JA, Allanson JE, et al. Supporting genetics in primary care: investigating how theory can inform professional education. Euro J Hum Gen. 2016;24:1541–6.

    Article  Google Scholar 

  27. Kohler JN, Turbitt E, Lewis KL, Wilfond BS, Jamal L, Peay HL, Bieskecker LG, Biesecker BB. Defining personal utility in genomics: a Delphi study. Clin Gen. 2017;92:290–7.

    Article  CAS  Google Scholar 

  28. Kaphingst KA, McBride CM, Wade C, Alford SH, Reid R, Lrason E, Baxevanis AD, Brody LC. Patient understanding of and responses to multiplex genetic susceptibility test results. Genet Med. 2012;14(7):681–7.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Kaphingst KA, McBride CM, Wade C, Alford SH, Brody LC, Baxevanis AD. Consumers’ use of web-based information and their decisiosn about multiplex genetic susceptibility testing. J Med Internet Res. 2010;12(3):e41.

  30. Carere DA, Kraft P, Kaphinst KA, Roberts S, Green RC. Consumers report lower confidence in their genetics knowledge following direct-to-consumer personal genomic testing. Genet Med. 2016;18(1):65–72.

    Article  PubMed  Google Scholar 

  31. Frieser MJ, Wilson S, Vrieze SI. Behavioral impact of return of genetic test results for complex disease: Systematic review and meta-analysis. Health Psychol. 2018;37(12):1134–44.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Visscher PM, Wray NR, Zhang Q, Sklar P, McCarthy MI, Brown MA, Yang J. 10 years of GWAS discovery: biology, function, and translation. Am J Hum Gen. 2016;101:5–22.

    Article  CAS  Google Scholar 

  33. Finn CT, Smoller JW. Genetic counseling in psychiatry. Harvard R Psych. 2006;14:109–21.

    Article  Google Scholar 

  34. Sperber NR, Carpetner JS, Cavallari LH, Damschroder LJ, Cooper-DeHoff RM, Denny JC, et al. Challenges and strategies for implementing genomic services in diverse settings: Experiences from the Implementing GeNomics in practice (IGNITE) network. BMC Med Gen. 2017;10(35).

  35. International Society of Psychiatric Genetics [ISPG]. Genetic testing statement. 2017;Available from https://ispg.net/genetic-testing-statement/.

  36. Ramoni RB, McGuire AL, Robinson JO, Morley DS, Plon SE, Joffe S. Experiences and attitudes of genome investigators regarding return of individual genetic test results. Gen Med. 2013;15(11):882–7.

    Google Scholar 

  37. Sullivan PK, Daly MJ, O’Donovan M. Genetic architectures of psychiatric disorders: the emerging picture and its implications. Nat R Gen. 2012. https://doi.org/10.1038/nrg3240.

    Article  Google Scholar 

  38. Kaphingst KA, Peterson E, Zhao J, Gaysynsky A, Elrick A, Hong SJ, Krakow M, et al. Cancer communication research in the area of genomics and precision medicine: a scoping review. Gen in Med. 2019;21(8):1691–8.

    Google Scholar 

  39. McBride CM, Graves KD, Kaphingst KA, Allen CG, Wang C, Arrendondo E, Klein WMP. Behavioral and social scientists’ reflections on genomics: a systematic evaluation within the Society of Behavioral Medicine. RBM. 2019;9:1012–9.

    Google Scholar 

  40. Klein WMP, McBride CM, Allen CG, Arrendondo EM, Bloss CS, Kaphingst KA, Sturm AC, Wang C. Optimal integration of behavioral medicine into clinical genetics and genomics. Am J Hum Gen. 2019;104:193–106.

    Article  CAS  Google Scholar 

  41. Senft N, Sanderson M, Selove R, Blot WJ, King S, Gilliam K, et al. Attitudes towards precision treatment of smoking in the Southern Community Cohort Study. Cancer Epi Biomark Prev. 2019;8:1345–52.

    Article  Google Scholar 

  42. Waters EA, Ball L, Ghlert S. ‘I don’t believe it.’ Acceptance and skepticism of genetic health information among African-American and White smokers. Soc Sci Med. 2017;184:153–160.

  43. Shepperd JA, Novell CA, O’Neill SC, Docherty SL, Sanderson SC, McBride CM, Lipkus IM. Contemplating genetic feedback regarding lung cancer susceptibility. Ann Behav Med. 2014;47(3):395–403.

    Article  PubMed  Google Scholar 

  44. Bloss CS, Jeste DV, Schork NJ. Genomics for disease treatment and prevention. Psych Clin North Am. 2011;34:147–66.

    Article  Google Scholar 

  45. Hollands GJ, French DP, Griffin SJ, Prevost AT, Sutton S, King S, et al. The impact of communicating genetic risks of disease on risk-reducing health behaviour: Systematic review with meta-analysis. BMJ. 2016;352:1–11.

    Google Scholar 

  46. Lipkus IM, Schwartz-Bloom R, Kelley MJ, Pan W. A Preliminary Exploration of College Smokers’ Reactions to Nicotine Dependence Genetic Susceptibility Feedback. Nic Tob Res. 2015;17(3):337–43.

    Article  Google Scholar 

  47. Sanderson SC, O’Neill SC, White DB, Depler G, Bastian L, Lipkus IM, McBride CM. Responses to online GSTM1 genetic test results among smokers related to patients with lung cancer: a pilot study. Cancer Epi Biomarkers Prev. 2009;18(8):1953–61.

    Article  CAS  Google Scholar 

  48. Kong C, Dunn M, Parker M. Psychiatric genomics and mental health treatment: setting the ethical agenda. Am J Bioethics. 2017;17:3–12.

    Article  Google Scholar 

  49. Roberts MC, Kennedy AE, Chambers DA, Khoury MJ. The current state of implementation science in genomic medicine: opportunities for improvement. Gen in Med. 2017;19(8):858–63.

    Google Scholar 

  50. Austin JC. Evidence-based genetic counseling for psychiatric disorders: A road map. Cold Spring Harbor P Med. 2019;1–14.

  51. Ramsey AT, Chen L-S, Hartz SM, Saccone NL, Fisher SL, Proctor EK, Bierut LJ. Toward the implementation of genomic applications for smoking cessation and smoking-related diseases. TBM. 2018;8:7–17.

    Google Scholar 

  52. Ramsey AT, Bourdon JL, Bray M, Dorsey A, Zalik M, Pietka A, et al. Proof of concept of a personalized genetic risk tool to promote smoking cessation: High acceptability and reduced cigarette smoking. Cancer Prev Res. 2021. https://doi.org/10.1158/1940-6207.CAPR-20-0328.

    Article  Google Scholar 

  53. Ramsey AT, Bray M, Laker PA, Bourdon JL, Dorsey A, Zalik M, et al. Participatory design of a personalized genetic risk tool to promote behavioral health. Cancer Prev Res. 2020;14:253–62. https://doi.org/10.1158/1940-6207.CAPR-20-0029.

    Article  Google Scholar 

  54. Strauss AL, Corbin JM. Basics of qualitative research: techniques and procedures for developing grounded theory. London: SAGE Publication; 1998.

    Google Scholar 

  55. Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Res. 2005;15:1277–88.

    Google Scholar 

  56. Myer DZ, Avery LM. Excel as a qualitative data analysis tool. Field Methods. 2009;21(1):91–112.

    Article  Google Scholar 

  57. Bree R, Gallagher G. Using Microsoft Excel to code and thematically analyze qualitative data: a simple, cost-effective approach. AISHE-J. 2016;8:2811–28114.

    Google Scholar 

  58. Moller AC, Merchant G, Conroy DE, West R, Hekler E, Kugler KC, et al. Applying and advancing behavior change theories and techniques in the context of a digital health revolution: Proposals for more effectively realizing untapped potential. J Behav Med. 2017;40:85–98.

    Article  PubMed  PubMed Central  Google Scholar 

  59. Zhang C-Q, Zhang R, Schwarzer R Hagger MS. A meta-analysis of the health action process approach. Health Psychol. Off. J. Div. Health Psych Am Psych Assoc. 2019;38:623–637.

  60. Chiu A, Hartz S, Smock N, Chen J, Qazi A, Onyeador J, et al. Most current smokers desire genetic susceptibility testing and genetically-efficacious medication. J Neuroimmune Pharm. 2018;13(4):430–7.

    Article  Google Scholar 

  61. Michie S, Van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Impl Sci. 2011;6(1):42.

    Article  Google Scholar 

  62. Rosenstock IM, Strecher VJ, Becker MH. Social learning theory and the health belief model. Health Edu Quarterly. 1988;15(2):175–83.

    Article  CAS  Google Scholar 

  63. Marteau TM, French DP, Griffin SJ, Prevost AT, Sutton S, Watkinson C, Attwood S, Hollands GJ. Effects of communicating DNA-based disease risk estimates on risk-reducing behaviors. Cochrane. 2010. https://doi.org/10.1002/14651858.CD007275.pub2.

    Article  Google Scholar 

  64. Cornetta K, Brown CG. Balancing personalized medicine and personalized care. Acad Med. 2013;88(3):309–13.

    Article  PubMed  PubMed Central  Google Scholar 

  65. Costain G, Esplen MJ, Toner B, Hodgkinson KA, Bassett AS. Evaluating genetic counseling for family members of Individuals with Schizophrenia in the molecular age. Schizophrenia Bull. 2014;41(1):88–99.

    Article  Google Scholar 

  66. Peterson RE, Kuchenbaecker K, Walters RK, et al. Genome-wide association studies in ancestrally diverse populations: Opportunities, methods, pitfalls, and recommendations. Cell. 2019;179.

Download references

Acknowledgements

We would like to extend a great thanks to the participants who took part in this study.

Funding

J.L. Bourdon was supported by the National Institute of Drug Abuse (NIDA) grant T32DA015035 at Washington University during this study before switching employers. L.J. Bierut was supported by NIDA grant K12DA041449, National Center for Advancing Translational Sciences grant UL1TR002345, and National Cancer Institute grant P30CA091842. M.J. Bray was supported by the NIH Training Grant 5T32MH014677. A.T. Ramsey was supported by the NIDA grants K12DA041449 and R34DA052928. Funders’ roles were to support the training and/or salary of the aforementioned co-authors; funders had no direct input in the coordination or execution of this study.

Author information

Authors and Affiliations

  1. Wellbridge Center for Addiction Treatment and Research, Center for Addiction Science, 525 Jan Way, Room 1523, Calverton, NY, 11922, USA

    Jessica L. Bourdon

  2. Department of Psychiatry, Washington University in St. Louis, St. Louis, MO, USA

    Amelia Dorsey, Maia Zalik, Amanda Pietka, Patricia Salyer, Michael J. Bray, Laura J. Bierut & Alex T. Ramsey

Authors
  1. Jessica L. Bourdon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Amelia Dorsey
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maia Zalik
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Amanda Pietka
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Patricia Salyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Michael J. Bray
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Laura J. Bierut
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Alex T. Ramsey
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors have read and approved the manuscript prior to submission. JLB assisted with data collection, led analyses, and led manuscript preparation. AD assisted with data collection, assisted with analyses, and provided edits on the manuscript. MZ assisted with data collection and provided edits on the manuscript. AP assisted with data collection and provided edits on the manuscript. PS made substantial contributions to the design of the work and data collection for this study. MJB assisted with data collection and provided edits on the manuscript. LJB helped with conceptualization of the project and provided edits on the manuscript. ATR led the scientific direction of this project and made substantial contributions to the conceptualization, design, data acquisition, analysis, data interpretation, and writing of the manuscript. All author read and approved the final manuscript.

Corresponding author

Correspondence to Jessica L. Bourdon.

Ethics declarations

Ethics approval and consent to participate

This study obtained institutional review board approval through the Human Research Protection Office at Washington University in St. Louis prior to data collection and consent by all participants during data collection. Consent was obtained via written format.

Consent for publication

Not applicable.

Competing interests

L.J. Bierut is listed as an inventor on Issued U.S. Patent 8,080,371 “Markers for Addiction” covering the use of certain SNPs in determining the diagnosis, prognosis, and treatment of addiction. This patent does not restrict research applications used in this study. The remaining authors declare no potential conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Supplemental Table 1. Overview of the RiskProfile’s contents. Supplemental Table 2. Breakdown of initial codes, categories, and themes across questions. Supplemental Text 1. Details about coding with Microsoft Excel. Supplementary Figure 1. Reproduced with permission from Ramsey et al., 2020 (35).

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bourdon, J.L., Dorsey, A., Zalik, M. et al. In-vivo design feedback and perceived utility of a genetically-informed smoking risk tool among current smokers in the community. BMC Med Genomics 14, 139 (2021). https://doi.org/10.1186/s12920-021-00976-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12920-021-00976-1

Keywords

  • Smoking
  • Genetics
  • Cessation
  • Implementation

BMC Medical Genomics

ISSN: 1755-8794

Contact us