Patients with baseline moderate-severe OSA (Apnea-hypopnea Index (AHI) ≥15), aged 18-75 years old, who were treated for OSA using a CPAP device for at least one month, and had insomnia complaints, were considered eligible. Insomnia was diagnosed by The International Classification of Sleep Disorders – Third Edition (ICSD-3) diagnostic criteria and ISI (Insomnia Severity Index) scale. Patients with ISI scores more than 15 or more than 6 in the first three questions, were also eligible.

PSQI (Pittsburgh Sleep Quality Index) ≥ 5 showed poor sleep quality and was considered as an inclusion criterion.

After baseline assessment, the exclusion criteria were considered as following: severe pulmonary diseases (COPD GOLD IV, severe uncontrolled bronchiectasis, ILD, and asthma), severe heart diseases (NYHA class IV and uncontrolled hypertension), neurologic disorders (Parkinson disease and Alzheimer's disease beyond stage III based on Hoehn & Yahr stage scale and Global Deterioration Scale respectively), psychiatric disorders (using drugs, psychiatric disorders at least for 3 months, and bipolar disorder), using a medication affecting sleep and respiration (benzodiazepins, Z-drugs, neuroleptics), being allergic to melatonin, pregnancy and lactation, hypothyroidism, narcolepsy, restless leg syndrome, and being a shift worker. Besides, if the tablets were not taken for 4 days, the patient was excluded from the study.

This study was a randomized, double-blind clinical trial, conducted in the sleep clinic of Emam Khomeini Hospital, Tehran, Iran. The forthcoming clinical trial was carried out adhering to the standards of the International Conference on Harmonization of Good Clinical Practice (ICH-GCP), the ethical principles of the Helsinki Declaration 2013, the general guidelines for ethics in human subject research in the country, the ethical guidelines for clinical trials in the country, and other national and university laws and regulations. Informed consent form and all questionnaires provided to all participants has been accepted by Tehran University of medical sciences ethics committee and this committee authorized this study (IR.TUMS.TIPS.REC.1400.177). Registration number IRCT20220105053635N1 was issued by the Iranian Registry of Clinical Trials (IRCT) on 19/04/2022.

After the eligible participants enrolled in the study, they were randomly assigned to the melatonin or placebo group. Random numbers of envelopes were prepared by a separate staff, not involved in assigning and assessment of patients, using an online randomization generator (www.sealedenvelopes.com). Random four blocks were used. Opaque-sealed sequentially numbered envelopes were used for allocation concealment. The envelopes and the tablets were similar in terms of color, shape and size, so all the people including researchers and patients, were blinded through the study.

A questionnaire was filled out at the time of enrollment, providing demographic information as follows: age, weight, height, BMI (Body Mass Index), marital status, education, job, PMH (Past Medical History which includes cardiovascular disorders, kidney diseases, pulmonary disorders, psychiatric disorders, and rheumatologic disorders), using medications associated with sleep disorders, baseline AHI, and duration of PAP therapy.

Insomnia severity was assessed by the ISI scale. ISI is a seven-item questionnaire that assesses nighttime and daytime consequences due to insomnia, during the last two weeks. The scores for each question were added- up to get the final score. The categorization of patients for insomnia was based on the total score. A score below 7 showed no clinically significant insomnia, scores ranged from 8-14, represented subthreshold insomnia, and scores more than 15, were considered as moderate to severe insomnia. The first three questions assess difficulty in falling asleep, maintaining sleep, and problems waking up too early. An ISI score of more than 6 for these first three questions, can probably help diagnose co-morbid insomnia [1].

Sleep quality was assessed by PSQI (Pittsburgh Sleep Quality Index) questionnaire. It is a self-reported instrument for the assessment of sleep quality and sleep disturbance in patients, over a one-month interval. This scale assesses 7 components: a)Total sleep quality, b) Sleep latency, c) sleep duration, d) sleep efficiency, e) sleep disturbance, f) use of sleep medications, and g) next morning dysfunction. The score of each component is ranged from 0 to 3. The total score resulted from adding the score of each component. A total PSQI score of more than 5, was eligible to enroll in the study.

ESS (Epworth Sleepiness Scale) is a sleep test questionnaire that assesses daytime sleepiness, in eight daily situations. The ESS score of less than 7, shows no sleepiness. A score between 8-9 represents the average amount of daytime sleepiness. A score of more than 10, shows excessive daytime sleepiness.

FOSQ-10 (Functional Outcomes of Sleep Questionnaire) includes 10 questions presenting day -time dysfunction, resulting from insomnia in 10 different situations. It shows the effect of poor sleep quality on daily activities, memory, and concentration. The total score is ranged from 10 to 40. The FOSQ-10 score below 15, shows daily activity disturbance in patients. Besides, Higher scores represent better daily functions.

All Persian versions of the questionnaires were checked for validity and reliability [27,28,29,30].

For the assessment of adverse effects, a questionnaire was filled out, which was about the type of adverse effects (Headache, Confusion, Dizziness, Somnolence, Nausea, Fatigue), frequency, date of adverse effects happening, and time of recovery. The causality was assessed by the Naranjo score.

Prior to this, participants were taught the principles of sleep hygiene. Following an initial assessment, eligible patients were asked to complete an informed consent form before being randomly assigned to either the placebo or melatonin group. Patients in the melatonin group received 10 mg melatonin tablets (produced by Jalinous company), 30- 60 minutes before sleep, for one month. Placebo tablets which were quite similar to the melatonin tablets in terms of color, shape, and taste, were given to the patients in the placebo group in the same order.

If a participant experienced poor sleep quality during the study despite receiving treatment appropriately, trazodone tablets were administered at a dose of 25-50mg. This was done to ensure ethical treatment of patients suffering from insomnia. The date, number of pills, and dose of trazodone usage were recorded. Although the data on Trazodone usage in obstructive sleep apnea is limited, studies and reviws showed that trazodone had no significant effect on Sleep Apnea Severity and Architecture include nadir oxygen saturation and duration of respiratory events. Moreover, trazodone at the dose of 100 mg could increase arousal thershold without impairing pharyngeal dilator muscle function and reduces the AHI. That is the resone of choosing trazodone in such patients.

The study's primary outcomes were improvements in sleep quality and daytime sleepiness, as assessed by four questionnaires: the ISI, PSQI, ESS, and FOSQ-10. Participants completed these questionnaires at both the baseline and the end of treatment [31, 32].

The study assessed adherence to PAP devices as a secondary outcome by measuring the number of days and hours patients used the device appropriately. This was defined as using the device for more than 4 hours and on at least 70% of days in the month prior to study enrollment. The data was extracted from reports generated by the PAP devices both at the baseline and at the end of the treatment.

The patients' medication and PAP device adherence, as well as their adherence to sleep hygiene principles, were re-evaluated after 15 days and at the end of the study. The same questionnaires were used to assess sleep quality, insomnia severity, daytime sleepiness, and impaired daily function.

The safety analyses included all patients who received therapy. Common adverse effects such as hypersensitivity reactions, headache, confusion, dizziness, somnolence, nausea and fatigue were evaluated during the study almost every 7 days and one month after the treatment was finished. The Naranjo score was used to evaluate causality for any adverse effects and the form was filled at the end of the treatment period. Patients with severe adverse effects were excluded. Unblinding would be performed for all patients if adverse effects occurred in more than 12.5% of participants (at least 7) or if they were caused by melatonin use. Serious adverse effects resulted in immediate unblinding for both the participants and the researchers.

Since there was lack of previous studies based on Iran in this population, the sample size was calculated according to other studies on comorbid insomnia and respiratory disorders [33], considering the error rate of 5%, power of 80%, α= 0.05 and β= 0.9 and using the following formula, a 50-subject sample size (25 in each group) was consider for this study:

IBM SPSS version 27 was applied for statistical analysis. Kolmogorov-Smirnov and Shapiro–Wilk tests were performed to examine the normality of distribution. Chi-square is performed to find out independency. Mean and standard deviations expressed continuous data and categorical data were represented by numbers and percentages. Independent sample t-test and Mann–Whitney was used to find out the differences between two arms, for normal and non-normal distribution data respectively and P-value less than 0.05 was considered as significant. The effect size was examined by Cohen's. One and two-way ANCOVA tests were performed to eliminate the covariates' effect.