Background: While speech analysis holds promise for mental health assessment, research often focuses on single symptoms, despite symptom co-occurrences and interactions. In addition, predictive models in Mental Health do not properly assess speech-based systems' limitations, such as uncertainty, or fairness for a safe clinical deployment. Objective: We investigated the predictive potential of mobile-collected speech data for detecting and estimating depression, anxiety, fatigue, and insomnia, focusing beyond mere accuracy, in the general population. Methods: We included n=435 healthy adults and recorded their answers concerning their perceived mental and sleep states. We asked them how they felt and if they had slept well lately. Clinically validated questionnaires measured depression, anxiety, insomnia, and fatigue severity. We developed a novel speech and machine learning pipeline involving voice activity detection, feature extraction, and model training. We detected voice activity automatically with a bidirectional neural network and examined participants' speech with a fully ML automatic pipeline to capture speech variability. Then, we modelled speech with a ThinResNet model that was pre-trained on a large open free database. Based on this speech modelling, we evaluated clinical threshold detection, individual score prediction, model uncertainty estimation, and performance fairness across demographics (age, sex, education). We employed a train-validation-test split for all evaluations: to develop our models, select the best ones and assess the generalizability of held-out data. Results: Our methods achieved high detection performance for all symptoms, particularly depression (PHQ-9 AP=0.77, BDI AP=0.83), insomnia (AIS AP=0.86), and fatigue (MFI Total Score AP=0.88). These strengths were maintained while ensuring high abstention rates for uncertain cases (Risk-Coverage AUCs < 0.1). Individual symptom scores were predicted with good accuracy (Correlations were all significant, with Pearson strengths between 0.59 and 0.74). Fairness analysis revealed that models were consistent for sex (average Disparity Ratio (DR) = 0.77), to a lesser extent for education level (average Disparity Ratio (DR) = 0.44) and worse for age groups (average Disparity Ratio (DR) = 0.26). Conclusions: This study demonstrates the potential of speech-based systems for multifaceted mental health assessment in the general population, not only for detecting clinical thresholds but also for estimating their severity. Addressing fairness and incorporating uncertainty estimation with selective classification are key contributions that can enhance the clinical utility and responsible implementation of such systems. This approach offers promise for more accurate and nuanced mental health assessments, potentially benefiting both patients and clinicians.

RR, XN, AL, MdG, MD, and AB are shareholders of Callyope and VO was an employee of Callyope.

This study was funded by Callyope.

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

After fully understanding the explanation of the study, all participants signed a consent form to participate in the study, in line with the Declaration of Helsinki, current Good Clinical Practice guidelines, and local laws and regulations. All procedures were approved by the French National Institutional Review Board, (identifier 23.00748.OOO2L7#I for the Committee for the Protection of Persons).

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes

The participants of this study did not give written consent for their data to be shared publicly, so due to the sensitive nature of the research supporting data is not available.