Background Melanoma is an aggressive form of skin cancer in which tumor-infiltrating lymphocytes (TILs) are a biomarker for recurrence and treatment response. Manual TIL assessment is prone to interobserver variability, and current deep learning models are not publicly accessible or have low performance. Deep learning models, however, have the potential of consistent spatial evaluation of TILs and other immune cell subsets with the potential of improved prognostic and predictive value. To make the development of these models possible, we created the Panoptic Segmentation of nUclei and tissue in advanced MelanomA (PUMA) dataset and assessed the performance of several state-of-the-art deep learning models. In addition, we show how to improve model performance further by using heuristic post-processing in which nuclei classes are updated based on their tissue localization. Results The PUMA dataset includes 155 primary and 155 metastatic melanoma H&E stained regions of interest with nuclei and tissue annotations from a single melanoma referral institution. The Hover-NeXt model, trained on the PUMA dataset, demonstrated the best performance for lymphocyte detection, approaching human interobserver agreement. In addition, heuristic post-processing of deep learning models improve the detection of non-common classes, such as epithelial nuclei. Conclusion The PUMA dataset is the first melanoma specific dataset that can be used to develop melanoma-specific nuclei and tissue segmentation models. These models can, in turn, be used for prognostic and predictive biomarker development. Incorporating tissue and nuclei segmentation is a step towards improved deep learning nuclei segmentation performance. We will use this dataset to organize the PUMA challenge in which the goal is to further improve model performance.

Karijn P.M. Suijkerbuijk reports a consulting/advisory relationship with Abbvie and Sairopa. She received honoraria from Bristol Myers Squibb and research funding from TigaTx, Bristol Myers Squibb, Philips, Genmab and Pierre Fabre. All paid to institution The remaining authors of this manuscript have no conflicts of interest to disclose.

This research was funded by an unrestricted grant of Stichting Hanarth Fonds, The Netherlands.

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

The Biobank Research Ethics Committee (TCBio) UMC Utrecht confirms that it has reviewed the release file in accordance with the UMC Utrecht Biobank Regulations and all other applicable regulations and laws. Based on the requirements as defined in these regulations and laws, the TCBio UMC Utrecht hereby issues an approval of the aforementioned dataset (reference number TCBio 23-270/U-B).

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes