# MutliGPU Training using split\_dataset\_per\_node with PyTorch Lightning

**URL:** <https://discuss.huggingface.co/t/mutligpu-training-using-split-dataset-per-node-with-pytorch-lightning/49402>\
**Category:** 🤗Datasets\
**Created:** [August 3, 2023, 10:47pm UTC](https://discuss.huggingface.co/t/mutligpu-training-using-split-dataset-per-node-with-pytorch-lightning/49402 "2023-08-03T22:47:09Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![mwalmsley](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mwalmsley/32/27314_2.png) [@mwalmsley](https://discuss.huggingface.co/u/mwalmsley)\
**Post date:** [May 24, 2024, 1:21pm UTC](https://discuss.huggingface.co/t/mutligpu-training-using-split-dataset-per-node-with-pytorch-lightning/49402/2 "2024-05-24T13:21:42Z")

</div>

So I’m not a pro here, but maybe relevant:

- webdataset solves the issue of distributing across nodes with a dataset(…, nodesplitter\_func=nodesplitter\_func) approach. This nodesplitter\_func can be anything but typically does torch.distributed.get\_rank etc, like you tried
- You can/should place the dataset() function inside a datamodule
- The distributed environment exists by the time datamodule.setup() is called, and functions called inside the datamodule get\_dataloader functions can read rank/world
- nodesplitter\_func is called when each worker is set up, but that’s okay as it returns the same list for workers on a given node (being deterministic)

I think the same thing might work for split\_dataset\_by\_node here. You would need to split the dataset inside webdatamodule and after webdatamodule.setup(), and then the distributed environment might be set up for you.

---

_[View the full topic](https://discuss.huggingface.co/t/mutligpu-training-using-split-dataset-per-node-with-pytorch-lightning/49402)._