# Train\_from\_iterator throwing TypeError: expected string or buffer errir

**URL:** <https://discuss.huggingface.co/t/train-from-iterator-throwing-typeerror-expected-string-or-buffer-errir/133871>\
**Category:** 🤗Tokenizers\
**Created:** [January 1, 2025, 7:09pm UTC](https://discuss.huggingface.co/t/train-from-iterator-throwing-typeerror-expected-string-or-buffer-errir/133871 "2025-01-01T19:09:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kevin1807](https://avatars.discourse-cdn.com/v4/letter/k/71c47a/32.png) [@Kevin1807](https://discuss.huggingface.co/u/Kevin1807)\
**Post date:** [January 1, 2025, 7:09pm UTC](https://discuss.huggingface.co/t/train-from-iterator-throwing-typeerror-expected-string-or-buffer-errir/133871/1 "2025-01-01T19:09:22Z")

</div>

im not able to figure out what im doing wrong here

```auto
class IngenxPreTokenizer:
    def __init__ (self):
        super(). __init__ ()
        self.base_tokenizer = IngenxTokenizer()
    
    def pre_tokenize(self, pretok: PreTokenizedString):
        processed = self.base_tokenizer.process_text(pretok)
        normalized_tokens = []
        current_offset = 0
        for token in processed:
            token_len = len(token)
            normalized_tokens.append((
                token,
                (current_offset, current_offset + token_len)
            ))
            current_offset += token_len + 1
            
        pretok.tokens = normalized_tokens
        return pretok
    

class IngenxTokenTrainer:
    def __init__ (self,df,size_dataset =240340,vocab_size=150000,min_freq = 5,batch_size=1000):
        self.tokenizer = IngenxTokenizer()
        self.df = df
        self.size_dataset = size_dataset
        self.vocab_size = vocab_size
        self.min_freq = min_freq
        self.batch_size=1000
        self.special_tokens = ["<|unk|>","<|pad|>","</|eos|>",
                               "<|var|>","</|var|>","<|val|>","<|val|>",
                               "<|func|>","<|func|>","<|op|>","</|op|>"
                              ]
        self.training_corpus = self.preprare_dataset()
        
    def preprare_dataset(self):
        X2 = np.random.choice(len(self.df), size=self.size_dataset, replace=False)
        examples = []
        for i in X2:
            # Convert to string and handle None/NaN values
            problem = str(self.df.iloc[i]['problem']) if pd.notna(self.df.iloc[i]['problem']) else ""
            solution = str(self.df.iloc[i]['solution']) if pd.notna(self.df.iloc[i]['solution']) else ""
            example = f"{problem} {solution}".strip()
            examples.append(example)
        return examples
        
    def get_training_corpus(self):
        dataset = self.training_corpus
        with tqdm(total=len(dataset), desc="Processing training corpus", unit="text") as pbar:
            for text in dataset:
                pbar.update(1)
                yield text
                
                
    def train_tokenizer(self):
        tokenizer = Tokenizer(BPE())
        tokenizer.pre_tokenizer = PreTokenizer.custom(IngenxPreTokenizer())

        trainer = BpeTrainer(
            vocab_size=self.vocab_size,
            min_frequency=self.min_freq,
            special_tokens=self.special_tokens
        )
        tokenizer.train_from_iterator(self.get_training_corpus(),trainer=trainer, length=len(self.training_corpus))
        tokenizer.save("ingenx_tokenizewr.json")
        return tokenizer

```

error

```auto
Exception Traceback (most recent call last)
<ipython-input-47-3f931020c7fd> in <cell line: 1>()
----> 1 a.train_tokenizer()

<ipython-input-44-3011da9bf75a> in train_tokenizer(self)
     41 special_tokens=self.special_tokens
     42 )
---> 43 tokenizer.train_from_iterator(self.get_training_corpus(),trainer=trainer, length=len(self.training_corpus))
     44 tokenizer.save("ingenx_tokenizewr.json")
     45 return tokenizer

Exception: TypeError: expected string or buffer

```

---

<div class="post-metadata">

**Author:** ![mahmutc](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mahmutc/32/52583_2.png) [@mahmutc](https://discuss.huggingface.co/u/mahmutc)\
**Post date:** [January 2, 2025, 6:46pm UTC](https://discuss.huggingface.co/t/train-from-iterator-throwing-typeerror-expected-string-or-buffer-errir/133871/2 "2025-01-02T18:46:17Z")

</div>

> [@Kevin1807](#):
>
> ```auto
> def get_training_corpus(self):
> dataset = self.training_corpus
> with tqdm(total=len(dataset), desc="Processing training corpus", unit="text") as pbar:
> for text in dataset:
> 
> ```

Could you please ensure that the dataset is not empty?

---

<div class="post-metadata">

**Author:** ![Kevin1807](https://avatars.discourse-cdn.com/v4/letter/k/71c47a/32.png) [@Kevin1807](https://discuss.huggingface.co/u/Kevin1807)\
**Post date:** [January 3, 2025, 2:18pm UTC](https://discuss.huggingface.co/t/train-from-iterator-throwing-typeerror-expected-string-or-buffer-errir/133871/3 "2025-01-03T14:18:41Z")

</div>

when i use Whitespace as a PreTokenizer it work just fine but when i try to do same with my custom PreTokenizer it throw’s that error

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/5/4/54ef5b8dc1fc4379480b2b15c334b934a1192d13.png)