Loading Documents

A RAG application can only answer questions about data it can read. Document loaders are the first step of that pipeline. They read data from a source (a text file, a PDF, a CSV, a web page, a database) and convert it into a standard Python object called a Document. In this lesson you will learn what a Document is, how the common loaders work, how to load lots of data without running out of memory, and how to write your own loader.

Overview: how loading works

Every data source has its own format. A PDF stores text in pages, a CSV stores rows, a web page stores HTML. If your splitter, embedding model and vector store had to understand all of these, your code would become messy. LangChain solves this with one common shape: the Document.

A Document (from langchain_core.documents) has two main fields:

• page_content: a string with the text.
• metadata: a dictionary with facts about the text, such as the file path (source), the page number, a row number or a URL.

A loader is a class that knows how to read one kind of source and produce a list of Documents. All loaders follow the same interface (BaseLoader in langchain_core.document_loaders):

• load() reads everything and returns a list[Document].
• lazy_load() returns an iterator that yields one Document at a time, so memory use stays small.
• alazy_load() is the async version of lazy loading.

Most ready-made loaders live in the langchain_community.document_loaders package. Some come from separate partner packages. Because every loader returns the same type, the rest of your pipeline (splitting, embedding, indexing) does not care where the data came from.

Why metadata matters

Metadata travels with the text through the whole pipeline. When a chunk is retrieved later, the metadata tells you which file and page it came from. This lets you show citations, filter results (for example only documents from one department) and debug wrong answers. Treat metadata as a first-class part of loading, not an afterthought.

Loading is not splitting

A loader usually returns big pieces: a whole text file is one Document, a PDF gives one Document per page. These are often too large for an embedding model or an LLM prompt. Splitting them into smaller chunks is the next lesson. Loaders only get the data in.

Syntax and key concepts

from langchain_community.document_loaders import SomeLoader

loader = SomeLoader(source, **options)
docs = loader.load()            # list of Document
for doc in loader.lazy_load():  # one Document at a time
    ...

• Constructor: takes the source (path, URL, folder) and options such as encoding.
• load(): convenient for small data; everything is held in memory.
• lazy_load(): preferred for big data; process and discard each Document.

These are the loaders you will meet most often:

Install what you need. Package names and extras can change, so check the LangChain documentation if an import fails.

pip install langchain-core langchain-community pypdf beautifulsoup4

None of the examples below need an API key or internet access. Loading is purely local work; keys only matter later, when you call an embedding model or an LLM. For that stage you can use a fake embedding class for offline practice.

Examples

Example 1: Creating a Document by hand

Before using loaders, build a Document yourself. This is also how you wrap data that no loader supports.

from langchain_core.documents import Document

doc = Document(
    page_content='LangChain loads data into Documents.',
    metadata={'source': 'notes.txt', 'page': 1},
)
print(type(doc).__name__)
print(doc.page_content)
print(doc.metadata)

Output:

Document
LangChain loads data into Documents.
{'source': 'notes.txt', 'page': 1}

The metadata dictionary is free-form. You can add any keys you like, such as author or department, as long as the values are simple types.

Example 2: TextLoader

This script creates a temporary file, loads it and inspects the result.

import os
import tempfile

from langchain_community.document_loaders import TextLoader

with tempfile.TemporaryDirectory() as tmp:
    path = os.path.join(tmp, 'welcome.txt')
    with open(path, 'w', encoding='utf-8') as f:
        f.write('Welcome to the RAG course.\nLoaders turn files into Documents.')

    docs = TextLoader(path, encoding='utf-8').load()
    print(len(docs))
    print(repr(docs[0].page_content))
    print(os.path.basename(docs[0].metadata['source']))

Output:

1
'Welcome to the RAG course.\nLoaders turn files into Documents.'
welcome.txt

TextLoader returns exactly one Document for the whole file, and sets metadata['source'] to the path. We pass encoding='utf-8' explicitly so the result does not depend on the operating system default.

Example 3: CSVLoader

For tabular data, each row becomes its own Document. That is useful for FAQ lists, product catalogues or support tickets.

import csv
import os
import tempfile

from langchain_community.document_loaders import CSVLoader

with tempfile.TemporaryDirectory() as tmp:
    path = os.path.join(tmp, 'team.csv')
    with open(path, 'w', encoding='utf-8', newline='') as f:
        writer = csv.writer(f)
        writer.writerow(['name', 'role'])
        writer.writerow(['Asha', 'Engineer'])
        writer.writerow(['Ravi', 'Designer'])

    docs = CSVLoader(file_path=path, encoding='utf-8').load()
    print(len(docs))
    print(docs[1].page_content)
    print(docs[1].metadata['row'])

Output:

2
name: Ravi
role: Designer
1

The header row is not a Document. Instead, each data row is written as column: value lines, and the metadata stores the zero-based row number. You can use the source_column option to take the source metadata from a column (for example a URL column) instead of the file path.

Example 4: DirectoryLoader with lazy loading

Real projects have folders of files. DirectoryLoader finds files with a glob pattern and uses an inner loader class for each one.

import os
import tempfile

from langchain_community.document_loaders import DirectoryLoader, TextLoader

with tempfile.TemporaryDirectory() as tmp:
    files = [('a.txt', 'Alpha'), ('b.txt', 'Beta'), ('skip.md', 'Ignored')]
    for name, text in files:
        with open(os.path.join(tmp, name), 'w', encoding='utf-8') as f:
            f.write(text)

    loader = DirectoryLoader(
        tmp,
        glob='**/*.txt',
        loader_cls=TextLoader,
        loader_kwargs={'encoding': 'utf-8'},
    )
    names = sorted(os.path.basename(d.metadata['source']) for d in loader.lazy_load())
    print(names)

Output:

['a.txt', 'b.txt']

The glob **/*.txt matches text files in the folder and all sub-folders, so skip.md is ignored. The order in which files are found is not guaranteed, which is why we sort the names before printing. Using lazy_load() means each file is read only when the loop asks for it.

Example 5: Writing a custom loader

If no loader fits (an internal API, a proprietary format), subclass BaseLoader and implement lazy_load(). You get load() for free.

from typing import Iterator

from langchain_core.document_loaders import BaseLoader
from langchain_core.documents import Document


class LineLoader(BaseLoader):
    def __init__(self, text: str, source: str) -> None:
        self.text = text
        self.source = source

    def lazy_load(self) -> Iterator[Document]:
        for number, line in enumerate(self.text.splitlines(), start=1):
            if line.strip():
                yield Document(
                    page_content=line.strip(),
                    metadata={'source': self.source, 'line': number},
                )


loader = LineLoader('First fact\n\nSecond fact', 'inline')
for doc in loader.load():
    print(doc.metadata['line'], doc.page_content)

Output:

1 First fact
3 Second fact

The blank second line is skipped, but the line numbers still count it, so the second fact keeps its true position (3). Good metadata points back to the original data exactly.

Under the hood: what happens on load()

1. You create the loader. Usually nothing is read yet; the constructor only stores the settings.
2. You call load(). In BaseLoader, this simply runs list(self.lazy_load()).
3. lazy_load() opens the source, parses it and yields Document objects one by one.
4. Your code receives the Documents and passes them to a text splitter.

So lazy_load() is the real implementation, and load() is a convenience wrapper. Because lazy loading is a Python generator, you can chain it with splitting and indexing so that only a small amount of data is in memory at any moment.

Other loaders: PDFs and web pages

These need an extra package (and, for web pages, network access), so we show them as snippets. Replace the placeholder path and URL with your own.

from langchain_community.document_loaders import PyPDFLoader, WebBaseLoader

pdf_docs = PyPDFLoader('reports/annual.pdf').load()
print(pdf_docs[0].metadata)   # contains 'source' and 'page'

web_docs = WebBaseLoader('https://example.com/').load()
print(web_docs[0].metadata)   # contains 'source' and often 'title'

The PDF loader gives one Document per page and records the page number, which is perfect for citations. The exact metadata keys can differ between loader versions, so print them once and check.

Common Mistakes

Mistake 1: Forgetting the encoding

docs = TextLoader('notes_in_hindi.txt').load()

Without an explicit encoding, the loader may use the system default and fail with a decode error on non-English text. Fix it with TextLoader('notes_in_hindi.txt', encoding='utf-8').load(). If you do not know the encoding, the loader has an option to auto-detect it, but it is slower and not perfect.

Mistake 2: Using load() on a huge folder

docs = DirectoryLoader('big_archive/', glob='**/*.pdf', loader_cls=PyPDFLoader).load()

This reads every page of every PDF into memory before you do anything. For large archives, iterate with lazy_load(), split and index in batches. Another problem here: one corrupt file can make the whole call fail, so wrap work in error handling or use the loader’s options for silently skipping errors (check your version’s documentation).

Mistake 3: Missing the optional dependency

from langchain_community.document_loaders import PyPDFLoader
PyPDFLoader('a.pdf').load()   # ImportError: pypdf is not installed

Many loaders only wrap a third-party library. Read the error message and install the named package, for example with pip install pypdf.

Mistake 4: Ignoring empty or noisy content

Scanned PDFs without OCR return empty text, and web pages include menus and footers. If you embed these, your index is filled with useless chunks. Always check len(doc.page_content) on a few samples and filter out empty Documents before indexing.

Best Practices

• Always inspect a few loaded Documents (page_content and metadata) before building the rest of the pipeline.
• Pass encoding='utf-8' explicitly to text-based loaders.
• Prefer lazy_load() for anything bigger than a few megabytes.
• Enrich metadata at load time: add fields like department, language or updated_at so you can filter in retrieval later.
• Keep source accurate. Citations in your final answers depend on it.
• Filter empty or duplicate Documents before they reach the vector store.
• Choose loaders by file type with a glob pattern per type, rather than one pattern for everything.
• Do not hard-code secrets in loaders for APIs or databases; read tokens from environment variables.
• Check the current LangChain documentation for the exact options of each loader, as they are updated often.

Practice Exercises

1. Metadata enrichment. Write a script that loads two temporary text files with TextLoader and adds a category key (for example 'faq') to each Document’s metadata. Print the final metadata of both Documents. Hint: doc.metadata['category'] = 'faq'.
2. Filter empty rows. Create a CSV with three rows, where one row has empty values. Load it with CSVLoader and print how many Documents have a page_content that contains real values. Decide how you would drop the useless one.
3. Custom loader. Write a BaseLoader subclass that takes a Python dictionary of {title: body} and yields one Document per entry, with the title stored in metadata['title']. Test it with load().

Summary

• A Document has page_content (text) and metadata (a dictionary), and is the common format for the whole RAG pipeline.
• Loaders convert a source into Documents; they share the load(), lazy_load() and alazy_load() interface.
• TextLoader gives one Document per file, CSVLoader one per row, PyPDFLoader one per page.
• DirectoryLoader combines a glob pattern with an inner loader class to read many files.
• load() wraps lazy_load(); use lazy loading for large data.
• You can write a custom loader by subclassing BaseLoader and implementing lazy_load().
• Good metadata and clean content at load time make retrieval, filtering and citations much easier later.