Reading and Writing Files

Reading and writing files is one of the most common tasks in real-world programs—configuration files, logs, CSV exports, and cached data all pass through the filesystem. Go’s standard library, mainly the os and bufio packages, gives you a simple one-call API for small files and a lower-level, streaming API for anything too large or too structured to load all at once. This lesson covers both approaches, how Go represents an open file under the hood, and the mistakes that trip up nearly every new Go developer.

Overview: How File I/O Works in Go

Every open file in Go is represented by a *os.File value from the os package. Under the hood, an *os.File is a thin wrapper around an operating-system file descriptor—an integer handle the kernel uses to track the open file, its current read/write position, and its access mode. Because *os.File implements the io.Reader and io.Writer interfaces (it has Read(p []byte) (int, error) and Write(p []byte) (int, error) methods), it works with the huge ecosystem of functions built around those two interfaces: io.Copy, bufio.NewScanner, encoding/json.NewDecoder, and more all accept any value that satisfies io.Reader or io.Writer. *os.File qualifies automatically—Go interfaces are satisfied implicitly, with no implements keyword, just matching method signatures.

Go gives you two levels of API. The first is whole-file convenience: os.ReadFile and os.WriteFile open, transfer, and close a file in one call, returning the entire contents as a []byte. These are perfect for small-to-moderate files (configs, templates, small data files) because the whole file is loaded into memory at once—reading a multi-gigabyte file this way would exhaust RAM. The second level is streaming: os.Open, os.Create, and os.OpenFile return a live *os.File that you read from or write to incrementally, so memory use stays constant no matter how large the file is.

Talking to the operating system is relatively expensive: each Read or Write call on a raw *os.File is a system call, which involves a costly transition from your program into kernel mode. Reading a file one byte, or one line, at a time this way would be very slow. The bufio package solves this by wrapping a reader or writer in an in-memory buffer: bufio.NewScanner and bufio.NewReader read a large chunk from the OS at once and then serve your program’s smaller reads out of memory, while bufio.NewWriter accumulates your writes in memory and only flushes them to the OS in large batches (or when you explicitly call Flush). This is why almost every real Go program that reads or writes files line-by-line or in small pieces wraps the raw *os.File in a bufio type.

When you create a file with os.Create, os.WriteFile, or os.OpenFile with the os.O_CREATE flag, you also supply a permission value such as 0644. On Unix-like systems this octal number controls read/write/execute bits for the owner, group, and others (0644 means the owner can read and write, everyone else can only read). On Windows the value is mostly ignored in favor of ACLs, but Go still requires you to pass one for portability. os.OpenFile is the most flexible constructor: it takes an explicit combination of flags such as os.O_RDONLY, os.O_WRONLY, os.O_RDWR, os.O_APPEND, os.O_CREATE, os.O_TRUNC, and os.O_EXCL, combined with the bitwise OR operator (|), giving you precise control over how the file is opened—for example, appending to a log file without erasing existing content.

Finally, every file you open must eventually be closed with file.Close(). The operating system limits how many file descriptors a single process may have open at once, so forgetting to close files is a resource leak that can eventually crash a long-running program. The idiomatic pattern is defer file.Close() immediately after a successful open, which guarantees the file is closed when the surrounding function returns, no matter which return path is taken.

Syntax

data, err := os.ReadFile(name string) ([]byte, error)
err := os.WriteFile(name string, data []byte, perm os.FileMode) error

file, err := os.Open(name string) (*os.File, error)          // read-only
file, err := os.Create(name string) (*os.File, error)         // write, truncate/create
file, err := os.OpenFile(name string, flag int, perm os.FileMode) (*os.File, error)

scanner := bufio.NewScanner(r io.Reader) *bufio.Scanner
writer  := bufio.NewWriter(w io.Writer) *bufio.Writer

Function	Purpose
os.ReadFile	Reads an entire file into memory in one call; opens and closes the file for you.
os.WriteFile	Writes a whole []byte to a file in one call, creating it if needed and truncating any existing content.
os.Open	Opens an existing file for reading only. Fails if the file does not exist.
os.Create	Creates a file for writing, truncating it if it already exists (or creating it if it doesn’t).
os.OpenFile	The general-purpose opener; lets you combine flags like append, create, and exclusive.
bufio.NewScanner	Wraps an io.Reader to read input line by line (or by other split functions).
bufio.NewWriter	Wraps an io.Writer to batch small writes; must be Flushed before the underlying file closes.

Examples

Example 1: Writing and reading a whole file

package main

import (
	"fmt"
	"os"
)

func main() {
	content := []byte("Hello, Go file I/O!\n")
	err := os.WriteFile("greeting.txt", content, 0644)
	if err != nil {
		fmt.Println("write error:", err)
		return
	}

	data, err := os.ReadFile("greeting.txt")
	if err != nil {
		fmt.Println("read error:", err)
		return
	}
	fmt.Print(string(data))
}

Output:

Hello, Go file I/O!

os.WriteFile creates greeting.txt (or overwrites it) with the given bytes and permission bits, then closes it. os.ReadFile opens the same file, reads every byte into a []byte, and closes it again. Converting the result with string(data) lets fmt.Print display it as text. Both functions return an error that must be checked—a missing directory or a permissions problem shows up there, not as a panic.

Example 2: Reading a file line by line

package main

import (
	"bufio"
	"fmt"
	"os"
)

func main() {
	err := os.WriteFile("numbers.txt", []byte("one\ntwo\nthree\n"), 0644)
	if err != nil {
		fmt.Println("write error:", err)
		return
	}

	file, err := os.Open("numbers.txt")
	if err != nil {
		fmt.Println("open error:", err)
		return
	}
	defer file.Close()

	scanner := bufio.NewScanner(file)
	lineNum := 1
	for scanner.Scan() {
		fmt.Printf("%d: %s\n", lineNum, scanner.Text())
		lineNum++
	}
	if err := scanner.Err(); err != nil {
		fmt.Println("scan error:", err)
	}
}

Output:

1: one
2: two
3: three

Here the file is opened with os.Open, which returns a live *os.File rather than the whole content. Wrapping it in bufio.NewScanner lets us pull one line at a time with scanner.Scan() and scanner.Text(), without loading the entire file into a single buffer. The defer file.Close() guarantees the file descriptor is released once main returns, and checking scanner.Err() after the loop catches any read error that isn’t simply end-of-file (which stops the loop silently and is not itself an error).

Example 3: Appending with a buffered writer

package main

import (
	"bufio"
	"fmt"
	"os"
)

func main() {
	err := os.WriteFile("log.txt", []byte("startup\n"), 0644)
	if err != nil {
		fmt.Println("write error:", err)
		return
	}

	file, err := os.OpenFile("log.txt", os.O_APPEND|os.O_WRONLY, 0644)
	if err != nil {
		fmt.Println("open error:", err)
		return
	}

	writer := bufio.NewWriter(file)
	for i := 1; i <= 3; i++ {
		fmt.Fprintf(writer, "event %d\n", i)
	}
	if err := writer.Flush(); err != nil {
		fmt.Println("flush error:", err)
	}
	if err := file.Close(); err != nil {
		fmt.Println("close error:", err)
	}

	data, err := os.ReadFile("log.txt")
	if err != nil {
		fmt.Println("read error:", err)
		return
	}
	fmt.Print(string(data))
}

Output:

startup
event 1
event 2
event 3

os.OpenFile is opened with os.O_APPEND|os.O_WRONLY, so every write lands after the existing content instead of overwriting it. The bufio.Writer collects the three fmt.Fprintf calls in memory; nothing reaches the disk until writer.Flush() runs. Only after flushing do we close the file and read it back to confirm the full log, including the original startup line, is present.

How It Works Step by Step

Walking through Example 3 in execution order:

1. os.WriteFile opens log.txt for writing (creating it if needed), writes "startup\n", and closes it—this is a full, self-contained open/write/close cycle.
2. os.OpenFile asks the OS for a new file descriptor on the same file, positioned at the end because of os.O_APPEND; the OS returns a fresh *os.File.
3. bufio.NewWriter allocates an in-memory buffer (4KB by default) and associates it with that *os.File; nothing has touched the disk yet.
4. Each fmt.Fprintf call appends formatted text to the in-memory buffer. If the buffer were to fill up, bufio would automatically flush it to the OS to make room—but for three short lines that never happens here.
5. writer.Flush() explicitly pushes the buffered bytes to the OS via the file descriptor's Write system call. This is the point at which the three event lines actually leave your program's memory.
6. file.Close() releases the file descriptor back to the OS. Closing does not itself flush a bufio.Writer—that already happened in the previous step, which is why the order (Flush then Close) matters.
7. os.ReadFile performs an independent open/read-all/close cycle and returns the full, now four-line, file contents.

Common Mistakes

Mistake 1: Ignoring the error from os.Open

Discarding the error with _ means you never learn whether the open actually succeeded, and a nil *os.File doesn't panic on later use—it just fails quietly.

file, _ := os.Open("data.txt")
scanner := bufio.NewScanner(file)
for scanner.Scan() {
	fmt.Println(scanner.Text())
}
// if data.txt doesn't exist, file is nil, Scan() fails immediately,
// and this loop silently prints nothing -- with no clue why

Check the error and bail out (or handle it) immediately, and always pair a successful open with defer file.Close():

file, err := os.Open("data.txt")
if err != nil {
	fmt.Println("open error:", err)
	return
}
defer file.Close()

scanner := bufio.NewScanner(file)
for scanner.Scan() {
	fmt.Println(scanner.Text())
}

Mistake 2: Forgetting to Flush a buffered writer

A bufio.Writer holds bytes in memory until its buffer fills or you call Flush. Closing the underlying file does not flush it for you, so unflushed data is simply lost.

file, err := os.Create("output.txt")
if err != nil {
	fmt.Println(err)
	return
}
defer file.Close()

writer := bufio.NewWriter(file)
writer.WriteString("important data\n")
// missing writer.Flush(): the buffered bytes are still only in
// memory when file.Close() runs, so output.txt ends up empty

Always flush before the function returns (and check the error, since a flush can fail if the disk is full):

file, err := os.Create("output.txt")
if err != nil {
	fmt.Println(err)
	return
}
defer file.Close()

writer := bufio.NewWriter(file)
writer.WriteString("important data\n")
if err := writer.Flush(); err != nil {
	fmt.Println("flush error:", err)
}

Best Practices

• Always check the error returned by every file operation—os.Open, os.Create, Read, Write, and Close can all fail independently.
• Call defer file.Close() immediately after a successful open so cleanup happens no matter which path the function returns through.
• Use os.ReadFile/os.WriteFile for small files where loading everything into memory is fine; switch to os.Open/os.OpenFile plus bufio for large files or line-oriented processing.
• Always Flush a bufio.Writer before the program relies on the data being on disk, and check the error Flush returns.
• When writing data you care about, also check the error from the explicit file.Close() call—on some filesystems the final flush to disk happens at close time and can fail.
• Use os.OpenFile with explicit flags (os.O_APPEND, os.O_CREATE, os.O_EXCL, and so on) whenever you need behavior beyond the defaults that os.Open or os.Create give you.
• Choose conservative permission bits like 0644 for regular files rather than overly permissive ones like 0777.
• Prefer io.Copy to stream data directly from one io.Reader to an io.Writer (for example, file to file) instead of reading a whole file into a byte slice just to write it out again.

Practice Exercises

• Write a program that reads a text file line by line and prints how many lines, words, and characters it contains, similar to the Unix wc command. Hint: use bufio.NewScanner for lines and strings.Fields to count words per line.
• Write a program that copies the contents of one file to another using os.Open, os.Create, and io.Copy, making sure to check every error and close both files.
• Write a program that appends a new line to a log file each time it runs, creating the file if it does not already exist. Hint: combine os.O_APPEND, os.O_CREATE, and os.O_WRONLY in a single call to os.OpenFile.

Summary

• *os.File wraps an OS file descriptor and implements io.Reader and io.Writer, so it works with the entire io/bufio ecosystem.
• os.ReadFile and os.WriteFile are one-call, whole-file convenience functions best suited to small files.
• os.Open, os.Create, and os.OpenFile return a live *os.File for streaming reads and writes without loading everything into memory.
• bufio.Scanner and bufio.Writer batch small reads and writes to avoid making a system call for every line or byte.
• A bufio.Writer must be explicitly Flushed—closing the file alone does not push buffered bytes to disk.
• os.OpenFile flags like os.O_APPEND, os.O_CREATE, and os.O_TRUNC, combined with |, give precise control over how a file is opened.
• Always check every returned error and pair a successful open with defer file.Close() to avoid descriptor leaks.