Files¶

Most of the time this will do:

infile = open("file.in", "rb")
outfile = open("file.out", "wb")
outfile.write(infile.read())

However for huge files you may want to do the reads/writes in pieces (or you may have to), and if you dig deeper you may find other technical problems.

Unfortunately, there’s no totally platform independent answer. On Unix, you can use os.system() to invoke the “cp” command (see your Unix manual for how it’s invoked). On DOS or Windows, use os.system() to invoke the “COPY” command. On the Mac, use macostools.copy(srcpath, dstpath). It will also copy the resource fork and Finder info.

There’s also the shutil module which contains a copyfile() function that implements the copy loop; but in Python 1.4 and earlier it opens files in text mode, and even in Python 1.5 it still isn’t good enough for the Macintosh: it doesn’t copy the resource fork and Finder info.

<<<

Note ~~~~ This should be fixed for modern Python on [[Mac OS X]], but needs to be confirmed (especially due to the way the OS now stores metadata). <<<

To delete (remove, unlink) a file, use os.remove(filename) or os.unlink(filename)

For documentation, see the posix section of the library manual. They are the same: unlink() is simply the Unix name for this function.

To remove a directory os.rmdir()

To create a directory os.mkdir()

To rename a file os.rename()

To truncate a file, open it using

f = open(filename, "r+")

, and use f.truncate(offset)

(The offset defaults to the current seek position. The ** “r+”** mode opens the file for reading and writing.)

There’s also os.ftruncate(fd, offset) for files opened with os.open() for advanced Unix hacks only.

The terminal has to be put into cbreak mode, in order to be able to read single characters from the terminal.

import sys
def getch():
  import os, tty
  fd = sys.stdin.fileno()
  tty_mode = tty.tcgetattr(fd)
  tty.setcbreak(fd)
  try:
    ch = os.read(fd, 1)
  finally:
    tty.tcsetattr(fd, tty.TCSAFLUSH, tty_mode)
  return ch

If you need to automatically save and restore objects and other data structures to files, the pickle library module solves this in a very general way (though you still can’t store things like open files, sockets or windows), and the library module shelve uses pickle and (g)dbm to create persistent mappings containing arbitrary Python objects. For possibly better performance, use the cPickle module.

A more awkward way of doing things is to use pickle‘s little sister, marshal. The marshal module provides very fast ways to store noncircular basic Python types to files and strings, and back again. Although marshal does not do fancy things like store instances or handle shared references properly, it does run extremely fast. For example loading a half megabyte of data may take less than a third of a second (on some machines). This often beats doing something more complex and general such as using gdbm with pickle/shelve.

You want to read through a file line by line. This is the most common way of processing text files. For example, if you’re writing a Python program that searches for text in a file, you’ll have to loop through the file contents on a line by line basis.

You can code the loop explicitly, like this:

file = open('/tmp/filename', 'r')
while True:
   line = file.readline()
   if line == "": break    # Check for end-of-file
   do_something(line)
file.close()

Or like this, which reads all of the lines of the file into a list:

file = open('/tmp/filename', 'r')
lines = file.readlines()
file.close()
for line in lines:
   do_something(line)

The fileinput module makes this even simpler by handling the loop for you:

import fileinput
for line in fileinput.input("/tmp/filename"):
   do_something(line)

For sheer simplicity, it’s hard to top reading all the lines from the file into a list, like this:

file = open('/tmp/filename', 'r')
lines = file.readlines()
file.close()
for line in lines:
   do_something(line)

Or like this:

file = open('/tmp/filename', 'r')
for line in file.readlines()
   do_something(line)

This is easy to code, but it does require reading the entire file into memory. These days, most systems will have enough memory to effortlessly handle files a few hundreds of kilobytes long. A 10 megabyte file will cause problems - swapping to disk, if not an actual crash - for many systems. Use your common sense; if you’re pretty sure you won’t need to handle large files, use file.readlines(); otherwise, use either of the two suggested solutions.

To loop over multiple files using fileinput, use a sequence of filenames (a list, tuple, etc) instead of just a single file.

   for filename in ("file1", "file2", "file3"):

      file  = open(filename, 'r')
      lines = file.readlines()
      file.close()

      for line in lines:
          do_something(line)





If you omit the names, they default to **sys.argv[1:]**, or to standard input

if no arguments were given.

You can use the sizehint parameter to readlines() to get some of the efficiency of readlines() without reading in the whole file. The following code isn’t optimized, but it shows the idea:

class BufferedFileReader:
  def __init__ (self, file):
    self.file = file
    self.lines = []
    self.numlines = 0
    self.index = 0

def readline (self):
  if (self.index >= self.numlines):
    self.lines = self.file.readlines(65536)
  self.numlines = len(self.lines)
  self.index = 0

  if (self.numlines == 0):
    return ""

  str = self.lines[self.index]
  self.index = self.index + 1
  return str

Use the struct module. It allows you to take a string read from a file containing binary data (usually numbers) and convert it to Python objects; and vice versa.

For example, the following code reads two 4-byte integers in big-endian format from a file:

 import struct
f = open(filename, "rb")  # Open in binary mode for portability
s = f.read(8)
x, y = struct.unpack(">ll", s)

The ‘>’ in the format string forces big-endian data; each letter ‘l’ reads one “long integer” (4 bytes) from the string.

You can refer to the Library Reference for more details about the struct module.

There is a problem that routinely bedevils new Python users who come from a Windows or DOS background. The problem is that on Windows, a backslash is a separator in filenames, whereas on Unix (where Python originated) the backslash has an entirely different function (as an escape character) and the //forward// slash is the filename separator character.

When a person from a Windows background starts having problems with filenames, and goes looking for a solution, the first thing that he (or she) usually finds is Python’s “raw” strings. However, from the perspective of a Windows user, raw strings aren’t truly raw they are only semi-raw, as you will rudely discover the first time that you use a raw string to specify a filename (or filename part) that ends with a single backslash.

A better strategy is to routinely code your filenames using forward slashes, and then use a Python function such as os.path.normcase to change the separator to whatever is appropriate to the local operating system. A nice bonus of ths strategy is that it is platform independent

import os.path
myFilename = "c:/mydir/myfile.txt"
myFilename = os.path.normcase(myFilename)