Flat files (GDBM manual)

13 Export and Import

GDBM databases can be converted into so-called flat format files. Such files cannot be used for searching, their sole purpose is to keep the data from the database for restoring it when the need arrives. There are two flat file formats, which differ in the way they represent the data and in the amount of meta-information stored. Both formats can be used, for example, to migrate between the different versions of GDBM databases. Generally speaking, flat files are safe to send over the network, and can be used to recreate the database on another machine. The recreated database is guaranteed to have the same format and contain the same set of key/value pairs as the database from which the flat file was created. However, it will not constitute a byte-to-byte equivalent of the latter. Various internal structures in the database can differ. In particular, ordering of key/value pairs can be different and the table of available file space will most probably differ, too. For databases in extended format, the numsync counter will be reset to 0 (see Numsync). These details are not visible to the application programmer, and are mentioned here only for completeness sake.

The fact that the restored database contains the same set of key/value pairs does not necessarily mean, however, that it can be used in the same way as the original one. For example, if the original database contained non-ASCII data (e.g. C structures, integers etc.), the recreated database can be of any use only if the target machine has the same integer size and byte ordering as the source one and if its C compiler uses the same packing conventions as the one which generated C which populated the original database. In general, such binary databases are not portable between machines, unless you follow some stringent rules on what data is written to them and how it is interpreted.

GDBM version 1.24 supports two flat file formats. The binary flat file format was first implemented in version 1.9.1. This format stores only key/data pairs, it does not keep information about the database file itself. As its name implies, files in this format are binary files. This format is supported for backward compatibility.

The ascii flat file format encodes all data in Base64 and stores not only key/data pairs, but also the original database file metadata, such as file name, mode and ownership. Files in this format can be sent without additional encapsulation over transmission channels that normally allow only ASCII data, such as, e.g. SMTP. Due to additional metadata they allow for restoring an exact copy of the database, including file ownership and privileges, which is especially important if the database in question contained some security-related data.

We call a process of creating a flat file from a database exporting or dumping this database. The reverse process, creating the database from a flat file is called importing or loading the database.

gdbm interface: int gdbm_dump (GDBM_FILE dbf, const char *filename, int format, int open_flags, int mode)

Dumps the database file to the named file in requested format. Arguments are:

dbf

A pointer to the source database, returned by a prior call to gdbm_open.

filename

Name of the dump file.

format

Output file format. Allowed values are: GDBM_DUMP_FMT_BINARY to create a binary dump and GDBM_DUMP_FMT_ASCII to create an ASCII dump file.

open_flags

How to create the output file. If flag is GDBM_WRCREAT the file will be created if it does not exist. If it does exist, the gdbm_dump will fail.

If flag is GDBM_NEWDB, the function will create a new output file, replacing it if it already exists.

mode

The permissions to use when creating the output file (see open a file in open(2) man page).

gdbm interface: int gdbm_load (GDBM_FILE *pdbf, const char *filename, int flag, int meta_mask, unsigned long *errline)

Loads data from the dump file filename into the database pointed to by pdbf. The latter can point to NULL, in which case the function will try to open an existing database, if flag is GDBM_REPLACE, or create a new one, if flag is GDBM_INSERT or if the database file does not exit. If it succeeds, the function will return, in the memory location pointed to by pdbf, a pointer to the newly created database. If the dump file carries no information about the original database file name, the function will set gdbm_errno to GDBM_NO_DBNAME and return -1, indicating failure.

The flag has the same meaning as the flag argument to the gdbm_store function (see Store).

The meta_mask argument can be used to disable restoring certain bits of file’s meta-data from the information in the input dump file. It is a binary OR of zero or more of the following:

GDBM_META_MASK_MODE: Do not restore file mode.
GDBM_META_MASK_OWNER: Do not restore file owner.

The function returns 0 upon successful completion or -1 on fatal errors and 1 on mild (non-fatal) errors.

If a fatal error occurs, gdbm_errno will be set to one of the following values:

GDBM_FILE_OPEN_ERROR

Input file (filename) cannot be opened. The errno variable can be used to get more detail about the failure.

GDBM_MALLOC_ERROR

Not enough memory to load data.

GDBM_FILE_READ_ERROR

Reading from filename failed. The errno variable can be used to get more detail about the failure.

GDBM_MALFORMED_DATA

GDBM_ILLEGAL_DATA

Input contained malformed data, i.e. it is not a valid GDBM dump file. This often means that the dump file got corrupted during the transfer.

The GDBM_ILLEGAL_DATA is an alias for this error code, maintained for backward compatibility.

GDBM_ITEM_NOT_FOUND

This error can occur only when the input file is in ASCII format. It indicates that the data part of the record about to be read lacked length specification. Application developers are advised to treat this error equally as GDBM_MALFORMED_DATA.

Mild errors mean that the function was able to successfully load and restore the data, but was unable to change the database file metadata afterwards. The table below lists possible values for gdbm_errno in this case. To get more detail, inspect the system errno variable.

GDBM_ERR_FILE_OWNER: The function was unable to restore database file owner.
GDBM_ERR_FILE_MODE: The function was unable to restore database file mode (permission bits).

If an error occurs while loading data from an input file in ASCII format, the number of line in which the error occurred will be stored in the location pointed to by the errline parameter, unless it is NULL.

If the line information is not available or applicable, errline will be set to 0.

gdbm interface: int gdbm_dump_to_file (GDBM_FILE dbf, FILE *fp, int format)

This is an alternative entry point to gdbm_dump (which see). Arguments are:

dbf: A pointer to the source database, returned by a call to gdbm_open.
fp: File to write the data to.
format: Format of the dump file. See the format argument to the gdbm_dump function.

gdbm interface: int gdbm_load_from_file_ext (GDBM_FILE *pdbf, FILE *fp, int flags, int replace, int meta_mask, unsigned long *line)

This is an alternative entry point to gdbm_load. It reads the dump from fp, which must be a file open for reading. The flags argument gives flags for gdbm_open call (see Open). It is used if pdbf points to NULL, i.e. the database has not been open yet. The rest of arguments is the same as for gdbm_load.

The function returns 0 on success. On error it returns -1 and sets gdbm_errno to one of error codes (see Error codes). In particular, GDBM_ERR_USAGE is returned if:

pdbf is NULL.
fp is NULL.
lower 7 bits of flags have the value GDBM_READER.

gdbm interface: int gdbm_load_from_file (GDBM_FILE *pdbf, FILE *fp, int replace, int meta_mask, unsigned long *line)

Yet another entry point to gdbm_load. It is equivalent to

gdbm_load_from_file (*pdbf, *fp,
                     replace ? GDBM_WRCREAT : GDBM_NEWDB,
                     replace, meta_mask, line);

gdbm interface: int gdbm_export (GDBM_FILE dbf, const char *exportfile, int flag, int mode)

This function is retained for compatibility with GDBM 1.10 and earlier. It dumps the database to a file in binary dump format and is equivalent to

gdbm_dump(dbf, exportfile, GDBM_DUMP_FMT_BINARY, flag, mode)

gdbm interface: int gdbm_export_to_file (GDBM_FILE dbf, FILE *fp): This is an alternative entry point to gdbm_export. This function writes to file fp a binary dump of the database dbf.

gdbm interface: int gdbm_import (GDBM_FILE dbf, const char *importfile, int flag)

This function is retained for compatibility with GDBM 1.10 and earlier. It loads the file importfile, which must be a binary flat file, into the database dbf and is equivalent to the following construct:

dbf = gdbm_open (importfile, 0,
                       flag == GDBM_REPLACE ?
                         GDBM_WRCREAT : GDBM_NEWDB,
                       0600, NULL);
gdbm_load (&dbf, exportfile, 0, flag, NULL)

gdbm interface: int gdbm_import_from_file (GDBM_FILE dbf, FILE *fp, int flag)

An alternative entry point to gdbm_import. Reads the binary dump from the file fp and stores the key/value pairs to dbf. See Store, for a description of flag.

This function is equivalent to:

dbf = gdbm_open (importfile, 0,
                       flag == GDBM_REPLACE ?
                         GDBM_WRCREAT : GDBM_NEWDB,
                       0600, NULL);
gdbm_load_from_file (dbf, fp, flag, 0, NULL);