1. Introduction

Joedb is a light-weight C++ relational database. Joedb keeps tabular data in memory, and writes a journal to a file. The whole data history is stored, so it is possible to re-create any past state of the database. Joedb has a network protocol, and can operate in a distributed fashion, a bit like git for structured data.

Joedb comes with a compiler that takes a database schema as input, and produces C++ code. The generated C++ data-manipulation code is convenient to use, efficient, and type-safe.

_images/joedb.svg

1.1. Pros and Cons

Joedb offers many nice features that may make it more attractive than typical alternatives such as Protocol Buffers, FlatBuffers, SQLite, XML, or JSON:

  • Unlike XML or JSON, joedb is a binary file format that does not require any parsing. So, joedb files are much smaller, and processing data is much faster. Joedb also comes with a text format that can be easily read or modified by humans, if necessary.

  • Unlike Protocol Buffers or FlatBuffers, joedb works like a database: it can incrementally update data stored on disk in a crash-safe way, and can handle concurrent connections of multiple clients to a single database.

  • The whole data history is stored. So, no old data can ever be lost. It is also possible to add time stamps and comments to the journal, and use it as a log of the application (if the history has to be forgotten for privacy or disk-space reasons, it is also possible to pack it).

  • If the database schema of an application changes over time, joedb can upgrade old files to the new version automatically. The upgrade includes changes to the schema as well as custom data manipulation (see Schema Upgrade).

  • The database schema is compiled into C++ code that allows convenient type-safe data manipulation. Many errors that would be detected at run time with SQL, XML, or JSON will be detected at compile time instead.

  • Joedb is very simple, light, and fast.

Joedb currently has some limitations that may be removed with future improvements:

  • The database is stored in memory. So it must be small enough to fit in RAM, and the full journal has to be replayed from scratch when opening a file. This may change with support of on-disk data storage. Also, blobs allow manipulating databases that are much bigger than available RAM.

  • C++ is the only supported programming language.

Compared to history-less databases, joedb has one fundamental drawback: frequently-updated values may make the joedb journal file grow very large.

So joedb might not be the best choice for every situation, but it is great if data fits in RAM, has to be stored safely on disk, and is manipulated by C++ code.

1.2. An Example

A simple example of how to use joedb is available in the doc/source/tutorial directory. The database schema is defined in the tutorial.joedbi file:

create_table city
add_field city name string
create_table person
add_field person first_name string
add_field person last_name string
add_field person home references city

Compiler instructions are in tutorial.joedbc:

namespace tutorial
create_unique_index city_by_name city name
create_index person_by_name person last_name,first_name

This tutorial database can be compiled into C++ source code with joedbc:

joedbc tutorial.joedbi tutorial.joedbc

This will produce various source files in the tutorial directory, that can be used to manipulate data conveniently in C++, as shown in the tutorial_main.cpp source file:

#include "tutorial/File_Database.h"
#include "joedb/io/main_exception_catcher.h"

#include <iostream>

/////////////////////////////////////////////////////////////////////////////
static int tutorial_main(int argc, char **argv)
/////////////////////////////////////////////////////////////////////////////
{
 //
 // Open the database
 //
 tutorial::File_Database db("tutorial.joedb");

 //
 // Simple data manipulation
 //
 db.new_city("Tokyo");
 db.new_city("New York");
 db.new_city("Paris");

 const auto Lille = db.new_city("Lille");
 const auto Amsterdam = db.new_city("Amsterdam");

 db.new_person("Rémi", "Coulom", Lille);
 db.new_person("Bertrand", "Picard", db.null_city());

 const auto Aristide = db.new_person("Aristide", "Martines", Amsterdam);

 db.set_last_name(Aristide, "Martinez");

 //
 // Use the index to display cities in alphabetical order
 //
 std::cout << "List of cities in alphabetical order:\n";
 for (auto city: db.get_index_of_city_by_name())
  std::cout << "  " << db.get_name(city.second) << '\n';

 //
 // Referring to another table
 //
 std::cout << "\nList of persons with their cities:\n";
 for (const auto person: db.get_person_table())
 {
  std::cout << "  " << db.get_first_name(person) << ' ';
  std::cout << db.get_last_name(person) << ' ';
  const auto city = db.get_home(person);
  if (city.is_null())
   std::cout << "is homeless\n";
  else
   std::cout << "lives in " << db.get_name(city) << '\n';
 }

 //
 // Deleting a record
 //
 db.delete_city(db.find_city_by_name("New York"));

 //
 // Time stamp and comment
 //
 db.write_timestamp();
 db.write_comment("The End");

 //
 // Writes to the database must be confirmed by an explicit checkpoint
 //
 db.checkpoint();

 return 0;
}

/////////////////////////////////////////////////////////////////////////////
int main(int argc, char **argv)
/////////////////////////////////////////////////////////////////////////////
{
 joedb::main_exception_catcher(tutorial_main, argc, argv);
}

Running the resulting program will produce this output:

List of cities in alphabetical order:
  Amsterdam
  Lille
  New York
  Paris
  Tokyo

List of persons with their cities:
  Rémi Coulom lives in Lille
  Bertrand Picard is homeless
  Aristide Martinez lives in Amsterdam

All the data was stored in the tutorial.joedb file. The database file is a binary file, so it is not convenient to inspect it directly. The joedb_logdump tool will produce a readable log:

comment "Automatic schema upgrade"
create_table city
add_field city name string
create_table person
add_field person first_name string
add_field person last_name string
add_field person home references city
valid_data
insert_into city 1
update city 1 name "Tokyo"
insert_into city 2
update city 2 name "New York"
insert_into city 3
update city 3 name "Paris"
insert_into city 4
update city 4 name "Lille"
insert_into city 5
update city 5 name "Amsterdam"
insert_into person 1
update person 1 first_name "Rémi"
update person 1 last_name "Coulom"
update person 1 home 4
insert_into person 2
update person 2 first_name "Bertrand"
update person 2 last_name "Picard"
update person 2 home 0
insert_into person 3
update person 3 first_name "Aristide"
update person 3 last_name "Martines"
update person 3 home 5
update person 3 last_name "Martinez"
delete_from city 2
timestamp 1730837216 2024-11-05 20:06:56 GMT
comment "The End"