3.3. Tools

Joedb comes with a collection of command-line tools.

3.3.1. `joedbi`

usage: joedbi <file> [<blob_file>]

<file> is one of:
 [file] [--<open_mode>] <file_name>
 <open_mode> is one of:
  read
  write
  new
  write_or_new (default)
  lock
 memory
 interpreted [--read] <file_name>
 sftp [--port p] [--verbosity v] <user> <host> <file_name>
 curl [--verbose] <URL>
 brotli [--read] <file_name>

joedbi is the joedb interpreter. It has commands to let you directly modify a file, and visualize the content of tables.

Below is a list of commands the interpreter understands:

General commands
~~~~~~~~~~~~~~~~
 about
 help
 quit
 abort
 echo on|off
 prompt on|off

Journal
~~~~~~~
 timestamp [<stamp>] (if no value is given, use current time)
 comment "<comment_string>"
 valid_data
 flush
 checkpoint
 write_blob <data_string>

Data definition
~~~~~~~~~~~~~~~
 create_table <table_name>
 drop_table <table_name>
 rename_table <old_table_name> <new_table_name>
 add_field <table_name> <field_name> <type> [default <value>]
 drop_field <table_name> <field_name>
 rename_field <table_name> <old_field_name> <new_field_name>
 custom <custom_name>

 <type> may be:
  string,
  blob,
  int8, int16, int32, int64,
  float32, float64,
  boolean,
  references <table_name>

Data manipulation
~~~~~~~~~~~~~~~~~
 insert_into <table_name> <record_id>
 insert_vector <table_name> <record_id> <size>
 update <table_name> <record_id> <field_name> <value>
 update_vector <table_name> <record_id> <field_name> <N> <v_1> ... <v_N>
 delete_from <table_name> <record_id>

Displaying data
~~~~~~~~~~~~~~~
 table <table_name> [<max_column_width>] [start] [length]
 record <table_name> [<record_id>]
 table_size <table_name>
 schema
 dump
 sql
 json [<base64>]

3.3.2. `joedbc`

Usage: joedbc <file.joedbi> <file.joedbc>

joedbc is the joedb compiler. It takes two file names as parameters. The first file should contain joedbi commands to create the database schema. The second file contains compiler options.

The joedbc file must at least contain a namespace option that indicates the namespace in which the code will be generated. It can also be used to define indexes.

3.3.3. `joedb_logdump`

usage: joedb_logdump [--sql] [--sqlite] [--raw] [--header] [--schema-only] [--ignore-errors] [--load] [--print-checkpoint] [--blob] <file.joedb>

joedb_logdump takes a joedb file name as parameter, and produces a sequence of joedbi commands. With the --sql option, it can produce SQL output. This way, joedb data can be easily imported into any system that understands SQL.

For instance, this is the SQL output of the tutorial database:

-- Automatic schema upgrade
CREATE TABLE "city"("__id" INTEGER PRIMARY KEY);
ALTER TABLE "city" ADD "name" TEXT;
CREATE TABLE "person"("__id" INTEGER PRIMARY KEY);
ALTER TABLE "person" ADD "first_name" TEXT;
ALTER TABLE "person" ADD "last_name" TEXT;
ALTER TABLE "person" ADD "home" INTEGER REFERENCES "city";
-- valid data
INSERT INTO "city"("__id") VALUES(1);
UPDATE "city" SET "name" = X'546f6b796f' WHERE "__id" = 1;
INSERT INTO "city"("__id") VALUES(2);
UPDATE "city" SET "name" = X'4e657720596f726b' WHERE "__id" = 2;
INSERT INTO "city"("__id") VALUES(3);
UPDATE "city" SET "name" = X'5061726973' WHERE "__id" = 3;
INSERT INTO "city"("__id") VALUES(4);
UPDATE "city" SET "name" = X'4c696c6c65' WHERE "__id" = 4;
INSERT INTO "city"("__id") VALUES(5);
UPDATE "city" SET "name" = X'416d7374657264616d' WHERE "__id" = 5;
INSERT INTO "person"("__id") VALUES(1);
UPDATE "person" SET "first_name" = X'52c3a96d69' WHERE "__id" = 1;
UPDATE "person" SET "last_name" = X'436f756c6f6d' WHERE "__id" = 1;
UPDATE "person" SET "home" = 4 WHERE "__id" = 1;
INSERT INTO "person"("__id") VALUES(2);
UPDATE "person" SET "first_name" = X'4265727472616e64' WHERE "__id" = 2;
UPDATE "person" SET "last_name" = X'506963617264' WHERE "__id" = 2;
UPDATE "person" SET "home" = NULL WHERE "__id" = 2;
INSERT INTO "person"("__id") VALUES(3);
UPDATE "person" SET "first_name" = X'4172697374696465' WHERE "__id" = 3;
UPDATE "person" SET "last_name" = X'4d617274696e6573' WHERE "__id" = 3;
UPDATE "person" SET "home" = 5 WHERE "__id" = 3;
UPDATE "person" SET "last_name" = X'4d617274696e657a' WHERE "__id" = 3;
DELETE FROM "city" WHERE "__id" = 2;
-- 2024-11-05 20:06:56 GMT
-- The End

3.3.4. `joedb_pack`

usage: joedb_pack [--ignore-errors] [--checkpoint N] <input.joedb> <output.joedb> 

Packing a file removes all its history, and keeps only the most recent data.

In order to support schema recognition (see Schema Upgrade), data-definition commands are not packed. They are left as-is, at the beginning of the log, in the same order as in the original file.

3.3.5. `joedb_convert`

usage: joedb_convert [--ignore-errors] [--checkpoint N] <input.joedb> <output.joedb> 

This copies the input to the output. --ignore-error sets the checkpoint value to the file length, and can be used to recover a damaged file.

In case the joedb file format ever changes in a way that is not compatible with the previous version, then this tool can be used to perform the conversion. The new format is first implemented in write functions. At this moment, joedb_convert is still able to read the old format, and writes the new format. This happened in the early days of joedb, but is not likely to happen again in the future.

3.3.6. `joedb_merge`

usage: joedb_merge <db_1.joedb> ... <db_N.joedb> <output.joedb>
or read file names from input stream: joedb_merge <output.joedb> <file_list.txt
Note: output file must not already exist

joedb_merge merges multiple files with the same schema into a single file that contains the concatenation of all tables. References are translated. Duplicates are not eliminated.

For instance, when merging those two databases:

{
 "city":
 {
  "__size": 2,
  "name": ["Lille", "Paris"]
 },
 "person":
 {
  "__size": 2,
  "first_name": ["Rémi", "Bill"],
  "last_name": ["Coulom", "Jordan"],
  "home": [0, 1]
 }
}

{
 "city":
 {
  "__size": 2,
  "name": ["Lille", "Barcelona"]
 },
 "person":
 {
  "__size": 2,
  "first_name": ["Albert", "Aristide"],
  "last_name": ["Dupont", "Martinez"],
  "home": [0, 1]
 }
}

joedb_merge produces this result:

{
 "city":
 {
  "__size": 4,
  "name": ["Lille", "Paris", "Lille", "Barcelona"]
 },
 "person":
 {
  "__size": 4,
  "first_name": ["Rémi", "Bill", "Albert", "Aristide"],
  "last_name": ["Coulom", "Jordan", "Dupont", "Martinez"],
  "home": [0, 1, 2, 3]
 }
}

3.3.7. `joedb_server`

usage: joedb_server [--port p] [--timeout t] [--share] [--nocheck] <file> <connection>

<file> is one of:
 [file] [--<open_mode>] <file_name>
 <open_mode> is one of:
  read
  write
  new
  write_or_new (default)
  shared
  lock
 memory
 interpreted [--read] <file_name>
 sftp [--port p] [--verbosity v] <user> <host> <file_name>
 curl [--verbose] <URL>
 brotli [--read] <file_name>

<connection> is one of:
 [dummy] (default) 
 dump [tail]
 sql [tail]
 file <file>
 network <host> <port>
 ssh <user> <host> <joedb_port> [<ssh_port> [<ssh_log_level>]]

The timeout is the time (in seconds) during which a client lock is kept.
0 (the default) means there is no timeout, and the lock is kept until the
client unlocks or is disconnected. A client that timed out is not disconnected,
and can still push data: the push will succeed only if there is no conflict.

Run a server to share a single database. See Concurrency for more information.

3.3.8. `joedb_multi_server`

usage: joedb_multi_server <config.joedbi>

Run a server to share multiple databases. The config file contains joedbi commands to set server parameters. For instance:

create_table server
add_field server file_name string
add_field server port int32

insert_into server 1 "test_1.joedb" 2001
insert_into server 2 "test_2.joedb" 2002

add_field server timeout int32 = 0

3.3.9. `joedb_client`

usage: joedb_client [--nocheck] <file> <connection>

<file> is one of:
 [file] [--<open_mode>] <file_name>
 <open_mode> is one of:
  read
  write
  new
  write_or_new
  shared (default)
  lock
 memory
 interpreted [--read] <file_name>
 sftp [--port p] [--verbosity v] <user> <host> <file_name>
 curl [--verbose] <URL>
 brotli [--read] <file_name>

<connection> is one of:
 [dummy] (default) 
 dump [tail]
 sql [tail]
 file <file>
 network <host> <port>
 ssh <user> <host> <joedb_port> [<ssh_port> [<ssh_log_level>]]

3.3.10. `joedb_push`

usage: joedb_push [--follow] <file> <connection>

<file> is one of:
 [file] <file_name>
 interpreted [--read] <file_name>
 sftp [--port p] [--verbosity v] <user> <host> <file_name>
 curl [--verbose] <URL>
 brotli <file_name>

<connection> is one of:
 [dump] (default) [tail]
 sql [tail]
 file <file>
 network <host> <port>
 ssh <user> <host> <joedb_port> [<ssh_port> [<ssh_log_level>]]

Pushes a local file to a connection. For example:

# Dump content of a file
joedb_push file.joedb dump

# Truncate a file containing an aborted transaction
joedb_push broken.joedb file fixed.joedb

# Follow additions to the end of a file
joedb_push --follow file.joedb tail

# Keep a backup server updated
joedb_push --follow database.joedb network backup_server 1234

3.3.11. `joedb_embed`

usage: joedb_embed [--base64|raw|utf8|ascii] <file_name.joedb> <namespace> <identifier>
output: <namespace>_<identifer>.{h,cpp}

joedb_embed compiles a joedb database file into a C++ string literal, and a function to open it as a Database.

3.3.12. `joedb_to_json`

joedb_to_json takes a joedb file name as parameter, and produces json output. Each column is represented as a vector, and references are indexes into the vector (-1 indicates the null reference). The --base64 option encodes strings in base64. Joedb considers a string as a vector of bytes that may take any value, but json strings are limited to UTF-8. So --base64 might be necessary for binary data or other special characters.

This is the json output of the tutorial database:

{
 "city":
 {
  "__size": 4,
  "name": ["Tokyo", "Paris", "Lille", "Amsterdam"]
 },
 "person":
 {
  "__size": 3,
  "first_name": ["Rémi", "Bertrand", "Aristide"],
  "last_name": ["Coulom", "Picard", "Martinez"],
  "home": [2, -1, 3]
 }
}

3.3.13. `joedb_edit`

joedb_edit is a bash script that dumps a joedb file to a temporary text file and opens the default text editor to modify it. Modifications will be pushed to the original if possible. Otherwise the user is given the option to overwrite the original file.

3.3.14. `joedb_browser`

joedb_browser is a bash script that produces an SQLite database, and invokes sqlitebrowser to browse it.

3.3. Tools

3.3.1. joedbi

3.3.2. joedbc

3.3.3. joedb_logdump

3.3.4. joedb_pack

3.3.5. joedb_convert

3.3.6. joedb_merge

3.3.7. joedb_server

3.3.8. joedb_multi_server

3.3.9. joedb_client

3.3.10. joedb_push

3.3.11. joedb_embed

3.3.12. joedb_to_json

3.3.13. joedb_edit

3.3.14. joedb_browser