Skip to content

Nessie export/import

Functionality to export a Nessie repository and import into another Nessie repository, allowing to migration from one backend database to another.

Usage

Nessie repository export + import requires direct access to the database used by Nessie. The necessary executable is the nessie-quarkus-cli-x.y.z-runner.jar can be downloaded from the release page on GitHub and is available for Nessie 0.43.0 or newer.

The Nessie Quarkus CLI tool nessie-quarkus-cli-x.y.z-runner.jar should use the same configuration settings as the Nessie Quarkus server.

Exporting

The following command (replace x.y.z with the version you’re using) exports your Nessie repository to a single ZIP file called my-export-file.zip,

java -jar nessie-quarkus-cli-x.y.z-runner.jar export --path my-export-file.zip

A ZIP file export contains all necessary repository information in a single, compressed file. Note that the export will only automatically generate a ZIP file, if the output path ends with .zip, otherwise it will export to a directory. You can force either option using the --output-format option.

Note

Please use the following command for advanced options. 

java -jar nessie-quarkus-cli-x.y.z-runner.jar help export

Importing

The following command (replace x.y.z with the version you’re using) imports your Nessie repository from a single ZIP file called my-export-file.zip,

java -jar nessie-quarkus-cli-x.y.z-runner.jar import --path my-export-file.zip

The import will fail, if the target Nessie repository exists and is not empty. If you intentionally want to overwrite an existing Nessie repository, then use the --erase-before-import option.

Note

Please use the following command for advanced options. 

java -jar nessie-quarkus-cli-x.y.z-runner.jar help import

Building blocks

  • Export functionality, based on AbstractNessieExporter to dump commits, named references, heads+fork points.
  • Import functionality, based on AbstractNessieImporter to load the exported data.
  • Commit log optimization to:
  • populate the list of parent-commits in all commits, according to the target Nessie repository’s configuration
  • populate the key-lists in the commits, according to the target Nessie repository’s configuration

Code examples

class CodeExamples {

  void exportExample(DatabaseAdapter databaseAdapter, Path exportZipFile) {

    ZipArchiveExporter.builder()
      .outputFile(exportZipFile)
      .databaseAdapter(databaseAdapter)
      .build()
      .exportNessieRepository();
  }

  void importExample(DatabaseAdapter databaseAdapter, Path importZipFile) {

    ImportResult importResult =
      ZipArchiveImporter.builder()
        .sourceZipFile(importZipFile)
        .databaseAdapter(databaseAdapter)
        .build()
        .importNessieRepository();

    CommitLogOptimization.builder()
      .headsAndForks(importResult.headsAndForkPoints())
      .databaseAdapter(databaseAdapter)
      .build()
      .optimize();
  }
}

ZipArchiveImporter can be replaced with FileImporter.

Export contents

Each export contains this information:

  • All commits (no specific order)
  • All named references including their heads
  • Heads + fork-points (used to feed commit-log optimization ran after a repository import)
  • Summary and inventory

Content, CommitMeta, global state, et al

A Nessie export contains all Content information without any database internal information. This means that there is no information contained whether the source repository stored Content using e.g. global state. All Contents and CommitMeta are exported in their public JSON representation.

As a side effect, an export from a Nessie repository with commits that were persisted using global state will be imported using on-reference-state. However, for content that was persisted using global state, there will multiple on-reference-states referring to the same Iceberg table-metadata.

Technical commit information

Exported commits do not contain key-lists or commit-parents or the like, because that is rather internal, implementation specific information and, additionally, the configuration of the target repository that controls the aggregated key-lists and commit-parent-lists might be different from the source repository.

However, exported commits do contain information about the commit-sequence-number and the technical created-at-timestamp.

Note

The nessie-quarkus-cli tool’s import command performs a commit-log optimization after all commits and named references have been created. This optimization populates missing aggregated key-lists and commit-parents. Running commit-log optimization is necessary for good performance to access contents and commit logs, but not strictly necessary. Commit-log optimization can be disabled.

Export contents consistency

Any Nessie export guarantees that the commits referenced by the named references and all their parent commits are contained in the exported data.

A Nessie export may contain unreferenced commits, for example commits that have been created while the export is running or commits that are otherwise unreferenced.

The HEADs of the named references and the heads in the HeadsAndForks structure may not be consistent, for example when commits have been created while the export is running.

Export formats

Exported data can be written either into an empty directory or as a compressed zip file.

Users can optionally zip the contents of an export to a directory and pass that to the zip-file based importer.