Skip to main content

Projects

At the core of Datacrafter is a project: a directory that is the single source of truth for how data is extracted, transformed, and loaded.

A project is ordinary files. You can store it in Git.

Initialize one with datacrafter init:

datacrafter init my-datacrafter-project
cd my-datacrafter-project

--path / --name still work (datacrafter init --path DIR --name NAME). If you pass a directory, that folder is created and project-name defaults to its basename.

datacrafter.yml​

A project must contain datacrafter.yml. Required properties:

  • version — currently "1"
  • project-name — human-readable name
  • project-id — unique id (generated by init)
  • extractor or extractors: — how data is obtained (required before datacrafter run / check succeed)

Optional blocks: processor, destination.

See the configuration reference for the full schema.

Project directories​

Typical layout after init and a successful run:

my-project/
├── datacrafter.yml      # Pipeline configuration
├── current/             # Files produced by the extractor
├── output/              # Processor destination files
├── temp/                # Working files
├── storage/             # APIBackuper config and similar
├── state.json           # Last run stages
└── datacrafter.log      # Execution log

There is no .datacrafter SQLite system database and no datacrafter elt command. Use run, status, and log.

Next steps​