Skip to main content

Running the Demo Project

In order to run the demo project, you will need to complete the setup steps and populate the db with test data.

1. Generate Test Data​

To generate test data run these commands

cd test_data_generator
python3 generate_e2e_test_data.py

That script will create several CSV files, containing sample model data (e.g. Account.csv, Contract.csv, etc). Move these into the import directory specified by IMPORT_DATA_VOLUME_PATH.

2. Open Airflow​

Once the tool is running and you have test data in the import directory, you can run the DAGs. Open the dashboard (http://localhost:8080/) and Select Airflow from the Developer Links dropdown at the top of the page. The Airflow dashboard is how you will run the tool. In Airflow a sequence of tasks is called a DAG. You should see several DAGs already in Airflow. We have included numbers in the dag names to help you run them in the right order.

3. Running the Setup DAG​

Click "01_setup". This will open the setup DAG. You will see a list of steps on the left and Dag details in the main page. You can select the "Graph" tab to easily view what steps are dependent on other steps completing.

Click the play button in the top right to trigger the setup DAG. You should see a column of little boxes appear next to the steps on the left side and a green bar above those boxes. Click on that bar and go to the graph tab. Each of the little boxes on the left and the steps on the graph have a color to indicate their status. Each of the steps should eventually turn dark green for success.

You can click on one of the tasks either in the graph view or in the little colored boxes to the left and then select the "Logs" tab at the top. This can be useful for diagnosing errors with a step in a DAG.

4. Running the Import And Conform to V12 DAG​

Once the setup DAG has run successfully it's time to start loading data. Normally we would load the data currently in salesforce into the tool first but for this example we will skip this step. Click the "DAGs" link next to the Airflow logo to go back to the DAGs overview. You can trigger the "02_import_and_conform_source1_to_v12" DAG directly from the overview screen by clicking the "Trigger DAG" button in the Actions column. Open the DAG by clicking its name and seeing what tasks are failing. The tasks ending in "_dbt_test" should fail because the example data has some quality issues but all other tasks should succeed. The little colored boxes on the left each represent one run of a task so if you run the DAG multiple times there will be a column of colored boxes.

In this example the "02_import_and_conform_source1_to_v12" DAG simply loads the test data CSVs into the database but this step would also be used to transform the source data into the V12 model during migration.