q02/dfg-cp-survey-pool
Watch
11
0
Fork
You've already forked dfg-cp-survey-pool
 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
dfg-cp-survey-pool/README.md

6 KiB
Raw Permalink Blame History

Data and code for the DFG-CP-SURVEY tool, 2026 edition

See also:

Summary of content

code

Scripts to convert existing data into structures that comply with the latest DFG-CP-SURVEY schema, see q02/concepts.trr379.de#9

config

Sample config for the dump-things-service to be deployed for the 2026 DFG-CP-SURVEY. Needs updates in order to be functional. Was tested successfully locally. The important requirement is that it should be fed with the updated DFG-CP-Survey schema, see q02/concepts.trr379.de#9.

data

Outputs from several API calls and subsequent script usage. These data records are to be fed into the public collection of the new dumpthings deployment.

Preparing the environment

uv venv --python 3.14
source .venv/bin/activate
uv pip install dump-things-pyclient

Preparing the data

The following was done for several classes in order to generate the JSONlines files in data:

uv run datalad run -m "Grab <class> records from 2025 dumpthings api" -o "data/<filename>.jsonl" "uv run dtc get-records --class <class> https://dfg-cp-survey.trr379.de/api 2025-public | uv run code/convert_records.py --delete-key annotations > data/<filename>.jsonl"

This code:

  • fetches all records of a specific class from the public collection of the 2025 deployment
  • removes the annotations field
  • populates the output file

It was done for classes:

  • City
  • ConsortiumProject
  • Country
  • Gender
  • Role
  • StatusGroup

Records for Discipline were taken from hub.psychoinformatics.de/msz/lehr-und-forschungsbereiche@bb8a63866b

Records for Property were added manually.

POSTing data to the new deployment

From data

Since we already have the JSON objects in the data directory, we can pipe a stream of lines into a command that POSTs these records to the correct new API endpoint of the dumpthings deployment. We want all of these records to go into the curated area of the public collection (or 2026-public if configured in the same pattern as the previous year), which means we need a curator token (most probably the one configured in the dumpthings config). And we also need to know the base URL of the dumpthings API:

export CURATOR_TOKEN=<insert-curator-token-here>
export DUMPTHINGS_API=<insert-dumpthings-api-url-here> # e.g. 'https://dfg-cp-survey.trr379.de/api'

Now we use dtc to POST the records, e.g. the Propertys:

cat data/properties.jsonl | uv run dtc --token $CURATOR_TOKEN post-records --curated $DUMPTHINGS_API 2026-public Property

This can be done following the same pattern for all other class records that need to go into the public collection.

DSCOrganizations

Since there are many thousands of organization records, it's probably best to GET and then POST records directly. Here we also need to remove specific keys:

  • annotations, same as for other classes
  • identifiers, because the list contains instances of IssuedIdentifier which is not a thing anymore, and the identifiers themselves do not seem to carry any specific weight when it comes to UI/UX functionality of the survey. The organizations' PIDs stay the same, which is what counts (especially for linkage to existing Contributor records, or linkage across pools e.g. to the TRR379 pool)
uv run dtc get-records --class DSCOrganization https://dfg-cp-survey.trr379.de/api 2025-public | uv run convert_records.py --delete-key annotations --delete-key identifiers | uv run dtc --token $CURATOR_TOKEN post-records --curated $DUMPTHINGS_API public DSCOrganization

Contributor records

Contributor records are at the core of this process. Existing Contributor records exist, and they are linked to specific people in the CRC based on email address. The new 2026 survey uses Forgejo-based authentication, and its integration with dumpthings means that a user authenticating via forgejo (where their account is also linked to an email address) will submit data into (and read existing data from) a forgejo-related incoming area. We need to make these incoming areas exist, with the existing data, in order for users to be able to see their existing data in the new deployment.

As an example, I (Stephan Heunis) submitted a Contributor record for the previous survey that went into my token-specific inbox in the previous dumpthings deployment. My data is known by an administrator to be backed up at a path along the likes of: ./2025/usr-sheunisfzjuelichde/Contributor/../<some-hash>.yaml. So we need to take this information, and information about my forgejo account (also linked to my email address), predict the correct incoming area label for my data, and then put my Contributor record in this correct incoming area in the new deployment.

This is done as follows:

  • get a list of all yaml files, i.e. paths to Contributor records
  • pass these into code/convert_contributors, which outputs the inbox label and the record, tab-separated
  • read these values into a curl command that POSTs the record to the specific inbox
find ./2025/usr-* -type f \( -name '*.yaml' -o -name '*.yml' \) |
  uv run code/convert_contributors.py |
  while IFS=$'\t' read -r inbox_label record; do
    printf '%s' "$record" |
      curl -X POST \
        "${DUMPTHINGS_API}/2026-protected/incoming/${inbox_label}/record/Contributor" \
        -H "Content-Type: application/json" \
        -H "X-DumpThings-Token: ${CURATOR_TOKEN}" \
        --data-binary @-
  done

The reason for using curl vs dtc is that the latter does not (yet) support using post-records for adding data to specific user inboxes.