112 lines
6 KiB
Markdown
112 lines
6 KiB
Markdown
# Data and code for the DFG-CP-SURVEY tool, 2026 edition
|
|
|
|
See also:
|
|
- UI updates:
|
|
- https://hub.trr379.de/q02/dfg-cp-survey-ui/issues/19
|
|
- https://hub.trr379.de/q02/dfg-cp-survey-ui/pulls/20
|
|
- Schema updates:
|
|
- https://hub.trr379.de/q02/concepts.trr379.de/issues/10
|
|
- https://hub.trr379.de/q02/concepts.trr379.de/pulls/9
|
|
- New disciplines: https://hub.psychoinformatics.de/msz/lehr-und-forschungsbereiche
|
|
|
|
## Summary of content
|
|
|
|
### `code`
|
|
|
|
Scripts to convert existing data into structures that comply with the latest DFG-CP-SURVEY schema, see https://hub.trr379.de/q02/concepts.trr379.de/pulls/9
|
|
|
|
### `config`
|
|
|
|
Sample config for the `dump-things-service` to be deployed for the 2026 DFG-CP-SURVEY. Needs updates in order to be functional. Was tested successfully locally. The important requirement is that it should be fed with the updated DFG-CP-Survey schema, see https://hub.trr379.de/q02/concepts.trr379.de/pulls/9.
|
|
|
|
### `data`
|
|
|
|
Outputs from several API calls and subsequent script usage. These data records are to be fed into the public collection of the new dumpthings deployment.
|
|
|
|
## Preparing the environment
|
|
|
|
```
|
|
uv venv --python 3.14
|
|
source .venv/bin/activate
|
|
uv pip install dump-things-pyclient
|
|
```
|
|
|
|
## Preparing the data
|
|
|
|
The following was done for several classes in order to generate the JSONlines files in `data`:
|
|
|
|
```
|
|
uv run datalad run -m "Grab <class> records from 2025 dumpthings api" -o "data/<filename>.jsonl" "uv run dtc get-records --class <class> https://dfg-cp-survey.trr379.de/api 2025-public | uv run code/convert_records.py --delete-key annotations > data/<filename>.jsonl"
|
|
```
|
|
|
|
This code:
|
|
- fetches all records of a specific class from the public collection of the 2025 deployment
|
|
- removes the `annotations` field
|
|
- populates the output file
|
|
|
|
It was done for classes:
|
|
- `City`
|
|
- `ConsortiumProject`
|
|
- `Country`
|
|
- `Gender`
|
|
- `Role`
|
|
- `StatusGroup`
|
|
|
|
Records for `Discipline` were taken from https://hub.psychoinformatics.de/msz/lehr-und-forschungsbereiche/commit/bb8a63866b55bca2cb9ab6873eddd038e924e9a6
|
|
|
|
Records for `Property` were added manually.
|
|
|
|
## POSTing data to the new deployment
|
|
|
|
### From `data`
|
|
|
|
Since we already have the JSON objects in the `data` directory, we can pipe a stream of lines into a command that POSTs these records to the correct new API endpoint of the dumpthings deployment. We want all of these records to go into the curated area of the public collection (or `2026-public` if configured in the same pattern as the previous year), which means we need a curator token (most probably the one configured in the [dumpthings config](./config/.dumpthings.yaml)). And we also need to know the base URL of the dumpthings API:
|
|
|
|
```
|
|
export CURATOR_TOKEN=<insert-curator-token-here>
|
|
export DUMPTHINGS_API=<insert-dumpthings-api-url-here> # e.g. 'https://dfg-cp-survey.trr379.de/api'
|
|
```
|
|
|
|
Now we use `dtc` to POST the records, e.g. the `Property`s:
|
|
|
|
```
|
|
cat data/properties.jsonl | uv run dtc --token $CURATOR_TOKEN post-records --curated $DUMPTHINGS_API 2026-public Property
|
|
```
|
|
|
|
This can be done following the same pattern for all other class records that need to go into the public collection.
|
|
|
|
### `DSCOrganization`s
|
|
|
|
Since there are many thousands of organization records, it's probably best to GET and then POST records directly. Here we also need to remove specific keys:
|
|
- `annotations`, same as for other classes
|
|
- `identifiers`, because the list contains instances of `IssuedIdentifier` which is not a thing anymore, and the identifiers themselves do not seem to carry any specific weight when it comes to UI/UX functionality of the survey. The organizations' PIDs stay the same, which is what counts (especially for linkage to existing `Contributor` records, or linkage across pools e.g. to the TRR379 pool)
|
|
|
|
```
|
|
uv run dtc get-records --class DSCOrganization https://dfg-cp-survey.trr379.de/api 2025-public | uv run convert_records.py --delete-key annotations --delete-key identifiers | uv run dtc --token $CURATOR_TOKEN post-records --curated $DUMPTHINGS_API public DSCOrganization
|
|
```
|
|
|
|
### `Contributor` records
|
|
|
|
`Contributor` records are at the core of this process. Existing `Contributor` records exist, and they are linked to specific people in the CRC based on email address. The new 2026 survey uses Forgejo-based authentication, and its integration with dumpthings means that a user authenticating via forgejo (where their account is also linked to an email address) will submit data into (and read existing data from) a forgejo-related incoming area. We need to make these incoming areas exist, with the existing data, in order for users to be able to see their existing data in the new deployment.
|
|
|
|
As an example, I (Stephan Heunis) submitted a `Contributor` record for the previous survey that went into my token-specific inbox in the previous dumpthings deployment. My data is known by an administrator to be backed up at a path along the likes of: `./2025/usr-sheunisfzjuelichde/Contributor/../<some-hash>.yaml`. So we need to take this information, and information about my forgejo account (also linked to my email address), predict the correct incoming area label for my data, and then put my `Contributor` record in this correct incoming area in the new deployment.
|
|
|
|
This is done as follows:
|
|
- get a list of all yaml files, i.e. paths to `Contributor` records
|
|
- pass these into `code/convert_contributors`, which outputs the inbox label and the record, tab-separated
|
|
- read these values into a `curl` command that POSTs the record to the specific inbox
|
|
|
|
```
|
|
find ./2025/usr-* -type f \( -name '*.yaml' -o -name '*.yml' \) |
|
|
uv run code/convert_contributors.py |
|
|
while IFS=$'\t' read -r inbox_label record; do
|
|
printf '%s' "$record" |
|
|
curl -X POST \
|
|
"${DUMPTHINGS_API}/2026-protected/incoming/${inbox_label}/record/Contributor" \
|
|
-H "Content-Type: application/json" \
|
|
-H "X-DumpThings-Token: ${CURATOR_TOKEN}" \
|
|
--data-binary @-
|
|
done
|
|
```
|
|
|
|
The reason for using `curl` vs `dtc` is that the latter does not (yet) support using `post-records` for adding data to specific user inboxes. |