acl22-identifying-the-human-values-behind-arguments

Code and docker containers employed in the ACL'22 publication "Identifying the Human Values behind Arguments". Use this repository to reproduce the experiments of the paper or apply the classifiers to new data.

Setup

Requirements:

• Docker for training/using the classifier
• CUDA and nvidia-docker for GPU support
• R for evaluation

Download the dataset from Zenodo and extract it in the main directory:

wget https://zenodo.org/record/6855004/files/webis-argvalues-22.zip
unzip webis-argvalues-22.zip

Download the models:

wget https://zenodo.org/record/6855004/files/models.zip
unzip models.zip

Or train them yourself.

Predict

Prediction on all arguments from webis-argvalues-22/arguments.tsv with test in the Usage column, or all arguments if no such column exists.

TAG=0.1.1-nocuda # or 'TAG=0.1.1-cuda11.3' if a GPU is available
GPUS="" # or 'GPUS="--gpus=all"' to use all GPUs

# Select classifiers with --classifier: "b" for BERT, "o" for one-baseline, and "s" for SVM
docker run --rm -it --init $GPUS \
  --volume "$PWD/webis-argvalues-22:/data" \
  --volume "$PWD/models:/models" \
  --volume "$PWD:/output" \
  ghcr.io/webis-de/acl22-value-classification:$TAG \
  python predict.py --classifier bos --levels "1,2,3,4a,4b"

Evaluate

Calculate for each model the label-wise and mean Precision, Recall, F1-Score, and Accuracy.

Rscript src/R/Evaluation.R --data-dir webis-argvalues-22/ --predictions predictions.tsv

Note that the result does vary for BERT after re-training due to randomness in the training process. We had to re-train our models after the publication, so expect to get slightly different results to the publication even with the models we published. In our retries, however, the conclusions we draw in the publication were still valid.

Train Classification Models

Training on all arguments from webis-argvalues-22/arguments.tsv with train in the Usage column, or all arguments if no such column exists.

mkdir models
TAG=0.1.1-nocuda # or 'TAG=0.1.1-cuda11.3' if a GPU is available
GPUS="" # or 'GPUS="--gpus=all"' to use all GPUs

# Select classifiers with --classifier: "b" for BERT and "s" for SVM
docker run --rm -it --init $GPUS \
  --volume "$PWD/webis-argvalues-22:/data" \
  --volume "$PWD/models:/models" \
  ghcr.io/webis-de/acl22-value-classification:$TAG \
  python training.py --classifier bs --levels "1,2,3,4a,4b"

Build Docker Images

The Docker images are hosted at ghcr.io and will be pulled automatically by docker run.

If you need to change them, you can also build them:

cd src/python/
docker build -t ghcr.io/webis-de/acl22-value-classification:0.1.1-cuda11.3 --build-arg CUDA=cuda11.3 .
docker build -t ghcr.io/webis-de/acl22-value-classification:0.1.1-nocuda --build-arg CUDA=nocuda .