One-Click Installation Universal Script for MIMIC-III Clinical Database CareVue subset with PostgreSQL on MAC OS, Windows, and Linux

[ningyile]

Dear Administrator of MIMIC Team:

I have completed the re-edition of SQL codes to install MIMIC-III Clinical Database CareVue subset(I delete the SQL query for Metavision subset and convert BigQuery function to Postgresql function), and I have written a shell script that can install the database and Concepts with Postgresql on Mac, Linux, and Windows.

• MIMIC new beginners only need to type bash install.sh in the terminal (For windows, the terminal was provided by bash.exe of Git software in C:\Program Files\Git\bin), and then the database can be automatically configured and installed. The whole process does not require any further operation.

• Sepsis3 has been added to the Concepts, which was edited based on A Comparative Analysis of Sepsis Identification Methods in an Electronic Database. sepsis3-mimic created byAlistair Johnson。

• Charlson Comorbidity Index has been added to the Concepts, which was edited based onCharlson in Concepts of MIMICIV。

• The readme profile contains detailed illustrations, which should be clear enough for beginners. Although the process is simple, I plan to upload a tutorial for MAC Linux, and Windows on Youtube. But the video tutorial is still being edited, and the video link will be added to the readme profile later. I hope the PR I submitted can be accepted, if you have any further questions, please get in touch with me. I'm looking forward to hearing back from you.

@alistairewj @tompollard

Chris Ning
[email protected]

[alistairewj]

This clearly took a lot of effort and I am very appreciative of the PR!

I like the idea of a very easy installation (we've always tried to make it simpler). We did start off with a Makefile for creating MIMIC, but in the end I found it was very difficult to keep things friendly across operating systems, and opted for a fully SQL approach.

With that in mind, after a quick look, there are some important changes.

• I'd rather not split mimic-iii into mimic-iii and mimic-iii-carevue. Since mimic-iii-carevue is a subset of mimic-iii, it's easier to keep the one folder and use it for mimic-iii-carevue. You should be able to use most of the mimic-iii scripts to build the data. The only catch is the validation script, which we might need a new version of.
• You'll have to use the existing concepts, and not create new SQL files. That should only require judicious use of relative paths.
• I don't like having a registry file to register a new menu item in Windows. Instead, I'd ask the user to install the bash software you need, and tell them they need to launch a terminal in the folder. That avoids the awkwardness of having a registry file and it means all the operating systems have a mostly equivalent starting point.
• For install.sh, I've made comments inline above
• Internationalization of bash scripts is possible with some additional work, which seems like a better approach than the back and forth if statements.

Before you go ahead and put more work into this though, I'd like to discuss this with some colleagues (@tompollard / @briangow) - I'm not 100% sure we'd want to commit to maintaining something like this. It is hard enough to keep this repo up to date as is!

[alistairewj]

I don't think it's a good idea to change the PostgreSQL security for inexperienced users to trust. Instead, try to get this to work for OS authentication (either via md5 or scram-sha-256). I think the best approach is:

1. Detect the type of auth. If it's incompatible (e.g. reject), ask the user if you can modify it to OS-level authentication (scram-sha-256). If they refuse, exit script.
2. If it's password, offer to export the PGPASSWORD env var at the start of the run.

[alistairewj]

Don't use "$os" == "Darwin" because users may install a different version of sed on their platform, instead you can check if --version has a valid return with gnu_sed=`sed --version 2>/dev/null` -> if $gnu_sed is non-empty, then it's GNU sed

[alistairewj]

Comments which repeat the below echo statement are redundant, and can be removed

[alistairewj]

Change this to carry on from the last sentence, e.g. "A new user will be created"

[alistairewj]

I would prompt the user for a password, rather than default it to the username.

[alistairewj]

The database name, schema name, and derived schema name should be environment variables set at the top of the script. This will allow you to add a --carevue flag which changes these variables to carevue specific ones.

I would suggest the database remains simple (mimic), but the schema is updated to mimiciii_cv, and the derived schema is also changed mimiciii_cv_derived.

[ningyile]

This clearly took a lot of effort and I am very appreciative of the PR!

I like the idea of a very easy installation (we've always tried to make it simpler). We did start off with a Makefile for creating MIMIC, but in the end I found it was very difficult to keep things friendly across operating systems, and opted for a fully SQL approach.

With that in mind, after a quick look, there are some important changes.

• I'd rather not split mimic-iii into mimic-iii and mimic-iii-carevue. Since mimic-iii-carevue is a subset of mimic-iii, it's easier to keep the one folder and use it for mimic-iii-carevue. You should be able to use most of the mimic-iii scripts to build the data. The only catch is the validation script, which we might need a new version of.
• You'll have to use the existing concepts, and not create new SQL files. That should only require judicious use of relative paths.
• I don't like having a registry file to register a new menu item in Windows. Instead, I'd ask the user to install the bash software you need, and tell them they need to launch a terminal in the folder. That avoids the awkwardness of having a registry file and it means all the operating systems have a mostly equivalent starting point.
• For install.sh, I've made comments inline above
• Internationalization of bash scripts is possible with some additional work, which seems like a better approach than the back and forth if statements.

Before you go ahead and put more work into this though, I'd like to discuss this with some colleagues (@tompollard / @briangow) - I'm not 100% sure we'd want to commit to maintaining something like this. It is hard enough to keep this repo up to date as is!

Thank you, Johnson! I think I can solve the problems you pointed out in the next version according to your requirements. But I need to check with you a few questions before I PR the next version of the shell script:

• The SQL codes of MIMIC-III contains queries for Metavsion tables, and I'm not sure whether it would cause several fatal errors or not. Anyway, I will execute the MIMIC-III SQL codes using relative paths in the shell scripts of next version, instead of creating new SQL codes. However, due to the number of rows for data will be changed, tables may not be checked unless the SQL code of check_data is re-edited or created with a new sub-version.

• For the issue of Bash Terminals on Windows, I'm going to ask users to decompress the portable version of Git (there is a download link for the portable version, just need to extract and use it without installing Git) and move it to the relative root directory of the MIMIC code, then change the directory where the shell script is located with cd command, and then start the installation script.

• The purpose of changing the auth of pg_hba.conf is to avoid entering passwords multiple times during the installation (more automatically and silently). So can I do this as the following: ① First, ask the user whether they agree to change the type of auth to trust during the installation (note that only change the auth to trust during the installation, and restore the previous settings after the installation) ②Second, if the user agrees, change the type of auth of pg_hba.conf as trust with sed -i.bk -E "$sed_ptn" "$pg_hba". This command sed -i.bk ...... is compatible with both GNU and FreeBSD version of sed(here we don't have to worry about the version of sed with the compatible command), which can back up pg_hba.conf as pg_hba.conf.bk before the edition of auth. The original backup settings file will be restored and reloaded when the installation is completed, so that pg_hba.conf remains unchanged after the installation.

• For the issue of internationalization of the bash script, I have tried to write the message in mo files(message object files, *.mo created from portable object files, *.po, need to copy to folder /usr/share/locale/zh_CN/LC_MESSAGES/, which is a problem for Windows) or two separate files (for example: en.ini and cn.ini), but I didn't want to generate additional files in codes of MIMIC-III, so I had to utilize if statements back and forth instead of the language profile mentioned above. Shell internationalization is an issue for cross-platform, but English doesn't seem to be a problem for most MIMIC users, so I'll consider using English only for the next version of the shell script.