feat: pre compute breached passwords (#37)

* feat: use dependency injection with callback function The database class should be implementation-agnostic, so the execute function needs to support arbitrary callbacks. There are currently two versions of execute in the event the user does not care about the result (e.g. when inserting), but a default value in the templated version is a sound alternative. The templated version is defined within the header file for C++ compiler reasons. The alternative is to explcitly define each possible ttype of T in the source file; right now, the only value is std::string, but this doesn't seem idiomatic. * fix: finalize SQlite statment after iteration * test: test Database class * fix: include vector in database.hpp * fix: include functional in database.hpp * test: use variable for path * docs: add some TODO statements * refactor: update cryptography test name * chore: update .gitignore * feat: allow rebuilding the database using command line * test: test allowing rebuild * build: remove make run from Makefile One or more command line arguments are now required. Use the command build/src/server build/passwords.db to achieve (almost) the same result as make run. * feat: adding logging * fix: only create table when rebuilding * docs: add comments for functions * fix: only abort if close fails * docs: add some TODO statements * feat: encode the encrypted breached passwords before inserting into the database * feat: use database for breached passwords * refactor: change rebuild flag to build and improve error handling * docs: update build instructions in readme * feat: wip store secret key b BREAKING CHANGE: trying to figure out how to properly retrieve value from secret key table * feat: insert key b into database so the user password can be encrypted with the same b as the breached passwords * test: fix the endpoint handler test by creating a table for the secret key b and storing the key * test: test that using the build flag on a populated database will produce a new empty database * test: test the contents of the names remain the same after reopening a database with the same file and build as default false BREAKING CHANGE: when the database file is initialized and reopened in the same test section then it passes, but not when the database file is reopened in a different section than initialization * refactor: use global variables for database tests * docs: update readmes: * docs: explain the --build flag in the private-data-lookup/README.md * docs: encourage the use of /data directory for databases * docs: use build instead of rebuild in constructor comment * refactor: move the code from the open() method in Database class to inside the Database constructor * refactor: throw invalid argument for an incorrect filepath instead of a runtime error * refactor: use snack_case for variables encoded_b and decoded_b * refactor: rename results vector to breached_passwords for readability * refactor: return the count of querying names tables instead of a boolean of the if the table exists * refactor: return the counts of the query rather than a boolean * docs: change description tense of test case * test: create own string list of passwords instead of using generatePasswords() * feat: add input sanitization to make sure the file ends in .db * refactor: use catch2 nested sections for database tests * feat: error check if the passwords table does not exist in the file passed in * refactor: throw runtime error rather than abort if the database can not close properly * refactor: remove encodePasswords() helper method * refactor: use abort instead of throwing error due to deconstructer conventions * feat: print error message in deconstructor * refactor: update import statements * refactor: improve database error handling * feat: use spdlog::error instead of print stderr * refactor: remove unused open definition * test: use int in std::function * docs: remove inline comment --------- Co-authored-by: Cedric Sirianni <[email protected]> Co-authored-by: stellaljung <[email protected]>
csirianni · Dec 7, 2023 · b9d8751 · b9d8751
1 parent 4368acd
commit b9d8751
Show file tree

Hide file tree

Showing 16 changed files with 327 additions and 75 deletions.
diff --git a/README.md b/README.md
@@ -30,10 +30,15 @@ cmake .. -DCMAKE_TOOLCHAIN_FILE=conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release
 cmake --build .
 ```
 
-Then, you can simply run
+Then, you can simply run this command on an existing database:
+Database file paths should end in .db, like "passwords.db"
+```console
+build/src/server <database filepath>
+```
 
+If you want to build a new database from a new or existing database, you can use the --build flag after the filepath. 
 ```console
-./src/server
+build/src/server <database filepath> --build
 ```
 
 Ensure that the backend is running with the frontend, otherwise you will see a message of server errors on the front-end website.
diff --git a/backend/.gitignore b/backend/.gitignore
@@ -1,2 +1,4 @@
 /build
-CMakeUserPresets.json
+CMakeUserPresets.json
+*.db
+/data
diff --git a/backend/Makefile b/backend/Makefile
@@ -6,9 +6,6 @@ build:
 	cd $(BUILD_DIR) && cmake .. -DCMAKE_TOOLCHAIN_FILE=conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON
 	cd $(BUILD_DIR) && cmake --build .
 
-run: build
-	cd ${BUILD_DIR} && ./src/server
-
 check: build
 	cd $(BUILD_DIR) && ./test/pdl_test	
 

diff --git a/backend/README.md b/backend/README.md
@@ -17,10 +17,14 @@ cmake .. -DCMAKE_TOOLCHAIN_FILE=conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release
 cmake --build .
 ```
 
-From `/backend`, start the server:
+From `/backend`, start the server. Use the `--build` flag to create or rebuild a database for the breached passwords or omit it to use an existing one:
 
 ```bash
-cd build && ./backend
+build/src/server <database filepath> --build
+```
+
+```bash
+build/src/server data/passwords.db 
 ```
 
 To fix VS Code import errors with Crow, try adding the following line to your `settings.json`:
@@ -36,5 +40,10 @@ To fix VS Code import errors with Crow, try adding the following line to your `s
 After building, you can run tests from `/backend`:
 
 ```bash
-cd build && ./tests
+cd build && ./test/pdl_test	
+```
+
+or alternatively:
+```bash
+make check
 ```
diff --git a/backend/conanfile.py b/backend/conanfile.py
@@ -11,6 +11,7 @@ def requirements(self):
         self.requires("catch2/3.4.0")
         self.requires("libsodium/1.0.18")
         self.requires("nlohmann_json/3.11.3")
+        self.requires("spdlog/1.12.0")
 
     def build_requirements(self):
         self.tool_requires("cmake/3.22.6")
diff --git a/backend/src/CMakeLists.txt b/backend/src/CMakeLists.txt
@@ -1,10 +1,11 @@
 find_package(Crow REQUIRED)
 find_package(SQLite3 REQUIRED)
 find_package(libsodium REQUIRED)
+find_package(spdlog REQUIRED)
 
 add_library(src database.cpp password.cpp server.cpp cryptography.cpp)
-target_link_libraries(src Crow::Crow libsodium::libsodium)
+target_link_libraries(src Crow::Crow libsodium::libsodium spdlog::spdlog)
 
 add_executable(server main.cpp)
-target_link_libraries(server Crow::Crow SQLite::SQLite3 libsodium::libsodium src)
+target_link_libraries(server Crow::Crow SQLite::SQLite3 libsodium::libsodium spdlog::spdlog src)
 target_include_directories(server PRIVATE src)
diff --git a/backend/src/cryptography.hpp b/backend/src/cryptography.hpp
@@ -6,7 +6,7 @@ namespace cryptography
 {
 
     /**
-     * @brief Encrypt the provided point using secret key b.
+     * @brief Hash and encrypt the provided password using secret key b.
      *
      * This function converts the provided password to a point on an elliptic curve and then encrypts it using the provided secret key.
      *
@@ -16,6 +16,15 @@ namespace cryptography
      */
     std::string hashAndEncryptPassword(const std::string &password, unsigned char *b);
 
+    /**
+     * @brief Encrypt the provided password using secret key b.
+     *
+     * This function does not hash the provided password before encrypting it.
+     *
+     * @param password the password to encrypt
+     * @param b the secret key
+     * @return std::string the encrypted password
+     */
     std::string encryptPassword(const std::string &password, unsigned char *b);
 
     /**

diff --git a/backend/src/database.cpp b/backend/src/database.cpp
@@ -1,5 +1,3 @@
-#include <filesystem>
-#include <sqlite3.h>
 #include "database.hpp"
 
 namespace
@@ -18,18 +16,34 @@ namespace
 
 namespace database
 {
-    Database::Database(const std::string &file_path) : is_closed_(false)
+    Database::Database(const std::string &file_path, bool build) : is_closed_(false)
     {
-        if (std::filesystem::exists(file_path))
+        // build the database:: if file exists, remove the file
+        if (build && std::filesystem::exists(file_path))
         {
-            std::remove(file_path.c_str());
+            if (std::remove(file_path.c_str()) != 0)
+            {
+                throw std::filesystem::filesystem_error("Unable to remove file", std::error_code(errno, std::generic_category()));
+            }
+        }
+        // use existing database: if file does not exist, throw an error
+        else if (!build && !std::filesystem::exists(file_path))
+        {
+            throw std::invalid_argument("File does not exist. Use --build to create a new database");
         }
 
+        // open the database (will create a file path if it does not exist)
         int result = sqlite3_open_v2(file_path.c_str(), &db_, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, NULL);
+        // database could not be opened
         if (result != SQLITE_OK)
         {
             const char *error_msg = sqlite3_errmsg(db_);
-            fprintf(stderr, "SQLite error: %s\n", error_msg);
+            throw std::runtime_error(std::string("SQLite error: ") + error_msg);
+        }
+        else
+        {
+            spdlog::info("Database created successfully");
+            spdlog::info("SQLite version: {}", sqlite3_libversion());
         }
     }
 
@@ -39,7 +53,7 @@ namespace database
         if (result != SQLITE_OK)
         {
             const char *error_msg = sqlite3_errmsg(db_);
-            fprintf(stderr, "SQLite error: %s\n", error_msg);
+            throw std::runtime_error(std::string("SQLite error: ") + error_msg);
         }
     }
 
@@ -50,7 +64,7 @@ namespace database
         if (result != SQLITE_OK)
         {
             const char *error_msg = sqlite3_errmsg(db_);
-            fprintf(stderr, "SQLite error: %s\n", error_msg);
+            throw std::runtime_error(std::string("SQLite error: ") + error_msg);
         }
     }
 
@@ -73,9 +87,9 @@ namespace database
             if (result != SQLITE_OK)
             {
                 const char *error_msg = sqlite3_errmsg(db_);
-                fprintf(stderr, "SQLite error: %s\n", error_msg);
+                spdlog::error("SQLite error: {}", error_msg);
+                std::abort();
             }
-            std::abort();
         }
     }
 }
diff --git a/backend/src/database.hpp b/backend/src/database.hpp
@@ -1,4 +1,9 @@
 #include <sqlite3.h>
+#include <vector>
+#include <stdexcept>
+#include <functional>
+#include <filesystem>
+#include "spdlog/spdlog.h"
 
 #ifndef DATABASE_H
 #define DATABASE_H
@@ -18,8 +23,9 @@ namespace database
          *
          * @warning If a file already exists at the provided path, it is replaced.
          * @param file_path The path for the .db file.
+         * @param build Whether or not to build the database.
          */
-        Database(const std::string &file_path);
+        Database(const std::string &file_path, bool build = false);
 
         /**
          * @brief Execute the provided SQL command.
@@ -28,6 +34,33 @@ namespace database
          */
         void execute(const std::string &command);
 
+        /**
+         * @brief Execute the provided SQL command and load each row into a vector using the callback function.
+         *
+         * @tparam T The type of the vector to be returned.
+         * @param command The command to be executed.
+         * @param callback The callback function to be used to load each row into the vector.
+         * @return std::vector<T> The vector containing the rows of the table.
+         */
+        template <typename T>
+        std::vector<T> execute(const std::string &command, std::function<T(sqlite3_stmt *)> callback)
+        {
+            sqlite3_stmt *stmt;
+            if (sqlite3_prepare_v2(db_, command.c_str(), -1, &stmt, NULL) != SQLITE_OK)
+            {
+                const char *error_msg = sqlite3_errmsg(db_);
+                throw std::runtime_error(std::string("SQLite error: ") + error_msg);
+            }
+            std::vector<T> result;
+            // TODO: vector reserve
+            while (sqlite3_step(stmt) == SQLITE_ROW)
+            {
+                result.push_back(callback(stmt));
+            }
+            sqlite3_finalize(stmt);
+            return result;
+        }
+
         /**
          * @brief Print the rows of the provided table.
          *

diff --git a/backend/src/main.cpp b/backend/src/main.cpp
@@ -9,29 +9,87 @@
 #include "server.hpp"
 #include "sodium.h"
 #include "cryptography.hpp"
+#include "spdlog/spdlog.h"
 
-int main()
+int main(int argc, char *argv[])
 {
-    database::Database db = database::Database("passwords.db");
-    db.execute("CREATE TABLE passwords (password TEXT);");
-
-    // generate and insert the passwords into the database
-    std::unordered_set<std::string> passwords = password::generatePasswords(100, 20);
-    passwords.insert("TestPass1&");
-    passwords.insert("ChocolateCake1!");
-    // 1. generate secret key b
-    unsigned char b[crypto_core_ristretto255_SCALARBYTES];
-    crypto_core_ristretto255_scalar_random(b);
-    // 2. encrypt each password with b (and hash to point)
-    std::vector<std::string> encrypted_passwords = cryptography::encrypt(passwords, b);
-
-    // 3. insert into database
-    for (const auto &password : passwords)
+    spdlog::set_level(spdlog::level::debug);
+
+    if (argc < 2 || argc > 3)
+    {
+        printf("Usage: %s <database file> [--build]\n", argv[0]);
+        return 1;
+    }
+
+    bool build = false;
+    if (argc == 3)
+    {
+        if (std::string(argv[2]) == "--build")
+        {
+            build = true;
+        }
+        else
+        {
+            printf("Usage: %s <database file> [--build]\n", argv[0]);
+            return 1;
+        }
+    }
+
+    // check if file does not end in .db
+    std::string file_path = argv[1];
+    if (file_path.find(".db") != file_path.length() - 3)
+    {
+        throw std::invalid_argument("File does not end in .db. Use the correct file extension to open the database.");
+    }
+
+    database::Database db = database::Database(argv[1], build);
+    if (build)
     {
-        db.execute("INSERT INTO passwords (password) VALUES ('" + password + "');");
+        // create password table
+        db.execute("CREATE TABLE passwords (password TEXT);");
+
+        // generate and insert the passwords into the database
+        std::unordered_set<std::string> passwords = password::generatePasswords(100, 20);
+        passwords.insert("TestPass1&");
+        passwords.insert("ChocolateCake1!");
+
+        // 1. generate secret key b
+        unsigned char b[crypto_core_ristretto255_SCALARBYTES];
+        crypto_core_ristretto255_scalar_random(b);
+
+        // 2. encrypt each password with b (and hash to point)
+        std::vector<std::string> encrypted_passwords = cryptography::encrypt(passwords, b);
+
+        // 3. insert into database
+        for (const auto &password : encrypted_passwords)
+        {
+            // encode password before inserting into database
+            db.execute("INSERT INTO passwords (password) VALUES ('" + crow::utility::base64encode(password, password.size()) + "');");
+        }
+
+        // create key table
+        db.execute("CREATE TABLE secret (key TEXT);");
+
+        // encode key b and insert into database
+        db.execute("INSERT INTO secret (key) VALUES ('" + crow::utility::base64encode(std::string(reinterpret_cast<const char *>(b), crypto_core_ristretto255_SCALARBYTES), crypto_core_ristretto255_SCALARBYTES) + "');");
+    }
+    else
+    {
+        // error check if !build but passwords table does not exist in the file passed in
+        std::function<bool(sqlite3_stmt *)> callback = [](sqlite3_stmt *stmt)
+        {
+            int count = atoi(reinterpret_cast<const char *>(sqlite3_column_text(stmt, 0)));
+            return count;
+        };
+
+        // check if passwords and key table exists
+        std::vector<bool> password_result = db.execute("SELECT COUNT(*) FROM sqlite_schema WHERE name = 'passwords';", callback);
+        std::vector<bool> secret_result = db.execute("SELECT COUNT(*) FROM sqlite_schema WHERE name = 'secret';", callback);
+        if (password_result.front() == 0 || secret_result.front() == 0) // no passwords or no key table exists
+        {
+            throw std::invalid_argument("Passwords and/or secret key table does not exist. Use --build to create a new database");
+        }
     }
-    // test password
-    db.execute("INSERT INTO passwords (password) VALUES ('TestPass1&');");
 
     // Enable CORS
     crow::App<crow::CORSHandler> app;
@@ -43,7 +101,7 @@ int main()
 
     // initialize endpoints
     server::root(app);
-    server::breachedPasswords(app, encrypted_passwords, b);
+    server::breachedPasswords(app, db);
 
     // set the port, set the app to run on multiple threads, and run the app
     app.port(18080).multithreaded().run();