Merge pull request #2 from okx/dev

Dev

.gitignore

@@ -8,3 +8,10 @@ pgo-data.profdata
 
 # MacOS nuisances
 .DS_Store
+
+**/generated
+docs/**bench**
+
+**/*/libposeidon-permute-c-mac.a
+**/*/go-iden3-crypto/
+.vscode

.gitmodules

@@ -0,0 +1,4 @@
+[submodule "depends/cryptography_cuda"]
+	path = depends/cryptography_cuda
+	url = [email protected]:okx/cryptography_cuda.git
+	branch =5c5c3ca7987125507c8eb17572ecb355ffed2256

Cargo.toml

@@ -1,5 +1,5 @@
 [workspace]
-members = ["evm", "field", "maybe_rayon", "plonky2", "starky", "util"]
+members = ["evm", "field", "maybe_rayon", "plonky2", "starky", "util",  "gen"]
 resolver = "2"
 
 [profile.release]
@@ -9,4 +9,4 @@ incremental = true
 #codegen-units = 1
 
 [profile.bench]
-opt-level = 3
+opt-level = 3

README.md

@@ -1,186 +1,17 @@
-# Plonky2 & more
-[![Discord](https://img.shields.io/discord/743511677072572486?logo=discord)](https://discord.gg/QZKRUpqCJ6)
+# description
+this repo is a fork of https://github.com/0xPolygonZero/plonky2. several optimizations were implemented to boost the computation speed.
 
-This repository was originally for Plonky2, a SNARK implementation based on techniques from PLONK and FRI. It has since expanded to include tools such as Starky, a highly performant STARK implementation.
+# optimizations
+- precompute of fft twiddle factors
+- cuda implementation of Goldilocks Field NTT (feature `cuda`)
 
-
-## Documentation
-
-For more details about the Plonky2 argument system, see this [writeup](plonky2/plonky2.pdf).
-
-Polymer Labs has written up a helpful tutorial [here](https://polymerlabs.medium.com/a-tutorial-on-writing-zk-proofs-with-plonky2-part-i-be5812f6b798)!
-
-
-## Examples
-
-A good starting point for how to use Plonky2 for simple applications is the included examples:
-
-* [`factorial`](plonky2/examples/factorial.rs): Proving knowledge of 100!
-* [`fibonacci`](plonky2/examples/fibonacci.rs): Proving knowledge of the hundredth Fibonacci number
-* [`range_check`](plonky2/examples/range_check.rs): Proving that a field element is in a given range
-* [`square_root`](plonky2/examples/square_root.rs): Proving knowledge of the square root of a given field element
-
-To run an example, use
-
-```sh
-cargo run --example <example_name>
-```
-
-
-## Building
-
-Plonky2 requires a recent nightly toolchain, although we plan to transition to stable in the future.
-
-To use a nightly toolchain for Plonky2 by default, you can run
+# dependencies
 ```
-rustup override set nightly
-```
-in the Plonky2 directory.
-
-
-## Running
-
-To see recursion performance, one can run this bench, which generates a chain of three recursion proofs:
-
-```sh
-RUSTFLAGS=-Ctarget-cpu=native cargo run --release --example bench_recursion -- -vv
+git submodule update --init --recursive
 ```
 
-## Jemalloc
-
-Plonky2 prefers the [Jemalloc](http://jemalloc.net) memory allocator due to its superior performance. To use it, include `jemallocator = "0.5.0"` in your `Cargo.toml` and add the following lines
-to your `main.rs`:
-
-```rust
-use jemallocator::Jemalloc;
-
-#[global_allocator]
-static GLOBAL: Jemalloc = Jemalloc;
+# run examples
+- cuda NTT
 ```
-
-Jemalloc is known to cause crashes when a binary compiled for x86 is run on an Apple silicon-based Mac under [Rosetta 2](https://support.apple.com/en-us/HT211861). If you are experiencing crashes on your Apple silicon Mac, run `rustc --print target-libdir`. The output should contain `aarch64-apple-darwin`. If the output contains `x86_64-apple-darwin`, then you are running the Rust toolchain for x86; we recommend switching to the native ARM version.
-
-
-## Guidance for external contributors
-
-Do you feel keen and able to help with Plonky2? That's great! We
-encourage external contributions!
-
-We want to make it easy for you to contribute, but at the same time we
-must manage the burden of reviewing external contributions. We are a
-small team, and the time we spend reviewing external contributions is
-time we are not developing ourselves.
-
-We also want to help you to avoid inadvertently duplicating work that
-is already underway, or building something that we will not
-want to incorporate.
-
-First and foremost, please keep in mind that this is a highly
-technical piece of software and contributing is only suitable for
-experienced mathematicians, cryptographers and software engineers.
-
-The Polygon Zero Team reserves the right to accept or reject any
-external contribution for any reason, including a simple lack of time
-to maintain it (now or in the future); we may even decline to review
-something that is not considered a sufficiently high priority for us.
-
-To avoid disappointment, please communicate your intention to
-contribute openly, while respecting the limited time and availability
-we have to review and provide guidance for external contributions. It
-is a good idea to drop a note in our public Discord #development
-channel of your intention to work on something, whether an issue, a
-new feature, or a performance improvement. This is probably all that's
-really required to avoid duplication of work with other contributors.
-
-What follows are some more specific requests for how to write PRs in a
-way that will make them easy for us to review. Deviating from these
-guidelines may result in your PR being rejected, ignored or forgotten.
-
-
-### General guidance for your PR
-
-Obviously PRs will not be considered unless they pass our Github
-CI. The Github CI is not executed for PRs from forks, but you can
-simulate the Github CI by running the commands in
-`.github/workflows/ci.yml`.
-
-Under no circumstances should a single PR mix different purposes: Your
-PR is either a bug fix, a new feature, or a performance improvement,
-never a combination. Nor should you include, for example, two
-unrelated performance improvements in one PR. Please just submit
-separate PRs. The goal is to make reviewing your PR as simple as
-possible, and you should be thinking about how to compose the PR to
-minimise the burden on the reviewer.
-
-Also note that any PR that depends on unstable features will be
-automatically rejected. The Polygon Zero Team may enable a small
-number of unstable features in the future for our exclusive use;
-nevertheless we aim to minimise the number of such features, and the
-number of uses of them, to the greatest extent possible.
-
-Here are a few specific guidelines for the three main categories of
-PRs that we expect:
-
-
-#### The PR fixes a bug
-
-In the PR description, please clearly but briefly describe
-
-1. the bug (could be a reference to a GH issue; if it is from a
-   discussion (on Discord/email/etc. for example), please copy in the
-   relevant parts of the discussion);
-2. what turned out to the cause the bug; and
-3. how the PR fixes the bug.
-
-Wherever possible, PRs that fix bugs should include additional tests
-that (i) trigger the original bug and (ii) pass after applying the PR.
-
-
-#### The PR implements a new feature
-
-If you plan to contribute an implementation of a new feature, please
-double-check with the Polygon Zero team that it is a sufficient
-priority for us that it will be reviewed and integrated.
-
-In the PR description, please clearly but briefly describe
-
-1. what the feature does
-2. the approach taken to implement it
-
-All PRs for new features must include a suitable test suite.
-
-
-#### The PR improves performance
-
-Performance improvements are particularly welcome! Please note that it
-can be quite difficult to establish true improvements for the
-workloads we care about. To help filter out false positives, the PR
-description for a performance improvement must clearly identify
-
-1. the target bottleneck (only one per PR to avoid confusing things!)
-2. how performance is measured
-3. characteristics of the machine used (CPU, OS, #threads if appropriate)
-4. performance before and after the PR
-
-
-## Licenses
-
-As this is a monorepo, see the individual crates within for license information.
-
-
-## Security
-
-This code has not yet been audited, and should not be used in any production systems.
-
-While Plonky2 is configurable, its defaults generally target 100 bits of security. The default FRI configuration targets 100 bits of *conjectured* security based on the conjecture in [ethSTARK](https://eprint.iacr.org/2021/582).
-
-Plonky2's default hash function is Poseidon, configured with 8 full rounds, 22 partial rounds, a width of 12 field elements (each ~64 bits), and an S-box of `x^7`. [BBLP22](https://tosc.iacr.org/index.php/ToSC/article/view/9850) suggests that this configuration may have around 95 bits of security, falling a bit short of our 100 bit target.
-
-
-## Links
-
-- [System Zero](https://github.com/0xPolygonZero/system-zero), a zkVM built on top of Starky (no longer maintained)
-- [Waksman](https://github.com/0xPolygonZero/plonky2-waksman), Plonky2 gadgets for permutation checking using Waksman networks (no longer maintained)
-- [Insertion](https://github.com/0xPolygonZero/plonky2-insertion), Plonky2 gadgets for insertion into a list (no longer maintained)
-- [u32](https://github.com/0xPolygonZero/plonky2-u32), Plonky2 gadgets for u32 arithmetic (no longer actively maintained)
-- [ECDSA](https://github.com/0xPolygonZero/plonky2-ecdsa), Plonky2 gadgets for the ECDSA algorithm (no longer actively maintained)
+cargo run --release -p plonky2_field --features=cuda --example fft
+```

docs/rust.md

@@ -0,0 +1,9 @@
+rustc 1.78.0-nightly will throw below error
+```
+error[E0635]: unknown feature `stdsimd`
+```
+
+```
+rustup toolchian install nightly-2024-02-04
+rustup override set nightly-2024-02-04
+```

evm/Cargo.toml

@@ -39,7 +39,7 @@ hashbrown = { version = "0.14.0" }
 tiny-keccak = "2.0.2"
 serde_json = "1.0"
 
-[target.'cfg(not(target_env = "msvc"))'.dependencies]
+[target.'cfg(not(target_os = "macos"))'.dependencies]
 jemallocator = "0.5.0"
 
 [dev-dependencies]

evm/src/lib.rs

@@ -197,10 +197,10 @@ mod stark_testing;
 
 use eth_trie_utils::partial_trie::HashedPartialTrie;
 // Set up Jemalloc
-#[cfg(not(target_env = "msvc"))]
+#[cfg(not(target_os = "macos"))]
 use jemallocator::Jemalloc;
 
-#[cfg(not(target_env = "msvc"))]
+#[cfg(not(target_os = "macos"))]
 #[global_allocator]
 static GLOBAL: Jemalloc = Jemalloc;
 

field/Cargo.toml

@@ -6,6 +6,9 @@ license = "MIT OR Apache-2.0"
 authors = ["Daniel Lubarov <[email protected]>", "William Borgeaud <[email protected]>", "Jacqueline Nabaglo <[email protected]>", "Hamish Ivey-Law <[email protected]>"]
 edition = "2021"
 
+[[example]]
+name = "fft"
+
 [dependencies]
 anyhow = { version = "1.0.40", default-features = false }
 itertools = { version = "0.11.0", default-features = false, features = ["use_alloc"] }
@@ -15,6 +18,24 @@ rand = { version = "0.8.5", default-features = false, features = ["getrandom"] }
 serde = { version = "1.0", default-features = false, features = ["alloc", "derive"] }
 static_assertions = { version = "1.1.0", default-features = false }
 unroll = { version = "0.1.5", default-features = false }
+lazy_static = "1.4.0"
+cryptography_cuda ={path="../depends/cryptography_cuda", optional=true}
+
+[dev-dependencies]
+rand = "*"
+
+[build-dependencies]
+plonky2_util = { path = "../util" }
+anyhow = { version = "1.0.40", default-features = false }
+syn = { version = "1", features = ["full"] }
+proc-macro2 = "1"
+quote = "1"
+
+[features]
+default = ["no_cuda"]
+cuda = ["cryptography_cuda"]
+precompile = []
+no_cuda = ["cryptography_cuda/no_cuda"]
 
 # Display math equations properly in documentation
 [package.metadata.docs.rs]

field/build.rs

@@ -0,0 +1,75 @@
+use std::env;
+use std::fs::write;
+use std::path::{Path};
+#[cfg(feature = "precompile")]
+use std::Path::{PathBuf};
+use std::process::Command;
+#[cfg(feature = "precompile")]
+use std::str::FromStr;
+
+use anyhow::Context;
+#[cfg(feature = "precompile")]
+use plonky2_util::pre_compute::{get_pre_compute_size, PRE_COMPUTE_END, PRE_COMPUTE_START};
+use proc_macro2::TokenStream;
+#[cfg(feature = "precompile")]
+use quote::quote;
+#[cfg(feature = "precompile")]
+use syn::Lit;
+
+#[cfg(feature = "precompile")]
+fn build_precompile() {
+    println!("cargo:rerun-if-changed=generated");
+    let cargo_manifest_dir = env::var("CARGO_MANIFEST_DIR").unwrap();
+
+    let path: PathBuf = PathBuf::from_str(&cargo_manifest_dir)
+        .unwrap()
+        .join("generated/goldilock_root_of_unity.rs");
+
+    let token_stream = build_token_stream(&path).expect("build token stream error");
+    _ = write_generated_file(token_stream, "goldilock_root_of_unity.rs");
+}
+fn main() {
+
+    #[cfg(feature = "precompile")]
+    build_precompile();
+}
+
+#[cfg(feature = "precompile")]
+fn build_token_stream(path: &PathBuf) -> anyhow::Result<TokenStream> {
+    let size = get_pre_compute_size(PRE_COMPUTE_START, PRE_COMPUTE_END);
+    let token = syn::parse_str::<Lit>(&format!("{}", size)).unwrap();
+    if path.exists() {
+        let stream: proc_macro2::TokenStream =
+            format!("\"{}\"", path.to_str().unwrap()).parse().unwrap();
+
+        Ok(quote! {
+            pub static PRE_COMPILED: [u64; #token] =
+
+            unsafe {
+               include!(
+                #stream
+                )
+            };
+
+        })
+    } else {
+        Ok(quote! {
+            pub static PRE_COMPILED: [u64; #token] = [0; #token];
+
+        })
+    }
+}
+
+pub fn write_generated_file(content: TokenStream, out_file: &str) -> anyhow::Result<()> {
+    let out_dir = env::var_os("OUT_DIR").context("failed to get OUT_DIR env var")?;
+    let path = Path::new(&out_dir).join(out_file);
+    let code = content.to_string();
+
+    _ = write(&path, code);
+
+    // Try to format the output for debugging purposes.
+    // Doesn't matter if rustfmt is unavailable.
+    let _ = Command::new("rustfmt").arg(path).output();
+
+    Ok(())
+}