Pr/scraping options #857

moshloop · 2024-08-21T21:16:35Z

No description provided.

moshloop · 2024-08-21T21:56:37Z

On small files there doesn't seem to be much difference, but only large files (6mb) there is a big change:

❯  go test -benchmem -run=^$ -bench ^BenchmarkDiffGenerator$ github.com/flanksource/config-db/db -count=2 -benchtime=10s -v
00:41:30.919 INF Loaded 3 change rules
goos: darwin
goarch: arm64
pkg: github.com/flanksource/config-db/db
BenchmarkDiffGenerator
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=go
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=go-12         	  730930	     17556 ns/op	       444.5 cpu%	         0.01559 gc/op	       110.0 jsonKB	        79.00 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=go-12         	  780650	     15941 ns/op	       476.1 cpu%	         0.01385 gc/op	       110.0 jsonKB	       287.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=go
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=go-12          	  660283	     18396 ns/op	       482.4 cpu%	         0.01758 gc/op	       110.0 jsonKB	       287.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=go-12          	  649162	     18304 ns/op	       488.9 cpu%	         0.01786 gc/op	       110.0 jsonKB	       417.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=go
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=go-12         	  890121	     13786 ns/op	       459.4 cpu%	         0.007871 gc/op	       110.0 jsonKB	       557.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=go-12         	  882524	     14096 ns/op	       435.2 cpu%	         0.007764 gc/op	       110.0 jsonKB	       557.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=jq
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=jq-12         	  709797	     17103 ns/op	       428.2 cpu%	         0.01360 gc/op	       110.0 jsonKB	       557.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=jq-12         	  675697	     17201 ns/op	       423.3 cpu%	         0.01371 gc/op	       110.0 jsonKB	       557.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=jq
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=jq-12          	  700491	     17734 ns/op	       424.1 cpu%	         0.01657 gc/op	       110.0 jsonKB	       557.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=jq-12          	  659250	     17740 ns/op	       424.6 cpu%	         0.01654 gc/op	       110.0 jsonKB	       557.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=jq
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=jq-12         	  891931	     14212 ns/op	       413.8 cpu%	         0.007742 gc/op	       110.0 jsonKB	       557.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=jq-12         	  817880	     14281 ns/op	       405.6 cpu%	         0.007730 gc/op	       110.0 jsonKB	       557.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=oj
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=oj-12         	  687028	     17102 ns/op	       404.1 cpu%	         0.01357 gc/op	       110.0 jsonKB	       558.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=oj-12         	  711387	     17201 ns/op	       402.6 cpu%	         0.01361 gc/op	       110.0 jsonKB	       558.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=oj
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=oj-12          	  674374	     17700 ns/op	       404.4 cpu%	         0.01645 gc/op	       110.0 jsonKB	       558.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=oj-12          	  661803	     17747 ns/op	       406.2 cpu%	         0.01658 gc/op	       110.0 jsonKB	       558.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=oj
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=oj-12         	  838864	     14357 ns/op	       400.7 cpu%	         0.007799 gc/op	       110.0 jsonKB	       558.0 rssMB	  229376 B/op       2 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=oj-12         	  830772	     14279 ns/op	       395.8 cpu%	         0.007725 gc/op	       110.0 jsonKB	       558.0 rssMB	  229376 B/op       2 allocs/op
PASS
ok  	github.com/flanksource/config-db/db	222.445s

☸ aws-flanksource (mission-control) in config-db on 🌱 feat/gc-on [$!?⇕] on ☁️   with
❯  go test -benchmem -run=^$ -bench ^BenchmarkDiffGenerator$ github.com/flanksource/config-db/db -count=2 -benchtime=10s -v
00:45:50.445 INF Loaded 3 change rules
goos: darwin
goarch: arm64
pkg: github.com/flanksource/config-db/db
BenchmarkDiffGenerator
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=go
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=go-12         	       2	5422736875 ns/op	       266.1 cpu%	      1050 gc/op	      6752 jsonKB	      9223 rssMB	23142717112 B/op	 1073824 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=go-12         	       2	5798839479 ns/op	       307.1 cpu%	      1038 gc/op	      6752 jsonKB	     10080 rssMB	23138518268 B/op	 1073791 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=go
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=go-12          	       2	5728356833 ns/op	       316.9 cpu%	      1027 gc/op	      6752 jsonKB	     10689 rssMB	23146918964 B/op	 1073860 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=go-12          	       2	5830607146 ns/op	       324.7 cpu%	      1012 gc/op	      6752 jsonKB	     10943 rssMB	23142721972 B/op	 1073831 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=go
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=go-12         	       2	5658092188 ns/op	       327.2 cpu%	      1006 gc/op	      6752 jsonKB	     11578 rssMB	23138525372 B/op	 1073824 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=go-12         	       2	5630181042 ns/op	       323.4 cpu%	      1002 gc/op	      6752 jsonKB	     11858 rssMB	23138513404 B/op	 1073783 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=jq
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=jq-12         	       2	6673121125 ns/op	       319.3 cpu%	      1140 gc/op	      6752 jsonKB	     11349 rssMB	23089799732 B/op	   25624 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=jq-12         	       2	6639794146 ns/op	       318.4 cpu%	      1152 gc/op	      6752 jsonKB	     11415 rssMB	23089798052 B/op	   25611 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=jq
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=jq-12          	       2	6883312792 ns/op	       317.4 cpu%	      1166 gc/op	      6752 jsonKB	     11268 rssMB	23089799360 B/op	   25617 allocs/op
^Csignal: interrupt
FAIL	github.com/flanksource/config-db/db	165.794s
FAIL

☸ aws-flanksource (mission-control) in config-db on 🌱 feat/gc-on [$!?⇕] on ☁️   with
❯  go test -benchmem -run=^$ -bench ^BenchmarkDiffGenerator$ github.com/flanksource/config-db/db -count=2 -benchtime=10s -v
00:48:58.028 INF Loaded 3 change rules
goos: darwin
goarch: arm64
pkg: github.com/flanksource/config-db/db
BenchmarkDiffGenerator
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=oj
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=oj-12         	     170	  70282715 ns/op	       124.8 cpu%	         2.418 gc/op	      6752 jsonKB	       200.0 rssMB	113480247 B/op	  954477 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=oj-12         	     170	  70170578 ns/op	       129.0 cpu%	         2.388 gc/op	      6752 jsonKB	       201.0 rssMB	113794348 B/op	  954478 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=oj
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=oj-12          	     157	  75352688 ns/op	       142.9 cpu%	         5.675 gc/op	      6752 jsonKB	       201.0 rssMB	122745917 B/op	  954503 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=oj-12          	     158	  75253792 ns/op	       151.4 cpu%	         5.734 gc/op	      6752 jsonKB	       201.0 rssMB	122336050 B/op	  954500 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=oj
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=oj-12         	     178	  66299580 ns/op	       144.8 cpu%	         1.107 gc/op	      6752 jsonKB	       267.0 rssMB	105964073 B/op	  954452 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=oj-12         	     177	  67574020 ns/op	       140.2 cpu%	         1.085 gc/op	      6752 jsonKB	       267.0 rssMB	104953817 B/op	  954452 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=jq
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=jq-12         	       2	6854568520 ns/op	       157.5 cpu%	      1142 gc/op	      6752 jsonKB	      7381 rssMB	23089798672 B/op	   25616 allocs/op
^Csignal: interrupt
FAIL	github.com/flanksource/config-db/db	154.595s

adityathebe · 2024-08-22T04:45:02Z

api/cache.go

-	aliases map[string]string
+	items    map[string]models.ConfigItem
+	aliases  map[string]string
+	notFound map[string]bool


I think map[string]struct{} is better for existence check as it allocates 0 bytes

adityathebe · 2024-08-22T04:48:59Z

cmd/run.go

@@ -33,11 +35,10 @@ var Run = &cobra.Command{

 		dutyCtx := context.New()
 		if dutyapi.DefaultConfig.ConnectionString != "" {
-			c, closer, err := duty.Start("config-db", duty.SkipMigrationByDefaultMode)
+			c, _, err := duty.Start("config-db", duty.ClientOnly)


why are we ignoring the closer? If you use embedded db, this was stopping it.

ClientOnly shouldn't be starting any db's

adityathebe · 2024-08-22T05:27:43Z

Posting my benchmark as well

BenchmarkDiffGenerator
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=go
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=go-32                49729            224709 ns/op               161.4 cpu%                0.01957 gc/op           3.000 jsonKB           86.00 rssMB       199097 B/op       1144 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=go-32                50074            233082 ns/op               161.1 cpu%                0.01905 gc/op           3.000 jsonKB           86.00 rssMB       199083 B/op       1144 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=go-32                54530            234314 ns/op               164.3 cpu%                0.01863 gc/op           3.000 jsonKB           85.00 rssMB       199101 B/op       1144 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=go-32                52185            224680 ns/op               164.1 cpu%                0.01817 gc/op           3.000 jsonKB           86.00 rssMB       199065 B/op       1144 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=go-32                53890            215719 ns/op               162.5 cpu%                0.01774 gc/op           3.000 jsonKB           85.00 rssMB       199041 B/op       1144 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=go
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=go-32                 39691            362867 ns/op               193.4 cpu%                0.03827 gc/op           3.000 jsonKB           80.00 rssMB       199714 B/op       1145 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=go-32                 58638            355942 ns/op               210.7 cpu%                0.03696 gc/op           3.000 jsonKB           81.00 rssMB       199680 B/op       1145 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=go-32                 31208            379914 ns/op               216.7 cpu%                0.03614 gc/op           3.000 jsonKB           81.00 rssMB       199668 B/op       1145 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=go-32                 28828            383956 ns/op               220.6 cpu%                0.03528 gc/op           3.000 jsonKB           82.00 rssMB       199624 B/op       1145 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=go-32                 43035            368322 ns/op               224.2 cpu%                0.03446 gc/op           3.000 jsonKB           83.00 rssMB       199667 B/op       1145 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=go
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=go-32                88465            129660 ns/op               217.6 cpu%                0.007088 gc/op          3.000 jsonKB          104.0 rssMB        198774 B/op       1144 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=go-32                91308            128682 ns/op               211.9 cpu%                0.006998 gc/op          3.000 jsonKB          106.0 rssMB        198784 B/op       1144 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=go-32                87444            129917 ns/op               206.9 cpu%                0.007102 gc/op          3.000 jsonKB          105.0 rssMB        198775 B/op       1144 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=go-32                91144            129632 ns/op               202.4 cpu%                0.007044 gc/op          3.000 jsonKB          104.0 rssMB        198779 B/op       1144 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=go-32                95020            130706 ns/op               198.1 cpu%                0.006967 gc/op          3.000 jsonKB          106.0 rssMB        198774 B/op       1144 allocs/op

BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=oj
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=oj-32               136387             81573 ns/op               149.5 cpu%                0.006335 gc/op          3.000 jsonKB           85.00 rssMB        65023 B/op        881 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=oj-32               142134             74918 ns/op               148.4 cpu%                0.006198 gc/op          3.000 jsonKB           85.00 rssMB        65020 B/op        881 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=oj-32               168786             76840 ns/op               146.1 cpu%                0.006043 gc/op          3.000 jsonKB           86.00 rssMB        65012 B/op        881 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=oj-32               157876             75910 ns/op               146.4 cpu%                0.005948 gc/op          3.000 jsonKB           85.00 rssMB        65017 B/op        881 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=oj-32               174801             74718 ns/op               145.3 cpu%                0.005835 gc/op          3.000 jsonKB           85.00 rssMB        65010 B/op        881 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=oj
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=oj-32                 76634            142449 ns/op               157.0 cpu%                0.01234 gc/op           3.000 jsonKB           79.00 rssMB        65207 B/op        881 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=oj-32                101796            138370 ns/op               165.9 cpu%                0.01184 gc/op           3.000 jsonKB           80.00 rssMB        65184 B/op        881 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=oj-32                 85009            144761 ns/op               172.0 cpu%                0.01180 gc/op           3.000 jsonKB           81.00 rssMB        65176 B/op        881 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=oj-32                 84258            133055 ns/op               175.4 cpu%                0.01162 gc/op           3.000 jsonKB           80.00 rssMB        65180 B/op        881 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=oj-32                 92614            132964 ns/op               178.4 cpu%                0.01147 gc/op           3.000 jsonKB           82.00 rssMB        65181 B/op        881 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=oj
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=oj-32               242865             48957 ns/op               173.6 cpu%                0.002635 gc/op          3.000 jsonKB          102.0 rssMB         64927 B/op        881 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=oj-32               234094             49551 ns/op               169.7 cpu%                0.002610 gc/op          3.000 jsonKB          100.0 rssMB         64928 B/op        881 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=oj-32               255454             48216 ns/op               166.1 cpu%                0.002588 gc/op          3.000 jsonKB          100.0 rssMB         64925 B/op        881 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=oj-32               251530             48292 ns/op               163.0 cpu%                0.002564 gc/op          3.000 jsonKB          100.0 rssMB         64925 B/op        881 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=oj-32               248845             48283 ns/op               160.4 cpu%                0.002548 gc/op          3.000 jsonKB          100.0 rssMB         64920 B/op        881 allocs/op

BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=jq
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=jq-32                 1141          10535530 ns/op               151.3 cpu%                0.02103 gc/op           3.000 jsonKB           88.00 rssMB       263422 B/op        399 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=jq-32                 1178          10528887 ns/op               143.2 cpu%                0.02122 gc/op           3.000 jsonKB           89.00 rssMB       263333 B/op        399 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=jq-32                  944          10830915 ns/op               137.0 cpu%                0.02119 gc/op           3.000 jsonKB           88.00 rssMB       263334 B/op        399 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=jq-32                  993          10659901 ns/op               131.2 cpu%                0.02014 gc/op           3.000 jsonKB           90.00 rssMB       263314 B/op        399 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=100,_NORMALIZER=jq-32                 1294          10709707 ns/op               124.7 cpu%                0.02087 gc/op           3.000 jsonKB           89.00 rssMB       263202 B/op        398 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=jq
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=jq-32                  1153          10441797 ns/op               116.5 cpu%                0.04337 gc/op           3.000 jsonKB           84.00 rssMB       263569 B/op        399 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=jq-32                  1164          10939633 ns/op               109.4 cpu%                0.04210 gc/op           3.000 jsonKB           85.00 rssMB       263586 B/op        399 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=jq-32                  1557           9122653 ns/op               105.5 cpu%                0.04239 gc/op           3.000 jsonKB           85.00 rssMB       263495 B/op        399 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=jq-32                  1472          10098870 ns/op               101.7 cpu%                0.04280 gc/op           3.000 jsonKB           85.00 rssMB       263543 B/op        399 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=50,_NORMALIZER=jq-32                  1525           9309040 ns/op                98.41 cpu%               0.04197 gc/op           3.000 jsonKB           85.00 rssMB       263476 B/op        399 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=jq
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=jq-32                 1494           8778351 ns/op                95.17 cpu%               0.009371 gc/op          3.000 jsonKB          108.0 rssMB        263022 B/op        398 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=jq-32                 1766           8791232 ns/op                91.73 cpu%               0.009060 gc/op          3.000 jsonKB          107.0 rssMB        263021 B/op        398 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=jq-32                 1392          10354904 ns/op                88.74 cpu%               0.009339 gc/op          3.000 jsonKB          106.0 rssMB        263040 B/op        398 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=jq-32                 1233           8596919 ns/op                84.92 cpu%               0.008921 gc/op          3.000 jsonKB          106.0 rssMB        263074 B/op        398 allocs/op
BenchmarkDiffGenerator/GOMEMLIMIT=1024mb_GOGC=200,_NORMALIZER=jq-32                 1431           8311349 ns/op                82.87 cpu%               0.009085 gc/op          3.000 jsonKB          106.0 rssMB        263012 B/op        398 allocs/op

moshloop added 5 commits August 22, 2024 00:15

chore: cache tempcache not found

75019c5

fix: config-db run hanging

22e926a

chore: support different json normalizers

ec03a15

chore: logging improvements

3b0f39b

chore: generate hack/generate-schema/go.mod

a77f085

moshloop requested review from yashmehrotra and adityathebe August 21, 2024 21:16

moshloop added 2 commits August 22, 2024 00:20

chore: fix make generate

f55caeb

chore: update diff benchmark

95fa0d2

chore: fix tests

d7c9285

adityathebe reviewed Aug 22, 2024

View reviewed changes

moshloop added 2 commits August 22, 2024 15:53

fix: switch to oj for json normalization

28d50e9

chore: switch to struct{} for cache value

b2d780f

moshloop closed this Aug 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pr/scraping options #857

Pr/scraping options #857

moshloop commented Aug 21, 2024

moshloop commented Aug 21, 2024

adityathebe Aug 22, 2024

adityathebe Aug 22, 2024

moshloop Aug 22, 2024

adityathebe commented Aug 22, 2024

Pr/scraping options #857

Pr/scraping options #857

Conversation

moshloop commented Aug 21, 2024

moshloop commented Aug 21, 2024

adityathebe Aug 22, 2024

Choose a reason for hiding this comment

adityathebe Aug 22, 2024

Choose a reason for hiding this comment

moshloop Aug 22, 2024

Choose a reason for hiding this comment

adityathebe commented Aug 22, 2024