- V0.42.1 (
e4b5387) - Reduce Authentication Requests to Keycloak (#281) (
8ac153c) - 0.41.34 (
fd55f11) - Bump wipac-rest-tools (#280) (
1494e0b) - Try another semantic release fix (
758c605) - Fix up ci actions (#278) (
7824614) - Use github container registry (#276) (
ff88925) - Fix monitoring script syntax (#277) (
187bb14) - Update to more modern tornado asyncio wait syntax (#275) (
42b64d4) - Paths and Pins (#274) (
7c5cd53) - Fix tests (#273) (
c9f9323) - Modify nersc_controller to avoid getting OOM killed (#272) (
6fa23c5) - Ltacmd catalog path (#270) (
1d39a31) - Update scrontab to prune slurm logs (#269) (
c53611b) - Metrics Reporting with Prometheus (#268) (
9a638a0) - Save status to original_status when moving to quarantine (#267) (
126867e) - Progress monitor (#257) (
52e86ff) - Bundler removes broken files of previous attempts (#261) (
d11e1df) - Bump py-versions CI release v2.1 (#259) (
bf64bb1) - Request minimum and maximum time for slurm jobs at NERSC (#258) (
3e2df4f) - Nersc-verifier should use configured hpss_avail path (#256) (
a2ab4cd) - LTA should use the long-term-archive client secret (#255) (
2888a26) - Updated call to hpss_avail command for Perlmutter (#254) (
be05c3d) - Added scrontab job to clean Slurm logs (#253) (
462224c) - Update LTA configuration to use Community File System instead of Perlmutter Scratch (#252) (
c334d15) - Cori Scratch is dead, long live Perlmutter Scratch (#250) (
8f6a12b) - Add -debug flag to grid-proxy-init (#249) (
5c4cf50) - Add Globus tools like globus-url-copy to Docker containers (#248) (
006880c) - Update File Catalog creds supplied to ltacmd (#247) (
fae7161) - Small changes that didn't make the last commit (#246) (
048487f) - Create new controller script for NERSC (#245) (
a030f8f) - New Auth Deployment (#244) (
3418b5a) - Support new authentication for File Catalog (#243) (
406945c) - Modified auth configuration; fixed logging (#242) (
ec23977) - Update LTA authentication (#241) (
96f7cbf) - Fix the return path (#240) (
54fe5fa) - Modify parameters for small transfer requests (#235) (
95913b2) - Jadetools from lta-vm-2 (#234) (
6c3bc11) - Use better event loop management (#233) (
bc5267b) - Make explicit dependency on pyjwt with crypto (
ae48e62) - 0.41.33 (
2babfce) - Bump requirements to latest WIPAC tools (#232) (
0e73531) - 0.41.32 (
abb232e) - Allow Locator to locate bundles with new style File Catalog records (#231) (
aea5078) - 0.41.31 (
b1aa0ec) - Adjust File Catalog queries to use a better index (#229) (
6be9806) - 0.41.30 (
eaea790) - Minimum needed for running site-move-verifier at DESY. Intended to replace pipe2-desy-move-verifier (#219) (
6693f50) - 0.41.29 (
7d80a7b) - Allow rate-limiter to ignore missing bundles in the destination check (#226) (
9d97899) - 0.41.28 (
9a3676e) - More logging for rate_limiter (#224) (
f46bc10) - 0.41.27 (
9013656) - Trim down some async warnings (#220) (
69f69a3) - 0.41.26 (
b7cb3c6) - Bump to rest-tools v1.1.23 (#218) (
73cdc00) - 0.41.25 (
5ddf68d) - Bumped rest-tools dependency to v1.1.22 (#217) (
ebeef43) - 0.41.24 (
1d997a6) - Clean up GET /Metadata query (#215) (
2985ff5) - 0.41.23 (
6645182) - Handle bundles with subdirectories (#213) (
4c2839b) - 0.41.22 (
36d362f) - Bump rest_tools dependency to v0.1.14 (#209) (
91637ed) - 0.41.21 (
afc733c) - Bump rest-tools dependency to 1.1.12 (#207) (
3f7203b) - 0.41.20 (
ec5e953) - First pass at adding telemetry to LTA (#206) (
4336a9a) - 0.41.19 (
09fa012) - Adds lta.date_archived field to bundle records in File Catalog (#205) (
c541380) - 0.41.18 (
9c5d699) - Add command: ltacmd catalog stats (#204) (
65889a4) - 0.41.17 (
c34eb05) - Setting up shop with SetupShop (#200) (
4a8b81a) - 0.41.16 (
f274bc4) - Add pipe-1 scripts for NERSC to WIPAC transfers (#198) (
d4bd39a) - 0.41.15 (
48035c9) - Merge branch 'master' of github.com:WIPACrepo/lta (
b1384b2) - I 1) don't want to change gridftp.py and 2) want this to be a pull request Revert "Add scripts for pipe1 (NERSC to WIPAC)" (
6ec3d8b) - 0.41.14 (
2828b9d) - Add scripts for pipe1 (NERSC to WIPAC) (
5ae80c8) - 0.41.13 (
d574a8a) - NERSC's HPSS has binaries in new locations (#196) (
f65b032) - 0.41.12 (
5da905d) - Unpacker can rewrite physical paths to logical paths (#197) (
835cde5) - 0.41.11 (
2a812cd) - Fix issues identified by LGTM (#194) (
dffaff6) - 0.41.10 (
b8da064) - Semantic Release on Pushes to Official Repository (#193) (
e7e8749) - 0.41.9 (
185fa77) - Make docker tags for releases (#192) (
dd3dee7) - 0.41.8 (
49b55c7) - Add customized versions of the scripts for pipe0 and pipe2. (#179) (
c90c31f) - 0.41.7 (
b5972f4) - Fix up Metadata handling error in DesyVerifier (#189) (
f4668dc) - 0.41.4 (
5243081) - Fix ltacmd copypasta error (#188) (
df054f2) - 0.41.3 (
1aea1de) - Fix up Metadata handling error in NerscVerifier (#187) (
188b34a) - 0.41.2 (
d0e6681) - Added some ltacmd metadata commands (#184) (
7baca55) - 0.41.1 (
6f4982d) - Add GitHub CI for building to DockerHub (#180) (
227e9a9) - Farewell Rucio (#171) (
ba61e05) - Standardize k8s components for deployment (#181) (
24ef7cd) - File Metadata lives in LTA DB not a Bundle spec (#172) (
e93ab24) - Add new Metadata entity to the LTA DB (#177) (
c83b632) - Fixed up some example scripts (#176) (
d218c7a) - Update base image of Dockerfile (#174) (
93c36b7) - Retool RucioStager as RateLimiter (#170) (
1c8833f) - Modify DesyVerifier to update File Catalog (#168) (
76fb25a) - Standardized module configuration (#167) (
febf140) - DesyMoveVerifier ready to verify at final destination (#162) (
01cc95e) - No, your other full bundle path (#157) (
84f3209) - Create a DesyStager component (#156) (
a24753a) - Fixing up GridFTP Replicator issues (#155) (
41a2c20) - Bump to v0.32 (#154) (
e39639c) - Add USE_FULL_BUNDLE_PATH configuration option (#153) (
fc3fe72) - GridFTP Replicator (#152) (
109456b) - Should not be here. (
2655ada) - Add first pass at history utility. First run will be slow. (
d6a9b92) - Improve GET /Bundles query performance (#150) (
a48f536) - Replicator tells Rucio to use Data Warehouse path (#149) (
cf41bd1) - SiteMoveVerifier waits for Rucio file completion (#148) (
06bd71f) - Adds new module DesyVerifier (#147) (
be654d3) - Added module DesyMoveVerifier (#146) (
27e455b) - Add the nersc return path deleter module. It and the nersc-retriever have 1 instance. Tune later (
92f32e0) - Fix unpacker location update in File Catalog (#145) (
f71caa8) - Adjust unpacker path handling (#144) (
f8ed17e) - Provide the arguments to hsi get in the correct order (#142) (
c195ebe) - Locator populates more bundle fields (#141) (
687fcb3) - Fixes for the return path (#140) (
589a852) - Added script to drive NerscRetriever at NERSC (
c8d092d) - Fix up File Catalog location parsing in Locator (#139) (
435ce2a) - Modify bundle path location to use full path (#138) (
cead4ea) - Use fixed size bundles and limit bundle file count (#137) (
65329e3) - Modify lta_cmd to use lazy initialization for commands (#136) (
ae37570) - Fixed up Locator querying LTA REST (#135) (
26d0bda) - Add Unpacker module (#134) (
3691193) - Improve quarantine reason messages (#133) (
1e5de51) - Adds NerscRetriever component (#132) (
f9b1cef) - Add performance logging around MongoDB queries in REST server (#131) (
984ca78) - Updates requirements, includes 503 backoff REST tools (#130) (
85c4079) - Do paging on file catalog searches (#129) (
d71bf11) - Removes JADE LTA code from Bundler (#128) (
cf296d7) - PATCH existing file catalog records for bundle (#127) (
dca0736) - Merge pull request #126 from blinkdog/dev-915-finish-him (
accb340) - Release v0.10.1 (
c894bfc) - Merge pull request #125 from blinkdog/lta-dashboard (
5aa4e5b) - Fixed up type specifications in a function (
a7a814d) - Added command: ltacmd dashboard (
a5edc56) - Merge pull request #124 from blinkdog/dev-924 (
36bd193) - SiteMoveVerifier will run its own checksums sometimes (
6200090) - Merge pull request #123 from blinkdog/master (
279d261) - Component additional status handled properly (
0afad10) - Merge pull request #122 from blinkdog/dev-923 (
0f05ece) - Bumped version to v0.9.0 (
3810d61) - Add /status/nersc route to retrieve cscratch quota (
ba9d1be) - Merge pull request #121 from blinkdog/dev-915-v0-8-1 (
4908ee3) - Pytest-asyncio==0.10.0 because 0.11.0 is a doozy (
c04e39c) - Best effort detachment from Rucio datasets (
1c3d5c5) - Train runs the three main modules in sequence; reduce latency (
fbf0763) - Add train use to supervisor, increase comments (
4740e6e) - Merge pull request #120 from blinkdog/master (
96b0659) - Fixed up f-strings with missing placeholders (
02af173) - Release v0.8.0 (
826468b) - Merge pull request #119 from blinkdog/dev-907 (
d2b65d7) - Creates TransferRequestFinisher and adds bin scripts (
5aa142f) - Merge pull request #118 from blinkdog/dev-906 (
37a7de9) - Added RucioStager component (
ac6c9dc) - Merge pull request #117 from blinkdog/dev-881 (
07ddf9c) - Added deleter component (
cbb56b0) - Merge pull request #116 from blinkdog/dev-896 (
1aea9b9) - Add check to prevent duplicate active requests (
a6bf73c) - Merge pull request #115 from blinkdog/dev-903 (
2cfd3ae) - Normalize paths and whitelist prefixes (
98dd925) - Merge pull request #114 from blinkdog/dev-880 (
f2786b4) - Deleter is now RucioDetacher (
243137a) - Merge pull request #113 from blinkdog/deleter-hotfix (
0b5f3fe) - Hotfix for Deleter updates to TransferRequests (
d6ed3a8) - Merge pull request #112 from blinkdog/dev-876 (
945fac3) - Display cleanup (
cea2010) - Make REST routes lean again (
cf11a85) - Added work_priority_timestamp to LTA DB (
e9fc9c0) - Merge pull request #111 from blinkdog/dev-873-part-two (
c6a4458) - Release v0.4.0 (
6de4405) - Merge pull request #110 from blinkdog/dev-873 (
76fc47a) - Refactor Deleter for k8s deployment (
301af0f) - Merge branch 'master' of github.com:WIPACrepo/lta (
25eada2) - Modify the scripts to use correct paths, tune operations. (
900fd31) - Merge pull request #109 from blinkdog/dev-864-part-deux (
bcc1e4d) - Merge branch 'master' into dev-864-part-deux (
02b4d5d) - Release v0.3.0 (
d828430) - Fix up source path for hsi taping (
1ee58f3) - Merge pull request #108 from blinkdog/dev-864-part-deux (
a317224) - Release v0.2.1 - Replicator Debugging (
f03e595) - Merge pull request #107 from WIPACrepo/move_config_py_to_rest_tools (
8b553a6) - Merge remote-tracking branch 'origin/move_config_py_to_rest_tools' into move_config_py_to_rest_tools (
6001029) - Add type ignore on from_enviromant imports (
878add6) - Update rest_tools commit hash (
6aa4dc2) - Rm config.py and update imports to point to rest_tools (
959a684) - Merge pull request #106 from blinkdog/master (
658212e) - Replicator gets RUCIO_PASSWORD from environment (
8493f82) - Change the location of the check for hsi modules. (
6aceb68) - Expand job-name display buffer for squeue info (
c026c47) - Add site-move-verifier to spawn list. Modify to count only hsi jobs (
83b2f58) - Merge pull request #105 from blinkdog/dev-862 (
6aeced2) - Picker now handles errors while querying File Catalog (
c900e86) - Merge pull request #103 from blinkdog/locator-module (
2cfe8a9) - Coverage tests and requirements update (
e1d5870) - Merge pull request #102 from blinkdog/dev-856 (
18d0533) - Let there be exit codes (
68333d0) - First pass at Locator module (
d3f8ad7) - Merge pull request #101 from blinkdog/dev-766 (
062bb7d) - Addressed Rucio integration issues; developer QoL changes (
3ab9539) - Merge pull request #100 from blinkdog/dev-845 (
6f70ad3) - Removed deprecated command ltacmd catalog load (
8d7da41) - Merge pull request #99 from blinkdog/dev-838 (
181c440) - Trim down bundle metadata recorded in the file catalog (
0678435) - Merge branch 'master' of github.com:WIPACrepo/lta (
a6354cf) - Merge pull request #98 from blinkdog/dev-837 (
688ce73) - Allow MongoDB to do the filtering too (
08a9ec9) - Merge pull request #97 from blinkdog/logical-name-lookup (
baaa7b9) - Change file catalog lookup to use logical_name (
d28aeb0) - Merge branch 'master' of github.com:WIPACrepo/lta (
ec6da6d) - Merge pull request #95 from blinkdog/hsi-full-path (
5997d13) - Modified subprocess argument to use full path to hsi (
ff25042) - Enable mongodb index profiling (#94) (
7b1ecae) - Nersc_supervisor.sh NERSC version (
02a4d00) - Merge branch 'master' of https://github.com/WIPACrepo/lta (
09cbe8f) - Merge pull request #93 from blinkdog/dev-833 (
f89042e) - Added error handling for conditions at NERSC (
3ab210a) - Merge pull request #91 from WIPACrepo/nersc-verifier-part-deux (
bb4c898) - Cleaning up the debug statements and fixing across codebase (
1003118) - Of course we need to decode() it (
c5b5393) - Because the REPL said so (
f3e69bf) - Charm not available in Python 3.6 (
8d942f0) - Third time is the charm (
4e23c67) - Maybe Popen mode works better (
2ddcd13) - Nersc-verifier-part-deux (
538fe82) - Merge pull request #90 from blinkdog/nersc-verifier-two-step (
a318794) - Fixed hsi commands for HPSS verification (
9ad466e) - Merge pull request #89 from blinkdog/minor-updates (
56b8f6f) - Development QoL and minor updates (
67b82e4) - Merge pull request #88 from blinkdog/release-v0.0.27 (
ce8b9af) - LTA Release v0.0.27 (
0f6f9f7) - Merge pull request #86 from blinkdog/master (
eca2e8e) - Added minimum size check to ltacmd request new (
61a4f65) - Merge pull request #85 from blinkdog/dev-802 (
d81389d) - Added --days filter for ltacmd status component (
de15375) - Merge pull request #84 from blinkdog/dev-803 (
ae8a6a5) - Moved scripts from root to bin (
9749d27) - Merge pull request #83 from blinkdog/dev-816 (
de46778) - Fixing up ltacmd and PATCH usage (
a3ec63b) - Merge pull request #80 from blinkdog/dev-815 (
03022ca) - Increase timeout to make CircleCI happy (
07aa081) - Fixed up some NerscMover bugs (
30d66c1) - Nersc_supervisor trim-down to something more like the final form (
7bd1027) - Merge pull request #79 from blinkdog/dev-794 (
720b351) - Bumped version to v0.0.23 (
acb5dbc) - Added command ltacmd bundle overdue (
d3b69d6) - Merge pull request #78 from blinkdog/dev-795 (
c4016b8) - Added archival components to ltacmd status whitelist (
b424346) - Merge pull request #77 from blinkdog/dev-793 (
ab2a099) - Added update-status subcommand to bundle and request (
9252c81) - Merge pull request #76 from blinkdog/master (
28acdf9) - SiteMoveVerifier unclaimed bundles not ready for verification (
4710922) - Merge pull request #75 from blinkdog/dev-790 (
ee8f3f9) - SiteMoveVerifier reports myquota as status (
33000a2) - Merge pull request #73 from blinkdog/dev-792 (
5c3d37f) - Refactor components for one-off runs (
83b5a1f) - Merge pull request #72 from blinkdog/dev-781 (
7dc6d52) - Removed check of concurrently running components (
92db97c) - Fixed up coverage tests (
1aa6f59) - Added preflight checks to NerscMover work cycle (
e5818c7) - Merge pull request #71 from blinkdog/dev-786 (
1f41069) - Picker should not use raw File Catalog entries (
b922951) - Merge pull request #70 from blinkdog/dev-785 (
42dd7c3) - Bumped version to v0.0.18 (
cd2d4b7) - Updated dependencies (
054cf3a) - Added route GET /status/component/count (
e9f051e) - Merge pull request #69 from blinkdog/dev-780 (
39a7ac1) - Removed two unnecessary variables (
96734a0) - Added new command: ltacmd request rm (
30e29b9) - Added docstring to a helper function (
e83cf44) - Refactored ltacmd query commands (
eab1663) - Merge pull request #68 from blinkdog/bundles-need-moar-files (
377b9c0) - Picker no longer drops TransferRequests (
af516db) - Merge pull request #67 from blinkdog/dev-782 (
451db29) - Fixed bug in human readable output of ltacmd request new (
d54dc51) - Merge pull request #65 from blinkdog/release-0.0.16 (
a1b3e95) - Bumped version and modified ChangeLog (
92a323c) - Merge pull request #64 from blinkdog/request-status-moar-details (
cd2ad7a) - Adds bundle details to ltacmd request status command (
c58912f) - Merge pull request #63 from blinkdog/lta-db-config-debug (
ed56bd3) - Now with more verbose configuration at start-up (
d325cdd) - Merge pull request #62 from blinkdog/nersc-integration-refactoring (
8dda8c9) - Modified /Bundles/actions/pop to allow query by destination (
76b1806) - Merge pull request #60 from blinkdog/jade-prefers-terrain (
9a7dddf) - One too many Step 7s (
af9edce) - Modified the order of bundling steps to accommodate JADE-LTA (
628c909) - Merge pull request #59 from blinkdog/more-verbose-ltacmds (
1bbb088) - Modified catalog check and catalog load commands to be more verbose: (
6eee0d3) - Merge pull request #58 from blinkdog/catalog-display-for-debug (
e7097ad) - Added new command catalog display (
23175f6) - Merge pull request #57 from blinkdog/os-is-for-regular-walking-not-for-fancy-walking (
87567ee) - Fixed bug in ltacmd disk file enumeration (
f1aa9f6) - Merge pull request #56 from blinkdog/make-it-so (
a87b2d3) - Added new request estimate command (
d373532) - Added catalog load command to ltacmd (
330fbc5) - Refactored ltacmd; added new catalog check command (
0d70e71) - Merge pull request #55 from blinkdog/all-your-routes-are-belong-to-admin (
8b1320a) - Updated rest server routes to include admin role (
b91fd92) - Merge pull request #53 from blinkdog/all-your-lta-not-belong-to-mongo (
886f4b9) - Fixed date in CHANGELOG (
ec510d4) - Fixed up Mongo authentication, hopefully (
6fcbd34) - Merge pull request #52 from blinkdog/add-mongo-auth (
043777d) - Add optional authentication for MongoDB (
6bd97aa) - Merge pull request #51 from blinkdog/lta-rest-containerify (
7d67f9b) - Single multipurpose container (
b1a2f88) - First draft at containerization (
07a39f2) - Merge pull request #50 from blinkdog/bundle-once-and-die (
5212990) - Added BUNDLE_ONCE_AND_DIE feature (
c7f27e0) - Fixed up lint errors (
0eab838) - Merge pull request #49 from blinkdog/bundler-mysql-hook (
9c7e2a5) - Merge branch 'master' into bundler-mysql-hook (
8e23a31) - Merge pull request #48 from blinkdog/nersc-verifier (
c415a4e) - Merge branch 'master' into nersc-verifier (
241e443) - Merge pull request #46 from blinkdog/nersc-mover (
17484af) - Merge pull request #45 from blinkdog/integration-refactoring (
5569221) - Finished integration testing; MySQL row populated (
48d991a) - Added support for integration testing in development (
c3bcbe6) - Fixed up existing unit tests for Bundler (
a8f4b43) - First pass at Bundler feeding JADE LTA (
b3cc4c2) - Update requirements to latest versions (
85e77a7) - Probably don't need 120 lines of commented out tests (
5616769) - Added coverage tests for NerscVerifier module (
dad4bd9) - Completed coverage tests for NERSC Mover component (
4c84bc9) - Using POST route to add locations (
03af946) - Corrected some issues identified by review feedback (
724fbe7) - First pass at NERSC Verifier module (
953e463) - First pass at NerscMover module (
529d201) - Integration testing refactoring (
31abff0) - Merge pull request #44 from blinkdog/dev584-lta-site-verifier (
4891a87) - Fixes error in RucioTransferService.status() reporting always complete (
b91cc4d) - Implement RucioTransferService.status() (
dc32de9) - Per site PFN and RSE and verification path (
bcc7a8f) - Now with coverage tests (
733dfa5) - Now with correct naming! (
a25a6a4) - First draft at SiteVerifier component (
c104639) - Merge pull request #43 from blinkdog/dev687-lta-deleter-refactor (
a84b581) - Removed _force_cleanup() as out of scope (
0bd8a5e) - Added unit tests for Deleter (
13a06ce) - Refactored Deleter file deletion into RucioTransferService (
030754c) - First pass at Deleter refactoring (
36dd7bb) - Merge pull request #42 from blinkdog/dev686-refactor-replicator (
7019de8) - Fixed up tests to use examples (
ed0c74f) - Added ignore for local testing configuration (
a811a88) - Added some configuration file examples (
6d3b3c5) - Unit tests for Replicator (
463a700) - First pass at Replicator refactoring (
86b00f6) - Merge pull request #41 from blinkdog/dev685-transfer-service (
d222dd3) - Fixed up a NotImplementedError error message (
79e5332) - First pass at TransferService refactoring (
6b14b7a) - Merge pull request #40 from blinkdog/dev688-refactor-bundler (
79a6ec2) - Bundler refactoring first draft (
601d18e) - Updated requirements and fixed D413 issue (
f370549) - Refactored LTA DB routes (
2fdb9d9) - Merge pull request #39 from blinkdog/dev681-refactor-picker (
b02c987) - More pythonic and less complicated (
34102be) - Refactored the Picker to create Bundles (
f2af4df) - Refactored the Picker to make use of changed LTA DB route (
4443dde) - Changes to TransferRequest pop action (
5a3546f) - Looks like somebody withdrew pyparsing 2.4.1 from circulation (
f9388fb) - Refactored Picker consumption of TransferRequests from the LTA DB (
d4a1536) - Merge pull request #36 from blinkdog/dev680-refactor-ltacmd-create-transfer-request (
5e6a90a) - Refactored command: ltacmd request new (
dd2af68) - Merge pull request #35 from blinkdog/dependency-upgrade (
c2f1638) - Updated dependencies to latest versions as of 22 July 2019 (
a1bbeb1) - Merge pull request #34 from blinkdog/deleter-module (
05b2545) - Implementation of Deleter module (
7302481) - Deleter class shell; stub implementation (
3c1ffd0) - Added adler32sum command script to project root (
ba1794e) - Merge pull request #32 from blinkdog/replicator-resync-try2 (
5c68c6e) - Final draft for Replicator module (
6f0dd7e) - Merge branch 'master' into replicator-resync-try2 (
96c3a5f) - Merge pull request #33 from WIPACrepo/token-svc (
c9fc0a4) - Poking at Rucio's REST API (
624260c) - Use fake token service in tests (
b68e8df) - Fix copypasta comment (
38fea66) - Initial component structure (
8d89196) - Merge pull request #31 from blinkdog/refactor-lta-modules (
cc283d5) - Refactoring Part 1: Bundler (
5fff800) - Refactoring Part 1: Picker (
827c034) - Merge pull request #30 from blinkdog/lta-lifecycle-commands (
f972fa6) - Used a better form of module import (
68ee197) - Ltacmd drain/kill/restart/start/stop (
160cfa6) - Added types and shored up tests to satisfy CircleCI (
fa95ea4) - Implemented drain/kill/start/stop commands (
7c293b7) - Moved lta_cmd.py from resources to lta (
3a3d9c3) - Merge pull request #28 from blinkdog/lta-command-line (
69470aa) - Merge branch 'master' into lta-command-line (
1f00e5b) - Merge pull request #27 from blinkdog/bundler-component (
cd069db) - Gotta refactor the unit tests for monitoring as well (
b725c84) - Refactored some things out to make them nicer (
7c7fb63) - Improvements to ltacmd, refactoring support scripts (
7929066) - Fix up unit tests for GET /status (
d9cba94) - Implement ltacmd status command (
7a8eb87) - Implemented ltacmd request command (
6048bfb) - Thank you CircleCI for keeping me on my toes (
a1d4ffe) - Implement Bundler component (
46db3ed) - Merge pull request #25 from blinkdog/bundle-route-support-merge (
efe8acf) - Refactor test fixtures, plagiarize trivial dependency (
76fc3e0) - Added Mongo indexes for sort fields (
fa01a20) - Pacify the cruel CircleCI (
efa18c1) - Added tests and fixed bugs in /Bundles routes (
8346f52) - Merge branch 'bundle-route-support' into bundle-route-support-merge (
79c22a3) - Add support for /Bundles routes (
3a4b6b4) - Update requirements.txt with better rest-tools handling (
545170c) - Merge pull request #24 from blinkdog/shards-are-the-secret-ingredient-in-the-web-scale-sauce (
ef3120e) - Another try with CircleCI (
24530bc) - Lift lion cub to complete the CircleCI of life (
76ece7f) - Result collection limits, not DB query limits (
ec83c1c) - Merged a small change from the review (
ffd4bcd) - Better handling for Mongo transaction semantics (
ca2ac81) - Update lta/rest_server.py (
17f18ef) - LTA DB now uses MongoDB as a data store (
b493849) - Update requirements to latest versions (
95d85c0) - Merge pull request #23 from blinkdog/files-actions-pop (
3481fc8) - PEP8 imports for rest_server.py (
1ab1d42) - File field 'status' doesn't contain a site (
6c483ce) - Removed binpacking tests (
4b3c077) - Removed unnecessary set_status(200) calls (
82e8e3e) - Implements /Files/actions/pop for the LTA DB (
9f66622) - Merge pull request #22 from blinkdog/lta-db-files-routes (
15bffe3) - Corrections to API for LTA DB; better tests (
9db9719) - Merge pull request #21 from blinkdog/dependency-management-helper (
804f3b5) - Implements most of the Files routes for the LTA DB (
f8e5e3c) - Updated documentation and helper scripts for dependency management (
4ff79b2) - Merge pull request #20 from blinkdog/delint-rest-server (
97048a0) - Fixed up lint issues in rest_server (
cc49b42) - Merge pull request #19 from blinkdog/delint-monitoring (
15d385c) - Fixed up lint issues with monitoring and tests (
73c3198) - Merge pull request #14 from blinkdog/implement-picker-guts (
c0b9596) - Merge branch 'implement-picker-guts' of github.com:blinkdog/lta into implement-picker-guts (
b375016) - Coverage for picker.py back to 100% (
4a83135) - Update README.md (
e56d62e) - Update lta/picker.py (
aa02c5f) - Circleci badge as a shield (
fd1b4e7) - Circleci badge (
ac5766a) - Add mypy to setupenv (
0bd1793) - Add flake8 to setupenv (
61d53f0) - Try again (
9446e23) - All in one shot (
903eb98) - Attempt to build with free circleci account (
9351699) - REST DB claimant and DELETE TransferRequest (
9193dd1) - Some housekeeping edits; remove lint, put testing code away properly (
f7bb4e0) - Added type annotations to internal Picker methods (
9122cf7) - Fun with Python iterables (
fa998ad) - Re-hire the mad scientist; mutation isn't all bad (
9a93fe5) - Use canonical timestamp format for File object (
2976398) - No more mad scientist mutation of innocent arguments (
24f6a3a) - Picker raises exception on deficient configuration (
0505ac8) - Picker provides some sane default configuration (
ee4fc25) - Setec Astronomy (
6e8a040) - Too many secrets (
8bba278) - Coverage tests for FilesActionsBulkCreateHandler (
ffc18f3) - Added 'prometheus-client' requirement; repaired broken tests (
cd8d734) - Most of the Picker logic complete sans tests (
111d7ca) - Merge pull request #13 from blinkdog/refactor-picker-rest-tools (
e86c47d) - Printing local tokens for fun and profit (
62e3383) - Changed exception type to make heartbeating great again (
ed9adb7) - Refactored heartbeat PATCH to use RestClient (
bafabb3) - Merge pull request #12 from WIPACrepo/monitoring-module (
e4ae754) - Clean up types (
6831847) - Monitoring module with prometheus output (
ca40425) - Merge pull request #11 from blinkdog/snake-molting (
731a2d2) - Smoothing out some of the rough edges in snake (
0426a95) - Merge pull request #10 from WIPACrepo/rest_api_status (
dad15a2) - Merge pull request #9 from blinkdog/lens-to-read-structured-logs (
398f2a1) - Hard to improve on ISO 8601 (
dc76228) - Basic status handling (
461899e) - Added a lens to make console debugging easier (
e0fbcc0) - Merge pull request #8 from WIPACrepo/api_transfer_requests (
ff98169) - Fixes from code review (
edb7244) - Get actions/pop working. put constraints on transfer request data. get coverage up (
f62a856) - More better linting (
6f7be88) - Merge from master (
b75817f) - Tranfer request api - first pass (
54b2c44) - Merge pull request #7 from blinkdog/flake8-docstrings (
569b915) - Committed too fast; so minor touch-ups (
9d9ddfd) - Merge pull request #6 from blinkdog/picker-enhancement-merge (
b23300e) - Added docstring linting to the project (
2188c74) - Fix up for code review (
29780b1) - Update lta/config.py (
775599b) - Fix up for xfail tests; config type massage (
c035b24) - Merge branch 'picker-rest-tools-refactor' into picker-enhancement-merge (
6a276f0) - Add fixture for testing rest server, to make it easier (
609dd60) - First pass at a bunch of improvements (
836d50a) - Merge pull request #5 from WIPACrepo/rest_api (
fdb23dc) - Add import for tornado.web so we can get the HTTPError (
54be937) - Fix copy/paste mistakes (
d430903) - Fix reachability test. this solves the coverage problem (
f56f013) - More better coverage (
5a5ebee) - Update tests/test_rest_server.py (
6c2eb99) - Update lta/rest_server.py (
12f1cf4) - Make NA private. if we want a required variable, use the string or list keys format. can always call the function multiple times (
5a52755) - Server infrastructure and test (
8e8e961) - Update config to take default values (
bd1acae) - Add rest-tools to env (
01d4289) - Merge pull request #4 from blinkdog/release-v0.0.4 (
41bba66) - Release v0.0.4 (
24989b9) - Heartbeat on a different thread with async/await (
a850b2c) - Merge pull request #3 from blinkdog/release-v0.0.3 (
bdb7b95) - Release v0.0.3 (
1250a7a) - Created Administrator documentation in doc/admin.md Created configuration dictionary creation in config.py Created picker.py to hold Picker component code Added pytest-mock and requests to requirements.txt Fixed developers e-mail in setup.py Fixed lots of little flake8 issues in setup.py Fixed hashbang in snake script Added another directory to clean task in snake script Removed formatting cruft in snake script Added unit tests for config.py and picker.py (
805d6fc) - Merge pull request #2 from blinkdog/release-v0.0.2 (
3c16c26) - Release v0.0.2 (
7eb4689) - Some more project structure work (
bb96775) - Merge pull request #1 from blinkdog/initial-import (
6c67851) - Project setup scripts (
336aeed) - Initial commit (
2ba008d)
- Bump wipac-rest-tools (#280) (
1494e0b) - Try another semantic release fix (
758c605) - Fix up ci actions (#278) (
7824614) - Use github container registry (#276) (
ff88925) - Fix monitoring script syntax (#277) (
187bb14) - Update to more modern tornado asyncio wait syntax (#275) (
42b64d4) - Paths and Pins (#274) (
7c5cd53) - Fix tests (#273) (
c9f9323) - Modify nersc_controller to avoid getting OOM killed (#272) (
6fa23c5) - Ltacmd catalog path (#270) (
1d39a31) - Update scrontab to prune slurm logs (#269) (
c53611b) - Metrics Reporting with Prometheus (#268) (
9a638a0) - Save status to original_status when moving to quarantine (#267) (
126867e) - Progress monitor (#257) (
52e86ff) - Bundler removes broken files of previous attempts (#261) (
d11e1df) - Bump py-versions CI release v2.1 (#259) (
bf64bb1) - Request minimum and maximum time for slurm jobs at NERSC (#258) (
3e2df4f) - Nersc-verifier should use configured hpss_avail path (#256) (
a2ab4cd) - LTA should use the long-term-archive client secret (#255) (
2888a26) - Updated call to hpss_avail command for Perlmutter (#254) (
be05c3d) - Added scrontab job to clean Slurm logs (#253) (
462224c) - Update LTA configuration to use Community File System instead of Perlmutter Scratch (#252) (
c334d15) - Cori Scratch is dead, long live Perlmutter Scratch (#250) (
8f6a12b) - Add -debug flag to grid-proxy-init (#249) (
5c4cf50) - Add Globus tools like globus-url-copy to Docker containers (#248) (
006880c) - Update File Catalog creds supplied to ltacmd (#247) (
fae7161) - Small changes that didn't make the last commit (#246) (
048487f) - Create new controller script for NERSC (#245) (
a030f8f) - New Auth Deployment (#244) (
3418b5a) - Support new authentication for File Catalog (#243) (
406945c) - Modified auth configuration; fixed logging (#242) (
ec23977) - Update LTA authentication (#241) (
96f7cbf) - Fix the return path (#240) (
54fe5fa) - Modify parameters for small transfer requests (#235) (
95913b2) - Jadetools from lta-vm-2 (#234) (
6c3bc11) - Use better event loop management (#233) (
bc5267b) - Make explicit dependency on pyjwt with crypto (
ae48e62)
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
- for new features
- for changes in existing functionality
- for soon-to-be removed features
- for now removed features
- for any bug fixes
- in case of vulnerabilities
- Removed deprecated Rucio components and their dependencies
- RucioStager has been renamed RateLimiter
- DesyVerifier component now populates the File Catalog from verified bundles
- DesyVerifier and NerscVerifier stop recording constituent files in the File Catalog record of the bundle
- Modified modules to use standardized configuration for interconnection
- DesyMoveVerifier verifies bundles at the final destination at DESY
- GridFTP Replicator now computes the correct path for USE_FULL_BUNDLE_PATH
- Added DesyStager component to stage files for transfer to DESY
- Modified the way GridFTP Replicator builds paths to use GridFTP
- Changed the way errors in globus-url-copy are handled due to spurious returncode
- Added DEST_SITE to GridFTP Replicator so instances can discriminate by destination
- GridFTP Replicator module for replicating files using GridFTP
- USE_FULL_BUNDLE_PATH option for GridFTP Replicator and SiteMoveVerifier
- LTA DB routes for GET /Bundles and GET /Bundles/{uuid} are more efficient
- Modified ltacmd script to take advantage of GET efficiencies in LTA DB
- Replicator can now register files at the Rucio destination with the Data Warehouse path.
- SiteMoveVerifier now waits for Rucio to indicate a file is complete
- DesyVerifier verifies files were properly copied to DESY
- DesyMoveVerifier verifies Rucio completion for bundles moved to DESY
- Unpacker specifies the correct file and path to update in the File Catalog
- Unpacker uses the correct field in the bundle to compute paths
- NerscRetriever now uses the correct ordering for arguments to hsi get
- Locator now populates bundle_path, checksum, size, and verified fields
- Locator now populates claimed and files fields
- NerscRetriever now looks for bundles in status located
- ltacmd status commands now work for return requests and bundles
- Script to drive NerscRetriever at NERSC
- Locator properly parses File Catalog locations when choosing bundles
- NerscVerifier now uses full path when adding locations to File Catalog records
- Fixed locator to properly query LTA REST server for TransferRequests
- Format of quarantine reason messages for Bundles
- Added NerscRetriever component for reading bundle files from HPSS at NERSC
- Added DEBUG logging for MongoDB queries in the REST server
0.14.0 - 2020-08-06
- New LTA_MAX_BODY_SIZE configuration; not yet functional in underlying library
- Updated requirements including new rest-tools with 503 backoff
0.13.0 - 2020-07-30
- Modified Picker and Locator to use File Catalog pagination for large record sets
- Coverage report no longer includes lta_cmd or transfer service implementations
- Bundler no longer adds a row to the deprecated JADE LTA database
- NerscVerifier will now PATCH a file catalog record that already exists for the bundle
- TransferRequestFinisher will actually finish Bundles and TransferRequests
0.10.0 - 2020-04-27
- SiteMoveVerifier will ignore Rucio and run checksums on its own in some cases
0.9.1 - 2020-04-24
- PATCH /status now adds component-specific status data properly
0.9.0 - 2020-04-23
- Added new /status/nersc route to LTA DB
- Relaxed RucioDetacher to act on a best effort basis
0.8.0 - 2020-04-10
- Created new Deleter module
- Created new RucioStager module
- Created new TransferRequestFinisher module
- Added scripts to bin directory to run new modules
- Added check in ltacmd to prevent duplicate transfer requests
- Added check in ltacmd to normalize transfer request path
0.7.0 - 2020-04-02
- Renamed Deleter module to RucioDetacher module
- Modified RucioDetacher to detact from both source and destination datasets
- Modified RucioDetacher to put bundles into "detached" status after processing
0.6.0 - 2020-03-29
- Deleter queries the LTA DB properly to fix TransferRequests
0.5.0 - 2020-03-26
- New work_priority_timestamp to LTA DB
- New commands: ltacmd {bundle,request} priority reset
- Modified command to display priority date for bundle and request
0.4.0 - 2020-03-26
- Refactored Deleter component to be make it ready for Kubernetes deployment
0.3.0 - 2020-03-24
- Fixed source location for taping in NerscMover component
0.2.1 - 2020-03-14
- Added better logging to RucioClient to help debug Rucio error codes
- Added --status flag to display status in ltacmd bundle ls
0.2.0 - 2020-03-11
- Replicator now takes RUCIO_PASSWORD from environment
- Replicator handling of errors while replicating Bundles
0.1.1 - 2020-03-09
- Picker handling of errors while querying the File Catalog
0.1.0 - 2020-02-25
- bin/site-move-verifier.sh was created to run a SiteMoveVerifier component
- Added /dump directory to .gitignore for developer convenience
- Modified docker-deps-{down,up}.sh to handle jade_lta_test container better
- Modified ltacmd to display the reason a TransferRequest or Bundle is quarantined
- replicator.py will now quarantine bundles that fail processing for some reason
- site_move_verifier.py will now quarantine bundles that fail processing for some reason
- Modified ltacmd script to have test/production configuration lines
- Updated Python requirements to latest available versions
- resources/rucio-workbench.sh has the correct information for production Rucio
- rucio_workbench.py had some minor development changes
- resources/test-data-reset.sh was modified to use resources directory scripts
- resources/test-data-reset.sh was modified to test Rucio-talking components
- bin/replicator.sh was sorely outdated, and has been brought up to date
- bundler.py not to fail hard when unable to reach MySQL dependency
- picker.py now awaits the function to quarantine a bad transfer request
- replicator.py now queries the LTA DB with the correct SOURCE_SITE
0.0.31 - 2020-02-11
- NerscVerifier trims down the bundle metadata added to the File Catalog
0.0.30 - 2020-02-11
- Allow MongoDB to use the hash index, but don't move the filtering to Python
0.0.29 - 2020-02-10
- ltacmd catalog commands use queries more friendly to MongoDB look up
0.0.28 - 2020-02-10
- NerscMover and NerscVerifier now use full path for hsi command
0.0.27 - 2020-01-30
- Updated Python package requirements to latest versions
0.0.26 - 2020-01-29
- Added --days to ltacmd status {component} to cull old status heartbeats
0.0.25 - 2020-01-27
- Fixed ltacmd bundle overdue not showing quarantined bundles
- Modified ltacmd bundle update-status to remove quarantine reason
- Fixed PATCH methods in Deleter, NerscVerifier, Picker, Replicator, and SiteMoveVerifier
0.0.24 - 2020-01-27
- nersc-mover.sh script now uses correct TAPE_BASE_PATH
- NerscMover component provides correct source path to HSI command
- PATCH methods in NerscMover only patch necessary fields in the bundle
- ltacmd bundle overdue no longer requires Python 3.7+
0.0.23 - 2020-01-23
- Added a new command: 'ltacmd bundle overdue'
- Added a new command: 'ltacmd bundle update-status'
- Added a new command: 'ltacmd request update-status'
- Whitelisted archival components in command: 'ltacmd status'
0.0.22 - 2020-01-22
- SiteMoveVerifier will unclaim bundles that are not yet ready for verification
0.0.21 - 2020-01-22
- SiteMoveVerifier now reports myquota command at NERSC as status
0.0.20 - 2020-01-21
- All components have new configuration RUN_ONCE_AND_DIE; default false
- Bundler no longer has BUNDLE_ONCE_AND_DIE
0.0.19 - 2020-01-15
- Picker now cooks catalog records before using them as bundle metadata
0.0.18 - 2020-01-14
- Added a new command: ltacmd request rm
- Added new LTA DB route: GET /status/{component-type}/count
- Refactored some commands: ltacmd {bundle,request} {ls,status}
0.0.17 - 2020-01-13
- Fixed bug in human readable output of command ltacmd request new
- Picker will quarantine a TransferRequest if it cannot make a bundle
0.0.16 - 2020-01-06
- Command: ltacmd request status now shows information about bundles
0.0.15 - 2020-01-03
- LTA DB logs its configuration for debugging purposes
0.0.14 - 2020-01-03
- Modified /Bundles/actions/pop to allow query by destination
0.0.13 - 2019-12-12
- Modified the order of bundling steps to accommodate JADE-LTA
0.0.12 - 2019-12-05
- Modified: ltacmd catalog check to be more verbose in non-JSON mode
- Modified: ltacmd catalog load to be more verbose in non-JSON mode
0.0.11 - 2019-12-05
- Command: ltacmd catalog display
0.0.10 - 2019-12-04
- Bug in the way ltacmd was enumerating files on disk
0.0.9 - 2019-12-04
- Command: ltacmd catalog check
- Command: ltacmd catalog load
- Command: ltacmd request estimate
- Removed component daemon stuff out of ltacmd
0.0.8 - 2019-11-22
- Normalized roles for routes in the LTA REST server; added 'admin' to all
- Updated requirements to latest versions
0.0.7 - 2019-11-19
- Fixed the way authentication credentials are provided to MongoDB
0.0.6 - 2019-11-15
- Authentication credentials now provided to MongoDB
0.0.5 - 2019-11-13
- Started tracking versions again because deployment to production
0.0.4 - 2019-01-03
- Status heartbeat reporting on an independent thread
- Independent sleep configuration for heartbeat and worker threads
- Configuration variable HEARTBEAT_SLEEP_DURATION_SECONDS added
- Configuration variable WORK_SLEEP_DURATION_SECONDS added
- Added application requirement requests-futures
- Added unit testing requirement pytest-asyncio
- Use of requests changed to FuturesSession of requests-futures
- Unit tests modified for async nature of heartbeat function
- Picker documentation in doc/admin.md
- Configuration variable SLEEP_DURATION_SECONDS removed
0.0.3 - 2018-12-18
- Administrator documentation in doc/admin.md
- Configuration dictionary creation in config.py
- First draft of Picker component in picker.py
- Requirements: pytest-mock and requests
- Clean task in snake script removes another directory
- developers@iwe e-mail in setup.py
- lots of little flake8 issues in setup.py
- hashbang in snake script
- formatting cruft in snake script
0.0.2 - 2018-12-12
- Changelog for the project
- Configuration for some tools to setup.cfg
- Project helper script: snake
- Updated documentation in README.md
- Project setup scripts