-
Notifications
You must be signed in to change notification settings - Fork 3
/
ChangeLog
441 lines (306 loc) · 15.3 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
2017-04-20 Frank Scheiner <[email protected]>
* 0.8.1 (gtransfer toolkit):
- Fixes
The halias(1) manpage used a wrong name for the halias configuration file
("halias.conf" instead of "aliases.conf"). This is now corrected.
2017-03-27 Frank Scheiner <[email protected]>
* 0.8.0 (gtransfer toolkit):
- New functionality
The (guc) sync level used by gtransfer for transfers can now be selected by
users. Syncing can also be disabled.
The host aliases can now be retrieved and installed from a file provided
locally or from a remote repository, just like dpaths and dparams.
Wrappers for uberftp (called gtools) were added that provide shorthand
functionality to users for common operations on remote GridFTP servers like
(g)cat, (g)ls, (g)mkdir, (g)mv and (g)rm. All wrappers support host aliases and
remote directory browsing just like gt.
2016-09-08 Frank Scheiner <[email protected]>
*0.7.1 (gtransfer toolkit):
- Fixes
Due to an erroneous comparison, data transfers with URLs that contained
host aliases that weren't prefixed with usernames failed. This because
a wrong replacement in the GridFTP URLs was applied due to the
erroneous comparison. This is now fixed both in gtransfer and the
gtransfer bash completion.
2016-09-01 Frank Scheiner <[email protected]>
*0.7.0 (gtransfer toolkit):
- New functionality
Gt now also supports "<USER>@" as prefix to host aliases. This was
already included in the Bash completion but not activated until now.
- Fixes
"helperFunctions/getURLWithoutPath()" did not anticipate user names in
front of host addresses in GridFTP URLs. This is now corrected.
Until now gt overwrote log files from unsuccessful tries and retries of
a transfer. This was a source of irritation during evaluation of
transfers as the start and stop timestamps did not resemble the start
and stop of the whole transfer. And it also meant a loss of possibly
helpful info. This was now changed so that older logfiles of the same
transfer are preserved and newer ones get an additional identifier for
each retry attached to their names.
2016-06-30 Frank Scheiner <[email protected]>
*0.6.0 (gtransfer toolkit):
- New functionality
The stall timeout setting for globus-url-copy is now configurable via
the gtransfer configuration file or from a dparam. If not configured in
a dparam, the setting from the gtransfer configuration file is used. The
old hard-coded default of 30 seconds is not necessarily suitable for
data transfers from sites which first need to stage in files from e.g.
tape before a data transfer can begin (e.g. sites using HPSS).
2016-04-12 Frank Scheiner <[email protected]>
*0.5.1 (gtransfer toolkit):
- Fixes
Using a "/" or "/*" in a source URL to transfer all files in a given dir
now no longer makes problems for "stat" if only one file existed in the
given dir. This fixes issue #41.
Special chars in URLs need to be percent or URL encoded when provided to
globus-url-copy on the commandline. This means that currently URLs with
special chars also have to be provided to gt percent or URL encoded.
This is now also mentioned in the gt manpage.
Before using stat, the path and file name in the source URL are now
decoded beforehand, which makes the call to stat also working for
URLs with special chars. This fixes issue #42.
2016-04-12 Frank Scheiner <[email protected]>
*0.5.0 (gtransfer toolkit):
- New functionality
Gt options can now be used multiple times on the command line, the last
setting of an option wins. This allows for setting gt default options
via shell aliases.
- Fixes
Added missing gt options to bash completion.
Enhanced the workaround for the guc bug (no size string in created
transfer lists if only a single file is to be transferred) to get the
size for `file://` source URLs with the `stat` tool locally.
2016-01-27 Frank Scheiner <[email protected]>
*0.4.1 (gtransfer toolkit):
- Fixes
Multipathing: A wrong variable name was corrected which prevented correct
functioning of multipathing when using a transfer list as input to gt.
Host aliases: Until now when trying to dealias host aliases that cannot be
accessed by the effective user, the halias tool didn't exit a while loop and
hence looped until interrupted by the user. This is now corrected.
2015-09-17 Frank Scheiner <[email protected]>
*0.4.0 (gtransfer toolkit):
- New functionality
Users can now enable data channel checksumming or data channel encryption with a
gtransfer option.
- Fixes
PID functionality: The rule for resolving a PID to an URL was adapted to backend
changes in EUDAT.
2015-04-23 Frank Scheiner <[email protected]>
*0.3.0 (gtransfer dist):
- New functionality
Data transfer using multipathing: Gtransfer can now distribute a data transfer
over multiple paths. This way users can benefit from the combined bandwidth of
multiple paths. For more details please have a look into the [README].
[README]: https://github.com/fr4nk5ch31n3r/gtransfer#data-transfer-using-multipathing
- Fixes
Included a workaround for an old `globus-url-copy` (guc) [bug]: A transfer list
created for a single file does not contain the size of the file. gt now
determines the file size in this case with `uberftp` and updates the transfer
list. The file size is needed for calculating the transfer rate and for the
bin packing heuristics used by the multipathing functionality.
[bug]: https://globus.atlassian.net/browse/GT-318
For native OS packages (like RPMs) all tools now use `/usr/share/gtransfer` as
base path for bash libraries. This fixes [issue #26].
Integrated a 1 second timeout to the ping command in
`helperFunctions/isValidUrl()` to keep wait time low when transferring between
hosts not answering echo requests.
[issue #26]: https://github.com/fr4nk5ch31n3r/gtransfer/issues/26
2013-10-14 Frank Scheiner <[email protected]>
gtransfer dist v0.2.1:
*0.1.1 (halias):
Fixed [issue #21]. This fix corrects a wrong variable name in halias that
prevented correct functioning after doing a system installation.
[issue #21]: https://github.com/fr4nk5ch31n3r/gtransfer/issues/21
2013-08-23 Frank Scheiner <[email protected]>
gtransfer dist v0.2.0:
*0.2.0 (gtransfer):
Added the following functionality:
- Support for host aliases.
Gtransfer can use host aliases as alternatives to host addresses. E.g. a user
can now use "myGridFTP:" and "gsiftp://host1.domain.tld:2811" synonymically.
- Support for persistent identifiers (EUDAT/EPIC) as source URL
Gtransfer can use persistent identifiers (PIDs) as used by EUDAT and provided by
EPIC as source of a data transfer.
Many thanks to Giuseppe Fiameni (CINECA) and Johannes Reetz (RZG) for explaining
the background and ideas of persistent identifiers (PIDs) and especially to
Giacomo Mariani (CINECA) for providing a test environment, the iRODS micro
services and helpful code examples that allowed me to get this extension to work
with gtransfer.
Also updated and reworked the documentation. Now using Markdown formatted files
as sources for manpages and PDFs.
2013-01-25 Frank Scheiner <[email protected]>
gtransfer dist v0.1.2:
*0.1.2 (gtransfer):
gt v0.1.1 always performed recursive transfers. This is now corrected and per
default gt now performs non-recursive transfers again. Fixes #5. Also defused a
guc call. Fixes #4.
*(install.sh):
The installer now also install the exemplary chunkConfig file. Fixes #6.
2013-01-24 Frank Scheiner <[email protected]>
gtransfer dist 0.1.1:
*0.1.1 (gtransfer):
The chunk creation in the auto-optimization of gtransfer v0.1.0 did not
correctly handle cases where no files of specific file size classes existed in
the list of files to transfer. This is now corrected. Fixes #1. Also added
functionality to remove the directory for gt temp files after a successful
transfer.
*0.1.1 (dpath):
dpath now correctly ignores empty dpath directories. Fixes #2.
*0.2.0 (dparam):
Updated dparam list logic to dpath counterpart.
2013-01-18 Frank Scheiner <[email protected]>
gtransfer dist 0.1.0:
*0.1.0 (gtransfer):
Merged dev version into master version of gtransfer. This added the following
functionality:
- Support for bash libraries, specific functionality is now contained in
libraries.
- Support for list transfers.
- Support for configurable auto-optimization depending on the file size.
- Support for reliabilty (incl. configurable number of retries).
- Support for interruptible and continuable transfers.
- Temporary files created during a transfer are now located below
"$HOME/.gtransfer/tmp/<SHA1_HASH_OF_COMMAND_LINE>/".
*0.1.0 (dpath):
Now supports dpath retrieval via globus-url-copy which allows to use "http://",
"https://", "gsiftp://" and "file://" URLs. Also integrated a reindexing
functionality to allow to use already existing dpaths and updating the indices
with sources and destinations from those dpaths.
*0.1.0 (dparam):
Now also supports dpath retrieval via globus-url-copy which allows to use
"http://", "https://", "gsiftp://" and "file://" URLs.
2012-11-19 Frank Scheiner <[email protected]>
gtransfer dist 0.0.10a:
*0.0.10a (gtransfer):
Corrected a call to the usageMsg() function that still used the old function
name.
2012-11-14 Frank Scheiner <[email protected]>
gtransfer dist 0.0.10:
*0.0.10 (gtransfer):
The connection check functionality (done prior to a transfer) can now be
deactivated from the config file. In addition the timeout used for the
connection check is now also configurable from the config file. The v0.0.10
connection check functionality behaves like in pre-v0.0.10 versions when a
pre-v0.0.10 config file is used.
*0.0.6 (dpath):
dpath now supports to create all possible dpaths between host addresses given
in a file (BATCH CREATE mode). It now can also use a specific dpath template in
CREATE and BATCH CREATE mode. In addition the list sources/destinations
functionality now can also use index files for sources (sources.index) and
destinations (destinations.index). This decreases response time of the remote
host completion of gtransfer considerably when many dpaths are available. The
index files are automatically created and updated during dpath creation.
(installer):
The installer script was reworked and now does not create links with absolute
paths for a system install. In addition the INSTALL document was renamed and
reformatted to Markdown syntax.
2012-11-03 Frank Scheiner <[email protected]>
gtransfer dist v0.0.9c:
Renamed modulefiles to generic names. Version strings inside of the modulefiles
are now configurable by the tarball build script (not part of the repo!). Names
of modulefiles are now also automatically renamed to the current dist version
when building tarballs.
Installer script adapted (included new name for README.md and bash completion
file).
Some rework on manual pages (consistency, config file search order, etc.).
2012-10-03 Frank Scheiner <[email protected]>
gtransfer dist v0.0.9b:
*0.0.9b (gtransfer):
Hotfix for a serious bug. Due to an unmodified call to an internal function,
which got a new interface in v0.0.8, v0.0.8 to v0.0.9a versions remove the
source directory after transferring the data (in a single step transfer) to the
destination directory if no dpath exists for the current source destination
combination. This is now fixed.
2012-10-03 Frank Scheiner <[email protected]>
gtransfer dist v0.0.9a:
*0.0.9a (gtransfer), 0.0.5b (dpath), 0.0.5a (dparam)
Added "/etc/gtransfer" search path for gtransfer configuration files to support
native OS packages. Removed non-standard "/opt/gtransfer/etc" search path
(although the FHS 2.3 is not fully clear on this). Adapted installer script and
tools to automatically add a config file search path for the gtransfer
installation dir structure. Updated man pages. Filename globbing is deactivated
inside of gtransfer.
2012-05-15 Frank Scheiner <[email protected]>
gtransfer distribution v0.0.9:
*0.0.9 (gtransfer)
Added a workaround for a uberftp "anomaly". Two or more subsequent calls to
uberftp don't work if the time frame in between is too short. The second call to
uberftp then fails and emits "Failed to connect to <IP_ADDRESS> port
<PORT_NUMBER>: Cannot assign requested address". The current workaround should
work for multi-step transfers that are not too short. If they're too short, the
post-command that removes the transit directory might be executed too early
after the pre-command that creates the transit directory and chmods it to 0700
was executed.
2012-05-11 Frank Scheiner <[email protected]>
gtransfer distribution v0.0.8:
*0.0.8 (gtransfer)
Added workaround for guc limitation. guc creates dirs with group and world r-x
permissions set. Because of this, temp dirs on transit sites were readable by
others than the owner during a transfer from source to transit site, transit
site to transit site and transit site to destination site.
The workaround now uses a tgftp pre-command to create the dir and then make it
only accessible by the owner (chmod 0700). This is done with two calls to
uberftp in the cases described above.
!IMPORTANT!
Additionally internal logic changes were introduced, that require a change in
the gtransfer configuration file. "verboseExec" is now activated when set to "1"
instead of "0". PLEASE CHANGE THIS IN YOUR CONF FILES!
2012-03-15 Frank Scheiner <[email protected]>
gtransfer distribution v0.0.7e:
Added PDF versions of the manual pages to the repo.
*0.0.7d (gtransfer)
Corrected a bug in the getPathFromURL() function that showed wrong behaviour for
URLs with files in the root directory ("/"). Example:
gsiftp://host.domain:2811/file.txt
This was parsed as follows:
protocol spec: gsiftp://
URL without path: gsiftp://host.domain:2811
path: "" (!) (=> should be "/")
file: file.txt
Thanks to Giacomo Mariani (CINECA) for detecting this bug.
2012-02-03 Frank Scheiner <[email protected]>
gtransfer distribution v0.0.7d:
Introduced own version number for gtransfer distribution. The version number of
the dist. is no longer determined by gtransfer itself, as updates to dpath and
dparam would not justify a new version number for gtransfer.
2012-01-19 Frank Scheiner <[email protected]>
*0.0.5a (dpath)
Corrected a wrong function call. The function name was still using the old
nomenclature.
2012-01-09 Frank Scheiner <[email protected]>
Minor changes in man pages and modulefile.
2011-12-18 Frank Scheiner <[email protected]>
*0.0.7c (gtransfer)
*0.0.4c (dparam)
*0.0.4d (dpath)
all:
Removed describing content about the configuration file contents from the
scripts. On a specific version of bash (3.1.17(1)) this didn't work and created
an error message. The following construct was used before:
"
VARIABLE=$( cat <<EOF
[...]
EOF
)
"
Improved infile documentation of "gtransfer.conf". Improved installation
documentation in INSTALL and "install.sh" script. The gtransfer tarball should
now conform to the FHS v2.3 standard.
*0.0.5 (dpath)
Dpath now supports listing dpaths for both system and user dirs. Listing of
sources and destinations now also includes data from system dpaths. Adapted help
output and manpage.
*0.0.5 (dparam)
Dparam now supports listing dparams for both system and user dirs. Adapted help
output and manpage.
2011-12-17 Frank Scheiner <[email protected]>
*0.0.7b (gtransfer)
*0.0.4b (dparam)
*0.0.4c (dpath)
all:
Enhanced builtin support for configuration files. The scripts now search for the
configuration files in three different locations:
- "/opt/gtransfer/etc/" (default for system install)
- "/etc/opt/gtransfer/"
- "$HOME/.gtransfer/" (default for user install)